scadaproj/docs/plans/2026-07-18-secrets-0.2.0-upgrade-and-clustering.md

# Secrets `0.2.0` Upgrade + Clustered Topology Adoption — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

**Goal:** Move all four family apps from `ZB.MOM.WW.Secrets` `0.1.2` to `0.2.0`, and adopt a **clustered secrets topology** in the two Akka-clustered apps (OtOpcUa, ScadaBridge) so a secret written on one node is resolvable on every node that needs it.

**Architecture:** No library construction — `0.2.0` is built, published, and restore-verified on the Gitea feed (5 packages). Two distinct workstreams: (A) a **pure version bump** for all four apps, which is behaviour-preserving and independently valuable (it picks up the G-8 KEK-rotation surface and a **high-severity transitive security fix**); (B) **topology adoption** for the two clustered apps, which is *configuration + one DI call*, no product-code change. The topology choice is **not the same for both apps** — see §2, this is the load-bearing design point.

**Tech Stack:** .NET 10, C#, `ZB.MOM.WW.Secrets{,.Abstractions,.Ui}` @ `0.2.0`, plus `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (OtOpcUa) / `ZB.MOM.WW.Secrets.Replicator.SqlServer` (ScadaBridge), all from the `dohertj2-gitea` feed. Akka.NET **1.5.62** in both clustered apps — an exact match with the replicator package's Akka dependency.

**Predecessors:** [`2026-07-16-secrets-adoption-design.md`](2026-07-16-secrets-adoption-design.md) (the adoption template — §5 rewritten by this work), the three executed per-app adoption plans, and [`2026-07-17-secrets-g7-clustered-replication-design.md`](2026-07-17-secrets-g7-clustered-replication-design.md) (the fork analysis, now a *deployment* decision rather than a build one).

---

## 1. Why bump at all

> **⚠️ CORRECTED TWICE during execution, 2026-07-18 — read this before citing the bump's value.**
> The original text claimed the bump "closes a high-severity advisory in all four apps." A first
> correction narrowed that to "two apps." **Both were wrong.** Measured by A/B against the `0.1.2`
> baseline in every repo: **all four already resolved the patched `SQLitePCLRaw.lib.e_sqlite3
> 2.1.12` before this bump.** The Secrets `0.2.0` upgrade delivers **no security change to any
> app.** The lesson — the same one G-7's post-build review taught — is that a plausible mechanism
> is not evidence. Both wrong claims came from reasoning about what `0.2.0`'s nuspec *should*
> deliver instead of measuring what each repo *already resolved*.

`0.2.0` does declare a direct dependency on the patched `SQLitePCLRaw.lib.e_sqlite3 2.1.12`
(GHSA-2m69-gcr7-jv3q, high). That is real — it is simply **redundant everywhere**, because each
repo had already arrived at ≥`2.1.12` by another route:

| App | How it already resolved ≥ 2.1.12 before the bump | Security delta from this bump |
|---|---|---|
| HistorianGateway | Direct pin `SQLitePCLRaw.lib.e_sqlite3 3.50.3` (cites a *different* CVE, CVE-2025-6965) | **None** |
| OtOpcUa | `SQLitePCLRaw.bundle_e_sqlite3` pinned `2.1.12` in `Directory.Packages.props` | **None** |
| MxAccessGateway | Transitively from the pre-existing `ZB.MOM.WW.Auth.ApiKeys 0.1.5` reference | **None** (A/B-verified on the 0.1.2 baseline) |
| ScadaBridge | Transitively from `ZB.MOM.WW.Auth.ApiKeys 0.1.5` (commit `50d79ed1`) | **None** (A/B-verified) |

**The honest justification for Tasks 1–4 is therefore version hygiene plus a precondition:** the
replication packages in Tasks 5–9 are `0.2.0` artifacts and cannot be consumed on `0.1.2`, and
`0.2.0` exposes the G-8 KEK-rotation surface (`secret rewrap-all`) the runbooks document but no app
can currently run. That is a legitimate reason to land them — but it is **weaker than originally
stated**, and it means Tasks 1–4 are *not* independently valuable in the way this plan first
claimed. If the clustering work is abandoned, the bumps buy only currency and rotation access.

### The real SQLite finding — and it was a live vulnerability

Chasing the suppressions turned up something the version bump had nothing to do with. **Fixed
2026-07-18**, in its own commits, separate from the Secrets bump:

**ScadaBridge was shipping a live high-severity vulnerability, masked by its own suppression.**
The earlier A/B only checked the *Host* project, which resolved a patched `2.1.12` transitively via
`Auth.ApiKeys`. Every *other* SQLite consumer — `AuditLog`, `SiteRuntime`, `StoreAndForward`,
`SiteEventLogging` and 11 test projects — still resolved the vulnerable **`2.1.11`**, and the
`NuGetAuditSuppress` hid NU1903 for all of them. Proven by removing the suppression and forcing a
restore: 12+ projects immediately reported NU1903.

Both suppressions rested on false premises: mxaccessgw's said *"no patched e_sqlite3 release exists
yet"*; ScadaBridge's said *"the only patched native lib is the SQLitePCLRaw 3.x line."* **`2.1.12`
patches this advisory within the 2.1.x line** — verified empirically (suppression removed + pin in
place → forced restore reports no NU1903). HistorianGateway's separate `3.50.3` pin cites
**CVE-2025-6965**, a *different* advisory; that is the likely source of the "only 3.x" confusion.

**Fix applied:** explicit `PackageReference` to `SQLitePCLRaw.lib.e_sqlite3 2.1.12` in every
SQLite-consuming project, suppressions deleted in both repos (ScadaBridge `2a0faeab`, mxaccessgw
`2f0cfe3`). HistorianGateway and OtOpcUa needed nothing — genuinely fixed already, no suppressions.

**Rejected alternative:** `CentralPackageTransitivePinningEnabled`, which clears it in one line but
makes every central version a ceiling for transitive resolution too — immediately demanding bumps
to `Google.Protobuf`, `Grpc.Net.Client`, `Microsoft.Data.SqlClient` and `Newtonsoft.Json`. That is
a gRPC/data-access change to a production SCADA platform and belongs in its own reviewed commit,
not smuggled in under a security fix. **Recorded as a genuine follow-up** — it is the better
long-term posture.

Verified: ScadaBridge 55 projects build 0 warn / 0 error, **1108 tests pass across the SQLite
layer**; mxaccessgw 0 warn / 0 error with test results identical to baseline.

Tasks 1–4 remain four independent, single-file, behaviour-preserving edits in four separate repos
— **fully parallelizable**.

### Build/test expectations — corrected against reality

The original text expected "0 warnings" everywhere. Measured, on the `0.1.2` baseline:

- **OtOpcUa** emits **861** pre-existing warnings (`OTOPCUA0001` / xUnit analyzers). Identical count before and after the bump. Do not expect 0.
- **HistorianGateway**'s full-solution restore **fails** on an AngleSharp `NU1902` advisory promoted to an error by `TreatWarningsAsErrors` — pre-existing, reproduced on the unmodified base branch.
- **MxAccessGateway** has no `src/MxGateway.sln`; the solutions are `src/ZB.MOM.WW.MxGateway.slnx` and `src/ZB.MOM.WW.MxGateway.NonWindows.slnx` (the macOS-safe subset omitting the x86 net48 worker). **`CLAUDE.md`'s recorded build command is stale** and should be corrected.
- Several suites have pre-existing environment-gated failures on macOS (live-historian DNS, Unix-domain-socket path length, shared Docker-fixture contention). **A/B against the baseline before attributing any failure to the bump.**

## 2. The topology fork — and why the two clustered apps resolve it differently

Verified against both repos on 2026-07-18. This is the single most important section; getting it wrong produces a cluster that looks healthy and silently fails to converge.

### OtOpcUa → **Akka peer-to-peer** (recommended)

- OtOpcUa is **one Akka cluster**. Every node shares a single `seed-nodes` list, so `DistributedPubSub` reaches every member. The app **already uses `DistributedPubSub` heavily** in product code (`DriverHostActor.cs:419` caches the mediator; driver-health, alarm, and deployment-ack topics all ride it), so the transport is proven in-app, not newly introduced.
- OtOpcUa registers its `ActorSystem` **in DI via `Akka.Hosting`** — which is exactly what `SecretReplicationActorProvider` requires.
- **Driver-role nodes resolve Layer-B driver secrets** (Galaxy API key, OpcUaClient passwords) at driver-session-open. Those nodes should keep starting drivers when central SQL is unreachable; peer-to-peer replication serves reads from local state, so a partitioned driver node keeps working on last-known-good and re-converges on heal.
- *Best-practice fit:* matches the app's existing cluster-communication idiom, adds no new infrastructure dependency, and preserves partition tolerance for the node role that most needs it.
- **Alternative:** shared SQL store against the existing ConfigDb (`AddZbSecretsSqlServerStore`) — simpler (zero distributed failure modes, one `rewrap-all`), but couples every driver node's secret resolution to ConfigDb reachability. Prefer this **only** if you decide driver nodes should hard-fail rather than run on cached secrets during a ConfigDb outage.

### ScadaBridge → **SQL-Server hub mode** (recommended) — Akka *cannot* do the job here

- ScadaBridge is **hub-and-spoke: one central cluster + N site clusters, each a SEPARATE Akka cluster** with its own `seed-nodes` list (`AkkaHostedService.BuildHocon`, `seed-nodes = [...]` per node). Central↔site traffic goes through `CentralCommunicationActor`, not cluster membership.
- **Therefore `DistributedPubSub` is cluster-scoped and does not cross the central↔site boundary.** An Akka replicator would converge the central pair among themselves and each site pair among themselves, producing **independent secret islands that never reconcile** — while every node reports healthy. This is the failure mode to avoid, and it is why "integrate with Akka" is the wrong answer for this app.
- Hub mode (`AddZbSecretsSqlServerReplication`) fits the topology exactly: each node keeps a **local** store and syncs bidirectionally with a shared central SQL hub. Site nodes keep resolving secrets **through a WAN outage** — which is the entire point of the hub-and-spoke design — and re-converge when the link returns.
- *Best-practice fit:* mirrors how ScadaBridge already treats central SQL (config, audit forwarding) — authoritative when reachable, locally buffered when not.
- **Alternative:** shared SQL store (`AddZbSecretsSqlServerStore`) — one store, no reconciliation, but a site that loses the WAN loses secret resolution past the resolver's short TTL cache. **Not recommended**: it contradicts the site-autonomy premise of the architecture.

### HistorianGateway, MxAccessGateway → **version bump only**

Both are single-box, non-clustered. No replication package, no topology. Tasks 1–2.

## 3. Constraints that apply to every topology

1. **Same KEK on every node — non-negotiable.** Rows are ciphertext-only; a node whose master key differs stores replicated rows fine and then **fails closed on every resolve** with a `kek_id` mismatch. It reads like data corruption but is a deployment error. Verify with the digest check in [`clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md) §0 **before** enabling replication.
2. **KEK rotation does not replicate.** A re-wrap deliberately leaves `revision`/`updated_utc` untouched so it stays invisible to LWW. `rewrap-all` runs **once per independent store** — once for a shared store; once per node *and* against the hub in hub mode; once per node in Akka mode.
3. **`nuget.config` needs no change.** Both clustered repos already map `ZB.MOM.WW.Secrets.*` to `dohertj2-gitea` (OtOpcUa `NuGet.config:29`, ScadaBridge `nuget.config:29`), and that pattern already covers `ZB.MOM.WW.Secrets.Replicator.SqlServer` / `.Replicator.AkkaDotNet`. **Verified** — do not add redundant patterns.
4. **Do not reconfigure Data Protection.** Unchanged from the original adoption: secrets envelope crypto is independent of DP, and touching DP risks invalidating cookies/hub tokens.

---

## Task 1 — HistorianGateway: bump `0.1.2` → `0.2.0`

**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 2, Task 3, Task 4 (different repos entirely)

**Files:**
- Modify: `~/Desktop/HistorianGateway/src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj:37-38`

**Steps:**

1. This repo does **not** use CPM for these — versions are inline on the `PackageReference`. Change both:
   ```xml
   <PackageReference Include="ZB.MOM.WW.Secrets" Version="0.2.0" />
   <PackageReference Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.0" />
   ```
2. Restore + build:
   ```bash
   cd ~/Desktop/HistorianGateway && dotnet build ZB.MOM.WW.HistorianGateway.slnx
   ```
   **Expected:** succeeds, 0 warnings. `0.2.0` is API-compatible with `0.1.2` for everything this app uses.
3. Confirm the security fix actually landed — this is the *point* of the task, so verify it rather than assuming:
   ```bash
   dotnet list ZB.MOM.WW.HistorianGateway.slnx package --include-transitive | grep -i sqlitepclraw
   ```
   **Expected:** `SQLitePCLRaw.lib.e_sqlite3` resolves to `2.1.12` (not `2.1.11`).
4. Run the offline suite: `dotnet test ZB.MOM.WW.HistorianGateway.slnx`
   **Expected:** green. **Known pre-existing:** an AngleSharp `NU1902` advisory warning in the test project — unrelated to this change; do not chase it, but confirm it is the *only* warning and that it predates the bump (`git stash` + rebuild to compare if unsure).
5. Commit:
   ```bash
   git add -A && git commit -m "chore(secrets): bump ZB.MOM.WW.Secrets 0.1.2 -> 0.2.0 (SQLitePCLRaw advisory fix + KEK rotation)"
   ```

---

## Task 2 — MxAccessGateway: bump `0.1.2` → `0.2.0`

**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 3, Task 4

**Files:**
- Modify: `~/Desktop/MxAccessGateway/src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj:20-22`

**Steps:**

1. Set all three inline versions to `0.2.0` (`ZB.MOM.WW.Secrets`, `.Abstractions`, `.Ui`).
2. Build: `cd ~/Desktop/MxAccessGateway && dotnet build src/MxGateway.sln` → **Expected:** 0 warnings.
3. Verify the transitive pin as in Task 1 step 3.
4. Test: `dotnet test src/MxGateway.Tests/MxGateway.Tests.csproj` → **Expected:** green (no MXAccess needed — fake worker).
5. Commit with the same message shape as Task 1.

---

## Task 3 — OtOpcUa: bump `0.1.2` → `0.2.0` (CPM)

**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 2, Task 4

**Files:**
- Modify: `~/Desktop/OtOpcUa/Directory.Packages.props:135-137`

**Steps:**

1. OtOpcUa **is** on Central Package Management — edit the three `<PackageVersion>` rows to `0.2.0`. Consuming csprojs carry bare `<PackageReference>` with no `Version`; **do not** add versions there.
2. Build: `cd ~/Desktop/OtOpcUa && dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **Expected:** 0 warnings (repo builds under `TreatWarningsAsErrors`).
3. Verify the transitive pin as in Task 1 step 3.
4. Test: `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → **Expected:** green.
5. Commit. Branch: `feat/secrets-0.2.0` off `origin/master` (**OtOpcUa's default is `master`**).

---

## Task 4 — ScadaBridge: bump `0.1.2` → `0.2.0` (CPM)

**Classification:** trivial
**Estimated implement time:** ~2 min
**Parallelizable with:** Task 1, Task 2, Task 3

**Files:**
- Modify: `~/Desktop/ScadaBridge/Directory.Packages.props:89-91`

**Steps:**

1. Edit the three `<PackageVersion>` rows to `0.2.0`.
2. Build: `cd ~/Desktop/ScadaBridge && dotnet build ZB.MOM.WW.ScadaBridge.slnx` → **Expected:** 0 warnings.
3. Verify the transitive pin as in Task 1 step 3.
4. Test: `dotnet test ZB.MOM.WW.ScadaBridge.slnx`
   **Expected:** green except the **known pre-existing** AngleSharp `NU1902` break in `CentralUI.Tests`. Confirm it predates the bump; do not fix it here.
5. Commit. Branch: `feat/secrets-0.2.0` off `origin/main`.

---

> **Checkpoint — review after Tasks 1–4.** All four apps are on `0.2.0` and the advisory is closed family-wide. This is a coherent, independently shippable slice: FF-merge and push each repo before starting Task 5. If the topology work is deferred, the plan still delivered its security value.

---

## Task 5 — OtOpcUa: add the Akka replicator package

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 7 (different repo)
**Depends on:** Task 3

**Files:**
- Modify: `~/Desktop/OtOpcUa/Directory.Packages.props` (ZB.MOM block, beside the rows edited in Task 3)
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`

**Steps:**

1. Add the CPM version row:
   ```xml
   <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.2.0" />
   ```
2. Add a bare reference in the **Host** csproj only (the Host owns the actor system and the secrets registration; driver projects keep referencing only `.Abstractions`):
   ```xml
   <PackageReference Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" />
   ```
3. Restore: `dotnet restore ZB.MOM.WW.OtOpcUa.slnx`
   **Expected:** resolves from `dohertj2-gitea` via the existing `ZB.MOM.WW.Secrets.*` mapping. It pulls `Akka.Cluster.Tools 1.5.62` — **the exact version OtOpcUa already pins**, so expect no downgrade/conflict. If NU1109 appears, stop: that means the Akka pins have drifted and the plan's premise needs rechecking.
4. Build → **Expected:** 0 warnings, nothing consumes the package yet.
5. Commit.

---

## Task 6 — OtOpcUa: wire Akka replication + shared-KEK config

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Depends on:** Task 5

> **High-risk because** it changes which `ISecretStore` implementation the container resolves on **every** node, including driver-role nodes that have no auth/AdminUI. A mistake here surfaces as drivers failing to open sessions, not as a test failure.

**Files:**
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (the existing unconditional `AddZbSecrets` call site added by the G-2 adoption)
- Modify: `~/Desktop/OtOpcUa/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
- Test: `~/Desktop/OtOpcUa/tests/.../Host.Tests/` — DI-resolution test (see step 4)

**Steps:**

1. **Replace** the runtime registration — `AddZbSecretsAkkaReplication` calls `AddZbSecrets` internally, so calling both would double-register:
   ```csharp
   // was: builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
   builder.Services.AddZbSecretsAkkaReplication(builder.Configuration, "Secrets", "Secrets:Replication");
   ```
   Using: `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection`.
   **Leave the pre-host expander untouched** — it builds a throwaway container that must stay a plain local SQLite store. Replication is a *runtime* concern; the bootstrap provider runs before the actor system exists.
2. Merge the serializer HOCON into the actor-system config so replication messages use the package's `SerializerWithStringManifest` rather than falling back to a default serializer:
   ```csharp
   // where the Akka HOCON is assembled:
   config = config.WithFallback(AkkaSecretsReplication.SerializationConfig);
   ```
3. Add the config block to `appsettings.json` beside the existing `Secrets` section:
   ```json
   "Secrets": {
     "Replication": { "AnnounceInterval": "00:00:30", "ActorName": "zb-secret-replication" }
   }
   ```
4. **Write a DI-resolution test before wiring** (this is the class of defect that shipped in the library itself — every unit test passed because nothing ever built a container):
   ```csharp
   [Fact]
   public void Secrets_store_resolves_as_the_replicating_decorator()
   {
       // Build the Host's service collection, then:
       var store = provider.GetRequiredService<ISecretStore>();
       Assert.IsType<ReplicatingSecretStore>(store);
       // And the undecorated concrete store must still resolve — the decorator depends on it:
       Assert.NotNull(provider.GetRequiredService<SqliteSecretStore>());
   }
   ```
   Run it first and confirm it **fails** before step 1's edit, passes after.
5. Build + full suite → **Expected:** 0 warnings, green.
6. **Verify the same KEK is on every node** before any multi-node run — per §3.1, use the digest check in `clustered-secrets.md` §0. Do not skip: a mismatch is silent until first resolve.
7. Commit.

---

## Task 7 — ScadaBridge: add the SQL-Server replicator package + hub schema

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 5 (different repo)
**Depends on:** Task 4

**Files:**
- Modify: `~/Desktop/ScadaBridge/Directory.Packages.props`
- Modify: `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/ZB.MOM.WW.ScadaBridge.Host.csproj`

**Steps:**

1. Add CPM row + bare reference for `ZB.MOM.WW.Secrets.Replicator.SqlServer` @ `0.2.0`.
2. Restore + build → **Expected:** resolves via the existing mapping; pulls `Microsoft.Data.SqlClient 6.0.2`. ScadaBridge already uses SqlClient — **if NU1109 appears, the repo pins an older SqlClient and you must reconcile the pin before continuing.** Report rather than force.
3. **Do not hand-provision the hub schema.** `AddZbSecretsSqlServerReplication` registers `SqlServerHubMigrationHostedService`, which creates it idempotently at startup. The DB and a login with `CREATE TABLE`/`CREATE SCHEMA` rights must exist — see `clustered-secrets.md` for the provisioning SQL.
4. Commit.

---

## Task 8 — ScadaBridge: wire hub replication + per-node config

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none
**Depends on:** Task 7

> **High-risk because** it introduces a background sweep that writes to a shared central database from every site node, and because a wrong `Secrets:SqlServer:ConnectionString` on a site node fails at startup (eager `Validate()`), taking the node down.

**Files:**
- Modify: `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (existing `AddZbSecrets` call site)
- Modify: `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json`
- Modify: the per-node docker appsettings under `~/Desktop/ScadaBridge/docker/`

**Steps:**

1. Replace the runtime registration (it calls `AddZbSecrets` internally — do not call both):
   ```csharp
   builder.Services.AddZbSecretsSqlServerReplication(
       builder.Configuration, "Secrets", "Secrets:SqlServer");
   ```
2. Config block — the connection string is itself a secret, so deliver it the way every other ScadaBridge connstr is delivered (a `${secret:}` token resolved by the pre-host expander, which runs *before* this registration binds):
   ```json
   "Secrets": {
     "SqlServer": {
       "ConnectionString": "${secret:sql/scadabridge/secrets-hub}",
       "SchemaName": "zbsecrets",
       "SyncInterval": "00:00:30",
       "SyncOnStartup": true
     }
   }
   ```
   **Ordering gotcha:** seed `sql/scadabridge/secrets-hub` in the **local** store on every node first. The hub connstr cannot itself come from the hub — that is a bootstrap cycle. This is the one config value that must stay locally seeded.
3. Add the same DI-resolution test as Task 6 step 4, asserting `ReplicatingSecretStore`.
4. Build + suite → **Expected:** 0 warnings; green modulo the known AngleSharp break.
5. Verify the shared KEK across central + site nodes (§3.1).
6. Commit.

---

## Task 9 — Live convergence gate (both clustered apps)

**Classification:** high-risk
**Estimated implement time:** ~5 min (excluding rig setup)
**Parallelizable with:** none
**Depends on:** Task 6, Task 8

> This is the **G-2-class live gate** the G-7 design called for. The library's offline suite proves the algorithms (2-node in-process Akka cluster, 15 live SQL-Server tests); it cannot prove *this app's* wiring. Do not mark the topology adopted without it.

**Steps — run per app:**

1. Stand up 2 nodes (OtOpcUa: docker-dev rig; ScadaBridge: central pair, then a site node against the central hub).
2. **Write on node A**, via the `/admin/secrets` UI or the `secret` CLI.
3. **Resolve on node B.** Expected: the value appears within one sweep/announce interval (~30 s).
4. **Delete on A → confirm the tombstone propagates to B** and B no longer resolves it. (Resurrection is the classic anti-entropy bug; this is the case that catches it.)
5. **Partition test** — the reason each topology was chosen:
   - OtOpcUa: stop node B's cluster link; confirm B still resolves its Layer-B driver secrets from local state; heal; confirm re-convergence.
   - ScadaBridge: stop the site node's route to the central hub; confirm it still resolves; restore; confirm the sweep re-converges both directions.
6. **Negative KEK control** (proves fail-closed rather than silent-wrong): point one node at a *different* KEK, confirm it fails closed with a `kek_id` mismatch rather than returning a wrong plaintext. Revert.
7. Record results in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md) — including anything that did **not** pass.

---

## Task 10 — Docs reconcile

**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** none
**Depends on:** Task 9

**Files:**
- Modify: `components/secrets/GAPS.md` (close the two Open items: version bump; topology adoption)
- Modify: `CLAUDE.md` (Secrets row: consumer pins `0.1.2` → `0.2.0`; record the adopted topology per app)
- Modify: `components/secrets/current-state/*/CURRENT-STATE.md` (per-app store/topology)

**Steps:** State what was **proven live** vs what was only built. If Task 9's partition or KEK-negative steps were skipped for lack of a rig, say so explicitly — an unproven claim in these docs is the exact failure mode [`component-status-claims-are-optimistic`](../../components/secrets/GAPS.md) already warns about.

---

## Rollback

Every task is a version/config change with a clean revert:
- Tasks 1–4: revert the version rows; `0.1.2` is still on the feed.
- Tasks 5–8: revert the `AddZbSecrets*Replication` call to plain `AddZbSecrets(config, "Secrets")` and drop the package reference. Local SQLite stores are untouched by replication — rows written while replication was on remain readable, since the KEK and schema are unchanged.