# Secrets replication — pair-local live gate on the three-mesh rig **Result: PASS — 4/4 checks.** Run 2026-08-07 on the docker-dev rig. ## Purpose Re-gate **pair-local** Akka peer-to-peer secret replication on the current three-mesh docker-dev rig, and **supersede the 2026-07-18 six-node gate ([lmxopcua#482](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482), closed at `master @ c878fbbd`) as evidence.** That gate proved a secret seeded on `site-a-1` converged to **all five** peers. It ran on the old single fleet-wide Akka mesh. **Mesh Phase 6** (`2839deb5`, 2026-07-24) partitioned the rig into three independent 2-node meshes, so `DistributedPubSub` — and with it the `zb-mom-ww-secrets` topic — is now **pair-local by construction**. The #482 result is therefore not merely stale, it asserts the opposite of the intended behaviour: six-node convergence would now be a *failure*. Phase 6's own live gate ([`2026-07-24-mesh-phase6-live-gate.md`](2026-07-24-mesh-phase6-live-gate.md)) recorded its secrets leg (leg 6) as **"PASS (by fallback criterion)"** — it observed zero secret errors fleet-wide and *inferred* pair-locality structurally from the mesh split. No secret was ever seeded. **This gate replaces that inference with direct evidence**, in both directions. This is the OtOpcUa half of the production-topology decision in [scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3) (design: `scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`), which fixes OtOpcUa production on **Akka pair-local** replication and required pair-local semantics to be re-gated on a real three-mesh rig before enabling. **Scope: config-only.** No product code was changed. One rig-config change was made (below). ## Rig state | Fact | Value | |---|---| | Repo / branch | `OtOpcUa` @ `master` | | Git SHA | `2100987453dfb44850f0deccbfb5583b49be6104` (`21009874`) | | Image | `otopcua-host:dev` = `sha256:384ce9aa7bab`, built 2026-08-07 09:49 UTC | | `OtOpcUa.Host.dll` | `a434b04f7d198e39…` — **byte-identical on all six nodes** | | Secrets libs | `ZB.MOM.WW.Secrets*` **0.2.3** (`Directory.Packages.props`) | | Replication | `Secrets__Replication__Enabled=true` on **all six** nodes (pre-existing, `x-secrets-env` anchor) | | Announce interval | `00:00:05` (rig override; product default 30 s) | | KEK | one shared committed **dev-only** key, identical on all six (`kek_id` sha256 `bb00d9490aa1…` on every row observed) | | Meshes | three independent 2-node: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2` | The rig was **rebuilt from current `master`** at the start of the run. It was found running **mixed images** — the site nodes were on a 2026-08-06 17:55 build, predating `5b9ff9d0` (#539) and `8b10d327` (#548), while the central pair was current. Rebuilding removed the ambiguity; all six now report the same `Host.dll` hash. (The rebuild also confirmed the build is deterministic: the freshly built `Host.dll` hash equals the one the already-current central pair carried.) Mesh formation after rebuild — each node handshakes **only** its pair partner: ``` site-a-1 Node [akka.tcp://otopcua@site-a-2:4053] is JOINING, roles [driver,cluster-SITE-A] site-b-1 Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [driver,cluster-SITE-B] central-1 Welcome from [akka.tcp://otopcua@central-2:4053] ``` ### Rig-config change made by this gate `docker-dev/docker-compose.yml` — `x-secrets-env` gained: ```yaml Secrets__SqlitePath: "/app/data/otopcua-secrets.db" ``` The appsettings default is the relative `otopcua-secrets.db`, which resolves to `/app` — **inside the image's writable container layer**. That cost two things, both of which obstructed this gate: the store is destroyed by any `up -d --build` container recreate, and it is unreachable from the host, so seeding or inspecting a node's store meant stopping the container and `docker cp`-ing the file out. `/app/data` is the node's **existing per-node volume** (the one LocalDb already uses) — still a private store per node, so replication is still what syncs them, but it survives recreate and a helper container can mount it while the node keeps running. Dev-rig config only; the product default is untouched. ## Method Both driver-only site nodes have **no AdminUI**, so there is no in-app authoring surface on them. Secrets were seeded and read with the `ZB.MOM.WW.Secrets.Cli` (`secret`) published from `scadaproj/ZB.MOM.WW.Secrets` and run in a throwaway container mounting **one node's** volume, with that node's KEK. This is the #482 method. This exercises the **anti-entropy path**, not the live-push path — see Residual 1. Anti-entropy is the library's own *correctness* path (`SecretReplicationActor`: every node publishes a `SecretManifestAnnounce` each interval; a peer computes push and pull sets in one pass), and it reads the node's live store, so a row written into that store by the CLI is announced by the running node on its next tick. Ciphertext identity is evidenced as a **SHA-256 over the encrypted columns only** (`ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag`), plus a hash of `kek_id`. No plaintext and no key material appears in this document. The secret values used were throwaway smoke strings. Two **isolation controls** were seeded alongside the subject secrets — one on `site-b-1`, one on `central-1`. Without them, "the site-a secret never appeared on site-b" would be equally consistent with site-b's replicator being dead. The controls prove each other pair's replication is *alive* while still not crossing. ## Results | # | Check | Result | Convergence | |---|---|---|---| | 1 | Seed on one site-a node ⇒ partner converges, byte-identical, decrypt-verified | **PASS** | **≤ 7 s** (budget 60 s) | | 2 | No cross-pair propagation to site-b or central | **PASS** | absent across 240 s / 48 announce intervals | | 3 | Delete propagates within the pair | **PASS** | **< 1 s** (budget 60 s) | | 4 | Clean pair restart — no resurrection, live secret survives | **PASS** | mesh re-formed in 7 s | Baseline before seeding: **all six stores empty** (clean slate from the rebuild). ### Check 1 — pair convergence · PASS Seeded on `site-a-1` only, 09:53:02–09:53:03Z. First poll of `site-a-2` at 09:53:10Z already had both rows, i.e. **≤ 7 s**, consistent with a single 5 s announce cycle. ``` site-a-1 pair-gate-smoke-1 rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00 pair-gate-survivor rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00 site-a-2 pair-gate-smoke-1 rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00 pair-gate-survivor rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00 ``` Ciphertext hashes, `kek_id` and `updated_utc` are **identical** on both nodes — the row is replicated verbatim, not re-encrypted, so the LWW ordering key is preserved. **Decrypt-verified on the partner**: `secret get` on `site-a-2` returned the exact plaintext seeded on `site-a-1` for both secrets. ### Check 2 — no cross-pair propagation · PASS `site-b-1` and `central-1` were each seeded with their own control secret at 09:53:50–09:53:52Z. `site-b-1/2` and `central-1/2` were then sampled **8 times over 240 s** (09:53:35Z → 09:57:46Z — 48 announce intervals at 5 s). Every sample was identical: ``` site-b-1 pair-gate-siteb-probe rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1 site-b-2 pair-gate-siteb-probe rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1 central-1 pair-gate-central-probe rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1 central-2 pair-gate-central-probe rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1 ``` - **Neither site-a secret ever appeared** on site-b or central, across all 8 samples. - **The controls converged within their own pairs** (byte-identical, same `updated_utc`) — so SITE-B and MAIN replication is demonstrably *alive*, and the absence above is isolation, not a dead replicator. - **Reverse direction also clean**: sampled at 09:54:25Z, the site-a pair held *only* its own two secrets — neither control had crossed into it. Isolation therefore holds **in every direction between all three pairs**. This is the correct post-Phase-6 behaviour, not a defect. ### Check 3 — delete propagates within the pair · PASS Deleted on **`site-a-2`** — deliberately the *receiving* node, so this also proves the reverse intra-pair direction (`site-a-2` → `site-a-1`), which Check 1 did not cover. Deleted 09:58:02Z; the tombstone was on `site-a-1` at the first poll, **< 1 s**: ``` site-a-1 pair-gate-smoke-1 rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00 site-a-2 pair-gate-smoke-1 rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00 ``` Delete is a **soft tombstone** (`is_deleted=1`, `revision` bumped, `updated_utc` restamped), which is what lets it win under last-writer-wins instead of being re-created by the peer's older live row. `pair-gate-survivor` was untouched on both nodes. ### Check 4 — clean pair restart · PASS `docker compose restart site-a-1 site-a-2` (both together — the supported pattern), 09:58:10Z → 09:58:20Z. The SITE-A mesh re-formed by 09:58:27Z (`site-a-1` founded, `site-a-2` joined, both to `Up`) — **7 s**. After restart, and re-confirmed at 10:06:41Z (> 60 s later, 12+ announce intervals): - **No resurrection.** `pair-gate-smoke-1` remained `rev=1 deleted=1` on both nodes with its `updated_utc` unchanged. `secret get` returns `{"error":"not-found"}`; `secret list` returns `['pair-gate-survivor']` on both nodes. - **Live secret survived and resolves.** `pair-gate-survivor` present on both nodes with its original ciphertext hash and `updated_utc`, and **decrypted to the correct plaintext on both**. Final fleet-wide sweep at 10:06:41Z: each pair holds exactly its own secrets and nothing else. **Zero** secret-related warnings or errors on any of the six nodes across the entire run. ## Conclusion Pair-local Akka secret replication is **live-proven on the three-mesh rig**: convergence and deletion within a pair are well inside the 60 s budget, isolation between all three pairs holds in every direction, and a clean pair restart neither resurrects a tombstone nor loses a live secret. This satisfies the scadaproj#3 precondition for the OtOpcUa production topology (Akka pair-local), and supersedes both the #482 six-node result and the Phase 6 leg-6 structural inference. **Unchanged by this gate:** `Secrets:Replication:Enabled` remains **default-false in the product**. It is enabled on the docker-dev rig only. Production enablement additionally requires one fleet-wide KEK supplied via appsettings/env and **never committed** — the rig's key is dev-only. ## Residuals 1. **The live-push path is not covered by this gate.** Seeding wrote into a node's store underneath the running app, so what converged is **anti-entropy** (`SecretManifestAnnounce` → push/pull), not the `PublishLocalWrite` fast path a write through the app's own `ReplicatingSecretStore` takes. Anti-entropy is the library's correctness path and the one that must hold, so this is the right thing to gate on a driver-only node — but the fast path remains covered only by the library's own two-node in-process cluster tests. Covering it live needs a write issued *through* a node's app, which on the site pairs means an authoring surface that does not exist there today; the central pair's AdminUI `/admin/secrets` page could cover it for MAIN. 2. **`appsettings.json` carries a stale, actively misleading comment.** The `Secrets:Replication` block still reads *"KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 … Do not enable until the library is fixed."* The repo has pinned **0.2.3** since 2026-07-19; `0.2.1` fixed the inert-registration defect and `0.2.2` fixed the DI deadlock (scadaproj#1). The comment now tells an operator not to enable a feature this gate just proved works. **Not corrected here** — this gate's mandate is config-only and that file is product config, not rig config. Worth a one-line follow-up. 3. **Working tree was not clean, and another session was editing the repo concurrently.** At the start of the run one unrelated file was dirty (`docs/plans/2026-05-29-adminui-followups.md.tasks.json`, a docs-bookkeeping edit); by the end a second had appeared (`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs`). Neither is product code on the gate path, and **the rig image was built from committed `master` (`21009874`)**, not from the working tree, so the gated artifact is unaffected. The commit carrying this gate is file-scoped and includes neither. Recorded for provenance — a concurrent editing session is the condition under which a "rebuild the rig" step could otherwise pick up someone else's in-flight change silently. 4. **Announce interval is 5 s on the rig, 30 s by product default.** The convergence figures here (≤ 7 s, < 1 s) are bounded by the rig's shortened interval. At the product default, expect worst-case convergence around one 30 s cycle — still inside the 60 s budget, but the measured numbers should not be quoted as production latencies. 5. **Not re-gated: pair restart with a delete issued while one node is down.** Check 4 restarts both nodes *after* both already held the tombstone. The harder resurrection case — deleting while a peer is offline, then bringing it back with a live copy of the row — is covered by the library's LWW tests but not live here.