lmxopcua/docs/plans/2026-08-07-secrets-pair-local-live-gate.md

Secrets replication — pair-local live gate on the three-mesh rig

Result: PASS — 4/4 checks. Run 2026-08-07 on the docker-dev rig.

Purpose

Re-gate pair-local Akka peer-to-peer secret replication on the current three-mesh docker-dev rig, and supersede the 2026-07-18 six-node gate (lmxopcua#482, closed at master @ c878fbbd) as evidence.

That gate proved a secret seeded on site-a-1 converged to all five peers. It ran on the old single fleet-wide Akka mesh. Mesh Phase 6 (2839deb5, 2026-07-24) partitioned the rig into three independent 2-node meshes, so DistributedPubSub — and with it the zb-mom-ww-secrets topic — is now pair-local by construction. The #482 result is therefore not merely stale, it asserts the opposite of the intended behaviour: six-node convergence would now be a failure.

Phase 6's own live gate (2026-07-24-mesh-phase6-live-gate.md) recorded its secrets leg (leg 6) as "PASS (by fallback criterion)" — it observed zero secret errors fleet-wide and inferred pair-locality structurally from the mesh split. No secret was ever seeded. This gate replaces that inference with direct evidence, in both directions.

This is the OtOpcUa half of the production-topology decision in scadaproj#3 (design: scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md), which fixes OtOpcUa production on Akka pair-local replication and required pair-local semantics to be re-gated on a real three-mesh rig before enabling.

Scope: config-only. No product code was changed. One rig-config change was made (below).

Rig state

Fact	Value
Repo / branch	OtOpcUa @ master
Git SHA	2100987453dfb44850f0deccbfb5583b49be6104 (21009874)
Image	otopcua-host:dev = sha256:384ce9aa7bab, built 2026-08-07 09:49 UTC
OtOpcUa.Host.dll	a434b04f7d198e39… — byte-identical on all six nodes
Secrets libs	ZB.MOM.WW.Secrets* 0.2.3 (Directory.Packages.props)
Replication	Secrets__Replication__Enabled=true on all six nodes (pre-existing, x-secrets-env anchor)
Announce interval	00:00:05 (rig override; product default 30 s)
KEK	one shared committed dev-only key, identical on all six (kek_id sha256 bb00d9490aa1… on every row observed)
Meshes	three independent 2-node: MAIN central-1/2, SITE-A site-a-1/2, SITE-B site-b-1/2

The rig was rebuilt from current master at the start of the run. It was found running mixed images — the site nodes were on a 2026-08-06 17:55 build, predating 5b9ff9d0 (#539) and 8b10d327 (#548), while the central pair was current. Rebuilding removed the ambiguity; all six now report the same Host.dll hash. (The rebuild also confirmed the build is deterministic: the freshly built Host.dll hash equals the one the already-current central pair carried.)

Mesh formation after rebuild — each node handshakes only its pair partner:

site-a-1  Node [akka.tcp://otopcua@site-a-2:4053] is JOINING, roles [driver,cluster-SITE-A]
site-b-1  Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [driver,cluster-SITE-B]
central-1 Welcome from [akka.tcp://otopcua@central-2:4053]

Rig-config change made by this gate

docker-dev/docker-compose.yml — x-secrets-env gained:

Secrets__SqlitePath: "/app/data/otopcua-secrets.db"

The appsettings default is the relative otopcua-secrets.db, which resolves to /app — inside the image's writable container layer. That cost two things, both of which obstructed this gate: the store is destroyed by any up -d --build container recreate, and it is unreachable from the host, so seeding or inspecting a node's store meant stopping the container and docker cp-ing the file out. /app/data is the node's existing per-node volume (the one LocalDb already uses) — still a private store per node, so replication is still what syncs them, but it survives recreate and a helper container can mount it while the node keeps running. Dev-rig config only; the product default is untouched.

Method

Both driver-only site nodes have no AdminUI, so there is no in-app authoring surface on them. Secrets were seeded and read with the ZB.MOM.WW.Secrets.Cli (secret) published from scadaproj/ZB.MOM.WW.Secrets and run in a throwaway container mounting one node's volume, with that node's KEK. This is the #482 method.

This exercises the anti-entropy path, not the live-push path — see Residual 1. Anti-entropy is the library's own correctness path (SecretReplicationActor: every node publishes a SecretManifestAnnounce each interval; a peer computes push and pull sets in one pass), and it reads the node's live store, so a row written into that store by the CLI is announced by the running node on its next tick.

Ciphertext identity is evidenced as a SHA-256 over the encrypted columns only (ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag), plus a hash of kek_id. No plaintext and no key material appears in this document. The secret values used were throwaway smoke strings.

Two isolation controls were seeded alongside the subject secrets — one on site-b-1, one on central-1. Without them, "the site-a secret never appeared on site-b" would be equally consistent with site-b's replicator being dead. The controls prove each other pair's replication is alive while still not crossing.

Results

#	Check	Result	Convergence
1	Seed on one site-a node ⇒ partner converges, byte-identical, decrypt-verified	PASS	≤ 7 s (budget 60 s)
2	No cross-pair propagation to site-b or central	PASS	absent across 240 s / 48 announce intervals
3	Delete propagates within the pair	PASS	< 1 s (budget 60 s)
4	Clean pair restart — no resurrection, live secret survives	PASS	mesh re-formed in 7 s

Baseline before seeding: all six stores empty (clean slate from the rebuild).

Check 1 — pair convergence · PASS

Seeded on site-a-1 only, 09:53:02–09:53:03Z. First poll of site-a-2 at 09:53:10Z already had both rows, i.e. ≤ 7 s, consistent with a single 5 s announce cycle.

site-a-1   pair-gate-smoke-1   rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00
           pair-gate-survivor  rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00
site-a-2   pair-gate-smoke-1   rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00
           pair-gate-survivor  rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00

Ciphertext hashes, kek_id and updated_utc are identical on both nodes — the row is replicated verbatim, not re-encrypted, so the LWW ordering key is preserved. Decrypt-verified on the partner: secret get on site-a-2 returned the exact plaintext seeded on site-a-1 for both secrets.

Check 2 — no cross-pair propagation · PASS

site-b-1 and central-1 were each seeded with their own control secret at 09:53:50–09:53:52Z. site-b-1/2 and central-1/2 were then sampled 8 times over 240 s (09:53:35Z → 09:57:46Z — 48 announce intervals at 5 s). Every sample was identical:

site-b-1   pair-gate-siteb-probe    rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1
site-b-2   pair-gate-siteb-probe    rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1
central-1  pair-gate-central-probe  rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1
central-2  pair-gate-central-probe  rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1

• Neither site-a secret ever appeared on site-b or central, across all 8 samples.
• The controls converged within their own pairs (byte-identical, same updated_utc) — so SITE-B and MAIN replication is demonstrably alive, and the absence above is isolation, not a dead replicator.
• Reverse direction also clean: sampled at 09:54:25Z, the site-a pair held only its own two secrets — neither control had crossed into it.

Isolation therefore holds in every direction between all three pairs. This is the correct post-Phase-6 behaviour, not a defect.

Check 3 — delete propagates within the pair · PASS

Deleted on site-a-2 — deliberately the receiving node, so this also proves the reverse intra-pair direction (site-a-2 → site-a-1), which Check 1 did not cover.

Deleted 09:58:02Z; the tombstone was on site-a-1 at the first poll, < 1 s:

site-a-1   pair-gate-smoke-1   rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00
site-a-2   pair-gate-smoke-1   rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00

Delete is a soft tombstone (is_deleted=1, revision bumped, updated_utc restamped), which is what lets it win under last-writer-wins instead of being re-created by the peer's older live row. pair-gate-survivor was untouched on both nodes.

Check 4 — clean pair restart · PASS

docker compose restart site-a-1 site-a-2 (both together — the supported pattern), 09:58:10Z → 09:58:20Z. The SITE-A mesh re-formed by 09:58:27Z (site-a-1 founded, site-a-2 joined, both to Up) — 7 s.

After restart, and re-confirmed at 10:06:41Z (> 60 s later, 12+ announce intervals):

• No resurrection. pair-gate-smoke-1 remained rev=1 deleted=1 on both nodes with its updated_utc unchanged. secret get returns {"error":"not-found"}; secret list returns ['pair-gate-survivor'] on both nodes.
• Live secret survived and resolves. pair-gate-survivor present on both nodes with its original ciphertext hash and updated_utc, and decrypted to the correct plaintext on both.

Final fleet-wide sweep at 10:06:41Z: each pair holds exactly its own secrets and nothing else. Zero secret-related warnings or errors on any of the six nodes across the entire run.

Conclusion

Pair-local Akka secret replication is live-proven on the three-mesh rig: convergence and deletion within a pair are well inside the 60 s budget, isolation between all three pairs holds in every direction, and a clean pair restart neither resurrects a tombstone nor loses a live secret. This satisfies the scadaproj#3 precondition for the OtOpcUa production topology (Akka pair-local), and supersedes both the #482 six-node result and the Phase 6 leg-6 structural inference.

Unchanged by this gate: Secrets:Replication:Enabled remains default-false in the product. It is enabled on the docker-dev rig only. Production enablement additionally requires one fleet-wide KEK supplied via appsettings/env and never committed — the rig's key is dev-only.

Residuals

1. The live-push path is not covered by this gate. Seeding wrote into a node's store underneath the running app, so what converged is anti-entropy (SecretManifestAnnounce → push/pull), not the PublishLocalWrite fast path a write through the app's own ReplicatingSecretStore takes. Anti-entropy is the library's correctness path and the one that must hold, so this is the right thing to gate on a driver-only node — but the fast path remains covered only by the library's own two-node in-process cluster tests. Covering it live needs a write issued through a node's app, which on the site pairs means an authoring surface that does not exist there today; the central pair's AdminUI /admin/secrets page could cover it for MAIN.

2. appsettings.json carries a stale, actively misleading comment. The Secrets:Replication block still reads "KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 … Do not enable until the library is fixed." The repo has pinned 0.2.3 since 2026-07-19; 0.2.1 fixed the inert-registration defect and 0.2.2 fixed the DI deadlock (scadaproj#1). The comment now tells an operator not to enable a feature this gate just proved works. Not corrected here — this gate's mandate is config-only and that file is product config, not rig config. Worth a one-line follow-up.

3. Working tree was not clean, and another session was editing the repo concurrently. At the start of the run one unrelated file was dirty (docs/plans/2026-05-29-adminui-followups.md.tasks.json, a docs-bookkeeping edit); by the end a second had appeared (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs). Neither is product code on the gate path, and the rig image was built from committed master (21009874), not from the working tree, so the gated artifact is unaffected. The commit carrying this gate is file-scoped and includes neither. Recorded for provenance — a concurrent editing session is the condition under which a "rebuild the rig" step could otherwise pick up someone else's in-flight change silently.

4. Announce interval is 5 s on the rig, 30 s by product default. The convergence figures here (≤ 7 s, < 1 s) are bounded by the rig's shortened interval. At the product default, expect worst-case convergence around one 30 s cycle — still inside the 60 s budget, but the measured numbers should not be quoted as production latencies.

5. Not re-gated: pair restart with a delete issued while one node is down. Check 4 restarts both nodes after both already held the tombstone. The harder resurrection case — deleting while a peer is offline, then bringing it back with a live copy of the row — is covered by the library's LWW tests but not live here.