lmxopcua/docs/plans/2026-08-07-secrets-pair-local-live-gate.md

# Secrets replication — pair-local live gate on the three-mesh rig

**Result: PASS — 4/4 checks.** Run 2026-08-07 on the docker-dev rig.

## Purpose

Re-gate **pair-local** Akka peer-to-peer secret replication on the current
three-mesh docker-dev rig, and **supersede the 2026-07-18 six-node gate
([lmxopcua#482](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482), closed at
`master @ c878fbbd`) as evidence.**

That gate proved a secret seeded on `site-a-1` converged to **all five** peers. It ran on
the old single fleet-wide Akka mesh. **Mesh Phase 6** (`2839deb5`, 2026-07-24) partitioned
the rig into three independent 2-node meshes, so `DistributedPubSub` — and with it the
`zb-mom-ww-secrets` topic — is now **pair-local by construction**. The #482 result is
therefore not merely stale, it asserts the opposite of the intended behaviour: six-node
convergence would now be a *failure*.

Phase 6's own live gate ([`2026-07-24-mesh-phase6-live-gate.md`](2026-07-24-mesh-phase6-live-gate.md))
recorded its secrets leg (leg 6) as **"PASS (by fallback criterion)"** — it observed zero
secret errors fleet-wide and *inferred* pair-locality structurally from the mesh split. No
secret was ever seeded. **This gate replaces that inference with direct evidence**, in both
directions.

This is the OtOpcUa half of the production-topology decision in
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3) (design:
`scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`), which fixes
OtOpcUa production on **Akka pair-local** replication and required pair-local semantics to
be re-gated on a real three-mesh rig before enabling.

**Scope: config-only.** No product code was changed. One rig-config change was made (below).

## Rig state

| Fact | Value |
|---|---|
| Repo / branch | `OtOpcUa` @ `master` |
| Git SHA | `2100987453dfb44850f0deccbfb5583b49be6104` (`21009874`) |
| Image | `otopcua-host:dev` = `sha256:384ce9aa7bab`, built 2026-08-07 09:49 UTC |
| `OtOpcUa.Host.dll` | `a434b04f7d198e39…` — **byte-identical on all six nodes** |
| Secrets libs | `ZB.MOM.WW.Secrets*` **0.2.3** (`Directory.Packages.props`) |
| Replication | `Secrets__Replication__Enabled=true` on **all six** nodes (pre-existing, `x-secrets-env` anchor) |
| Announce interval | `00:00:05` (rig override; product default 30 s) |
| KEK | one shared committed **dev-only** key, identical on all six (`kek_id` sha256 `bb00d9490aa1…` on every row observed) |
| Meshes | three independent 2-node: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2` |

The rig was **rebuilt from current `master`** at the start of the run. It was found running
**mixed images** — the site nodes were on a 2026-08-06 17:55 build, predating `5b9ff9d0`
(#539) and `8b10d327` (#548), while the central pair was current. Rebuilding removed the
ambiguity; all six now report the same `Host.dll` hash. (The rebuild also confirmed the
build is deterministic: the freshly built `Host.dll` hash equals the one the already-current
central pair carried.)

Mesh formation after rebuild — each node handshakes **only** its pair partner:

```
site-a-1  Node [akka.tcp://otopcua@site-a-2:4053] is JOINING, roles [driver,cluster-SITE-A]
site-b-1  Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [driver,cluster-SITE-B]
central-1 Welcome from [akka.tcp://otopcua@central-2:4053]
```

### Rig-config change made by this gate

`docker-dev/docker-compose.yml` — `x-secrets-env` gained:

```yaml
Secrets__SqlitePath: "/app/data/otopcua-secrets.db"
```

The appsettings default is the relative `otopcua-secrets.db`, which resolves to `/app` —
**inside the image's writable container layer**. That cost two things, both of which
obstructed this gate: the store is destroyed by any `up -d --build` container recreate, and
it is unreachable from the host, so seeding or inspecting a node's store meant stopping the
container and `docker cp`-ing the file out. `/app/data` is the node's **existing per-node
volume** (the one LocalDb already uses) — still a private store per node, so replication is
still what syncs them, but it survives recreate and a helper container can mount it while
the node keeps running. Dev-rig config only; the product default is untouched.

## Method

Both driver-only site nodes have **no AdminUI**, so there is no in-app authoring surface on
them. Secrets were seeded and read with the `ZB.MOM.WW.Secrets.Cli` (`secret`) published
from `scadaproj/ZB.MOM.WW.Secrets` and run in a throwaway container mounting **one node's**
volume, with that node's KEK. This is the #482 method.

This exercises the **anti-entropy path**, not the live-push path — see Residual 1. Anti-entropy
is the library's own *correctness* path (`SecretReplicationActor`: every node publishes a
`SecretManifestAnnounce` each interval; a peer computes push and pull sets in one pass), and
it reads the node's live store, so a row written into that store by the CLI is announced by
the running node on its next tick.

Ciphertext identity is evidenced as a **SHA-256 over the encrypted columns only**
(`ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag`), plus a hash of `kek_id`. No
plaintext and no key material appears in this document. The secret values used were
throwaway smoke strings.

Two **isolation controls** were seeded alongside the subject secrets — one on `site-b-1`,
one on `central-1`. Without them, "the site-a secret never appeared on site-b" would be
equally consistent with site-b's replicator being dead. The controls prove each other pair's
replication is *alive* while still not crossing.

## Results

| # | Check | Result | Convergence |
|---|---|---|---|
| 1 | Seed on one site-a node ⇒ partner converges, byte-identical, decrypt-verified | **PASS** | **≤ 7 s** (budget 60 s) |
| 2 | No cross-pair propagation to site-b or central | **PASS** | absent across 240 s / 48 announce intervals |
| 3 | Delete propagates within the pair | **PASS** | **< 1 s** (budget 60 s) |
| 4 | Clean pair restart — no resurrection, live secret survives | **PASS** | mesh re-formed in 7 s |

Baseline before seeding: **all six stores empty** (clean slate from the rebuild).

### Check 1 — pair convergence · PASS

Seeded on `site-a-1` only, 09:53:02–09:53:03Z. First poll of `site-a-2` at 09:53:10Z already
had both rows, i.e. **≤ 7 s**, consistent with a single 5 s announce cycle.

```
site-a-1   pair-gate-smoke-1   rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00
           pair-gate-survivor  rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00
site-a-2   pair-gate-smoke-1   rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00
           pair-gate-survivor  rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00
```

Ciphertext hashes, `kek_id` and `updated_utc` are **identical** on both nodes — the row is
replicated verbatim, not re-encrypted, so the LWW ordering key is preserved. **Decrypt-verified
on the partner**: `secret get` on `site-a-2` returned the exact plaintext seeded on `site-a-1`
for both secrets.

### Check 2 — no cross-pair propagation · PASS

`site-b-1` and `central-1` were each seeded with their own control secret at 09:53:50–09:53:52Z.
`site-b-1/2` and `central-1/2` were then sampled **8 times over 240 s** (09:53:35Z → 09:57:46Z
— 48 announce intervals at 5 s). Every sample was identical:

```
site-b-1   pair-gate-siteb-probe    rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1
site-b-2   pair-gate-siteb-probe    rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1
central-1  pair-gate-central-probe  rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1
central-2  pair-gate-central-probe  rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1
```

- **Neither site-a secret ever appeared** on site-b or central, across all 8 samples.
- **The controls converged within their own pairs** (byte-identical, same `updated_utc`) —
  so SITE-B and MAIN replication is demonstrably *alive*, and the absence above is isolation,
  not a dead replicator.
- **Reverse direction also clean**: sampled at 09:54:25Z, the site-a pair held *only* its own
  two secrets — neither control had crossed into it.

Isolation therefore holds **in every direction between all three pairs**. This is the correct
post-Phase-6 behaviour, not a defect.

### Check 3 — delete propagates within the pair · PASS

Deleted on **`site-a-2`** — deliberately the *receiving* node, so this also proves the
reverse intra-pair direction (`site-a-2` → `site-a-1`), which Check 1 did not cover.

Deleted 09:58:02Z; the tombstone was on `site-a-1` at the first poll, **< 1 s**:

```
site-a-1   pair-gate-smoke-1   rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00
site-a-2   pair-gate-smoke-1   rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00
```

Delete is a **soft tombstone** (`is_deleted=1`, `revision` bumped, `updated_utc` restamped),
which is what lets it win under last-writer-wins instead of being re-created by the peer's
older live row. `pair-gate-survivor` was untouched on both nodes.

### Check 4 — clean pair restart · PASS

`docker compose restart site-a-1 site-a-2` (both together — the supported pattern),
09:58:10Z → 09:58:20Z. The SITE-A mesh re-formed by 09:58:27Z (`site-a-1` founded,
`site-a-2` joined, both to `Up`) — **7 s**.

After restart, and re-confirmed at 10:06:41Z (> 60 s later, 12+ announce intervals):

- **No resurrection.** `pair-gate-smoke-1` remained `rev=1 deleted=1` on both nodes with its
  `updated_utc` unchanged. `secret get` returns `{"error":"not-found"}`; `secret list` returns
  `['pair-gate-survivor']` on both nodes.
- **Live secret survived and resolves.** `pair-gate-survivor` present on both nodes with its
  original ciphertext hash and `updated_utc`, and **decrypted to the correct plaintext on both**.

Final fleet-wide sweep at 10:06:41Z: each pair holds exactly its own secrets and nothing else.
**Zero** secret-related warnings or errors on any of the six nodes across the entire run.

## Conclusion

Pair-local Akka secret replication is **live-proven on the three-mesh rig**: convergence and
deletion within a pair are well inside the 60 s budget, isolation between all three pairs
holds in every direction, and a clean pair restart neither resurrects a tombstone nor loses a
live secret. This satisfies the scadaproj#3 precondition for the OtOpcUa production topology
(Akka pair-local), and supersedes both the #482 six-node result and the Phase 6 leg-6
structural inference.

**Unchanged by this gate:** `Secrets:Replication:Enabled` remains **default-false in the
product**. It is enabled on the docker-dev rig only. Production enablement additionally
requires one fleet-wide KEK supplied via appsettings/env and **never committed** — the rig's
key is dev-only.

## Residuals

1. **The live-push path is not covered by this gate.** Seeding wrote into a node's store
   underneath the running app, so what converged is **anti-entropy**
   (`SecretManifestAnnounce` → push/pull), not the `PublishLocalWrite` fast path a write
   through the app's own `ReplicatingSecretStore` takes. Anti-entropy is the library's
   correctness path and the one that must hold, so this is the right thing to gate on a
   driver-only node — but the fast path remains covered only by the library's own two-node
   in-process cluster tests. Covering it live needs a write issued *through* a node's app,
   which on the site pairs means an authoring surface that does not exist there today; the
   central pair's AdminUI `/admin/secrets` page could cover it for MAIN.

2. **`appsettings.json` carries a stale, actively misleading comment.** The `Secrets:Replication`
   block still reads *"KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet
   0.2.0 … Do not enable until the library is fixed."* The repo has pinned **0.2.3** since
   2026-07-19; `0.2.1` fixed the inert-registration defect and `0.2.2` fixed the DI deadlock
   (scadaproj#1). The comment now tells an operator not to enable a feature this gate just
   proved works. **Not corrected here** — this gate's mandate is config-only and that file is
   product config, not rig config. Worth a one-line follow-up.

3. **Working tree was not clean, and another session was editing the repo concurrently.**
   At the start of the run one unrelated file was dirty
   (`docs/plans/2026-05-29-adminui-followups.md.tasks.json`, a docs-bookkeeping edit); by the
   end a second had appeared
   (`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs`).
   Neither is product code on the gate path, and **the rig image was built from committed
   `master` (`21009874`)**, not from the working tree, so the gated artifact is unaffected.
   The commit carrying this gate is file-scoped and includes neither. Recorded for provenance
   — a concurrent editing session is the condition under which a "rebuild the rig" step could
   otherwise pick up someone else's in-flight change silently.

4. **Announce interval is 5 s on the rig, 30 s by product default.** The convergence figures
   here (≤ 7 s, < 1 s) are bounded by the rig's shortened interval. At the product default,
   expect worst-case convergence around one 30 s cycle — still inside the 60 s budget, but the
   measured numbers should not be quoted as production latencies.

5. **Not re-gated: pair restart with a delete issued while one node is down.** Check 4 restarts
   both nodes *after* both already held the tombstone. The harder resurrection case — deleting
   while a peer is offline, then bringing it back with a live copy of the row — is covered by
   the library's LWW tests but not live here.