dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(secrets): pair-local replication live gate on the three-mesh rig — PASS 4/4
v2-ci / build (push) Successful in 6m33s
Details
v2-ci / unit-tests (push) Failing after 17m17s
Details

Browse Source 
Re-gates Akka peer-to-peer secret replication on the current three-mesh docker-dev
rig and supersedes the 2026-07-18 six-node gate (lmxopcua#482, c878fbbd) as evidence.
That gate proved convergence to all five peers on the old single fleet-wide mesh;
after mesh Phase 6 (2839deb5) partitioned the rig into three independent 2-node
meshes, six-node convergence would be a FAILURE, not a pass. Phase 6's own leg 6
recorded "PASS (by fallback criterion)" — it inferred pair-locality structurally and
never seeded a secret. This replaces that inference with direct evidence.

All four checks PASS, on master @ 21009874 with one image (OtOpcUa.Host.dll
a434b04f byte-identical on all six nodes), Secrets libs 0.2.3:

  1. Pair convergence — seeded on site-a-1 alone, on site-a-2 in <=7s (budget 60s),
     byte-identical ciphertext + kek_id + updated_utc, decrypt-verified on the partner.
  2. No cross-pair propagation — absent from site-b and central across 8 samples /
     240s / 48 announce intervals, and clean in the reverse direction too. Isolation
     controls seeded on site-b-1 and central-1 converged within their OWN pairs, so
     the absence is isolation rather than a dead replicator.
  3. Delete propagates within the pair — issued on site-a-2 (the receiving node, so
     this also covers the reverse intra-pair direction); tombstone on site-a-1 in <1s.
  4. Clean pair restart (both together) — mesh re-formed in 7s, no resurrection of the
     tombstone, live secret survived and decrypted correctly on both nodes.

Zero secret-related warnings or errors fleet-wide across the whole run. Satisfies the
scadaproj#3 precondition for the OtOpcUa production topology (Akka pair-local).
Secrets:Replication:Enabled remains default-false in the product; the rig KEK is dev-only.

Rig config: SqlitePath moved onto each node's existing per-node /app/data volume. The
appsettings default resolves to /app, inside the image's writable container layer, so
the store was destroyed by any `up -d --build` recreate and was unreachable from the
host — seeding or inspecting a node meant stopping the container to docker cp the file
out. Still one private store per node; replication is still what syncs them.

Residuals in the gate doc: the live-push path is not covered (anti-entropy is what a
driver-only node can be gated on), and appsettings.json still carries a stale
"KNOWN NON-FUNCTIONAL against 0.2.0 / do not enable" comment that this gate disproves
— left uncorrected here because this run is config-only.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 06:08:50 -04:00
parent 2100987453
commit 1bd79c8e4a
2 changed files with 253 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-dev/docker-compose.yml
+14 -2
View File
@@ -115,8 +115,19 @@ name: otopcua-dev
# OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
# NEVER reuse this key outside this local dev rig.
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
# observable quickly during dev exercise. Each node keeps its own local SQLite store
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
# observable quickly during dev exercise. Each node keeps its OWN local SQLite store;
# replication syncs them.
#
# SqlitePath is overridden onto the node's existing per-node `/app/data` volume (the one
# LocalDb already uses) instead of the appsettings default `otopcua-secrets.db`, which
# resolves to /app — inside the image's writable container layer. Two things that costs,
# both of which bit the 2026-08-07 pair-local gate: the store is DESTROYED by any
# `up -d --build` container recreate (so a seeded row cannot outlive a rig rebuild), and
# it is unreachable from the host, so seeding or inspecting a node's store meant stopping
# the container and `docker cp`-ing the file out. On the volume, each node still has a
# private store (one volume per node — this is NOT shared storage, and replication is
# still what syncs them), but it survives recreate and a helper container can mount it
# read/write while the node keeps running.
#
# Per-cluster mesh Phase 6: this DPS topic now rides each pair's OWN 2-node mesh, not one
# shared six-node mesh — so secret replication is PAIR-LOCAL. central-1/central-2 replicate
@@ -127,6 +138,7 @@ name: otopcua-dev
# was needed for this — Enabled=true + the same KEK on every node is still correct; it just now
# converges within three independent 2-node groups instead of one six-node group.
x-secrets-env: &secrets-env
Secrets__SqlitePath: "/app/data/otopcua-secrets.db"
Secrets__Replication__Enabled: "true"
Secrets__Replication__AnnounceInterval: "00:00:05"
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
docs/plans/2026-08-07-secrets-pair-local-live-gate.md
+239
View File
@@ -0,0 +1,239 @@
# Secrets replication — pair-local live gate on the three-mesh rig
**Result: PASS — 4/4 checks.** Run 2026-08-07 on the docker-dev rig.
## Purpose
Re-gate **pair-local** Akka peer-to-peer secret replication on the current
three-mesh docker-dev rig, and **supersede the 2026-07-18 six-node gate
([lmxopcua#482](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482), closed at
`master @ c878fbbd`) as evidence.**
That gate proved a secret seeded on `site-a-1` converged to **all five** peers. It ran on
the old single fleet-wide Akka mesh. **Mesh Phase 6** (`2839deb5`, 2026-07-24) partitioned
the rig into three independent 2-node meshes, so `DistributedPubSub` — and with it the
`zb-mom-ww-secrets` topic — is now **pair-local by construction**. The #482 result is
therefore not merely stale, it asserts the opposite of the intended behaviour: six-node
convergence would now be a *failure*.
Phase 6's own live gate ([`2026-07-24-mesh-phase6-live-gate.md`](2026-07-24-mesh-phase6-live-gate.md))
recorded its secrets leg (leg 6) as **"PASS (by fallback criterion)"** — it observed zero
secret errors fleet-wide and *inferred* pair-locality structurally from the mesh split. No
secret was ever seeded. **This gate replaces that inference with direct evidence**, in both
directions.
This is the OtOpcUa half of the production-topology decision in
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3) (design:
`scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`), which fixes
OtOpcUa production on **Akka pair-local** replication and required pair-local semantics to
be re-gated on a real three-mesh rig before enabling.
**Scope: config-only.** No product code was changed. One rig-config change was made (below).
## Rig state
| Fact | Value |
|---|---|
| Repo / branch | `OtOpcUa` @ `master` |
| Git SHA | `2100987453dfb44850f0deccbfb5583b49be6104` (`21009874`) |
| Image | `otopcua-host:dev` = `sha256:384ce9aa7bab`, built 2026-08-07 09:49 UTC |
| `OtOpcUa.Host.dll` | `a434b04f7d198e39…` — **byte-identical on all six nodes** |
| Secrets libs | `ZB.MOM.WW.Secrets*` **0.2.3** (`Directory.Packages.props`) |
| Replication | `Secrets__Replication__Enabled=true` on **all six** nodes (pre-existing, `x-secrets-env` anchor) |
| Announce interval | `00:00:05` (rig override; product default 30 s) |
| KEK | one shared committed **dev-only** key, identical on all six (`kek_id` sha256 `bb00d9490aa1…` on every row observed) |
| Meshes | three independent 2-node: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2` |
The rig was **rebuilt from current `master`** at the start of the run. It was found running
**mixed images** — the site nodes were on a 2026-08-06 17:55 build, predating `5b9ff9d0`
(#539) and `8b10d327` (#548), while the central pair was current. Rebuilding removed the
ambiguity; all six now report the same `Host.dll` hash. (The rebuild also confirmed the
build is deterministic: the freshly built `Host.dll` hash equals the one the already-current
central pair carried.)
Mesh formation after rebuild — each node handshakes **only** its pair partner:
```
site-a-1 Node [akka.tcp://otopcua@site-a-2:4053] is JOINING, roles [driver,cluster-SITE-A]
site-b-1 Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [driver,cluster-SITE-B]
central-1 Welcome from [akka.tcp://otopcua@central-2:4053]
```
### Rig-config change made by this gate
`docker-dev/docker-compose.yml` — `x-secrets-env` gained:
```yaml
Secrets__SqlitePath: "/app/data/otopcua-secrets.db"
```
The appsettings default is the relative `otopcua-secrets.db`, which resolves to `/app` —
**inside the image's writable container layer**. That cost two things, both of which
obstructed this gate: the store is destroyed by any `up -d --build` container recreate, and
it is unreachable from the host, so seeding or inspecting a node's store meant stopping the
container and `docker cp`-ing the file out. `/app/data` is the node's **existing per-node
volume** (the one LocalDb already uses) — still a private store per node, so replication is
still what syncs them, but it survives recreate and a helper container can mount it while
the node keeps running. Dev-rig config only; the product default is untouched.
## Method
Both driver-only site nodes have **no AdminUI**, so there is no in-app authoring surface on
them. Secrets were seeded and read with the `ZB.MOM.WW.Secrets.Cli` (`secret`) published
from `scadaproj/ZB.MOM.WW.Secrets` and run in a throwaway container mounting **one node's**
volume, with that node's KEK. This is the #482 method.
This exercises the **anti-entropy path**, not the live-push path — see Residual 1. Anti-entropy
is the library's own *correctness* path (`SecretReplicationActor`: every node publishes a
`SecretManifestAnnounce` each interval; a peer computes push and pull sets in one pass), and
it reads the node's live store, so a row written into that store by the CLI is announced by
the running node on its next tick.
Ciphertext identity is evidenced as a **SHA-256 over the encrypted columns only**
(`ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag`), plus a hash of `kek_id`. No
plaintext and no key material appears in this document. The secret values used were
throwaway smoke strings.
Two **isolation controls** were seeded alongside the subject secrets — one on `site-b-1`,
one on `central-1`. Without them, "the site-a secret never appeared on site-b" would be
equally consistent with site-b's replicator being dead. The controls prove each other pair's
replication is *alive* while still not crossing.
## Results
| # | Check | Result | Convergence |
|---|---|---|---|
| 1 | Seed on one site-a node ⇒ partner converges, byte-identical, decrypt-verified | **PASS** | **≤ 7 s** (budget 60 s) |
| 2 | No cross-pair propagation to site-b or central | **PASS** | absent across 240 s / 48 announce intervals |
| 3 | Delete propagates within the pair | **PASS** | **< 1 s** (budget 60 s) |
| 4 | Clean pair restart — no resurrection, live secret survives | **PASS** | mesh re-formed in 7 s |
Baseline before seeding: **all six stores empty** (clean slate from the rebuild).
### Check 1 — pair convergence · PASS
Seeded on `site-a-1` only, 09:53:02–09:53:03Z. First poll of `site-a-2` at 09:53:10Z already
had both rows, i.e. **≤ 7 s**, consistent with a single 5 s announce cycle.
```
site-a-1 pair-gate-smoke-1 rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00
pair-gate-survivor rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00
site-a-2 pair-gate-smoke-1 rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00
pair-gate-survivor rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00
```
Ciphertext hashes, `kek_id` and `updated_utc` are **identical** on both nodes — the row is
replicated verbatim, not re-encrypted, so the LWW ordering key is preserved. **Decrypt-verified
on the partner**: `secret get` on `site-a-2` returned the exact plaintext seeded on `site-a-1`
for both secrets.
### Check 2 — no cross-pair propagation · PASS
`site-b-1` and `central-1` were each seeded with their own control secret at 09:53:50–09:53:52Z.
`site-b-1/2` and `central-1/2` were then sampled **8 times over 240 s** (09:53:35Z → 09:57:46Z
— 48 announce intervals at 5 s). Every sample was identical:
```
site-b-1 pair-gate-siteb-probe rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1
site-b-2 pair-gate-siteb-probe rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1
central-1 pair-gate-central-probe rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1
central-2 pair-gate-central-probe rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1
```
- **Neither site-a secret ever appeared** on site-b or central, across all 8 samples.
- **The controls converged within their own pairs** (byte-identical, same `updated_utc`) —
so SITE-B and MAIN replication is demonstrably *alive*, and the absence above is isolation,
not a dead replicator.
- **Reverse direction also clean**: sampled at 09:54:25Z, the site-a pair held *only* its own
two secrets — neither control had crossed into it.
Isolation therefore holds **in every direction between all three pairs**. This is the correct
post-Phase-6 behaviour, not a defect.
### Check 3 — delete propagates within the pair · PASS
Deleted on **`site-a-2`** — deliberately the *receiving* node, so this also proves the
reverse intra-pair direction (`site-a-2` → `site-a-1`), which Check 1 did not cover.
Deleted 09:58:02Z; the tombstone was on `site-a-1` at the first poll, **< 1 s**:
```
site-a-1 pair-gate-smoke-1 rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00
site-a-2 pair-gate-smoke-1 rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00
```
Delete is a **soft tombstone** (`is_deleted=1`, `revision` bumped, `updated_utc` restamped),
which is what lets it win under last-writer-wins instead of being re-created by the peer's
older live row. `pair-gate-survivor` was untouched on both nodes.
### Check 4 — clean pair restart · PASS
`docker compose restart site-a-1 site-a-2` (both together — the supported pattern),
09:58:10Z → 09:58:20Z. The SITE-A mesh re-formed by 09:58:27Z (`site-a-1` founded,
`site-a-2` joined, both to `Up`) — **7 s**.
After restart, and re-confirmed at 10:06:41Z (> 60 s later, 12+ announce intervals):
- **No resurrection.** `pair-gate-smoke-1` remained `rev=1 deleted=1` on both nodes with its
`updated_utc` unchanged. `secret get` returns `{"error":"not-found"}`; `secret list` returns
`['pair-gate-survivor']` on both nodes.
- **Live secret survived and resolves.** `pair-gate-survivor` present on both nodes with its
original ciphertext hash and `updated_utc`, and **decrypted to the correct plaintext on both**.
Final fleet-wide sweep at 10:06:41Z: each pair holds exactly its own secrets and nothing else.
**Zero** secret-related warnings or errors on any of the six nodes across the entire run.
## Conclusion
Pair-local Akka secret replication is **live-proven on the three-mesh rig**: convergence and
deletion within a pair are well inside the 60 s budget, isolation between all three pairs
holds in every direction, and a clean pair restart neither resurrects a tombstone nor loses a
live secret. This satisfies the scadaproj#3 precondition for the OtOpcUa production topology
(Akka pair-local), and supersedes both the #482 six-node result and the Phase 6 leg-6
structural inference.
**Unchanged by this gate:** `Secrets:Replication:Enabled` remains **default-false in the
product**. It is enabled on the docker-dev rig only. Production enablement additionally
requires one fleet-wide KEK supplied via appsettings/env and **never committed** — the rig's
key is dev-only.
## Residuals
1. **The live-push path is not covered by this gate.** Seeding wrote into a node's store
underneath the running app, so what converged is **anti-entropy**
(`SecretManifestAnnounce` → push/pull), not the `PublishLocalWrite` fast path a write
through the app's own `ReplicatingSecretStore` takes. Anti-entropy is the library's
correctness path and the one that must hold, so this is the right thing to gate on a
driver-only node — but the fast path remains covered only by the library's own two-node
in-process cluster tests. Covering it live needs a write issued *through* a node's app,
which on the site pairs means an authoring surface that does not exist there today; the
central pair's AdminUI `/admin/secrets` page could cover it for MAIN.
2. **`appsettings.json` carries a stale, actively misleading comment.** The `Secrets:Replication`
block still reads *"KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet
0.2.0 … Do not enable until the library is fixed."* The repo has pinned **0.2.3** since
2026-07-19; `0.2.1` fixed the inert-registration defect and `0.2.2` fixed the DI deadlock
(scadaproj#1). The comment now tells an operator not to enable a feature this gate just
proved works. **Not corrected here** — this gate's mandate is config-only and that file is
product config, not rig config. Worth a one-line follow-up.
3. **Working tree was not clean, and another session was editing the repo concurrently.**
At the start of the run one unrelated file was dirty
(`docs/plans/2026-05-29-adminui-followups.md.tasks.json`, a docs-bookkeeping edit); by the
end a second had appeared
(`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs`).
Neither is product code on the gate path, and **the rig image was built from committed
`master` (`21009874`)**, not from the working tree, so the gated artifact is unaffected.
The commit carrying this gate is file-scoped and includes neither. Recorded for provenance
— a concurrent editing session is the condition under which a "rebuild the rig" step could
otherwise pick up someone else's in-flight change silently.
4. **Announce interval is 5 s on the rig, 30 s by product default.** The convergence figures
here (≤ 7 s, < 1 s) are bounded by the rig's shortened interval. At the product default,
expect worst-case convergence around one 30 s cycle — still inside the 60 s budget, but the
measured numbers should not be quoted as production latencies.
5. **Not re-gated: pair restart with a delete issued while one node is down.** Check 4 restarts
both nodes *after* both already held the tombstone. The harder resurrection case — deleting
while a peer is offline, then bringing it back with a live copy of the row — is covered by
the library's LWW tests but not live here.