diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 3e6cbe4f..c4c69027 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -115,8 +115,19 @@ name: otopcua-dev # OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d # NEVER reuse this key outside this local dev rig. # AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is -# observable quickly during dev exercise. Each node keeps its own local SQLite store -# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them. +# observable quickly during dev exercise. Each node keeps its OWN local SQLite store; +# replication syncs them. +# +# SqlitePath is overridden onto the node's existing per-node `/app/data` volume (the one +# LocalDb already uses) instead of the appsettings default `otopcua-secrets.db`, which +# resolves to /app — inside the image's writable container layer. Two things that costs, +# both of which bit the 2026-08-07 pair-local gate: the store is DESTROYED by any +# `up -d --build` container recreate (so a seeded row cannot outlive a rig rebuild), and +# it is unreachable from the host, so seeding or inspecting a node's store meant stopping +# the container and `docker cp`-ing the file out. On the volume, each node still has a +# private store (one volume per node — this is NOT shared storage, and replication is +# still what syncs them), but it survives recreate and a helper container can mount it +# read/write while the node keeps running. # # Per-cluster mesh Phase 6: this DPS topic now rides each pair's OWN 2-node mesh, not one # shared six-node mesh — so secret replication is PAIR-LOCAL. central-1/central-2 replicate @@ -127,6 +138,7 @@ name: otopcua-dev # was needed for this — Enabled=true + the same KEK on every node is still correct; it just now # converges within three independent 2-node groups instead of one six-node group. x-secrets-env: &secrets-env + Secrets__SqlitePath: "/app/data/otopcua-secrets.db" Secrets__Replication__Enabled: "true" Secrets__Replication__AnnounceInterval: "00:00:05" ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}" diff --git a/docs/plans/2026-08-07-secrets-pair-local-live-gate.md b/docs/plans/2026-08-07-secrets-pair-local-live-gate.md new file mode 100644 index 00000000..b3c56460 --- /dev/null +++ b/docs/plans/2026-08-07-secrets-pair-local-live-gate.md @@ -0,0 +1,239 @@ +# Secrets replication — pair-local live gate on the three-mesh rig + +**Result: PASS — 4/4 checks.** Run 2026-08-07 on the docker-dev rig. + +## Purpose + +Re-gate **pair-local** Akka peer-to-peer secret replication on the current +three-mesh docker-dev rig, and **supersede the 2026-07-18 six-node gate +([lmxopcua#482](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/482), closed at +`master @ c878fbbd`) as evidence.** + +That gate proved a secret seeded on `site-a-1` converged to **all five** peers. It ran on +the old single fleet-wide Akka mesh. **Mesh Phase 6** (`2839deb5`, 2026-07-24) partitioned +the rig into three independent 2-node meshes, so `DistributedPubSub` — and with it the +`zb-mom-ww-secrets` topic — is now **pair-local by construction**. The #482 result is +therefore not merely stale, it asserts the opposite of the intended behaviour: six-node +convergence would now be a *failure*. + +Phase 6's own live gate ([`2026-07-24-mesh-phase6-live-gate.md`](2026-07-24-mesh-phase6-live-gate.md)) +recorded its secrets leg (leg 6) as **"PASS (by fallback criterion)"** — it observed zero +secret errors fleet-wide and *inferred* pair-locality structurally from the mesh split. No +secret was ever seeded. **This gate replaces that inference with direct evidence**, in both +directions. + +This is the OtOpcUa half of the production-topology decision in +[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3) (design: +`scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`), which fixes +OtOpcUa production on **Akka pair-local** replication and required pair-local semantics to +be re-gated on a real three-mesh rig before enabling. + +**Scope: config-only.** No product code was changed. One rig-config change was made (below). + +## Rig state + +| Fact | Value | +|---|---| +| Repo / branch | `OtOpcUa` @ `master` | +| Git SHA | `2100987453dfb44850f0deccbfb5583b49be6104` (`21009874`) | +| Image | `otopcua-host:dev` = `sha256:384ce9aa7bab`, built 2026-08-07 09:49 UTC | +| `OtOpcUa.Host.dll` | `a434b04f7d198e39…` — **byte-identical on all six nodes** | +| Secrets libs | `ZB.MOM.WW.Secrets*` **0.2.3** (`Directory.Packages.props`) | +| Replication | `Secrets__Replication__Enabled=true` on **all six** nodes (pre-existing, `x-secrets-env` anchor) | +| Announce interval | `00:00:05` (rig override; product default 30 s) | +| KEK | one shared committed **dev-only** key, identical on all six (`kek_id` sha256 `bb00d9490aa1…` on every row observed) | +| Meshes | three independent 2-node: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2` | + +The rig was **rebuilt from current `master`** at the start of the run. It was found running +**mixed images** — the site nodes were on a 2026-08-06 17:55 build, predating `5b9ff9d0` +(#539) and `8b10d327` (#548), while the central pair was current. Rebuilding removed the +ambiguity; all six now report the same `Host.dll` hash. (The rebuild also confirmed the +build is deterministic: the freshly built `Host.dll` hash equals the one the already-current +central pair carried.) + +Mesh formation after rebuild — each node handshakes **only** its pair partner: + +``` +site-a-1 Node [akka.tcp://otopcua@site-a-2:4053] is JOINING, roles [driver,cluster-SITE-A] +site-b-1 Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [driver,cluster-SITE-B] +central-1 Welcome from [akka.tcp://otopcua@central-2:4053] +``` + +### Rig-config change made by this gate + +`docker-dev/docker-compose.yml` — `x-secrets-env` gained: + +```yaml +Secrets__SqlitePath: "/app/data/otopcua-secrets.db" +``` + +The appsettings default is the relative `otopcua-secrets.db`, which resolves to `/app` — +**inside the image's writable container layer**. That cost two things, both of which +obstructed this gate: the store is destroyed by any `up -d --build` container recreate, and +it is unreachable from the host, so seeding or inspecting a node's store meant stopping the +container and `docker cp`-ing the file out. `/app/data` is the node's **existing per-node +volume** (the one LocalDb already uses) — still a private store per node, so replication is +still what syncs them, but it survives recreate and a helper container can mount it while +the node keeps running. Dev-rig config only; the product default is untouched. + +## Method + +Both driver-only site nodes have **no AdminUI**, so there is no in-app authoring surface on +them. Secrets were seeded and read with the `ZB.MOM.WW.Secrets.Cli` (`secret`) published +from `scadaproj/ZB.MOM.WW.Secrets` and run in a throwaway container mounting **one node's** +volume, with that node's KEK. This is the #482 method. + +This exercises the **anti-entropy path**, not the live-push path — see Residual 1. Anti-entropy +is the library's own *correctness* path (`SecretReplicationActor`: every node publishes a +`SecretManifestAnnounce` each interval; a peer computes push and pull sets in one pass), and +it reads the node's live store, so a row written into that store by the CLI is announced by +the running node on its next tick. + +Ciphertext identity is evidenced as a **SHA-256 over the encrypted columns only** +(`ciphertext‖nonce‖tag‖wrapped_dek‖wrap_nonce‖wrap_tag`), plus a hash of `kek_id`. No +plaintext and no key material appears in this document. The secret values used were +throwaway smoke strings. + +Two **isolation controls** were seeded alongside the subject secrets — one on `site-b-1`, +one on `central-1`. Without them, "the site-a secret never appeared on site-b" would be +equally consistent with site-b's replicator being dead. The controls prove each other pair's +replication is *alive* while still not crossing. + +## Results + +| # | Check | Result | Convergence | +|---|---|---|---| +| 1 | Seed on one site-a node ⇒ partner converges, byte-identical, decrypt-verified | **PASS** | **≤ 7 s** (budget 60 s) | +| 2 | No cross-pair propagation to site-b or central | **PASS** | absent across 240 s / 48 announce intervals | +| 3 | Delete propagates within the pair | **PASS** | **< 1 s** (budget 60 s) | +| 4 | Clean pair restart — no resurrection, live secret survives | **PASS** | mesh re-formed in 7 s | + +Baseline before seeding: **all six stores empty** (clean slate from the rebuild). + +### Check 1 — pair convergence · PASS + +Seeded on `site-a-1` only, 09:53:02–09:53:03Z. First poll of `site-a-2` at 09:53:10Z already +had both rows, i.e. **≤ 7 s**, consistent with a single 5 s announce cycle. + +``` +site-a-1 pair-gate-smoke-1 rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00 + pair-gate-survivor rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00 +site-a-2 pair-gate-smoke-1 rev=0 deleted=0 cipher_sha256=dd7c08f4c6e1fda7cae900e6 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.0367742+00:00 + pair-gate-survivor rev=0 deleted=0 cipher_sha256=a373b6f2b985315f70b3ac34 kek=bb00d9490aa1 updated=2026-08-07T09:53:02.9838162+00:00 +``` + +Ciphertext hashes, `kek_id` and `updated_utc` are **identical** on both nodes — the row is +replicated verbatim, not re-encrypted, so the LWW ordering key is preserved. **Decrypt-verified +on the partner**: `secret get` on `site-a-2` returned the exact plaintext seeded on `site-a-1` +for both secrets. + +### Check 2 — no cross-pair propagation · PASS + +`site-b-1` and `central-1` were each seeded with their own control secret at 09:53:50–09:53:52Z. +`site-b-1/2` and `central-1/2` were then sampled **8 times over 240 s** (09:53:35Z → 09:57:46Z +— 48 announce intervals at 5 s). Every sample was identical: + +``` +site-b-1 pair-gate-siteb-probe rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1 +site-b-2 pair-gate-siteb-probe rev=0 deleted=0 cipher_sha256=87ca3d3ee8b00df59ef51b73 kek=bb00d9490aa1 +central-1 pair-gate-central-probe rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1 +central-2 pair-gate-central-probe rev=0 deleted=0 cipher_sha256=4f09d442b742907322546f94 kek=bb00d9490aa1 +``` + +- **Neither site-a secret ever appeared** on site-b or central, across all 8 samples. +- **The controls converged within their own pairs** (byte-identical, same `updated_utc`) — + so SITE-B and MAIN replication is demonstrably *alive*, and the absence above is isolation, + not a dead replicator. +- **Reverse direction also clean**: sampled at 09:54:25Z, the site-a pair held *only* its own + two secrets — neither control had crossed into it. + +Isolation therefore holds **in every direction between all three pairs**. This is the correct +post-Phase-6 behaviour, not a defect. + +### Check 3 — delete propagates within the pair · PASS + +Deleted on **`site-a-2`** — deliberately the *receiving* node, so this also proves the +reverse intra-pair direction (`site-a-2` → `site-a-1`), which Check 1 did not cover. + +Deleted 09:58:02Z; the tombstone was on `site-a-1` at the first poll, **< 1 s**: + +``` +site-a-1 pair-gate-smoke-1 rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00 +site-a-2 pair-gate-smoke-1 rev=1 deleted=1 cipher_sha256=dd7c08f4c6e1fda7cae900e6 updated=2026-08-07T09:58:02.1938781+00:00 +``` + +Delete is a **soft tombstone** (`is_deleted=1`, `revision` bumped, `updated_utc` restamped), +which is what lets it win under last-writer-wins instead of being re-created by the peer's +older live row. `pair-gate-survivor` was untouched on both nodes. + +### Check 4 — clean pair restart · PASS + +`docker compose restart site-a-1 site-a-2` (both together — the supported pattern), +09:58:10Z → 09:58:20Z. The SITE-A mesh re-formed by 09:58:27Z (`site-a-1` founded, +`site-a-2` joined, both to `Up`) — **7 s**. + +After restart, and re-confirmed at 10:06:41Z (> 60 s later, 12+ announce intervals): + +- **No resurrection.** `pair-gate-smoke-1` remained `rev=1 deleted=1` on both nodes with its + `updated_utc` unchanged. `secret get` returns `{"error":"not-found"}`; `secret list` returns + `['pair-gate-survivor']` on both nodes. +- **Live secret survived and resolves.** `pair-gate-survivor` present on both nodes with its + original ciphertext hash and `updated_utc`, and **decrypted to the correct plaintext on both**. + +Final fleet-wide sweep at 10:06:41Z: each pair holds exactly its own secrets and nothing else. +**Zero** secret-related warnings or errors on any of the six nodes across the entire run. + +## Conclusion + +Pair-local Akka secret replication is **live-proven on the three-mesh rig**: convergence and +deletion within a pair are well inside the 60 s budget, isolation between all three pairs +holds in every direction, and a clean pair restart neither resurrects a tombstone nor loses a +live secret. This satisfies the scadaproj#3 precondition for the OtOpcUa production topology +(Akka pair-local), and supersedes both the #482 six-node result and the Phase 6 leg-6 +structural inference. + +**Unchanged by this gate:** `Secrets:Replication:Enabled` remains **default-false in the +product**. It is enabled on the docker-dev rig only. Production enablement additionally +requires one fleet-wide KEK supplied via appsettings/env and **never committed** — the rig's +key is dev-only. + +## Residuals + +1. **The live-push path is not covered by this gate.** Seeding wrote into a node's store + underneath the running app, so what converged is **anti-entropy** + (`SecretManifestAnnounce` → push/pull), not the `PublishLocalWrite` fast path a write + through the app's own `ReplicatingSecretStore` takes. Anti-entropy is the library's + correctness path and the one that must hold, so this is the right thing to gate on a + driver-only node — but the fast path remains covered only by the library's own two-node + in-process cluster tests. Covering it live needs a write issued *through* a node's app, + which on the site pairs means an authoring surface that does not exist there today; the + central pair's AdminUI `/admin/secrets` page could cover it for MAIN. + +2. **`appsettings.json` carries a stale, actively misleading comment.** The `Secrets:Replication` + block still reads *"KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet + 0.2.0 … Do not enable until the library is fixed."* The repo has pinned **0.2.3** since + 2026-07-19; `0.2.1` fixed the inert-registration defect and `0.2.2` fixed the DI deadlock + (scadaproj#1). The comment now tells an operator not to enable a feature this gate just + proved works. **Not corrected here** — this gate's mandate is config-only and that file is + product config, not rig config. Worth a one-line follow-up. + +3. **Working tree was not clean, and another session was editing the repo concurrently.** + At the start of the run one unrelated file was dirty + (`docs/plans/2026-05-29-adminui-followups.md.tasks.json`, a docs-bookkeeping edit); by the + end a second had appeared + (`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs`). + Neither is product code on the gate path, and **the rig image was built from committed + `master` (`21009874`)**, not from the working tree, so the gated artifact is unaffected. + The commit carrying this gate is file-scoped and includes neither. Recorded for provenance + — a concurrent editing session is the condition under which a "rebuild the rig" step could + otherwise pick up someone else's in-flight change silently. + +4. **Announce interval is 5 s on the rig, 30 s by product default.** The convergence figures + here (≤ 7 s, < 1 s) are bounded by the rig's shortened interval. At the product default, + expect worst-case convergence around one 30 s cycle — still inside the 60 s budget, but the + measured numbers should not be quoted as production latencies. + +5. **Not re-gated: pair restart with a delete issued while one node is down.** Check 4 restarts + both nodes *after* both already held the tombstone. The harder resurrection case — deleting + while a peer is offline, then bringing it back with a live copy of the row — is covered by + the library's LWW tests but not live here.