docs(secrets): pair-local replication live gate on the three-mesh rig — PASS 4/4
v2-ci / build (push) Successful in 6m33s
v2-ci / unit-tests (push) Failing after 17m17s

Re-gates Akka peer-to-peer secret replication on the current three-mesh docker-dev
rig and supersedes the 2026-07-18 six-node gate (lmxopcua#482, c878fbbd) as evidence.
That gate proved convergence to all five peers on the old single fleet-wide mesh;
after mesh Phase 6 (2839deb5) partitioned the rig into three independent 2-node
meshes, six-node convergence would be a FAILURE, not a pass. Phase 6's own leg 6
recorded "PASS (by fallback criterion)" — it inferred pair-locality structurally and
never seeded a secret. This replaces that inference with direct evidence.

All four checks PASS, on master @ 21009874 with one image (OtOpcUa.Host.dll
a434b04f byte-identical on all six nodes), Secrets libs 0.2.3:

  1. Pair convergence — seeded on site-a-1 alone, on site-a-2 in <=7s (budget 60s),
     byte-identical ciphertext + kek_id + updated_utc, decrypt-verified on the partner.
  2. No cross-pair propagation — absent from site-b and central across 8 samples /
     240s / 48 announce intervals, and clean in the reverse direction too. Isolation
     controls seeded on site-b-1 and central-1 converged within their OWN pairs, so
     the absence is isolation rather than a dead replicator.
  3. Delete propagates within the pair — issued on site-a-2 (the receiving node, so
     this also covers the reverse intra-pair direction); tombstone on site-a-1 in <1s.
  4. Clean pair restart (both together) — mesh re-formed in 7s, no resurrection of the
     tombstone, live secret survived and decrypted correctly on both nodes.

Zero secret-related warnings or errors fleet-wide across the whole run. Satisfies the
scadaproj#3 precondition for the OtOpcUa production topology (Akka pair-local).
Secrets:Replication:Enabled remains default-false in the product; the rig KEK is dev-only.

Rig config: SqlitePath moved onto each node's existing per-node /app/data volume. The
appsettings default resolves to /app, inside the image's writable container layer, so
the store was destroyed by any `up -d --build` recreate and was unreachable from the
host — seeding or inspecting a node meant stopping the container to docker cp the file
out. Still one private store per node; replication is still what syncs them.

Residuals in the gate doc: the live-push path is not covered (anti-entropy is what a
driver-only node can be gated on), and appsettings.json still carries a stale
"KNOWN NON-FUNCTIONAL against 0.2.0 / do not enable" comment that this gate disproves
— left uncorrected here because this run is config-only.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 06:08:50 -04:00
parent 2100987453
commit 1bd79c8e4a
2 changed files with 253 additions and 2 deletions
+14 -2
View File
@@ -115,8 +115,19 @@ name: otopcua-dev
# OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
# NEVER reuse this key outside this local dev rig.
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
# observable quickly during dev exercise. Each node keeps its own local SQLite store
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
# observable quickly during dev exercise. Each node keeps its OWN local SQLite store;
# replication syncs them.
#
# SqlitePath is overridden onto the node's existing per-node `/app/data` volume (the one
# LocalDb already uses) instead of the appsettings default `otopcua-secrets.db`, which
# resolves to /app — inside the image's writable container layer. Two things that costs,
# both of which bit the 2026-08-07 pair-local gate: the store is DESTROYED by any
# `up -d --build` container recreate (so a seeded row cannot outlive a rig rebuild), and
# it is unreachable from the host, so seeding or inspecting a node's store meant stopping
# the container and `docker cp`-ing the file out. On the volume, each node still has a
# private store (one volume per node — this is NOT shared storage, and replication is
# still what syncs them), but it survives recreate and a helper container can mount it
# read/write while the node keeps running.
#
# Per-cluster mesh Phase 6: this DPS topic now rides each pair's OWN 2-node mesh, not one
# shared six-node mesh — so secret replication is PAIR-LOCAL. central-1/central-2 replicate
@@ -127,6 +138,7 @@ name: otopcua-dev
# was needed for this — Enabled=true + the same KEK on every node is still correct; it just now
# converges within three independent 2-node groups instead of one six-node group.
x-secrets-env: &secrets-env
Secrets__SqlitePath: "/app/data/otopcua-secrets.db"
Secrets__Replication__Enabled: "true"
Secrets__Replication__AnnounceInterval: "00:00:05"
ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"