docs(secrets): close out scadaproj#3 — gRPC hub shipped 0.4.1, both live gates PASS; residual filed as #4

Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.

- clustered-secrets.md, Production enablement: the gRPC hub is no longer
  pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
  main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
  "until that ships, ScadaBridge replication stays off" line is replaced with
  the truth: the code path is merged and gated, and production enablement is a
  separate operator action behind a still-default-off flag. OtOpcUa's
  pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
  rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
  BOTH central nodes plus both nodes of every site pair, because each central
  node has its own independent store.

- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
  0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
  residual. Existing narrative left intact and amended, not rewritten.

- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
  corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
  still pin 0.2.3); the "production topologies DECIDED" region now records
  delivery, both gates, default-off, and the residual.

- Plan ledger: task 9 completed.

Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 09:00:11 -04:00
parent c86cead5c2
commit 5198d3dc7f
4 changed files with 91 additions and 16 deletions
@@ -196,21 +196,44 @@ Which topology each app runs in production is decided — tracked as
- **OtOpcUa: Akka peer-to-peer, pair-local.** Each application cluster is its own 2-node Akka
mesh (mesh Phase 6), so replication converges within a pair and nowhere else — seed each pair
independently (central via `/admin/secrets`; site pairs via the `secret` CLI or ciphertext-only
bundle import).
bundle import). **This topology is live-gated as such** — PASS 4/4 on the three-mesh rig
(`OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`, `lmxopcua master @ 1bd79c8e`):
pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair
propagation proven with seeded isolation controls on the other pairs** rather than by mere
absence. It supersedes the 2026-07-18 six-node gate as evidence, which predates the mesh
Phase 6 partition.
- **ScadaBridge: central gRPC hub (§2's SQL-hub mode is NOT used).** Site servers never talk to
the DB directly — they talk to central for everything — so the SqlServer hub's site-side
connection string violates the topology rule. A `ZB.MOM.WW.Secrets.Replicator.Grpc` package
(central-hosted `secrets_hub.v1`, **pull-only: updates originate at central only, by design**)
replaces it; until that ships, ScadaBridge replication stays off. The gRPC mode is **cached on
the site clusters** exactly like §2's hub mode — each site node keeps a full local store,
serves last-known-good secrets while central is unreachable, and converges on the first
successful sweep.
connection string violates the topology rule. The `ZB.MOM.WW.Secrets.Replicator.Grpc` package
(central-hosted `secrets_hub.v1`, **pull-only: updates originate at central only, by design** —
exactly two RPCs, `GetManifest`/`GetSecrets`, with no write RPC at the contract level) replaces
it. **It is BUILT, SHIPPED (`0.4.1`) and LIVE-GATED PASS 4/4** — see
`ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md` (`ScadaBridge main @ f6c3f7c5`):
convergence in 5 s, byte-identical and decrypt-verified on both site-a nodes; a cold boot with
central down serves last-known-good and resumes; tombstones propagate ≤9 s with no resurrection;
auth negatives deny uniformly with zero credential material in the logs. The gRPC mode is
**cached on the site clusters** exactly like §2's hub mode — each site node keeps a full local
store, serves last-known-good secrets while central is unreachable, and converges on the first
successful sweep. **The code path is merged and gated; production enablement is still a separate
operator action** — `Secrets:Replication:Enabled` remains default-off, and `Secrets:Replication:Mode`
defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately. Do not
enable ScadaBridge in production until
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is settled — the central
pair does not converge with itself and the site client dials a single endpoint.
- **KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override,
never committed.** The committed dev KEK is dev-only. Verify per pair with the §0 digest
compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
- **Rotation fan-out:** re-wraps do not replicate, so `secret rewrap-all` runs against each
node's store individually (both nodes of every pair; on ScadaBridge also central). The KEK
doctor is the recovery path for a missed node.
node's store individually. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge**
it is **both central nodes plus both nodes of every site pair** — every central node has its
own independent local store and nothing replicates between them (live-observed: `central-b`
answered an authenticated `GetManifest` with an empty manifest for a whole gate run while
`central-a` held every secret;
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)), so skipping the
second central leaves a store that will be wrong the moment authority or endpoint failover
moves to it. Note that the pull-only channel does not re-push re-wrapped rows either: a site
node's own store must be re-wrapped directly. The KEK doctor is the recovery path for a
missed node.
---