docs(secrets): close scadaproj#4 — central shared SQL store + failover shipped

Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).

- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
  never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
  FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 11:43:53 -04:00
parent 31ca9407a6
commit 46523db2bd
4 changed files with 81 additions and 22 deletions
@@ -216,24 +216,41 @@ Which topology each app runs in production is decided — tracked as
store, serves last-known-good secrets while central is unreachable, and converges on the first
successful sweep. **The code path is merged and gated; production enablement is still a separate
operator action** — `Secrets:Replication:Enabled` remains default-off, and `Secrets:Replication:Mode`
defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately. Do not
enable ScadaBridge in production until
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is settled — the central
pair does not converge with itself and the site client dials a single endpoint.
defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately.
- **The central pair runs on ONE shared SQL-Server store** (§1's `AddZbSecretsSqlServerStore`;
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4), CLOSED 2026-08-07,
Secrets family `0.5.x`, `ScadaBridge main @ e697477c`). Central in Grpc mode **requires**
`Secrets:SqlServer:ConnectionString` — the boot fails closed naming the key if it is blank,
and refuses a `${secret:}` value (the store's own connection string can never be a secret
reference — bootstrap circularity; deliver it as a literal or environment value, never
committed). Both hub instances serve the one store, so their manifests are identical **by
construction** — the #4 divergence (an authenticated hub answering an empty manifest while
its partner held every secret) is structurally unreproducible. Sites are unchanged (local
SQLite, pull-only sweep). Live gate PASS 5/5:
`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`.
- **Site-side hub endpoint failover: `Secrets:GrpcHub:FallbackEndpoints`** (family `0.5.0`).
List the **second central node's** hub endpoint there. The failover reader tries endpoints
in rotation on transport-level gRPC errors, with sticky preference for the last endpoint
that answered. **Same-store-only rule:** every listed endpoint must serve the SAME store —
for ScadaBridge that is the central pair over the one shared SQL store. Never list endpoints
backed by independent stores: failing over to an emptier hub is a silent convergence stop,
the exact defect #4 recorded.
- **Minimum version for central pairs: `0.5.1`.** `0.5.0`'s boot-time
`SqlServerSecretsStoreMigrator` lost a concurrent first-boot race on a virgin database
(error 2714 crashed one node when both centrals provisioned the empty DB simultaneously —
the normal first deploy of a pair). `0.5.1` retries the race (1205/2714/1913/2627) and was
re-drilled live: both centrals booting in the same second against a virgin DB, clean.
- **KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override,
never committed.** The committed dev KEK is dev-only. Verify per pair with the §0 digest
compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
- **Rotation fan-out:** re-wraps do not replicate, so `secret rewrap-all` runs against each
node's store individually. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge**
it is **both central nodes plus both nodes of every site pair** — every central node has its
own independent local store and nothing replicates between them (live-observed: `central-b`
answered an authenticated `GetManifest` with an empty manifest for a whole gate run while
`central-a` held every secret;
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)), so skipping the
second central leaves a store that will be wrong the moment authority or endpoint failover
moves to it. Note that the pull-only channel does not re-push re-wrapped rows either: a site
node's own store must be re-wrapped directly. The KEK doctor is the recovery path for a
missed node.
independent store. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge** it is
**ONE run against central's shared SQL store** — the CLI auto-targets it by reading central's
own appsettings (`Secrets:SqlServer:ConnectionString`), and one run covers both central nodes
because there is only one store — **plus both nodes of every site pair**. (The old "both
central nodes" caveat is gone with the shared store; scadaproj#4.) Note that the pull-only
channel does not re-push re-wrapped rows either: a site node's own store must be re-wrapped
directly. The KEK doctor is the recovery path for a missed node.
---