docs(secrets): close scadaproj#4 — central shared SQL store + failover shipped

Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).

- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
  never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
  FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 11:43:53 -04:00
parent 31ca9407a6
commit 46523db2bd
4 changed files with 81 additions and 22 deletions
+43 -1
View File
@@ -235,10 +235,52 @@ project 38/38 green.
never deletes) but a **silent convergence stop**, and secrets authored on the central node the
sites do not dial reach nobody. Folded into the same issue: delete-while-follower-offline is
untested, and alerting on the denial Warning must sum per-window deltas because the warning is
deliberately rate-limited. **ScadaBridge production enablement should wait on #4.**
deliberately rate-limited. ~~**ScadaBridge production enablement should wait on #4.**~~
*(Superseded — #4 CLOSED 2026-08-07; see the next section.)*
Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).
### Central shared SQL store + endpoint failover — scadaproj#4 CLOSED 2026-08-07
Plan: `docs/plans/2026-08-07-secrets-central-shared-sql-store.md` (scadaproj). Resolution:
divergence removed **structurally**, not by replication — in `Secrets:Replication:Mode=Grpc`
both ScadaBridge central nodes' `ISecretStore` is the **one shared SQL-Server store**
(`AddZbSecretsSqlServerStore`), so both hubs serve identical manifests by construction and
endpoint failover between them is safe.
- **Library `0.5.0`** (scadaproj `main @ 7a2980e` → released `e62b4a5`, 6 packages on the feed,
restore-verified): `Secrets:GrpcHub:FallbackEndpoints` + `FailoverSecretsHubReader` — ordered
rotation on `RpcException` only, sticky preference, same-store-only rule documented at the
option; zero fallbacks keeps the registration graph byte-identical.
- **Library `0.5.1`** (`main @ 31ca940`): `SqlServerSecretsStoreMigrator` virgin-DB concurrent
bring-up race fixed (retry filter 1205/2714/1913/2627) — found live when the 0.5.0 first pass
crashed central-a on the pair's simultaneous first boot. **Minimum version for central pairs.**
- **ScadaBridge** `main @ 68f812ea` (shared store + Layer-A expander swap + fail-closed
`EnsureCentralSharedStoreConnectionString`: central+Grpc requires
`Secrets:SqlServer:ConnectionString`, refuses blank and `${secret:}` values; 472 offline
tests) → `4d7f09d5` (gate doc + rig compose) → `e697477c` (0.5.1 pins). All pushed.
- **Live gate PASS 5/5 + 0.5.1 re-drill**
(`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair
parity byte-identical (the #4 empty-manifest defect unreproducible by construction); failover
both directions (18 s convergence with central-a down; recovered-primary wrap in one sweep
tick; sticky, no flapping); **delete-while-follower-offline — the #4 residual — GATED**
(tombstone on first sweep back, no resurrection); Layer-A expander proven reading the SHARED
SQL store by three-way value discrimination; fail-closed negatives (boot refusal naming the
key; SQL down → hub errors, never empty; sites keep last-known-good). Re-drill: virgin-DB
simultaneous central bring-up clean on 0.5.1 (0.5.0 crashed reproducibly).
- **Rewrap fan-out collapsed:** on ScadaBridge, `secret rewrap-all` is now ONE run against
central's shared SQL store + both nodes of every site pair (runbook updated; the "both central
nodes" caveat deleted). The denial-warning alerting note (sum per-window deltas) stands.
- **Residual carried out:** [ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)
— a pre-Serilog unhandled boot exception **wedges the container** (100% CPU spin, stays
`running`, restart policy never fires). Pre-existing (old SQLite path too); reproduced twice
during the gate. Not a secrets defect, but every pre-host fail-closed refusal lands in this
window.
ScadaBridge pins `0.5.1`; the other three apps stay on the `0.2.3` libraries. Replication
remains default-OFF in the product — production enablement is an operator action and no longer
waits on #4.
### Upstream defect found during adoption — `0.2.0` Akka replication was inert
- ✅ **Fixed in `0.2.1`, published to the feed 2026-07-18** (all 5 packages; all four apps consume it).