docs(secrets): close scadaproj#4 — central shared SQL store + failover shipped
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
@@ -216,24 +216,41 @@ Which topology each app runs in production is decided — tracked as
|
||||
store, serves last-known-good secrets while central is unreachable, and converges on the first
|
||||
successful sweep. **The code path is merged and gated; production enablement is still a separate
|
||||
operator action** — `Secrets:Replication:Enabled` remains default-off, and `Secrets:Replication:Mode`
|
||||
defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately. Do not
|
||||
enable ScadaBridge in production until
|
||||
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is settled — the central
|
||||
pair does not converge with itself and the site client dials a single endpoint.
|
||||
defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately.
|
||||
- **The central pair runs on ONE shared SQL-Server store** (§1's `AddZbSecretsSqlServerStore`;
|
||||
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4), CLOSED 2026-08-07,
|
||||
Secrets family `0.5.x`, `ScadaBridge main @ e697477c`). Central in Grpc mode **requires**
|
||||
`Secrets:SqlServer:ConnectionString` — the boot fails closed naming the key if it is blank,
|
||||
and refuses a `${secret:}` value (the store's own connection string can never be a secret
|
||||
reference — bootstrap circularity; deliver it as a literal or environment value, never
|
||||
committed). Both hub instances serve the one store, so their manifests are identical **by
|
||||
construction** — the #4 divergence (an authenticated hub answering an empty manifest while
|
||||
its partner held every secret) is structurally unreproducible. Sites are unchanged (local
|
||||
SQLite, pull-only sweep). Live gate PASS 5/5:
|
||||
`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`.
|
||||
- **Site-side hub endpoint failover: `Secrets:GrpcHub:FallbackEndpoints`** (family `0.5.0`).
|
||||
List the **second central node's** hub endpoint there. The failover reader tries endpoints
|
||||
in rotation on transport-level gRPC errors, with sticky preference for the last endpoint
|
||||
that answered. **Same-store-only rule:** every listed endpoint must serve the SAME store —
|
||||
for ScadaBridge that is the central pair over the one shared SQL store. Never list endpoints
|
||||
backed by independent stores: failing over to an emptier hub is a silent convergence stop,
|
||||
the exact defect #4 recorded.
|
||||
- **Minimum version for central pairs: `0.5.1`.** `0.5.0`'s boot-time
|
||||
`SqlServerSecretsStoreMigrator` lost a concurrent first-boot race on a virgin database
|
||||
(error 2714 crashed one node when both centrals provisioned the empty DB simultaneously —
|
||||
the normal first deploy of a pair). `0.5.1` retries the race (1205/2714/1913/2627) and was
|
||||
re-drilled live: both centrals booting in the same second against a virgin DB, clean.
|
||||
- **KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override,
|
||||
never committed.** The committed dev KEK is dev-only. Verify per pair with the §0 digest
|
||||
compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
|
||||
- **Rotation fan-out:** re-wraps do not replicate, so `secret rewrap-all` runs against each
|
||||
node's store individually. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge**
|
||||
it is **both central nodes plus both nodes of every site pair** — every central node has its
|
||||
own independent local store and nothing replicates between them (live-observed: `central-b`
|
||||
answered an authenticated `GetManifest` with an empty manifest for a whole gate run while
|
||||
`central-a` held every secret;
|
||||
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)), so skipping the
|
||||
second central leaves a store that will be wrong the moment authority or endpoint failover
|
||||
moves to it. Note that the pull-only channel does not re-push re-wrapped rows either: a site
|
||||
node's own store must be re-wrapped directly. The KEK doctor is the recovery path for a
|
||||
missed node.
|
||||
independent store. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge** it is
|
||||
**ONE run against central's shared SQL store** — the CLI auto-targets it by reading central's
|
||||
own appsettings (`Secrets:SqlServer:ConnectionString`), and one run covers both central nodes
|
||||
because there is only one store — **plus both nodes of every site pair**. (The old "both
|
||||
central nodes" caveat is gone with the shared store; scadaproj#4.) Note that the pull-only
|
||||
channel does not re-push re-wrapped rows either: a site node's own store must be re-wrapped
|
||||
directly. The KEK doctor is the recovery path for a missed node.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -235,10 +235,52 @@ project 38/38 green.
|
||||
never deletes) but a **silent convergence stop**, and secrets authored on the central node the
|
||||
sites do not dial reach nobody. Folded into the same issue: delete-while-follower-offline is
|
||||
untested, and alerting on the denial Warning must sum per-window deltas because the warning is
|
||||
deliberately rate-limited. **ScadaBridge production enablement should wait on #4.**
|
||||
deliberately rate-limited. ~~**ScadaBridge production enablement should wait on #4.**~~
|
||||
*(Superseded — #4 CLOSED 2026-08-07; see the next section.)*
|
||||
|
||||
Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).
|
||||
|
||||
### Central shared SQL store + endpoint failover — scadaproj#4 CLOSED 2026-08-07
|
||||
|
||||
Plan: `docs/plans/2026-08-07-secrets-central-shared-sql-store.md` (scadaproj). Resolution:
|
||||
divergence removed **structurally**, not by replication — in `Secrets:Replication:Mode=Grpc`
|
||||
both ScadaBridge central nodes' `ISecretStore` is the **one shared SQL-Server store**
|
||||
(`AddZbSecretsSqlServerStore`), so both hubs serve identical manifests by construction and
|
||||
endpoint failover between them is safe.
|
||||
|
||||
- **Library `0.5.0`** (scadaproj `main @ 7a2980e` → released `e62b4a5`, 6 packages on the feed,
|
||||
restore-verified): `Secrets:GrpcHub:FallbackEndpoints` + `FailoverSecretsHubReader` — ordered
|
||||
rotation on `RpcException` only, sticky preference, same-store-only rule documented at the
|
||||
option; zero fallbacks keeps the registration graph byte-identical.
|
||||
- **Library `0.5.1`** (`main @ 31ca940`): `SqlServerSecretsStoreMigrator` virgin-DB concurrent
|
||||
bring-up race fixed (retry filter 1205/2714/1913/2627) — found live when the 0.5.0 first pass
|
||||
crashed central-a on the pair's simultaneous first boot. **Minimum version for central pairs.**
|
||||
- **ScadaBridge** `main @ 68f812ea` (shared store + Layer-A expander swap + fail-closed
|
||||
`EnsureCentralSharedStoreConnectionString`: central+Grpc requires
|
||||
`Secrets:SqlServer:ConnectionString`, refuses blank and `${secret:}` values; 472 offline
|
||||
tests) → `4d7f09d5` (gate doc + rig compose) → `e697477c` (0.5.1 pins). All pushed.
|
||||
- **Live gate PASS 5/5 + 0.5.1 re-drill**
|
||||
(`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair
|
||||
parity byte-identical (the #4 empty-manifest defect unreproducible by construction); failover
|
||||
both directions (18 s convergence with central-a down; recovered-primary wrap in one sweep
|
||||
tick; sticky, no flapping); **delete-while-follower-offline — the #4 residual — GATED**
|
||||
(tombstone on first sweep back, no resurrection); Layer-A expander proven reading the SHARED
|
||||
SQL store by three-way value discrimination; fail-closed negatives (boot refusal naming the
|
||||
key; SQL down → hub errors, never empty; sites keep last-known-good). Re-drill: virgin-DB
|
||||
simultaneous central bring-up clean on 0.5.1 (0.5.0 crashed reproducibly).
|
||||
- **Rewrap fan-out collapsed:** on ScadaBridge, `secret rewrap-all` is now ONE run against
|
||||
central's shared SQL store + both nodes of every site pair (runbook updated; the "both central
|
||||
nodes" caveat deleted). The denial-warning alerting note (sum per-window deltas) stands.
|
||||
- **Residual carried out:** [ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)
|
||||
— a pre-Serilog unhandled boot exception **wedges the container** (100% CPU spin, stays
|
||||
`running`, restart policy never fires). Pre-existing (old SQLite path too); reproduced twice
|
||||
during the gate. Not a secrets defect, but every pre-host fail-closed refusal lands in this
|
||||
window.
|
||||
|
||||
ScadaBridge pins `0.5.1`; the other three apps stay on the `0.2.3` libraries. Replication
|
||||
remains default-OFF in the product — production enablement is an operator action and no longer
|
||||
waits on #4.
|
||||
|
||||
### Upstream defect found during adoption — `0.2.0` Akka replication was inert
|
||||
|
||||
- ✅ **Fixed in `0.2.1`, published to the feed 2026-07-18** (all 5 packages; all four apps consume it).
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-08-07-secrets-central-shared-sql-store.md",
|
||||
"tasks": [
|
||||
{"id": 1, "subject": "Task 1: Library — FallbackEndpoints + failover hub reader", "status": "pending", "classification": "high-risk", "parallelizableWith": [2]},
|
||||
{"id": 2, "subject": "Task 2: ScadaBridge — central shared SQL store + Layer-A expander", "status": "pending", "classification": "high-risk", "parallelizableWith": [1]},
|
||||
{"id": 3, "subject": "Task 3: Publish family 0.5.0", "status": "pending", "classification": "small", "blockedBy": [1]},
|
||||
{"id": 4, "subject": "Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring", "status": "pending", "classification": "standard", "blockedBy": [2, 3]},
|
||||
{"id": 5, "subject": "Task 5: Live gate on the docker cluster", "status": "pending", "classification": "high-risk", "blockedBy": [4]},
|
||||
{"id": 6, "subject": "Task 6: Closeout — #4 close, runbook, GAPS, index", "status": "pending", "classification": "small", "blockedBy": [5]}
|
||||
{"id": 1, "subject": "Task 1: Library — FallbackEndpoints + failover hub reader", "status": "completed", "classification": "high-risk", "parallelizableWith": [2], "note": "FailoverSecretsHubReader + FallbackEndpoints validation + DI composition, zero-fallback shape byte-identical; scadaproj main @ 7a2980e."},
|
||||
{"id": 2, "subject": "Task 2: ScadaBridge — central shared SQL store + Layer-A expander", "status": "completed", "classification": "high-risk", "parallelizableWith": [1], "note": "AddZbSecretsSqlServerStore on central + EnsureCentralSharedStoreConnectionString fail-closed pre-checks + expander swap; 472 offline tests; ScadaBridge main @ 68f812ea."},
|
||||
{"id": 3, "subject": "Task 3: Publish family 0.5.0", "status": "completed", "classification": "small", "blockedBy": [1], "note": "0.5.0 published (6 pkgs, restore-verified) @ e62b4a5; 0.5.1 followed @ 31ca940 (virgin-DB concurrent migrator race fix found by the live gate)."},
|
||||
{"id": 4, "subject": "Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring", "status": "completed", "classification": "standard", "blockedBy": [2, 3], "note": "Pins + FallbackEndpoints wiring in 68f812ea; bumped to 0.5.1 @ e697477c after the gate. All pushed."},
|
||||
{"id": 5, "subject": "Task 5: Live gate on the docker cluster", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "PASS 5/5 + 0.5.1 virgin-DB re-drill (ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md, @ 4d7f09d5/e697477c); found the 0.5.1 migrator race + the pre-Serilog wedge (ScadaBridge#34)."},
|
||||
{"id": 6, "subject": "Task 6: Closeout — #4 close, runbook, GAPS, index", "status": "completed", "classification": "small", "blockedBy": [5], "note": "scadaproj#4 CLOSED (issuecomment-5751); wedge filed as ScadaBridge#34; clustered-secrets.md rewrap/topology rewrite; GAPS.md + CLAUDE.md Secrets row updated."}
|
||||
],
|
||||
"lastUpdated": "2026-08-07"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user