ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md

# Secrets central shared SQL store + hub failover — live gate on the docker cluster

**Result: PASS — 5/5 checks.** Run 2026-08-07 on the local 8-node docker cluster.
Two defects were found and are **documented, not patched** (standing rule): one in the
`ZB.MOM.WW.Secrets.Replicator.SqlServer` library (concurrent first-boot migration race),
one in the ScadaBridge Host's pre-Serilog crash path (unhandled boot exception wedges the
process instead of exiting). Neither is a failure of a gate check — every check's own
criterion passed — but both need a controller decision. See [Defects](#defects-found).

## Purpose

Terminal gate (Task 5) for the central-shared-SQL-store program
(`scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md`, resolving
[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4)): in
`Secrets:Replication:Mode=Grpc`, both central nodes' `ISecretStore` is now the **shared
SQL-Server store** (`AddZbSecretsSqlServerStore`), so the two hub instances serve identical
manifests **by construction**, and the site-side hub client gains ordered
`Secrets:GrpcHub:FallbackEndpoints` with sticky preference (Secrets family **0.5.0**).

This gate also **discharges the code-review residual that the `Program.cs` SQL expander
path has no offline test**: check 4 exercises the `expanderUsesSharedSqlStore` branch
positively (Layer-A `${secret:}` resolution from the shared SQL store, including the
`SqlServerSecretsStoreMigrator` boot run), and check 5(a) exercises its fail-closed
negative (`EnsureCentralSharedStoreConnectionString` throwing the designed message from
`Program.cs` line 84 — the expander block — before any `SqlConnection` sees the value).

**Scope: rig config only.** No product code was changed by this gate.

## Rig state

| Fact | Value |
|---|---|
| Repo / branch | `ScadaBridge` @ `main` |
| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) |
| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z |
| `ZB.MOM.WW.ScadaBridge.Host.dll` | `9dba0869ff4dfa7c…` — byte-identical on all 4 participating nodes |
| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…`, informational version **`0.5.0+7a2980e`**, `FailoverSecretsHubReader` present — byte-identical on all 4 |
| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** |
| Hub enabled on | `central-a`, `central-b` (hub, **shared SQL store**) and `site-a-a`, `site-a-b` (followers, local SQLite) |
| Hub NOT enabled on | `site-b-*`, `site-c-*` — verified: zero `Secrets__*` env vars, zero hub/sweep log lines |
| Sweep interval | `00:00:30` — the product default, not shortened; timings below are real |
| KEK | the same shared committed **dev-only** key as the 2026-08-07 hub gate (`kek_id` sha256 `7451bcbc1f1f` on every row observed) |
| Working tree at build | only `docker/docker-compose.yml` modified (rig config, not in the image) + untracked `.claude/` |

### Rig DB choice (record)

Central's shared store is a **dedicated database `ZbSecretsHub`** on the rig's existing
`scadabridge-mssql` container (the same SQL Server ScadaBridge central already uses),
default `zbsecrets` schema, tables created by the boot-time `SqlServerSecretsStoreMigrator`
(`schema_version` = 1, `secret`). The database itself and the `scadabridge_app` user grant
(db_owner on that DB only) were provisioned once via `sqlcmd` in the SQL container. The
connection string uses the rig's **existing committed dev credentials**
(`User Id=scadabridge_app`, password as already committed in the compose/infra files —
redacted here), `TrustServerCertificate=true`.

### Rig-config change made by this gate (committed)

`docker/docker-compose.yml`:

- New anchor `x-secrets-hub-central-env` → `Secrets__SqlServer__ConnectionString`
  pointing at `scadabridge-mssql,1433 / ZbSecretsHub`. Applied to **central-a and
  central-b only** (`<<: [*secrets-hub-env, *secrets-hub-central-env]`). Site nodes must
  never carry this key — sites talk to central, not to central's database.
- `x-secrets-hub-site-env` gains
  `Secrets__GrpcHub__FallbackEndpoints__0: "http://scadabridge-central-b:8083"` (site-a
  pair only), with the same-store-only warning in the comment.
- The stale central-volume comments (SQLite as central's secret store) updated: the
  `scadabridge-secrets.db` under `central-node-*/data` is now a pre-0.5.0 residue; the
  volume survives for `inbound-api-keys.sqlite`.

All values remain DEV-ONLY under the same committed-rig-credential exception as before.

## Method

Central-side seeding/deleting used a throwaway helper console (scratchpad, not committed)
that composes **exactly** what central's Layer-A expander composes —
`AddZbSecretsSqlServerStore` + `SqlServerSecretsStoreMigrator` + the CLI's own
`SecretCommands` — against `ZbSecretsHub` over TCP with the rig dev KEK. This is the
plan's "CLI → shared SQL" seeding path; with one shared store, a row written by any client
is on exactly the path a row written via central-b would be (proven bidirectionally by
check 1's byte-identical hub reads). The `/admin/secrets` page was again not used —
covered separately by ScadaBridge#22.

Site-store assertions honored the gate-method rule from last time: **every** SQLite
access ran in a throwaway container on the guest kernel (`mcr.microsoft.com/dotnet/aspnet:10.0`
+ the published `secret` CLI, mounting the node's `data` volume). SQL Server, being
client-server, has no virtiofs coherence problem, so host-side access to the shared store
is safe.

gRPC calls were made with `fullstorydev/grpcurl` on `scadabridge-net` against the
committed `secrets_hub.proto`. No token, KEK, secret plaintext, or connection-string
password appears in this document; ciphertext identity is a SHA-256 over the encrypted
columns only.

## Results

| # | Check | Result | Timing |
|---|---|---|---|
| 1 | Central-pair parity by construction | **PASS** | byte-identical hub reads; 1 SQL row |
| 2 | Failover (`FallbackEndpoints`, sticky) | **PASS** | converged 18 s with central-a down; reverse failover converged same tick |
| 3 | Delete-while-follower-offline | **PASS** | tombstone on returning node's first sweep; no resurrection |
| 4 | Layer-A expander on SQL | **PASS** | value provably landed from the shared store |
| 5 | Fail-closed negatives | **PASS** | designed boot refusal; hub throws on SQL outage, never serves empty |

Baseline before seeding: `ZbSecretsHub..secret` empty; site-a stores held the previous
gate's three `hub-gate-smoke-*` rows (pull-only never deletes — they persist as
last-known-good residue); central-a's residual SQLite held the same three;
**central-b's residual SQLite was empty — the literal scadaproj#4 divergence, still on
disk**, which check 4 turns into a decoy.

### Check 1 — central-pair parity by construction · PASS

Seeded `shared-gate-1` into the shared store at 15:10:53Z. Then, authenticated:

- `GetManifest` on **both** `scadabridge-central-a:8083` and `scadabridge-central-b:8083`
  returned the identical single entry (`shared-gate-1`,
  `updatedUtc=2026-08-07T15:10:53.8689010+00:00`).
- `GetSecrets` full-row responses from the two hubs are **byte-identical**
  (sha256 `ec1f270c9404471d…` for both), `cipher_sha256=40e5a8df9faa8e189c9f0dc6`,
  `kek sha256 7451bcbc1f1f`.
- `SELECT COUNT(*) FROM zbsecrets.secret` → **1**. There is one copy of the row; "same
  HLC/ciphertext" is not a replication outcome to verify but an identity.

The 2026-08-07 hub gate's empty-manifest-from-central-b defect is **unreproducible by
construction**: central-b has no store of its own to be empty. Both site nodes converged
the row in **2 s** (seed 15:10:53Z → `pulled 1 row(s)` 15:10:55Z).

### Check 2 — failover · PASS

The failover machinery was exercised in **both directions**, plus once unplanned:

**Boot window (unplanned, evidence kept).** During bring-up central-a was wedged (defect 1)
and central-b not yet serving. Both site nodes logged the 0.5.0 failover Warning — naming
both endpoint URIs and nothing else — then the sweep-level failure:

```
site-a-a [15:04:55 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083.
site-a-a [15:05:25 WRN] Secrets hub endpoint http://scadabridge-central-a:8083 did not answer (Unavailable); failing over to http://scadabridge-central-b:8083.
```

At 15:05:25 central-b answered → both clients went **sticky on central-b**.

**Primary down (the check proper).** central-a stopped 15:11:43Z. **No new warning
appeared — correct, and worth understanding:** the clients were already sticky on
central-b, so the primary's outage cost zero failed calls, which is the documented sticky
rationale working as designed. `shared-gate-2` was seeded at 15:15:07Z with central-a
down; **both** site nodes converged it at 15:15:25Z (**18 s**) and decrypted it to the
exact seeded plaintext (in-container `secret get`).

**Reverse drill (rotation wrap + recovered-primary reuse).** central-a restarted healthy
(15:16:15Z), then central-b stopped (15:16:24Z) and `shared-gate-2b` seeded (15:16:35Z):

```
site-a-a [15:16:55 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unavailable); failing over to http://scadabridge-central-a:8083.
site-a-a [15:16:55 INF] Secret hub sync converged: pulled 1 row(s) from the central hub.
```

The sticky client hit the dead fallback, warned naming both endpoints, wrapped to the
recovered primary and converged **in the same sweep tick** — proving the rotation and
that central-a serves the same shared store. `site-a-b`'s 15:16:55Z sweep failed over but
still failed as a whole (stale gRPC subchannel view of the restarted central-a — the same
bounded post-restart artifact the previous gate recorded) and converged unaided on the
next tick (15:17:25Z); decrypt-verified. So worst-case post-outage convergence remains
**two** sweep intervals.

**No flapping**: after central-b returned (15:17:06Z) no further hub warnings appeared on
either site node until the deliberate SQL outage in check 5.

### Check 3 — delete-while-follower-offline · PASS

This was the untested residual from scadaproj#4 (the previous gate's residual 5).

- 15:17:58Z — `site-a-b` stopped, holding **live** `shared-gate-1` (rev=0).
- 15:18:29Z — `shared-gate-1` deleted centrally → SQL row `revision=1, is_deleted=1,
  updated_utc=deleted_utc=2026-08-07T15:18:29.3213100+00:00`. `shared-gate-2`/`2b` left
  live as controls.
- 15:18:55Z — **`site-a-a` (the online follower) converged the tombstone** (≤26 s):
  `rev=1 del=True`, identical timestamp; `secret get` → `{"error":"not-found"}`.
- 15:19:13Z — `site-a-b` restarted; its `SyncOnStartup` sweep **in the boot second**
  pulled the tombstone: `rev=1 del=True`, identical timestamp, `get` → not-found. Its
  stale live copy did **not** resurrect — and structurally cannot push anywhere (no write
  RPC on the wire).
- Re-verified ≥10 sweeps later (15:25Z final-state table below): still tombstoned on
  both followers, controls still live.

### Check 4 — Layer-A expander on SQL · PASS

Designed as a three-way discrimination so "the expanded value landed from the SHARED
store" is proven by **value**, not by boot success:

- Shared SQL store: `expander-gate-pepper` = a **5-character** value.
- central-a's **stale residual SQLite** store: same name, a **41-character decoy** value.
- central-a env (temporary probe): `ScadaBridge__InboundApi__ApiKeyPepper` =
  `${secret:expander-gate-pepper}`. The unexpanded literal is 30 characters.

`StartupValidator` (which runs immediately **after** the Layer-A expander) requires the
pepper to be ≥16 characters. So of the three possible sources, **only the shared SQL
store's value fails validation**. central-a was recreated at 15:20:36Z and its boot
failed with exactly:

```
Unhandled exception. System.InvalidOperationException: Configuration validation failed:
  - ScadaBridge:InboundApi:ApiKeyPepper is required and must be at least 16 characters for Central (backs the inbound API-key peppered-HMAC verifier)
```

A boot reading the SQLite decoy or leaving the token unexpanded would have passed. The
expander therefore resolved the reference **from the shared SQL store** — the store the
running node serves — not from the stale local one (which is the exact divergence the
store-swap exists to prevent, sitting right there on disk as a live decoy).

Positive half: the store row was then rotated to a valid ≥16-char value (the same literal
central-b carries, preserving pair pepper parity) and central-a restarted → clean boot
15:20:54Z, `Now listening` on 5000/8083, `/health/ready` 200. Probe reverted, both probe
rows tombstoned/removed. Together with check 5(a), this **discharges the offline-test
residual for the `Program.cs` SQL expander path** — both its resolve path and its
fail-closed guard ran live.

Seeding was via CLI → shared SQL; check 1 proved both centrals serve that store
byte-identically, so this is equivalent to the plan's "seed via central-b".

### Check 5 — fail-closed negatives · PASS

**(a) Blank connection string.** central-a recreated with
`Secrets__SqlServer__ConnectionString: ""` (still Central + `Mode=Grpc`). Boot refused
with the designed message, thrown from the **expander block** (`Program.cs` line 84 →
`SecretsRegistration.EnsureCentralSharedStoreConnectionString`), before any
`SqlConnection` could bury it:

```
Unhandled exception. System.InvalidOperationException: Secrets:SqlServer:ConnectionString is empty, but this node is Central with Secrets:Replication:Mode=Grpc. Central in Grpc mode requires the SHARED SQL-Server secret store — an independent local store per central node is the divergence scadaproj#4 recorded (a hub failover would 'succeed' against an empty manifest). Supply the connection string via appsettings or the environment (Secrets__SqlServer__ConnectionString).
```

It names the key, explains why, and echoes no configured value. Probe reverted; node
healthy again. (The refusal-to-boot criterion PASSED; *how* the refused process then
lingers is defect 2 below.)

**(b) SQL Server stopped.** `scadabridge-mssql` stopped 15:22:06Z (accepted transient for
central's core DBs and the sibling env2 rig; both recovered).

- Authenticated `GetManifest` against central-a → **gRPC error**
  (`Code: Unknown / Exception was thrown by handler`) — **the hub throws; it never
  serves an empty manifest.** No credential or connection-string material in the error.
- Site sweeps failed loudly, trying **both** hubs (both depend on the one store):

  ```
  site-a-a [15:22:40 WRN] Secrets hub endpoint http://scadabridge-central-b:8083 did not answer (Unknown); failing over to http://scadabridge-central-a:8083.
  site-a-a [15:22:54 WRN] Secret hub sync failed; the node continues serving its local store and will retry on the next interval.
  ```

- Last-known-good held: `secret get shared-gate-2` on `site-a-a` returned the correct
  plaintext mid-outage.
- SQL restarted 15:23:10Z → the hub answered the full 4-entry manifest immediately, no
  node restarts. Recovery made positively observable: `shared-gate-final` seeded 15:24:17Z
  converged on `site-a-a` in **8 s** and `site-a-b` in **26 s**, decrypt-verified on both.

### Final state — three-way parity

At 15:25Z the shared SQL store and both site stores held the identical five rows
(name / revision / tombstone / `updated_utc` all equal):

```
expander-gate-pepper rev=2 del=1 updated=2026-08-07T15:21:26.9497970+00:00
shared-gate-1        rev=1 del=1 updated=2026-08-07T15:18:29.3213100+00:00
shared-gate-2        rev=0 del=0 updated=2026-08-07T15:15:07.5685210+00:00
shared-gate-2b       rev=0 del=0 updated=2026-08-07T15:16:35.0265220+00:00
shared-gate-final    rev=0 del=0 updated=2026-08-07T15:24:18.1206230+00:00
```

(Site stores additionally keep the previous gate's `hub-gate-smoke-*` residue — pull-only
never deletes rows absent from the manifest, by design.)

**Log hygiene — zero hits, fleet-wide.** All 8 nodes' `docker logs` and every on-disk
Serilog file under `docker/*/logs/` grepped for: the dev bearer token, the dev KEK, all
four smoke plaintexts, the decoy pepper value, **and the SQL dev password** (notably: the
migrator crash stack in defect 1 did NOT leak the connection string):

```
central-a/b, site-a-a/b, site-b-a/b, site-c-a/b: token=0 kek=0 p1..p4=0 decoy=0 sqlpw=0
on-disk log files: 0 files matched, for each of the 8 search strings
```

Default-OFF pin: `site-b-a`/`site-c-a` show zero `Secrets__*` env vars and zero hub log
lines.

## Defects found

**None against any check's criterion.** Two real defects surfaced around the checks —
per the gate's standing rule, **documented and NOT patched**; both need a controller
decision.

### 1. Library — `SqlServerSecretsStoreMigrator` concurrent first-boot race (error 2714)

`ZB.MOM.WW.Secrets.Replicator.SqlServer` (present since 0.2.0, including 0.5.0). The
class doc claims *"Idempotent: safe to run on every node at every startup, including
concurrently"* and that Serializable + `IF NOT EXISTS` guards make simultaneous provision
of a virgin database converge. Live, the ordinary `docker compose up` recreated both
central nodes at once, both ran the boot migrator against the empty `ZbSecretsHub`,
central-b won, and central-a died at 15:04:55Z with:

```
Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an object named 'zbsecrets' in the database.
CREATE SCHEMA failed due to previous errors.
   at ZB.MOM.WW.Secrets.Replicator.SqlServer.SqlServerSecretsStoreMigrator.MigrateOnceAsync(...)
Error Number:2714,State:6,Class:16
```

The retry loop in `MigrateAsync` catches **only** deadlock victims (error 1205,
`MaxDeadlockRetries=5`); the object-exists race surfaces as **2714**, which propagates
and aborts the boot. Window: virgin database + N nodes booting together — which is the
NORMAL first deploy of a central pair, exactly the situation the class comment says it
handles. Once the schema exists the migrator is genuinely idempotent (central-a's manual
restart at 15:09:59Z and every recreate since booted clean), so the blast radius is
one-time-per-virgin-DB — but it lands on the very first production bring-up, and combined
with defect 2 it left the node wedged rather than restart-recovered.

**Suggested direction (for the controller, not applied):** include 2714 (and arguably
2617/1913-class "already exists" errors) in the retry filter, or catch-and-recheck —
retrying is exactly as correct for this race as for the deadlock it already retries.

### 2. ScadaBridge Host — pre-Serilog unhandled boot exception wedges the process instead of exiting

The plan and the compose comments document the pre-Serilog failure window as: container
"exits with a bare stderr trace and restarts". **Observed twice, it does not exit.** After
printing the crash banner the main thread spins at ~100% CPU indefinitely; the container
stays `running` (`RestartCount=0`), `restart: unless-stopped` never fires, and the node
serves nothing. Reproduced on **both** pre-host failure shapes this gate produced:

- the defect-1 `SqlException` from the expander's migrator run (spun for ~5 minutes,
  main thread confirmed as the busy thread, until manually restarted);
- the check-4 `StartupValidator` `InvalidOperationException` (same signature: banner
  printed, 100% CPU, still `running`).

So it is not specific to SqlClient or to the new SQL path — any throw in the pre-host
window (including the pre-existing SQLite path's failure modes) turns a clean fail-fast
into a silent wedge that needs a manual restart, and monitoring that watches container
state sees a healthy-looking `running` container. Mechanism not root-caused here (gate
scope); the observable and two reproductions are recorded. Worth a dedicated issue —
fail-closed is only honest if the process actually exits.

### Gate-method notes (not defects)

- docker compose interpolates `${…}` in YAML itself: a literal `${secret:…}` env value in
  the compose file must be written `$${secret:…}` or the service silently keeps its old
  env (compose warns `invalid interpolation format`). Caught before any result was
  recorded; the check-4 probe used the escaped form and the applied env was verified with
  `docker inspect`.
- `docker logs --since` windows overlap pre-restart lines; every "converged after X"
  assertion above was re-anchored on timestamps, not on grep hits alone.

## Residuals

1. **Defects 1 and 2 above** — filed for controller disposition; neither blocks the
   topology (defect 1 is one-time-per-virgin-DB and recoverable by restart; defect 2 is
   pre-existing behavior of the pre-host window, now with two live reproductions).
2. **`site-a-b`'s one-tick sweep failure after a central restart** (stale subchannel) —
   same bounded, self-healing artifact the previous gate recorded; worst-case post-outage
   convergence is two sweep intervals.
3. **The `/admin/secrets` write path was again not exercised** — same coverage note as
   the previous gate; ScadaBridge#22 covers the page itself.
4. **Replication stays default-OFF in the product.** Enabled on this rig only (4 of 8
   nodes), dev KEK/token/credentials. Production enablement needs a real KEK out of band,
   a real hub token, a real SQL credential — and, per the plan's closeout, `rewrap-all`
   on ScadaBridge now runs **once against central's shared SQL store** plus both nodes of
   every site pair.
5. **Central secret resolution now depends on SQL Server availability** — the accepted
   trade from the decision record, observed live in check 5(b): during the outage the hub
   throws (sites keep last-known-good), and a central node cannot *boot* through the
   Layer-A expander while SQL is down (container-restart-retryable — subject to defect 2).

## Conclusion

The shared-store topology does what the design says: there is **one** copy of every
central secret, both hubs serve it byte-identically, site-side failover between the
central pair is safe *because* of that (proven in both directions, including the
recovered-primary wrap in a single sweep tick), a delete issued while a follower is down
lands as a tombstone on its very first sweep back with no resurrection, central's
pre-host `${secret:}` expander provably reads the shared store rather than the stale
local file sitting next to it, and every failure mode observed failed **closed and
loud** — designed refusal messages naming keys, hub errors instead of empty manifests,
sweep warnings instead of silent stalls — with zero credential material in any log on
any of the eight nodes. **5/5.** scadaproj#4's divergence is gone by construction; the
two defects the gate flushed out are recorded above for disposition.