scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md

# Secrets: Central Shared SQL Store + Hub Endpoint Failover (scadaproj#4)

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development to implement this plan task-by-task.

**Goal:** Resolve [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) — ScadaBridge's central pair diverging on secrets — by pointing both central nodes' `ISecretStore` at one shared SQL Server store (the SqlServer replicator's existing shared-store mode), then making site-side hub endpoint failover safe and shipping it (`FallbackEndpoints`, family `0.5.0`).

**Architecture:** In `Secrets:Replication:Mode=Grpc`, central's store becomes `AddZbSecretsSqlServerStore` (one copy of every row — divergence is impossible by construction, so both hub instances serve identical manifests). Sites are unchanged (local SQLite + pull-only sweep) except the sweep client gains ordered fallback endpoints with sticky preference. The Layer-A pre-host `${secret:}` expander on central must follow the store swap or pre-host refs resolve from a stale local SQLite store.

**Tech stack:** .NET 10, Grpc.Net.Client 2.76.0, existing `ZB.MOM.WW.Secrets.Replicator.SqlServer` shared-store mode (live-tested vs SQL Server 2022 since 0.2.0), docker-dev cluster for the live gate.

**Decision record (user-approved 2026-08-07):** shared SQL store chosen over central-pair convergence/replication because it removes divergence *structurally* (no replication-lag window), makes endpoint failover trivially safe (either central answers with the truth), and collapses central's `rewrap-all` to one run. Accepted trade: central secret resolution now depends on SQL Server availability — acceptable because ScadaBridge central is already SQL-coupled to its core, and a SQL outage makes the hub *throw* (sites keep last-known-good), never serve empty.

---

## Standing rules (every task, every subagent)

- **Never** echo PSK/bearer/key/KEK material in logs, exception messages, status output, commit messages, or reports. Error messages name exact config keys instead.
- Fail closed on every unresolvable auth/config case.
- gRPC auth interceptors keep **exactly one public constructor**.
- Do NOT touch ScadaBridge `test.txt`/`sqllogin.txt` (live production credentials) or the `deploy/` overlay.
- Gitea API token via `git credential fill` (protocol=https, host=gitea.dohertylan.com, username=dohertj2) — never echo it.
- Every commit ends with trailer: `Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1`
- scadaproj repo: work branches off `main` @ `294eb16`. ScadaBridge repo: off `main` @ `f6c3f7c5`. Both currently clean.
- Full offline suites must be green (0 warnings, `TreatWarningsAsErrors`) before any merge.

---

### Task 1: Library — `FallbackEndpoints` + failover hub reader

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2 (different repo)

**Repo:** scadaproj (`~/Desktop/scadaproj`), worktree branch `feat/secrets-hub-failover` off `main`.

**Files:**
- Modify: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/SecretsGrpcHubClientOptions.cs`
- Create: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/FailoverSecretsHubReader.cs`
- Modify: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/DependencyInjection/SecretsGrpcHubClientExtensions.cs`
- Modify: `ZB.MOM.WW.Secrets/CHANGELOG.md` (0.5.0 Unreleased entry)
- Test: `ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/` (new `FailoverSecretsHubReaderTests.cs`, extend `SecretsGrpcHubClientOptionsTests`, extend client DI tests)

**Spec:**

1. `SecretsGrpcHubClientOptions` gains:
   ```csharp
   /// <summary>
   /// Additional hub addresses tried, in order, when a call to <see cref="Endpoint"/> (or an
   /// earlier fallback) fails with a transport-level gRPC error. Optional; empty by default.
   /// Meaningful only when every listed endpoint serves the SAME store — for ScadaBridge that
   /// is the central pair over one shared SQL Server store. Do NOT list endpoints backed by
   /// independent stores: failing over to an emptier hub is a silent convergence stop, which is
   /// the exact defect (scadaproj#4) the shared store exists to prevent.
   /// </summary>
   public IList<string> FallbackEndpoints { get; } = [];
   ```
   (Getter-only `IList<string>` — the config binder adds to it. Same http-cleartext caveat as `Endpoint` applies; reference it.)
2. `Validate(sectionPath)`: each fallback entry must be an absolute http/https URI (same check as `Endpoint`, message naming `{sectionPath}:FallbackEndpoints`); reject blank entries; reject an entry equal (OrdinalIgnoreCase) to `Endpoint` or a duplicate of another fallback — a duplicated endpoint silently halves the failover the operator thinks they configured.
3. New `internal sealed partial class FailoverSecretsHubReader : ISecretsHubReader`:
   - Constructed with `IReadOnlyList<ISecretsHubReader> readers` (ordered: primary first) + `IReadOnlyList<string> endpointLabels` (for logging) + `ILogger`.
   - Both `GetManifestAsync` and `GetManyAsync`: start at the **sticky index** (last endpoint that answered, `Volatile.Read`/`Volatile.Write` on an `int` field, initial 0), try each reader in rotation order; on `RpcException` log a Warning naming the failed endpoint URI and the next one tried (URIs are not secrets; the token must never appear), continue; when one answers, store its index as sticky and return. All readers failed → rethrow the **last** `RpcException` (the sweep's catch-all handles it as today).
   - Only `RpcException` triggers failover. `OperationCanceledException` and anything else propagates immediately — cancellation is the host stopping, not an endpoint fault.
   - Sticky rationale (must be in the class doc): a sweep is GetManifest + N GetSecrets batches; without stickiness every call after a primary outage would re-pay the primary's failure (a full CallDeadline of latency) before reaching the healthy fallback.
4. DI (`AddZbSecretsGrpcHubClient`):
   - **Zero fallbacks → the registration graph is byte-identical to today** (single keyed channel under `ChannelServiceKey`, single `GrpcSecretsHubClient` as `ISecretsHubReader`). Existing consumers and ScadaBridge's 31 wiring pins must pass unchanged.
   - With fallbacks: keyed channel per endpoint — primary keeps `ChannelServiceKey`, fallback *i* uses `$"{ChannelServiceKey}:fallback:{i}"` — each wrapped in its own `GrpcSecretsHubClient`, composed into `FailoverSecretsHubReader` registered as the `ISecretsHubReader`. Channels stay container-owned singletons (disposal).
   - `GrpcSecretsHubClient` itself is **unchanged**.
5. Tests (TDD, red first):
   - Options: valid fallback accepted; relative/blank/duplicate-of-primary/duplicate-fallback rejected with messages naming `…:FallbackEndpoints`; zero fallbacks still valid.
   - Failover reader (fake `ISecretsHubReader`s): primary healthy → fallback never called; primary throws `RpcException(Unavailable)` → fallback answers + Warning logged naming both endpoints; sticky — next call goes to the fallback FIRST; recovered primary is reused after the fallback fails in turn (rotation wraps); all fail → last RpcException rethrown; `OperationCanceledException` does NOT fail over; concurrent calls safe (no torn sticky state).
   - Log-content test: no token material in any failover log message (reuse the existing no-secret-leak assertion pattern from the interceptor tests).
   - DI: zero-fallback shape unchanged (resolve `ISecretsHubReader`, assert `GrpcSecretsHubClient`); with fallbacks assert `FailoverSecretsHubReader` + one keyed channel per endpoint.
6. CHANGELOG: add `## 0.5.0` entry under the existing format (feature: FallbackEndpoints/failover; note the same-store requirement).

**Steps:** red tests → implement → `dotnet test ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx` full green, 0 warnings → commit on the worktree branch.

---

### Task 2: ScadaBridge — central shared SQL store + Layer-A expander

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 1 (different repo)

**Repo:** ScadaBridge (`~/Desktop/ScadaBridge`), branch `feat/secrets-central-shared-store` off `main` @ `f6c3f7c5`. Uses the **existing 0.4.1 packages** (shared-store mode has been in `Replicator.SqlServer` since 0.2.0) — no dependency on Task 1/3.

**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs`
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (Layer-A expander block, ~lines 53–65)
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (comment/placeholder only)
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs`

**Spec:**

1. `SecretsRegistration.AddScadaBridgeSecrets`, `UsesGrpcHub(config)` branch, `case SecretsNodeRole.Central`:
   - **Before** any registration, read `HubConnectionStringKey` (`Secrets:SqlServer:ConnectionString`). If blank → `throw new InvalidOperationException` naming the key and why: central in Grpc mode REQUIRES the shared SQL store — an independent local store per central node is exactly the divergence scadaproj#4 recorded (a hub failover would "succeed" against an empty manifest). Fail-closed, matching the mode's philosophy.
   - If the value contains `${secret:` → throw naming the key: the secrets store's own connection string can never be a secret reference (the expander needs the store to resolve it — bootstrap circularity). Same rule the class remarks already document for the hub bearer token.
   - Replace `services.AddZbSecrets(config, SecretsSectionPath)` with `services.AddZbSecretsSqlServerStore(config, SecretsSectionPath, SqlServerSectionPath)`, then `AddZbSecretsGrpcHub` as today. (`AddZbSecretsSqlServerStore` registers `SqlServerSecretStore` as `ISecretStore` *before* calling `AddZbSecrets`, displacing SQLite — the hub service then serves the shared store automatically.)
   - `case SecretsNodeRole.Site`: **byte-identical to today** (`AddZbSecrets` + `AddZbSecretsGrpcHubClient`).
   - Update the class-level remarks: Grpc-mode paragraph now states central's store is the shared SQL store and why; bootstrap-constraint paragraph adds the connection-string-is-never-a-ref rule; note the Layer-A expander follows the swap (point at Program.cs).
2. `Program.cs` Layer-A expander: when `configuration["ScadaBridge:Node:Role"]` is `Central` (OrdinalIgnoreCase) AND `SecretsRegistration.UsesGrpcHub(configuration)` AND `configuration[SecretsRegistration.HubConnectionStringKey]` is non-blank → build the throwaway provider with `.AddZbSecretsSqlServerStore(configuration, "Secrets")` and run `SqlServerSecretsStoreMigrator.MigrateAsync` instead of the SQLite migrator. Every other case keeps the current SQLite path byte-identical. Comment must say WHY: with central's runtime store on SQL, an expander left on SQLite would resolve pre-host `${secret:}` refs from a stale/empty local store — values silently diverging from what the running node serves. (Central+Grpc with a blank connection string deliberately falls through to the SQLite path: the boot is about to fail in `AddScadaBridgeSecrets` with the clear message; don't duplicate the throw here.)
3. `appsettings.json`: in the existing Secrets area add `"SqlServer": { "ConnectionString": "" }` with a `_comment` stating it is required on central when `Replication:Mode=Grpc`, must be a literal/env value (never `${secret:}`), and sites leave it empty.
4. Wiring pins (extend `SecretsReplicationWiringTests`, red first):
   - Central + Grpc + connection string → `ISecretStore` resolves to `SqlServerSecretStore`; hub service still registered.
   - Central + Grpc + blank connection string → registration throws, message names `Secrets:SqlServer:ConnectionString`.
   - Central + Grpc + `${secret:…}` connection string → throws, message names the key and the circularity.
   - Site + Grpc → `ISecretStore` still SQLite; SqlServer package types NOT in the container (sites-don't-talk-to-SQL pin).
   - SqlServer-mode and replication-off paths byte-identical (existing pins keep passing).
5. Suite: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` full green, 0 warnings. Commit on the branch; do NOT merge yet (merge happens with Task 4).

---

### Task 3: Publish family 0.5.0

**Classification:** small
**Estimated implement time:** ~3 min
**Blocked by:** Task 1

**Repo:** scadaproj. After Task 1's review approves and its branch merges to `main`:

1. `ZB.MOM.WW.Secrets/Directory.Build.props`: `0.4.1` → `0.5.0`. CHANGELOG: retitle the Unreleased/0.5.0 entry with the date.
2. `dotnet test` full suite green → `dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o artifacts` → push all 6 packages to the `dohertj2-gitea` feed (same `dotnet nuget push` invocation as 0.4.1; API key via `git credential fill`).
3. Restore-verify from a scratch cache (same probe as 0.4.1).
4. Commit + push `main`.

---

### Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring

**Classification:** standard
**Estimated implement time:** ~3 min
**Blocked by:** Tasks 2, 3

**Repo:** ScadaBridge, continues `feat/secrets-central-shared-store`.

1. `Directory.Packages.props`: all `ZB.MOM.WW.Secrets*` pins `0.4.1` → `0.5.0`.
2. `appsettings.json` `Secrets:GrpcHub`: add `"FallbackEndpoints": []` placeholder + `_comment`: list the SECOND central node here; safe only because both centrals serve one shared SQL store.
3. Wiring pins: site + Grpc + one fallback endpoint → `ISecretsHubReader` resolves to the failover reader; zero fallbacks → plain `GrpcSecretsHubClient` (pin the unchanged-default shape).
4. Full suite green, 0 warnings. Merge `feat/secrets-central-shared-store` → `main` after review, push.

---

### Task 5: Live gate on the docker cluster

**Classification:** high-risk
**Estimated implement time:** gate run, ~15 min wall
**Blocked by:** Task 4

**Repo:** ScadaBridge docker rig (same rig as the 2026-08-07 hub gate; central pair + site-a pair with `Secrets:Replication:Mode=Grpc`). Produce `ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md` recording every check with evidence, then merge/push any rig-compose changes.

Rig prep: central-a/central-b gain `Secrets__SqlServer__ConnectionString` → the rig's SQL Server container (dedicated database or schema; default `SchemaName` from `SqlServerSecretsOptions` is fine); site-a nodes gain `Secrets__GrpcHub__FallbackEndpoints__0` → central-b's hub endpoint. Rig secrets/tokens are the existing dev values — never real credentials.

**Checks (all must PASS):**
1. **Central-pair parity by construction:** seed a secret through central-a's store (CLI against the SQL store, or `/admin/secrets`); `GetManifest` from BOTH central hubs returns it identically (same HLC/ciphertext). The 2026-08-07 gate's empty-manifest-from-central-b defect must be unreproducible.
2. **Failover:** stop central-a; site-a's next sweep logs the failover Warning (naming both endpoints, no token material) and converges a NEW secret written while central-a is down (written via central-b `/admin/secrets` or CLI → shared SQL). Restart central-a; sweeps keep succeeding (sticky on central-b is acceptable — verify no flapping).
3. **Delete-while-follower-offline** (the untested residual from #4): stop one site-a node; delete a secret centrally; restart the node → tombstone applied, row does NOT resurrect, and the OTHER site node also converged the delete.
4. **Layer-A expander on SQL:** a `${secret:}` config reference on central resolves at boot from the SHARED store (seed the secret via central-b, boot central-a, verify the expanded value landed) — proving the expander followed the store swap.
5. **Fail-closed negatives:** central with Grpc mode + blank `Secrets:SqlServer:ConnectionString` refuses to boot naming the key; SQL container stopped → hub calls fail (site logs sweep failure, keeps serving last-known-good; NO empty manifest served).

Gate-method rule from last time: never seed a bind-mounted SQLite store from the macOS host — for site-side assertions use exec-in-container or throwaway containers. (Central's store is now SQL, so central-side seeding is just SQL/CLI/UI.)

---

### Task 6: Closeout

**Classification:** small
**Estimated implement time:** ~4 min
**Blocked by:** Task 5

1. **scadaproj#4**: close with a comment recording the resolution (shared SQL store + FallbackEndpoints, 0.5.0, gate SHAs), including that the delete-while-follower-offline residual was gated (check 3) and the denial-warning-alerting note (sum per-window deltas) which stands independently.
2. `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`: ScadaBridge topology = central pair on one shared SQL store + Grpc hub; **rewrap fan-out: ONE run against central's SQL store** (the CLI auto-targets it via `Secrets:SqlServer:ConnectionString` in central's appsettings) **+ both nodes of every site pair** — delete the "both central nodes" caveat. Document the FallbackEndpoints same-store-only rule.
3. `components/secrets/GAPS.md`: append the resolution section.
4. scadaproj `CLAUDE.md` Secrets row: 0.5.0, ScadaBridge pins 0.5.0, central shared SQL store, #4 CLOSED, gate reference.
5. Update this plan's `.tasks.json` ledger; commit scadaproj `main` + push (index-edit rule: commit on `main`).