diff --git a/docs/plans/2026-08-07-secrets-central-shared-sql-store.md b/docs/plans/2026-08-07-secrets-central-shared-sql-store.md new file mode 100644 index 0000000..b41c744 --- /dev/null +++ b/docs/plans/2026-08-07-secrets-central-shared-sql-store.md @@ -0,0 +1,174 @@ +# Secrets: Central Shared SQL Store + Hub Endpoint Failover (scadaproj#4) + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development to implement this plan task-by-task. + +**Goal:** Resolve [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) — ScadaBridge's central pair diverging on secrets — by pointing both central nodes' `ISecretStore` at one shared SQL Server store (the SqlServer replicator's existing shared-store mode), then making site-side hub endpoint failover safe and shipping it (`FallbackEndpoints`, family `0.5.0`). + +**Architecture:** In `Secrets:Replication:Mode=Grpc`, central's store becomes `AddZbSecretsSqlServerStore` (one copy of every row — divergence is impossible by construction, so both hub instances serve identical manifests). Sites are unchanged (local SQLite + pull-only sweep) except the sweep client gains ordered fallback endpoints with sticky preference. The Layer-A pre-host `${secret:}` expander on central must follow the store swap or pre-host refs resolve from a stale local SQLite store. + +**Tech stack:** .NET 10, Grpc.Net.Client 2.76.0, existing `ZB.MOM.WW.Secrets.Replicator.SqlServer` shared-store mode (live-tested vs SQL Server 2022 since 0.2.0), docker-dev cluster for the live gate. + +**Decision record (user-approved 2026-08-07):** shared SQL store chosen over central-pair convergence/replication because it removes divergence *structurally* (no replication-lag window), makes endpoint failover trivially safe (either central answers with the truth), and collapses central's `rewrap-all` to one run. Accepted trade: central secret resolution now depends on SQL Server availability — acceptable because ScadaBridge central is already SQL-coupled to its core, and a SQL outage makes the hub *throw* (sites keep last-known-good), never serve empty. + +--- + +## Standing rules (every task, every subagent) + +- **Never** echo PSK/bearer/key/KEK material in logs, exception messages, status output, commit messages, or reports. Error messages name exact config keys instead. +- Fail closed on every unresolvable auth/config case. +- gRPC auth interceptors keep **exactly one public constructor**. +- Do NOT touch ScadaBridge `test.txt`/`sqllogin.txt` (live production credentials) or the `deploy/` overlay. +- Gitea API token via `git credential fill` (protocol=https, host=gitea.dohertylan.com, username=dohertj2) — never echo it. +- Every commit ends with trailer: `Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1` +- scadaproj repo: work branches off `main` @ `294eb16`. ScadaBridge repo: off `main` @ `f6c3f7c5`. Both currently clean. +- Full offline suites must be green (0 warnings, `TreatWarningsAsErrors`) before any merge. + +--- + +### Task 1: Library — `FallbackEndpoints` + failover hub reader + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 2 (different repo) + +**Repo:** scadaproj (`~/Desktop/scadaproj`), worktree branch `feat/secrets-hub-failover` off `main`. + +**Files:** +- Modify: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/SecretsGrpcHubClientOptions.cs` +- Create: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/FailoverSecretsHubReader.cs` +- Modify: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/DependencyInjection/SecretsGrpcHubClientExtensions.cs` +- Modify: `ZB.MOM.WW.Secrets/CHANGELOG.md` (0.5.0 Unreleased entry) +- Test: `ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/` (new `FailoverSecretsHubReaderTests.cs`, extend `SecretsGrpcHubClientOptionsTests`, extend client DI tests) + +**Spec:** + +1. `SecretsGrpcHubClientOptions` gains: + ```csharp + /// + /// Additional hub addresses tried, in order, when a call to (or an + /// earlier fallback) fails with a transport-level gRPC error. Optional; empty by default. + /// Meaningful only when every listed endpoint serves the SAME store — for ScadaBridge that + /// is the central pair over one shared SQL Server store. Do NOT list endpoints backed by + /// independent stores: failing over to an emptier hub is a silent convergence stop, which is + /// the exact defect (scadaproj#4) the shared store exists to prevent. + /// + public IList FallbackEndpoints { get; } = []; + ``` + (Getter-only `IList` — the config binder adds to it. Same http-cleartext caveat as `Endpoint` applies; reference it.) +2. `Validate(sectionPath)`: each fallback entry must be an absolute http/https URI (same check as `Endpoint`, message naming `{sectionPath}:FallbackEndpoints`); reject blank entries; reject an entry equal (OrdinalIgnoreCase) to `Endpoint` or a duplicate of another fallback — a duplicated endpoint silently halves the failover the operator thinks they configured. +3. New `internal sealed partial class FailoverSecretsHubReader : ISecretsHubReader`: + - Constructed with `IReadOnlyList readers` (ordered: primary first) + `IReadOnlyList endpointLabels` (for logging) + `ILogger`. + - Both `GetManifestAsync` and `GetManyAsync`: start at the **sticky index** (last endpoint that answered, `Volatile.Read`/`Volatile.Write` on an `int` field, initial 0), try each reader in rotation order; on `RpcException` log a Warning naming the failed endpoint URI and the next one tried (URIs are not secrets; the token must never appear), continue; when one answers, store its index as sticky and return. All readers failed → rethrow the **last** `RpcException` (the sweep's catch-all handles it as today). + - Only `RpcException` triggers failover. `OperationCanceledException` and anything else propagates immediately — cancellation is the host stopping, not an endpoint fault. + - Sticky rationale (must be in the class doc): a sweep is GetManifest + N GetSecrets batches; without stickiness every call after a primary outage would re-pay the primary's failure (a full CallDeadline of latency) before reaching the healthy fallback. +4. DI (`AddZbSecretsGrpcHubClient`): + - **Zero fallbacks → the registration graph is byte-identical to today** (single keyed channel under `ChannelServiceKey`, single `GrpcSecretsHubClient` as `ISecretsHubReader`). Existing consumers and ScadaBridge's 31 wiring pins must pass unchanged. + - With fallbacks: keyed channel per endpoint — primary keeps `ChannelServiceKey`, fallback *i* uses `$"{ChannelServiceKey}:fallback:{i}"` — each wrapped in its own `GrpcSecretsHubClient`, composed into `FailoverSecretsHubReader` registered as the `ISecretsHubReader`. Channels stay container-owned singletons (disposal). + - `GrpcSecretsHubClient` itself is **unchanged**. +5. Tests (TDD, red first): + - Options: valid fallback accepted; relative/blank/duplicate-of-primary/duplicate-fallback rejected with messages naming `…:FallbackEndpoints`; zero fallbacks still valid. + - Failover reader (fake `ISecretsHubReader`s): primary healthy → fallback never called; primary throws `RpcException(Unavailable)` → fallback answers + Warning logged naming both endpoints; sticky — next call goes to the fallback FIRST; recovered primary is reused after the fallback fails in turn (rotation wraps); all fail → last RpcException rethrown; `OperationCanceledException` does NOT fail over; concurrent calls safe (no torn sticky state). + - Log-content test: no token material in any failover log message (reuse the existing no-secret-leak assertion pattern from the interceptor tests). + - DI: zero-fallback shape unchanged (resolve `ISecretsHubReader`, assert `GrpcSecretsHubClient`); with fallbacks assert `FailoverSecretsHubReader` + one keyed channel per endpoint. +6. CHANGELOG: add `## 0.5.0` entry under the existing format (feature: FallbackEndpoints/failover; note the same-store requirement). + +**Steps:** red tests → implement → `dotnet test ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx` full green, 0 warnings → commit on the worktree branch. + +--- + +### Task 2: ScadaBridge — central shared SQL store + Layer-A expander + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 1 (different repo) + +**Repo:** ScadaBridge (`~/Desktop/ScadaBridge`), branch `feat/secrets-central-shared-store` off `main` @ `f6c3f7c5`. Uses the **existing 0.4.1 packages** (shared-store mode has been in `Replicator.SqlServer` since 0.2.0) — no dependency on Task 1/3. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (Layer-A expander block, ~lines 53–65) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (comment/placeholder only) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs` + +**Spec:** + +1. `SecretsRegistration.AddScadaBridgeSecrets`, `UsesGrpcHub(config)` branch, `case SecretsNodeRole.Central`: + - **Before** any registration, read `HubConnectionStringKey` (`Secrets:SqlServer:ConnectionString`). If blank → `throw new InvalidOperationException` naming the key and why: central in Grpc mode REQUIRES the shared SQL store — an independent local store per central node is exactly the divergence scadaproj#4 recorded (a hub failover would "succeed" against an empty manifest). Fail-closed, matching the mode's philosophy. + - If the value contains `${secret:` → throw naming the key: the secrets store's own connection string can never be a secret reference (the expander needs the store to resolve it — bootstrap circularity). Same rule the class remarks already document for the hub bearer token. + - Replace `services.AddZbSecrets(config, SecretsSectionPath)` with `services.AddZbSecretsSqlServerStore(config, SecretsSectionPath, SqlServerSectionPath)`, then `AddZbSecretsGrpcHub` as today. (`AddZbSecretsSqlServerStore` registers `SqlServerSecretStore` as `ISecretStore` *before* calling `AddZbSecrets`, displacing SQLite — the hub service then serves the shared store automatically.) + - `case SecretsNodeRole.Site`: **byte-identical to today** (`AddZbSecrets` + `AddZbSecretsGrpcHubClient`). + - Update the class-level remarks: Grpc-mode paragraph now states central's store is the shared SQL store and why; bootstrap-constraint paragraph adds the connection-string-is-never-a-ref rule; note the Layer-A expander follows the swap (point at Program.cs). +2. `Program.cs` Layer-A expander: when `configuration["ScadaBridge:Node:Role"]` is `Central` (OrdinalIgnoreCase) AND `SecretsRegistration.UsesGrpcHub(configuration)` AND `configuration[SecretsRegistration.HubConnectionStringKey]` is non-blank → build the throwaway provider with `.AddZbSecretsSqlServerStore(configuration, "Secrets")` and run `SqlServerSecretsStoreMigrator.MigrateAsync` instead of the SQLite migrator. Every other case keeps the current SQLite path byte-identical. Comment must say WHY: with central's runtime store on SQL, an expander left on SQLite would resolve pre-host `${secret:}` refs from a stale/empty local store — values silently diverging from what the running node serves. (Central+Grpc with a blank connection string deliberately falls through to the SQLite path: the boot is about to fail in `AddScadaBridgeSecrets` with the clear message; don't duplicate the throw here.) +3. `appsettings.json`: in the existing Secrets area add `"SqlServer": { "ConnectionString": "" }` with a `_comment` stating it is required on central when `Replication:Mode=Grpc`, must be a literal/env value (never `${secret:}`), and sites leave it empty. +4. Wiring pins (extend `SecretsReplicationWiringTests`, red first): + - Central + Grpc + connection string → `ISecretStore` resolves to `SqlServerSecretStore`; hub service still registered. + - Central + Grpc + blank connection string → registration throws, message names `Secrets:SqlServer:ConnectionString`. + - Central + Grpc + `${secret:…}` connection string → throws, message names the key and the circularity. + - Site + Grpc → `ISecretStore` still SQLite; SqlServer package types NOT in the container (sites-don't-talk-to-SQL pin). + - SqlServer-mode and replication-off paths byte-identical (existing pins keep passing). +5. Suite: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` full green, 0 warnings. Commit on the branch; do NOT merge yet (merge happens with Task 4). + +--- + +### Task 3: Publish family 0.5.0 + +**Classification:** small +**Estimated implement time:** ~3 min +**Blocked by:** Task 1 + +**Repo:** scadaproj. After Task 1's review approves and its branch merges to `main`: + +1. `ZB.MOM.WW.Secrets/Directory.Build.props`: `0.4.1` → `0.5.0`. CHANGELOG: retitle the Unreleased/0.5.0 entry with the date. +2. `dotnet test` full suite green → `dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o artifacts` → push all 6 packages to the `dohertj2-gitea` feed (same `dotnet nuget push` invocation as 0.4.1; API key via `git credential fill`). +3. Restore-verify from a scratch cache (same probe as 0.4.1). +4. Commit + push `main`. + +--- + +### Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring + +**Classification:** standard +**Estimated implement time:** ~3 min +**Blocked by:** Tasks 2, 3 + +**Repo:** ScadaBridge, continues `feat/secrets-central-shared-store`. + +1. `Directory.Packages.props`: all `ZB.MOM.WW.Secrets*` pins `0.4.1` → `0.5.0`. +2. `appsettings.json` `Secrets:GrpcHub`: add `"FallbackEndpoints": []` placeholder + `_comment`: list the SECOND central node here; safe only because both centrals serve one shared SQL store. +3. Wiring pins: site + Grpc + one fallback endpoint → `ISecretsHubReader` resolves to the failover reader; zero fallbacks → plain `GrpcSecretsHubClient` (pin the unchanged-default shape). +4. Full suite green, 0 warnings. Merge `feat/secrets-central-shared-store` → `main` after review, push. + +--- + +### Task 5: Live gate on the docker cluster + +**Classification:** high-risk +**Estimated implement time:** gate run, ~15 min wall +**Blocked by:** Task 4 + +**Repo:** ScadaBridge docker rig (same rig as the 2026-08-07 hub gate; central pair + site-a pair with `Secrets:Replication:Mode=Grpc`). Produce `ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md` recording every check with evidence, then merge/push any rig-compose changes. + +Rig prep: central-a/central-b gain `Secrets__SqlServer__ConnectionString` → the rig's SQL Server container (dedicated database or schema; default `SchemaName` from `SqlServerSecretsOptions` is fine); site-a nodes gain `Secrets__GrpcHub__FallbackEndpoints__0` → central-b's hub endpoint. Rig secrets/tokens are the existing dev values — never real credentials. + +**Checks (all must PASS):** +1. **Central-pair parity by construction:** seed a secret through central-a's store (CLI against the SQL store, or `/admin/secrets`); `GetManifest` from BOTH central hubs returns it identically (same HLC/ciphertext). The 2026-08-07 gate's empty-manifest-from-central-b defect must be unreproducible. +2. **Failover:** stop central-a; site-a's next sweep logs the failover Warning (naming both endpoints, no token material) and converges a NEW secret written while central-a is down (written via central-b `/admin/secrets` or CLI → shared SQL). Restart central-a; sweeps keep succeeding (sticky on central-b is acceptable — verify no flapping). +3. **Delete-while-follower-offline** (the untested residual from #4): stop one site-a node; delete a secret centrally; restart the node → tombstone applied, row does NOT resurrect, and the OTHER site node also converged the delete. +4. **Layer-A expander on SQL:** a `${secret:}` config reference on central resolves at boot from the SHARED store (seed the secret via central-b, boot central-a, verify the expanded value landed) — proving the expander followed the store swap. +5. **Fail-closed negatives:** central with Grpc mode + blank `Secrets:SqlServer:ConnectionString` refuses to boot naming the key; SQL container stopped → hub calls fail (site logs sweep failure, keeps serving last-known-good; NO empty manifest served). + +Gate-method rule from last time: never seed a bind-mounted SQLite store from the macOS host — for site-side assertions use exec-in-container or throwaway containers. (Central's store is now SQL, so central-side seeding is just SQL/CLI/UI.) + +--- + +### Task 6: Closeout + +**Classification:** small +**Estimated implement time:** ~4 min +**Blocked by:** Task 5 + +1. **scadaproj#4**: close with a comment recording the resolution (shared SQL store + FallbackEndpoints, 0.5.0, gate SHAs), including that the delete-while-follower-offline residual was gated (check 3) and the denial-warning-alerting note (sum per-window deltas) which stands independently. +2. `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`: ScadaBridge topology = central pair on one shared SQL store + Grpc hub; **rewrap fan-out: ONE run against central's SQL store** (the CLI auto-targets it via `Secrets:SqlServer:ConnectionString` in central's appsettings) **+ both nodes of every site pair** — delete the "both central nodes" caveat. Document the FallbackEndpoints same-store-only rule. +3. `components/secrets/GAPS.md`: append the resolution section. +4. scadaproj `CLAUDE.md` Secrets row: 0.5.0, ScadaBridge pins 0.5.0, central shared SQL store, #4 CLOSED, gate reference. +5. Update this plan's `.tasks.json` ledger; commit scadaproj `main` + push (index-edit rule: commit on `main`). diff --git a/docs/plans/2026-08-07-secrets-central-shared-sql-store.md.tasks.json b/docs/plans/2026-08-07-secrets-central-shared-sql-store.md.tasks.json new file mode 100644 index 0000000..9b8d601 --- /dev/null +++ b/docs/plans/2026-08-07-secrets-central-shared-sql-store.md.tasks.json @@ -0,0 +1,12 @@ +{ + "planPath": "docs/plans/2026-08-07-secrets-central-shared-sql-store.md", + "tasks": [ + {"id": 1, "subject": "Task 1: Library — FallbackEndpoints + failover hub reader", "status": "pending", "classification": "high-risk", "parallelizableWith": [2]}, + {"id": 2, "subject": "Task 2: ScadaBridge — central shared SQL store + Layer-A expander", "status": "pending", "classification": "high-risk", "parallelizableWith": [1]}, + {"id": 3, "subject": "Task 3: Publish family 0.5.0", "status": "pending", "classification": "small", "blockedBy": [1]}, + {"id": 4, "subject": "Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring", "status": "pending", "classification": "standard", "blockedBy": [2, 3]}, + {"id": 5, "subject": "Task 5: Live gate on the docker cluster", "status": "pending", "classification": "high-risk", "blockedBy": [4]}, + {"id": 6, "subject": "Task 6: Closeout — #4 close, runbook, GAPS, index", "status": "pending", "classification": "small", "blockedBy": [5]} + ], + "lastUpdated": "2026-08-07" +}