docs(plans): central shared SQL secrets store + hub endpoint failover (scadaproj#4)
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,174 @@
|
||||
# Secrets: Central Shared SQL Store + Hub Endpoint Failover (scadaproj#4)
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Resolve [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) — ScadaBridge's central pair diverging on secrets — by pointing both central nodes' `ISecretStore` at one shared SQL Server store (the SqlServer replicator's existing shared-store mode), then making site-side hub endpoint failover safe and shipping it (`FallbackEndpoints`, family `0.5.0`).
|
||||
|
||||
**Architecture:** In `Secrets:Replication:Mode=Grpc`, central's store becomes `AddZbSecretsSqlServerStore` (one copy of every row — divergence is impossible by construction, so both hub instances serve identical manifests). Sites are unchanged (local SQLite + pull-only sweep) except the sweep client gains ordered fallback endpoints with sticky preference. The Layer-A pre-host `${secret:}` expander on central must follow the store swap or pre-host refs resolve from a stale local SQLite store.
|
||||
|
||||
**Tech stack:** .NET 10, Grpc.Net.Client 2.76.0, existing `ZB.MOM.WW.Secrets.Replicator.SqlServer` shared-store mode (live-tested vs SQL Server 2022 since 0.2.0), docker-dev cluster for the live gate.
|
||||
|
||||
**Decision record (user-approved 2026-08-07):** shared SQL store chosen over central-pair convergence/replication because it removes divergence *structurally* (no replication-lag window), makes endpoint failover trivially safe (either central answers with the truth), and collapses central's `rewrap-all` to one run. Accepted trade: central secret resolution now depends on SQL Server availability — acceptable because ScadaBridge central is already SQL-coupled to its core, and a SQL outage makes the hub *throw* (sites keep last-known-good), never serve empty.
|
||||
|
||||
---
|
||||
|
||||
## Standing rules (every task, every subagent)
|
||||
|
||||
- **Never** echo PSK/bearer/key/KEK material in logs, exception messages, status output, commit messages, or reports. Error messages name exact config keys instead.
|
||||
- Fail closed on every unresolvable auth/config case.
|
||||
- gRPC auth interceptors keep **exactly one public constructor**.
|
||||
- Do NOT touch ScadaBridge `test.txt`/`sqllogin.txt` (live production credentials) or the `deploy/` overlay.
|
||||
- Gitea API token via `git credential fill` (protocol=https, host=gitea.dohertylan.com, username=dohertj2) — never echo it.
|
||||
- Every commit ends with trailer: `Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1`
|
||||
- scadaproj repo: work branches off `main` @ `294eb16`. ScadaBridge repo: off `main` @ `f6c3f7c5`. Both currently clean.
|
||||
- Full offline suites must be green (0 warnings, `TreatWarningsAsErrors`) before any merge.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Library — `FallbackEndpoints` + failover hub reader
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 2 (different repo)
|
||||
|
||||
**Repo:** scadaproj (`~/Desktop/scadaproj`), worktree branch `feat/secrets-hub-failover` off `main`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/SecretsGrpcHubClientOptions.cs`
|
||||
- Create: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/FailoverSecretsHubReader.cs`
|
||||
- Modify: `ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/DependencyInjection/SecretsGrpcHubClientExtensions.cs`
|
||||
- Modify: `ZB.MOM.WW.Secrets/CHANGELOG.md` (0.5.0 Unreleased entry)
|
||||
- Test: `ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/` (new `FailoverSecretsHubReaderTests.cs`, extend `SecretsGrpcHubClientOptionsTests`, extend client DI tests)
|
||||
|
||||
**Spec:**
|
||||
|
||||
1. `SecretsGrpcHubClientOptions` gains:
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Additional hub addresses tried, in order, when a call to <see cref="Endpoint"/> (or an
|
||||
/// earlier fallback) fails with a transport-level gRPC error. Optional; empty by default.
|
||||
/// Meaningful only when every listed endpoint serves the SAME store — for ScadaBridge that
|
||||
/// is the central pair over one shared SQL Server store. Do NOT list endpoints backed by
|
||||
/// independent stores: failing over to an emptier hub is a silent convergence stop, which is
|
||||
/// the exact defect (scadaproj#4) the shared store exists to prevent.
|
||||
/// </summary>
|
||||
public IList<string> FallbackEndpoints { get; } = [];
|
||||
```
|
||||
(Getter-only `IList<string>` — the config binder adds to it. Same http-cleartext caveat as `Endpoint` applies; reference it.)
|
||||
2. `Validate(sectionPath)`: each fallback entry must be an absolute http/https URI (same check as `Endpoint`, message naming `{sectionPath}:FallbackEndpoints`); reject blank entries; reject an entry equal (OrdinalIgnoreCase) to `Endpoint` or a duplicate of another fallback — a duplicated endpoint silently halves the failover the operator thinks they configured.
|
||||
3. New `internal sealed partial class FailoverSecretsHubReader : ISecretsHubReader`:
|
||||
- Constructed with `IReadOnlyList<ISecretsHubReader> readers` (ordered: primary first) + `IReadOnlyList<string> endpointLabels` (for logging) + `ILogger`.
|
||||
- Both `GetManifestAsync` and `GetManyAsync`: start at the **sticky index** (last endpoint that answered, `Volatile.Read`/`Volatile.Write` on an `int` field, initial 0), try each reader in rotation order; on `RpcException` log a Warning naming the failed endpoint URI and the next one tried (URIs are not secrets; the token must never appear), continue; when one answers, store its index as sticky and return. All readers failed → rethrow the **last** `RpcException` (the sweep's catch-all handles it as today).
|
||||
- Only `RpcException` triggers failover. `OperationCanceledException` and anything else propagates immediately — cancellation is the host stopping, not an endpoint fault.
|
||||
- Sticky rationale (must be in the class doc): a sweep is GetManifest + N GetSecrets batches; without stickiness every call after a primary outage would re-pay the primary's failure (a full CallDeadline of latency) before reaching the healthy fallback.
|
||||
4. DI (`AddZbSecretsGrpcHubClient`):
|
||||
- **Zero fallbacks → the registration graph is byte-identical to today** (single keyed channel under `ChannelServiceKey`, single `GrpcSecretsHubClient` as `ISecretsHubReader`). Existing consumers and ScadaBridge's 31 wiring pins must pass unchanged.
|
||||
- With fallbacks: keyed channel per endpoint — primary keeps `ChannelServiceKey`, fallback *i* uses `$"{ChannelServiceKey}:fallback:{i}"` — each wrapped in its own `GrpcSecretsHubClient`, composed into `FailoverSecretsHubReader` registered as the `ISecretsHubReader`. Channels stay container-owned singletons (disposal).
|
||||
- `GrpcSecretsHubClient` itself is **unchanged**.
|
||||
5. Tests (TDD, red first):
|
||||
- Options: valid fallback accepted; relative/blank/duplicate-of-primary/duplicate-fallback rejected with messages naming `…:FallbackEndpoints`; zero fallbacks still valid.
|
||||
- Failover reader (fake `ISecretsHubReader`s): primary healthy → fallback never called; primary throws `RpcException(Unavailable)` → fallback answers + Warning logged naming both endpoints; sticky — next call goes to the fallback FIRST; recovered primary is reused after the fallback fails in turn (rotation wraps); all fail → last RpcException rethrown; `OperationCanceledException` does NOT fail over; concurrent calls safe (no torn sticky state).
|
||||
- Log-content test: no token material in any failover log message (reuse the existing no-secret-leak assertion pattern from the interceptor tests).
|
||||
- DI: zero-fallback shape unchanged (resolve `ISecretsHubReader`, assert `GrpcSecretsHubClient`); with fallbacks assert `FailoverSecretsHubReader` + one keyed channel per endpoint.
|
||||
6. CHANGELOG: add `## 0.5.0` entry under the existing format (feature: FallbackEndpoints/failover; note the same-store requirement).
|
||||
|
||||
**Steps:** red tests → implement → `dotnet test ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx` full green, 0 warnings → commit on the worktree branch.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: ScadaBridge — central shared SQL store + Layer-A expander
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 1 (different repo)
|
||||
|
||||
**Repo:** ScadaBridge (`~/Desktop/ScadaBridge`), branch `feat/secrets-central-shared-store` off `main` @ `f6c3f7c5`. Uses the **existing 0.4.1 packages** (shared-store mode has been in `Replicator.SqlServer` since 0.2.0) — no dependency on Task 1/3.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Program.cs` (Layer-A expander block, ~lines 53–65)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json` (comment/placeholder only)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs`
|
||||
|
||||
**Spec:**
|
||||
|
||||
1. `SecretsRegistration.AddScadaBridgeSecrets`, `UsesGrpcHub(config)` branch, `case SecretsNodeRole.Central`:
|
||||
- **Before** any registration, read `HubConnectionStringKey` (`Secrets:SqlServer:ConnectionString`). If blank → `throw new InvalidOperationException` naming the key and why: central in Grpc mode REQUIRES the shared SQL store — an independent local store per central node is exactly the divergence scadaproj#4 recorded (a hub failover would "succeed" against an empty manifest). Fail-closed, matching the mode's philosophy.
|
||||
- If the value contains `${secret:` → throw naming the key: the secrets store's own connection string can never be a secret reference (the expander needs the store to resolve it — bootstrap circularity). Same rule the class remarks already document for the hub bearer token.
|
||||
- Replace `services.AddZbSecrets(config, SecretsSectionPath)` with `services.AddZbSecretsSqlServerStore(config, SecretsSectionPath, SqlServerSectionPath)`, then `AddZbSecretsGrpcHub` as today. (`AddZbSecretsSqlServerStore` registers `SqlServerSecretStore` as `ISecretStore` *before* calling `AddZbSecrets`, displacing SQLite — the hub service then serves the shared store automatically.)
|
||||
- `case SecretsNodeRole.Site`: **byte-identical to today** (`AddZbSecrets` + `AddZbSecretsGrpcHubClient`).
|
||||
- Update the class-level remarks: Grpc-mode paragraph now states central's store is the shared SQL store and why; bootstrap-constraint paragraph adds the connection-string-is-never-a-ref rule; note the Layer-A expander follows the swap (point at Program.cs).
|
||||
2. `Program.cs` Layer-A expander: when `configuration["ScadaBridge:Node:Role"]` is `Central` (OrdinalIgnoreCase) AND `SecretsRegistration.UsesGrpcHub(configuration)` AND `configuration[SecretsRegistration.HubConnectionStringKey]` is non-blank → build the throwaway provider with `.AddZbSecretsSqlServerStore(configuration, "Secrets")` and run `SqlServerSecretsStoreMigrator.MigrateAsync` instead of the SQLite migrator. Every other case keeps the current SQLite path byte-identical. Comment must say WHY: with central's runtime store on SQL, an expander left on SQLite would resolve pre-host `${secret:}` refs from a stale/empty local store — values silently diverging from what the running node serves. (Central+Grpc with a blank connection string deliberately falls through to the SQLite path: the boot is about to fail in `AddScadaBridgeSecrets` with the clear message; don't duplicate the throw here.)
|
||||
3. `appsettings.json`: in the existing Secrets area add `"SqlServer": { "ConnectionString": "" }` with a `_comment` stating it is required on central when `Replication:Mode=Grpc`, must be a literal/env value (never `${secret:}`), and sites leave it empty.
|
||||
4. Wiring pins (extend `SecretsReplicationWiringTests`, red first):
|
||||
- Central + Grpc + connection string → `ISecretStore` resolves to `SqlServerSecretStore`; hub service still registered.
|
||||
- Central + Grpc + blank connection string → registration throws, message names `Secrets:SqlServer:ConnectionString`.
|
||||
- Central + Grpc + `${secret:…}` connection string → throws, message names the key and the circularity.
|
||||
- Site + Grpc → `ISecretStore` still SQLite; SqlServer package types NOT in the container (sites-don't-talk-to-SQL pin).
|
||||
- SqlServer-mode and replication-off paths byte-identical (existing pins keep passing).
|
||||
5. Suite: `dotnet test ZB.MOM.WW.ScadaBridge.slnx` full green, 0 warnings. Commit on the branch; do NOT merge yet (merge happens with Task 4).
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Publish family 0.5.0
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~3 min
|
||||
**Blocked by:** Task 1
|
||||
|
||||
**Repo:** scadaproj. After Task 1's review approves and its branch merges to `main`:
|
||||
|
||||
1. `ZB.MOM.WW.Secrets/Directory.Build.props`: `0.4.1` → `0.5.0`. CHANGELOG: retitle the Unreleased/0.5.0 entry with the date.
|
||||
2. `dotnet test` full suite green → `dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o artifacts` → push all 6 packages to the `dohertj2-gitea` feed (same `dotnet nuget push` invocation as 0.4.1; API key via `git credential fill`).
|
||||
3. Restore-verify from a scratch cache (same probe as 0.4.1).
|
||||
4. Commit + push `main`.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~3 min
|
||||
**Blocked by:** Tasks 2, 3
|
||||
|
||||
**Repo:** ScadaBridge, continues `feat/secrets-central-shared-store`.
|
||||
|
||||
1. `Directory.Packages.props`: all `ZB.MOM.WW.Secrets*` pins `0.4.1` → `0.5.0`.
|
||||
2. `appsettings.json` `Secrets:GrpcHub`: add `"FallbackEndpoints": []` placeholder + `_comment`: list the SECOND central node here; safe only because both centrals serve one shared SQL store.
|
||||
3. Wiring pins: site + Grpc + one fallback endpoint → `ISecretsHubReader` resolves to the failover reader; zero fallbacks → plain `GrpcSecretsHubClient` (pin the unchanged-default shape).
|
||||
4. Full suite green, 0 warnings. Merge `feat/secrets-central-shared-store` → `main` after review, push.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Live gate on the docker cluster
|
||||
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** gate run, ~15 min wall
|
||||
**Blocked by:** Task 4
|
||||
|
||||
**Repo:** ScadaBridge docker rig (same rig as the 2026-08-07 hub gate; central pair + site-a pair with `Secrets:Replication:Mode=Grpc`). Produce `ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md` recording every check with evidence, then merge/push any rig-compose changes.
|
||||
|
||||
Rig prep: central-a/central-b gain `Secrets__SqlServer__ConnectionString` → the rig's SQL Server container (dedicated database or schema; default `SchemaName` from `SqlServerSecretsOptions` is fine); site-a nodes gain `Secrets__GrpcHub__FallbackEndpoints__0` → central-b's hub endpoint. Rig secrets/tokens are the existing dev values — never real credentials.
|
||||
|
||||
**Checks (all must PASS):**
|
||||
1. **Central-pair parity by construction:** seed a secret through central-a's store (CLI against the SQL store, or `/admin/secrets`); `GetManifest` from BOTH central hubs returns it identically (same HLC/ciphertext). The 2026-08-07 gate's empty-manifest-from-central-b defect must be unreproducible.
|
||||
2. **Failover:** stop central-a; site-a's next sweep logs the failover Warning (naming both endpoints, no token material) and converges a NEW secret written while central-a is down (written via central-b `/admin/secrets` or CLI → shared SQL). Restart central-a; sweeps keep succeeding (sticky on central-b is acceptable — verify no flapping).
|
||||
3. **Delete-while-follower-offline** (the untested residual from #4): stop one site-a node; delete a secret centrally; restart the node → tombstone applied, row does NOT resurrect, and the OTHER site node also converged the delete.
|
||||
4. **Layer-A expander on SQL:** a `${secret:}` config reference on central resolves at boot from the SHARED store (seed the secret via central-b, boot central-a, verify the expanded value landed) — proving the expander followed the store swap.
|
||||
5. **Fail-closed negatives:** central with Grpc mode + blank `Secrets:SqlServer:ConnectionString` refuses to boot naming the key; SQL container stopped → hub calls fail (site logs sweep failure, keeps serving last-known-good; NO empty manifest served).
|
||||
|
||||
Gate-method rule from last time: never seed a bind-mounted SQLite store from the macOS host — for site-side assertions use exec-in-container or throwaway containers. (Central's store is now SQL, so central-side seeding is just SQL/CLI/UI.)
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Closeout
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~4 min
|
||||
**Blocked by:** Task 5
|
||||
|
||||
1. **scadaproj#4**: close with a comment recording the resolution (shared SQL store + FallbackEndpoints, 0.5.0, gate SHAs), including that the delete-while-follower-offline residual was gated (check 3) and the denial-warning-alerting note (sum per-window deltas) which stands independently.
|
||||
2. `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`: ScadaBridge topology = central pair on one shared SQL store + Grpc hub; **rewrap fan-out: ONE run against central's SQL store** (the CLI auto-targets it via `Secrets:SqlServer:ConnectionString` in central's appsettings) **+ both nodes of every site pair** — delete the "both central nodes" caveat. Document the FallbackEndpoints same-store-only rule.
|
||||
3. `components/secrets/GAPS.md`: append the resolution section.
|
||||
4. scadaproj `CLAUDE.md` Secrets row: 0.5.0, ScadaBridge pins 0.5.0, central shared SQL store, #4 CLOSED, gate reference.
|
||||
5. Update this plan's `.tasks.json` ledger; commit scadaproj `main` + push (index-edit rule: commit on `main`).
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-08-07-secrets-central-shared-sql-store.md",
|
||||
"tasks": [
|
||||
{"id": 1, "subject": "Task 1: Library — FallbackEndpoints + failover hub reader", "status": "pending", "classification": "high-risk", "parallelizableWith": [2]},
|
||||
{"id": 2, "subject": "Task 2: ScadaBridge — central shared SQL store + Layer-A expander", "status": "pending", "classification": "high-risk", "parallelizableWith": [1]},
|
||||
{"id": 3, "subject": "Task 3: Publish family 0.5.0", "status": "pending", "classification": "small", "blockedBy": [1]},
|
||||
{"id": 4, "subject": "Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring", "status": "pending", "classification": "standard", "blockedBy": [2, 3]},
|
||||
{"id": 5, "subject": "Task 5: Live gate on the docker cluster", "status": "pending", "classification": "high-risk", "blockedBy": [4]},
|
||||
{"id": 6, "subject": "Task 6: Closeout — #4 close, runbook, GAPS, index", "status": "pending", "classification": "small", "blockedBy": [5]}
|
||||
],
|
||||
"lastUpdated": "2026-08-07"
|
||||
}
|
||||
Reference in New Issue
Block a user