dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
92ff6a0d8cdcad932a5be13732e551b758890902
scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md
T

17 KiB
Raw Blame History

Secrets: Central Shared SQL Store + Hub Endpoint Failover (scadaproj#4)

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development to implement this plan task-by-task.

Goal: Resolve scadaproj#4 — ScadaBridge's central pair diverging on secrets — by pointing both central nodes' ISecretStore at one shared SQL Server store (the SqlServer replicator's existing shared-store mode), then making site-side hub endpoint failover safe and shipping it (FallbackEndpoints, family 0.5.0).

Architecture: In Secrets:Replication:Mode=Grpc, central's store becomes AddZbSecretsSqlServerStore (one copy of every row — divergence is impossible by construction, so both hub instances serve identical manifests). Sites are unchanged (local SQLite + pull-only sweep) except the sweep client gains ordered fallback endpoints with sticky preference. The Layer-A pre-host ${secret:} expander on central must follow the store swap or pre-host refs resolve from a stale local SQLite store.

Tech stack: .NET 10, Grpc.Net.Client 2.76.0, existing ZB.MOM.WW.Secrets.Replicator.SqlServer shared-store mode (live-tested vs SQL Server 2022 since 0.2.0), docker-dev cluster for the live gate.

Decision record (user-approved 2026-08-07): shared SQL store chosen over central-pair convergence/replication because it removes divergence structurally (no replication-lag window), makes endpoint failover trivially safe (either central answers with the truth), and collapses central's rewrap-all to one run. Accepted trade: central secret resolution now depends on SQL Server availability — acceptable because ScadaBridge central is already SQL-coupled to its core, and a SQL outage makes the hub throw (sites keep last-known-good), never serve empty.

Standing rules (every task, every subagent)

  • Never echo PSK/bearer/key/KEK material in logs, exception messages, status output, commit messages, or reports. Error messages name exact config keys instead.
  • Fail closed on every unresolvable auth/config case.
  • gRPC auth interceptors keep exactly one public constructor.
  • Do NOT touch ScadaBridge test.txt/sqllogin.txt (live production credentials) or the deploy/ overlay.
  • Gitea API token via git credential fill (protocol=https, host=gitea.dohertylan.com, username=dohertj2) — never echo it.
  • Every commit ends with trailer: Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
  • scadaproj repo: work branches off main @ 294eb16. ScadaBridge repo: off main @ f6c3f7c5. Both currently clean.
  • Full offline suites must be green (0 warnings, TreatWarningsAsErrors) before any merge.

Task 1: Library — FallbackEndpoints + failover hub reader

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 2 (different repo)

Repo: scadaproj (~/Desktop/scadaproj), worktree branch feat/secrets-hub-failover off main.

Files:

  • Modify: ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/SecretsGrpcHubClientOptions.cs
  • Create: ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/FailoverSecretsHubReader.cs
  • Modify: ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.Grpc/DependencyInjection/SecretsGrpcHubClientExtensions.cs
  • Modify: ZB.MOM.WW.Secrets/CHANGELOG.md (0.5.0 Unreleased entry)
  • Test: ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests/ (new FailoverSecretsHubReaderTests.cs, extend SecretsGrpcHubClientOptionsTests, extend client DI tests)

Spec:

  1. SecretsGrpcHubClientOptions gains: 
    /// <summary>
/// Additional hub addresses tried, in order, when a call to <see cref="Endpoint"/> (or an
/// earlier fallback) fails with a transport-level gRPC error. Optional; empty by default.
/// Meaningful only when every listed endpoint serves the SAME store — for ScadaBridge that
/// is the central pair over one shared SQL Server store. Do NOT list endpoints backed by
/// independent stores: failing over to an emptier hub is a silent convergence stop, which is
/// the exact defect (scadaproj#4) the shared store exists to prevent.
/// </summary>
public IList<string> FallbackEndpoints { get; } = [];
    (Getter-only IList<string> — the config binder adds to it. Same http-cleartext caveat as Endpoint applies; reference it.)
  2. Validate(sectionPath): each fallback entry must be an absolute http/https URI (same check as Endpoint, message naming {sectionPath}:FallbackEndpoints); reject blank entries; reject an entry equal (OrdinalIgnoreCase) to Endpoint or a duplicate of another fallback — a duplicated endpoint silently halves the failover the operator thinks they configured.
  3. New internal sealed partial class FailoverSecretsHubReader : ISecretsHubReader:
    • Constructed with IReadOnlyList<ISecretsHubReader> readers (ordered: primary first) + IReadOnlyList<string> endpointLabels (for logging) + ILogger.
    • Both GetManifestAsync and GetManyAsync: start at the sticky index (last endpoint that answered, Volatile.Read/Volatile.Write on an int field, initial 0), try each reader in rotation order; on RpcException log a Warning naming the failed endpoint URI and the next one tried (URIs are not secrets; the token must never appear), continue; when one answers, store its index as sticky and return. All readers failed → rethrow the last RpcException (the sweep's catch-all handles it as today).
    • Only RpcException triggers failover. OperationCanceledException and anything else propagates immediately — cancellation is the host stopping, not an endpoint fault.
    • Sticky rationale (must be in the class doc): a sweep is GetManifest + N GetSecrets batches; without stickiness every call after a primary outage would re-pay the primary's failure (a full CallDeadline of latency) before reaching the healthy fallback.
  4. DI (AddZbSecretsGrpcHubClient):
    • Zero fallbacks → the registration graph is byte-identical to today (single keyed channel under ChannelServiceKey, single GrpcSecretsHubClient as ISecretsHubReader). Existing consumers and ScadaBridge's 31 wiring pins must pass unchanged.
    • With fallbacks: keyed channel per endpoint — primary keeps ChannelServiceKey, fallback i uses $"{ChannelServiceKey}:fallback:{i}" — each wrapped in its own GrpcSecretsHubClient, composed into FailoverSecretsHubReader registered as the ISecretsHubReader. Channels stay container-owned singletons (disposal).
    • GrpcSecretsHubClient itself is unchanged.
  5. Tests (TDD, red first):
    • Options: valid fallback accepted; relative/blank/duplicate-of-primary/duplicate-fallback rejected with messages naming …:FallbackEndpoints; zero fallbacks still valid.
    • Failover reader (fake ISecretsHubReaders): primary healthy → fallback never called; primary throws RpcException(Unavailable) → fallback answers + Warning logged naming both endpoints; sticky — next call goes to the fallback FIRST; recovered primary is reused after the fallback fails in turn (rotation wraps); all fail → last RpcException rethrown; OperationCanceledException does NOT fail over; concurrent calls safe (no torn sticky state).
    • Log-content test: no token material in any failover log message (reuse the existing no-secret-leak assertion pattern from the interceptor tests).
    • DI: zero-fallback shape unchanged (resolve ISecretsHubReader, assert GrpcSecretsHubClient); with fallbacks assert FailoverSecretsHubReader + one keyed channel per endpoint.
  6. CHANGELOG: add ## 0.5.0 entry under the existing format (feature: FallbackEndpoints/failover; note the same-store requirement).

Steps: red tests → implement → dotnet test ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx full green, 0 warnings → commit on the worktree branch.

Task 2: ScadaBridge — central shared SQL store + Layer-A expander

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 1 (different repo)

Repo: ScadaBridge (~/Desktop/ScadaBridge), branch feat/secrets-central-shared-store off main @ f6c3f7c5. Uses the existing 0.4.1 packages (shared-store mode has been in Replicator.SqlServer since 0.2.0) — no dependency on Task 1/3.

Files:

  • Modify: src/ZB.MOM.WW.ScadaBridge.Host/SecretsRegistration.cs
  • Modify: src/ZB.MOM.WW.ScadaBridge.Host/Program.cs (Layer-A expander block, ~lines 53–65)
  • Modify: src/ZB.MOM.WW.ScadaBridge.Host/appsettings.json (comment/placeholder only)
  • Test: tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SecretsReplicationWiringTests.cs

Spec:

  1. SecretsRegistration.AddScadaBridgeSecrets, UsesGrpcHub(config) branch, case SecretsNodeRole.Central:
    • Before any registration, read HubConnectionStringKey (Secrets:SqlServer:ConnectionString). If blank → throw new InvalidOperationException naming the key and why: central in Grpc mode REQUIRES the shared SQL store — an independent local store per central node is exactly the divergence scadaproj#4 recorded (a hub failover would "succeed" against an empty manifest). Fail-closed, matching the mode's philosophy.
    • If the value contains ${secret: → throw naming the key: the secrets store's own connection string can never be a secret reference (the expander needs the store to resolve it — bootstrap circularity). Same rule the class remarks already document for the hub bearer token.
    • Replace services.AddZbSecrets(config, SecretsSectionPath) with services.AddZbSecretsSqlServerStore(config, SecretsSectionPath, SqlServerSectionPath), then AddZbSecretsGrpcHub as today. (AddZbSecretsSqlServerStore registers SqlServerSecretStore as ISecretStore before calling AddZbSecrets, displacing SQLite — the hub service then serves the shared store automatically.)
    • case SecretsNodeRole.Site: byte-identical to today (AddZbSecrets + AddZbSecretsGrpcHubClient).
    • Update the class-level remarks: Grpc-mode paragraph now states central's store is the shared SQL store and why; bootstrap-constraint paragraph adds the connection-string-is-never-a-ref rule; note the Layer-A expander follows the swap (point at Program.cs).
  2. Program.cs Layer-A expander: when configuration["ScadaBridge:Node:Role"] is Central (OrdinalIgnoreCase) AND SecretsRegistration.UsesGrpcHub(configuration) AND configuration[SecretsRegistration.HubConnectionStringKey] is non-blank → build the throwaway provider with .AddZbSecretsSqlServerStore(configuration, "Secrets") and run SqlServerSecretsStoreMigrator.MigrateAsync instead of the SQLite migrator. Every other case keeps the current SQLite path byte-identical. Comment must say WHY: with central's runtime store on SQL, an expander left on SQLite would resolve pre-host ${secret:} refs from a stale/empty local store — values silently diverging from what the running node serves. (Central+Grpc with a blank connection string deliberately falls through to the SQLite path: the boot is about to fail in AddScadaBridgeSecrets with the clear message; don't duplicate the throw here.)
  3. appsettings.json: in the existing Secrets area add "SqlServer": { "ConnectionString": "" } with a _comment stating it is required on central when Replication:Mode=Grpc, must be a literal/env value (never ${secret:}), and sites leave it empty.
  4. Wiring pins (extend SecretsReplicationWiringTests, red first):
    • Central + Grpc + connection string → ISecretStore resolves to SqlServerSecretStore; hub service still registered.
    • Central + Grpc + blank connection string → registration throws, message names Secrets:SqlServer:ConnectionString.
    • Central + Grpc + ${secret:…} connection string → throws, message names the key and the circularity.
    • Site + Grpc → ISecretStore still SQLite; SqlServer package types NOT in the container (sites-don't-talk-to-SQL pin).
    • SqlServer-mode and replication-off paths byte-identical (existing pins keep passing).
  5. Suite: dotnet test ZB.MOM.WW.ScadaBridge.slnx full green, 0 warnings. Commit on the branch; do NOT merge yet (merge happens with Task 4).

Task 3: Publish family 0.5.0

Classification: small Estimated implement time: ~3 min Blocked by: Task 1

Repo: scadaproj. After Task 1's review approves and its branch merges to main:

  1. ZB.MOM.WW.Secrets/Directory.Build.props: 0.4.1 → 0.5.0. CHANGELOG: retitle the Unreleased/0.5.0 entry with the date.
  2. dotnet test full suite green → dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o artifacts → push all 6 packages to the dohertj2-gitea feed (same dotnet nuget push invocation as 0.4.1; API key via git credential fill).
  3. Restore-verify from a scratch cache (same probe as 0.4.1).
  4. Commit + push main.

Task 4: ScadaBridge — pin bump 0.5.0 + fallback endpoint wiring

Classification: standard Estimated implement time: ~3 min Blocked by: Tasks 2, 3

Repo: ScadaBridge, continues feat/secrets-central-shared-store.

  1. Directory.Packages.props: all ZB.MOM.WW.Secrets* pins 0.4.1 → 0.5.0.
  2. appsettings.json Secrets:GrpcHub: add "FallbackEndpoints": [] placeholder + _comment: list the SECOND central node here; safe only because both centrals serve one shared SQL store.
  3. Wiring pins: site + Grpc + one fallback endpoint → ISecretsHubReader resolves to the failover reader; zero fallbacks → plain GrpcSecretsHubClient (pin the unchanged-default shape).
  4. Full suite green, 0 warnings. Merge feat/secrets-central-shared-store → main after review, push.

Task 5: Live gate on the docker cluster

Classification: high-risk Estimated implement time: gate run, ~15 min wall Blocked by: Task 4

Repo: ScadaBridge docker rig (same rig as the 2026-08-07 hub gate; central pair + site-a pair with Secrets:Replication:Mode=Grpc). Produce ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md recording every check with evidence, then merge/push any rig-compose changes.

Rig prep: central-a/central-b gain Secrets__SqlServer__ConnectionString → the rig's SQL Server container (dedicated database or schema; default SchemaName from SqlServerSecretsOptions is fine); site-a nodes gain Secrets__GrpcHub__FallbackEndpoints__0 → central-b's hub endpoint. Rig secrets/tokens are the existing dev values — never real credentials.

Checks (all must PASS):

  1. Central-pair parity by construction: seed a secret through central-a's store (CLI against the SQL store, or /admin/secrets); GetManifest from BOTH central hubs returns it identically (same HLC/ciphertext). The 2026-08-07 gate's empty-manifest-from-central-b defect must be unreproducible.
  2. Failover: stop central-a; site-a's next sweep logs the failover Warning (naming both endpoints, no token material) and converges a NEW secret written while central-a is down (written via central-b /admin/secrets or CLI → shared SQL). Restart central-a; sweeps keep succeeding (sticky on central-b is acceptable — verify no flapping).
  3. Delete-while-follower-offline (the untested residual from #4): stop one site-a node; delete a secret centrally; restart the node → tombstone applied, row does NOT resurrect, and the OTHER site node also converged the delete.
  4. Layer-A expander on SQL: a ${secret:} config reference on central resolves at boot from the SHARED store (seed the secret via central-b, boot central-a, verify the expanded value landed) — proving the expander followed the store swap.
  5. Fail-closed negatives: central with Grpc mode + blank Secrets:SqlServer:ConnectionString refuses to boot naming the key; SQL container stopped → hub calls fail (site logs sweep failure, keeps serving last-known-good; NO empty manifest served).

Gate-method rule from last time: never seed a bind-mounted SQLite store from the macOS host — for site-side assertions use exec-in-container or throwaway containers. (Central's store is now SQL, so central-side seeding is just SQL/CLI/UI.)

Task 6: Closeout

Classification: small Estimated implement time: ~4 min Blocked by: Task 5

  1. scadaproj#4: close with a comment recording the resolution (shared SQL store + FallbackEndpoints, 0.5.0, gate SHAs), including that the delete-while-follower-offline residual was gated (check 3) and the denial-warning-alerting note (sum per-window deltas) which stands independently.
  2. ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md: ScadaBridge topology = central pair on one shared SQL store + Grpc hub; rewrap fan-out: ONE run against central's SQL store (the CLI auto-targets it via Secrets:SqlServer:ConnectionString in central's appsettings) + both nodes of every site pair — delete the "both central nodes" caveat. Document the FallbackEndpoints same-store-only rule.
  3. components/secrets/GAPS.md: append the resolution section.
  4. scadaproj CLAUDE.md Secrets row: 0.5.0, ScadaBridge pins 0.5.0, central shared SQL store, #4 CLOSED, gate reference.
  5. Update this plan's .tasks.json ledger; commit scadaproj main + push (index-edit rule: commit on main).
Reference in New Issue View Git Blame Copy Permalink