dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Secrets gRPC hub: central-pair convergence + client endpoint failover are one design question #4

New Issue
Closed
opened 2026-08-07 08:54:52 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-08-07 08:54:52 -04:00
Owner
Copy Link
Copy Source

Found during the ScadaBridge gRPC-hub live gate (ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md, PASS 4/4, ScadaBridge main @ f6c3f7c5). Neither observation is a gate failure — the gate exercised the central-a hub only, which is the topology the rig enables — but together they leave one unanswered design question that must be settled before production enablement.

Follow-on to scadaproj#3; design at docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md.

Observed (live, verified)

1. The central pair does not converge with itself. Each central node has an independent local secret store, and nothing replicates between them. During the gate, central-b answered an authenticated GetManifest with an empty manifest for the entire run, while central-a held every secret. The hub is mapped on both central nodes (it is wired to the central role, not to a leader), so a healthy-looking, correctly-authenticating hub can serve nothing.

2. The site client dials a single Endpoint. GrpcSecretSyncService targets one configured Secrets:GrpcHub:Endpoint with no failover to the second central node.

Why these are one question, not two

Adding endpoint failover on its own makes the problem worse, not better: a client that failed over from central-a to central-b today would succeed — a valid TLS/h2c connection, a valid bearer token, a 200 response — and receive an empty manifest. Because the channel is pull-only and never deletes, that is not data loss; sites keep serving last-known-good. It is a silent convergence stop: sites stop receiving new and rotated secrets with no error anywhere, and secrets authored on the node the sites do not dial reach nobody at all.

So "which central node is authoritative for secrets" and "what does a site do when that node is unreachable" have to be answered together. Sketch of the option space (not a decision):

  • Replicate within the central pair (its own channel, or reuse an existing one) and then add client failover — both centrals answer identically, failover is safe.
  • Pin the hub to a single authoritative central node (active-node gate, as the LocalDb/active-node work does) so the non-authoritative node refuses rather than answering empty — failover then means "wait for the authority", and an empty answer becomes impossible.
  • Front both centrals behind the existing control-plane routing so the endpoint is already an abstraction over the pair, and solve authority at the routing layer.

Whichever is chosen, the fail-closed rule should extend to this case: a hub that is not authoritative must refuse, never answer empty. An empty manifest is currently indistinguishable from "central legitimately has no secrets."

Minor notes from the same gate (fold in here rather than filing separately)

  • Delete-while-follower-offline is untested. The gate proved tombstone propagation to online followers (<= 9 s, no resurrection) and cold-boot catch-up separately, but not the combination: a secret deleted at central while a site node is down, then that node returning. Pull-only + tombstones say it should be correct; it has not been observed.
  • Alerting on the 0.4.1 denial warning must sum per-window deltas. The warning is deliberately rate-limited (defer-not-drop, proven in the gate), so any alert rule that samples a single window's count will under-report a sustained credential-probing attempt. Sum the deltas across windows.

Not affected

  • Site-pair behaviour: both site-a nodes converged byte-identically, decrypt-verified (gate check 1).
  • Offline posture: cold boot with central down serves last-known-good and resumes (gate check 2).
  • Auth: uniform fail-closed denial, rate limit proven, zero credential material in logs including the rejected tokens themselves (gate check 4).
  • OtOpcUa, which uses the Akka pair-local topology, not this hub (OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md, PASS 4/4, master @ 1bd79c8e).

The Secrets:Replication:Enabled flag remains default-off in both products; production enablement is a separate operator action and should not happen for ScadaBridge until this is settled.

Found during the ScadaBridge gRPC-hub live gate (`ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`, PASS 4/4, `ScadaBridge main @ f6c3f7c5`). Neither observation is a gate failure — the gate exercised the central-a hub only, which is the topology the rig enables — but together they leave one unanswered design question that must be settled before production enablement. Follow-on to [scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3); design at `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`. ## Observed (live, verified) **1. The central pair does not converge with itself.** Each central node has an independent local secret store, and nothing replicates between them. During the gate, `central-b` answered an *authenticated* `GetManifest` with an **empty manifest** for the entire run, while `central-a` held every secret. The hub is mapped on both central nodes (it is wired to the central role, not to a leader), so a healthy-looking, correctly-authenticating hub can serve nothing. **2. The site client dials a single `Endpoint`.** `GrpcSecretSyncService` targets one configured `Secrets:GrpcHub:Endpoint` with no failover to the second central node. ## Why these are one question, not two Adding endpoint failover on its own makes the problem *worse*, not better: a client that failed over from `central-a` to `central-b` today would succeed — a valid TLS/h2c connection, a valid bearer token, a 200 response — and receive an empty manifest. Because the channel is **pull-only and never deletes**, that is not data loss; sites keep serving last-known-good. It is a **silent convergence stop**: sites stop receiving new and rotated secrets with no error anywhere, and secrets authored on the node the sites do not dial reach nobody at all. So "which central node is authoritative for secrets" and "what does a site do when that node is unreachable" have to be answered together. Sketch of the option space (not a decision): - **Replicate within the central pair** (its own channel, or reuse an existing one) and then add client failover — both centrals answer identically, failover is safe. - **Pin the hub to a single authoritative central node** (active-node gate, as the LocalDb/active-node work does) so the non-authoritative node refuses rather than answering empty — failover then means "wait for the authority", and an empty answer becomes impossible. - **Front both centrals behind the existing control-plane routing** so the endpoint is already an abstraction over the pair, and solve authority at the routing layer. Whichever is chosen, the fail-closed rule should extend to this case: **a hub that is not authoritative must refuse, never answer empty.** An empty manifest is currently indistinguishable from "central legitimately has no secrets." ## Minor notes from the same gate (fold in here rather than filing separately) - **Delete-while-follower-offline is untested.** The gate proved tombstone propagation to online followers (<= 9 s, no resurrection) and cold-boot catch-up separately, but not the combination: a secret deleted at central while a site node is down, then that node returning. Pull-only + tombstones say it should be correct; it has not been observed. - **Alerting on the 0.4.1 denial warning must sum per-window deltas.** The warning is deliberately rate-limited (defer-not-drop, proven in the gate), so any alert rule that samples a single window's count will under-report a sustained credential-probing attempt. Sum the deltas across windows. ## Not affected - Site-pair behaviour: both site-a nodes converged byte-identically, decrypt-verified (gate check 1). - Offline posture: cold boot with central down serves last-known-good and resumes (gate check 2). - Auth: uniform fail-closed denial, rate limit proven, zero credential material in logs including the rejected tokens themselves (gate check 4). - OtOpcUa, which uses the Akka pair-local topology, not this hub (`OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`, PASS 4/4, `master @ 1bd79c8e`). The `Secrets:Replication:Enabled` flag remains **default-off in both products**; production enablement is a separate operator action and should not happen for ScadaBridge until this is settled.
dohertj2 commented 2026-08-07 11:41:50 -04:00
Author
Owner
Copy Link
Copy Source

RESOLVED — shipped as the central-shared-SQL-store program (plan scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md, decision record user-approved 2026-08-07).

Decision

Central-pair convergence/replication was rejected in favour of moving both ScadaBridge central nodes' ISecretStore onto ONE shared SQL Server store (AddZbSecretsSqlServerStore, the SqlServer replicator's shared-store mode) in Secrets:Replication:Mode=Grpc. Divergence is removed structurally — there is one copy of every row, so both hub instances serve identical manifests by construction (no replication-lag window), endpoint failover becomes trivially safe (either central answers with the truth), and central's rewrap-all collapses to one run. Accepted trade: central secret resolution now depends on SQL Server availability — acceptable because central is already SQL-coupled, and a SQL outage makes the hub throw (sites keep last-known-good), never serve empty.

Shipped

  • Library 0.5.0 (scadaproj main @ 7a2980e failover reader → e62b4a5 release, 6 packages on the feed, restore-verified): Secrets:GrpcHub:FallbackEndpoints + FailoverSecretsHubReader — ordered fallback with sticky preference, RpcException-only failover, same-store-only rule documented at the option (listing independently-stored endpoints is exactly this issue's silent-convergence-stop).
  • Library 0.5.1 (scadaproj main @ 31ca940): SqlServerSecretsStoreMigrator virgin-DB concurrent bring-up race fixed — retry filter now covers 1205/2714/1913/2627 (found live: 0.5.0 crashed central-a on the pair's simultaneous first boot against an empty DB).
  • ScadaBridge (main @ 68f812ea → 4d7f09d5 gate doc/rig → e697477c 0.5.1 pins, all pushed): central+Grpc now REQUIRES Secrets:SqlServer:ConnectionString (fail-closed via shared EnsureCentralSharedStoreConnectionString, refusing blank and ${secret:} values), Layer-A pre-host expander follows the store swap, sites unchanged, FallbackEndpoints wired; 472 offline tests.

Live gate — PASS 5/5 + re-drill

ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md:

  1. Central-pair parity by construction — byte-identical hub responses from both centrals (sha256-matched GetSecrets), one SQL row; this issue's empty-manifest-from-central-b defect is unreproducible (central-b has no store of its own to be empty).
  2. Failover both directions — 18 s convergence via central-b with central-a down; reverse drill wrapped to the recovered primary in one sweep tick; sticky, no flapping.
  3. Delete-while-follower-offline — the untested residual recorded on this issue, now GATED: tombstone applied on the returning node's first sweep, no resurrection, online follower converged too.
  4. Layer-A expander on SQL — proven by three-way value discrimination (shared store vs stale SQLite decoy vs unexpanded literal) that central's pre-host ${secret:} resolution reads the SHARED store.
  5. Fail-closed negatives — blank connstr refused naming the key from the expander block; SQL down → hub errors, never an empty manifest; sites keep last-known-good. Zero credential material in any log on all 8 nodes.

Re-drill on 0.5.1: virgin-DB simultaneous central bring-up clean (both booted in the same second, schema provisioned once) where 0.5.0 crashed reproducibly.

Residuals

  • Denial-warning alerting note stands independently: the hub's auth-denial Warning is deliberately rate-limited, so alerting must sum per-window deltas, not count log lines.
  • New defect carried out of the gate: ScadaBridge#34 — a pre-Serilog unhandled boot exception wedges the container (100% CPU spin, stays running, restart policy never fires). Pre-existing (affects the old SQLite path too), reproduced twice during this gate.

Runbook (ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md) updated: ScadaBridge rewrap fan-out is now ONE central run against the shared SQL store + both nodes of every site pair; the "both central nodes" caveat this issue created is deleted. ScadaBridge production enablement no longer waits on this issue.

**RESOLVED — shipped as the central-shared-SQL-store program** (plan `scadaproj/docs/plans/2026-08-07-secrets-central-shared-sql-store.md`, decision record user-approved 2026-08-07). ## Decision Central-pair convergence/replication was **rejected** in favour of moving both ScadaBridge central nodes' `ISecretStore` onto **ONE shared SQL Server store** (`AddZbSecretsSqlServerStore`, the SqlServer replicator's shared-store mode) in `Secrets:Replication:Mode=Grpc`. Divergence is removed **structurally** — there is one copy of every row, so both hub instances serve identical manifests by construction (no replication-lag window), endpoint failover becomes trivially safe (either central answers with the truth), and central's `rewrap-all` collapses to one run. Accepted trade: central secret resolution now depends on SQL Server availability — acceptable because central is already SQL-coupled, and a SQL outage makes the hub **throw** (sites keep last-known-good), never serve empty. ## Shipped - **Library `0.5.0`** (scadaproj `main @ 7a2980e` failover reader → `e62b4a5` release, 6 packages on the feed, restore-verified): `Secrets:GrpcHub:FallbackEndpoints` + `FailoverSecretsHubReader` — ordered fallback with sticky preference, `RpcException`-only failover, same-store-only rule documented at the option (listing independently-stored endpoints is exactly this issue's silent-convergence-stop). - **Library `0.5.1`** (scadaproj `main @ 31ca940`): `SqlServerSecretsStoreMigrator` virgin-DB concurrent bring-up race fixed — retry filter now covers 1205/2714/1913/2627 (found live: 0.5.0 crashed central-a on the pair's simultaneous first boot against an empty DB). - **ScadaBridge** (`main @ 68f812ea` → `4d7f09d5` gate doc/rig → `e697477c` 0.5.1 pins, all pushed): central+Grpc now REQUIRES `Secrets:SqlServer:ConnectionString` (fail-closed via shared `EnsureCentralSharedStoreConnectionString`, refusing blank and `${secret:}` values), Layer-A pre-host expander follows the store swap, sites unchanged, `FallbackEndpoints` wired; 472 offline tests. ## Live gate — PASS 5/5 + re-drill `ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`: 1. **Central-pair parity by construction** — byte-identical hub responses from both centrals (sha256-matched `GetSecrets`), one SQL row; **this issue's empty-manifest-from-central-b defect is unreproducible** (central-b has no store of its own to be empty). 2. **Failover both directions** — 18 s convergence via central-b with central-a down; reverse drill wrapped to the recovered primary in one sweep tick; sticky, no flapping. 3. **Delete-while-follower-offline** — the untested residual recorded on this issue, now GATED: tombstone applied on the returning node's first sweep, no resurrection, online follower converged too. 4. **Layer-A expander on SQL** — proven by three-way value discrimination (shared store vs stale SQLite decoy vs unexpanded literal) that central's pre-host `${secret:}` resolution reads the SHARED store. 5. **Fail-closed negatives** — blank connstr refused naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good. Zero credential material in any log on all 8 nodes. Re-drill on 0.5.1: virgin-DB simultaneous central bring-up clean (both booted in the same second, schema provisioned once) where 0.5.0 crashed reproducibly. ## Residuals - **Denial-warning alerting note stands independently**: the hub's auth-denial Warning is deliberately rate-limited, so alerting must **sum per-window deltas**, not count log lines. - **New defect carried out of the gate**: [ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34) — a pre-Serilog unhandled boot exception **wedges the container** (100% CPU spin, stays `running`, restart policy never fires). Pre-existing (affects the old SQLite path too), reproduced twice during this gate. Runbook (`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`) updated: ScadaBridge rewrap fan-out is now ONE central run against the shared SQL store + both nodes of every site pair; the "both central nodes" caveat this issue created is deleted. ScadaBridge production enablement no longer waits on this issue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/scadaproj#4
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?