dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Secrets: production replication topologies decided — OtOpcUa Akka pair-local; ScadaBridge needs a central gRPC hub replicator (pull-only) #3

New Issue
Closed
opened 2026-08-07 05:27:14 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-08-07 05:27:14 -04:00
Owner
Copy Link
Copy Source

Closes the previously-untracked production-enablement decision for ZB.MOM.WW.Secrets clustered replication (flag Secrets:Replication:Enabled, default off everywhere; live-validated only on dev rigs with the committed dev KEK).

Decisions (owner, 2026-08-07)

  1. OtOpcUa production topology = Akka peer-to-peer, pair-local. Fits the post-mesh-Phase-6 reality (one 2-node Akka mesh per application cluster; sites have no SQL). Replication is pair-local by construction; each pair is seeded independently (central pair via /admin/secrets, site pairs via the secret CLI or ciphertext-only bundle import).
  2. One fleet-wide production KEK, delivered per node via appsettings/env — never committed. A single KEK keeps bundle-based seeding possible between pairs and makes rotation one procedure. Verify per pair with the digest-compare check (runbook §0).
  3. ScadaBridge does NOT use the SqlServer hub replicator. Site servers never talk to central SQL directly — they talk to central for everything (templates, config, deploys), and secrets must follow the same path. The shipped hub-replication mode has site nodes opening a connection string to central SQL, which violates that topology rule.
  4. Build a central-hosted gRPC secrets hub instead — new shared package ZB.MOM.WW.Secrets.Replicator.Grpc: central hosts a secrets_hub.v1 gRPC service backed by its own store; site nodes run the existing reconciler sweep against it over the established central↔site gRPC plane (bearer auth, fail-closed). Pull-only by design — updates originate at central only (owner-confirmed invariant, matching how all site-bound configuration flows). Offline posture unchanged from the hub mode: a site booting with central unreachable serves last-known-good local secrets and converges on the first successful sweep.

Work items

  • Design doc: (committed 591307c) docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md (this repo)
  • Runbook: (committed 591307c) add a Production enablement section to ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md
  • Implement ZB.MOM.WW.Secrets.Replicator.Grpc (proto + central hub service + pull-only sweep client), offline suite — (shipped 0.4.0 @ 321ab22, 0.4.1 @ c86cead; 6 packages on the feed, restore-verified. Pull-only enforced at the contract: exactly two RPCs, GetManifest/GetSecrets, no write RPC — pinned by a descriptor-reflection test. Fail-closed constant-time bearer auth, uniform denial, GetSecrets batch cap. Grpc suite 166 tests; solution 444/0/15. ZB.MOM.WW.Secrets/CHANGELOG.md created.)
  • ScadaBridge wiring both ends + live gate on the docker cluster (site boots with central down => serves cached secrets; central add/delete converges <= sweep interval) — (ScadaBridge main @ f6c3f7c5, fast-forward merged + pushed. Gate PASS 4/4, ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md: convergence 5 s byte-identical + decrypt-verified on both site-a nodes; cold boot with central down serves last-known-good and resumes (14 s / 43 s); tombstone <=9 s, no resurrection; auth negatives uniform, 0.4.1 denial Warning fired, rate limit proven defer-not-drop, zero credential material in any log. Role is a method parameter (SecretsNodeRole.Central|Site) so config cannot flip a site into hosting the hub; mode key Secrets:Replication:Mode default SqlServer; hub on central's existing h2c listener CentralGrpcPort 8083; 31 DI/endpoint pins. Flag default-off; rig enables central pair + site-a only.)
  • OtOpcUa: re-run the Akka live gate on the post-Phase-6 three-mesh rig (the 2026-07-18 6-node convergence gate predates the mesh partition; pair-local semantics never gated as such: converge within pair, no cross-pair propagation, clean pair restart) — (lmxopcua master @ 1bd79c8e. PASS 4/4, OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md: pair convergence <=7 s; intra-pair delete <1 s; clean pair restart; no cross-pair propagation proven with seeded isolation controls on the other pairs, not by mere absence. Supersedes the 2026-07-18 six-node gate as evidence. Rig fix landed with it: Secrets__SqlitePath moved onto each node's per-node /app/data volume. Stale appsettings _comment claiming the feature is non-functional corrected in lmxopcua master @ 01256344.)
  • Rotation: document per-node rewrap-all fan-out under both topologies (re-wraps deliberately do not replicate) — (ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md, Production-enablement section. OtOpcUa = both nodes of every pair. ScadaBridge = both central nodes plus both nodes of every site pair, because each central node has its own independent store — see the residual below.)

Side benefit (not in scope)

A ciphertext-only central->site secrets channel is what the permanently-unreplicated notification_lists/smtp_configurations tables have been waiting for: an SMTP password stored as a ${secret:} ref rides this channel as ciphertext, removing the plaintext-password objection that blocked replicating that config.

Status 2026-08-07: all work items done; closing. One residual is carried forward as scadaproj#4 — ScadaBridge's central pair does not converge with itself and the site client dials a single endpoint, which are one design question. Secrets:Replication:Enabled remains default-off in both products; production enablement is a separate operator action, and for ScadaBridge it should wait on #4.

Closes the previously-untracked production-enablement decision for `ZB.MOM.WW.Secrets` clustered replication (flag `Secrets:Replication:Enabled`, default off everywhere; live-validated only on dev rigs with the committed dev KEK). ## Decisions (owner, 2026-08-07) 1. **OtOpcUa production topology = Akka peer-to-peer, pair-local.** Fits the post-mesh-Phase-6 reality (one 2-node Akka mesh per application cluster; sites have no SQL). Replication is pair-local by construction; each pair is seeded independently (central pair via `/admin/secrets`, site pairs via the `secret` CLI or ciphertext-only bundle import). 2. **One fleet-wide production KEK, delivered per node via appsettings/env — never committed.** A single KEK keeps bundle-based seeding possible between pairs and makes rotation one procedure. Verify per pair with the digest-compare check (runbook §0). 3. **ScadaBridge does NOT use the SqlServer hub replicator.** Site servers never talk to central SQL directly — they talk to central for everything (templates, config, deploys), and secrets must follow the same path. The shipped hub-replication mode has site nodes opening a connection string to central SQL, which violates that topology rule. 4. **Build a central-hosted gRPC secrets hub instead** — new shared package `ZB.MOM.WW.Secrets.Replicator.Grpc`: central hosts a `secrets_hub.v1` gRPC service backed by its own store; site nodes run the existing reconciler sweep against it over the established central↔site gRPC plane (bearer auth, fail-closed). **Pull-only by design — updates originate at central only** (owner-confirmed invariant, matching how all site-bound configuration flows). Offline posture unchanged from the hub mode: a site booting with central unreachable serves last-known-good local secrets and converges on the first successful sweep. ## Work items - [x] Design doc: (committed 591307c) `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md` (this repo) - [x] Runbook: (committed 591307c) add a Production enablement section to `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md` - [x] Implement `ZB.MOM.WW.Secrets.Replicator.Grpc` (proto + central hub service + pull-only sweep client), offline suite — (shipped `0.4.0` @ `321ab22`, `0.4.1` @ `c86cead`; 6 packages on the feed, restore-verified. Pull-only enforced at the contract: exactly two RPCs, `GetManifest`/`GetSecrets`, no write RPC — pinned by a descriptor-reflection test. Fail-closed constant-time bearer auth, uniform denial, `GetSecrets` batch cap. Grpc suite 166 tests; solution 444/0/15. `ZB.MOM.WW.Secrets/CHANGELOG.md` created.) - [x] ScadaBridge wiring both ends + live gate on the docker cluster (site boots with central down => serves cached secrets; central add/delete converges <= sweep interval) — (`ScadaBridge main @ f6c3f7c5`, fast-forward merged + pushed. **Gate PASS 4/4**, `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`: convergence 5 s byte-identical + decrypt-verified on both site-a nodes; cold boot with central down serves last-known-good and resumes (14 s / 43 s); tombstone <=9 s, no resurrection; auth negatives uniform, 0.4.1 denial Warning fired, rate limit proven defer-not-drop, zero credential material in any log. Role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub; mode key `Secrets:Replication:Mode` default `SqlServer`; hub on central's existing h2c listener `CentralGrpcPort` 8083; 31 DI/endpoint pins. Flag default-off; rig enables central pair + site-a only.) - [x] OtOpcUa: re-run the Akka live gate on the post-Phase-6 three-mesh rig (the 2026-07-18 6-node convergence gate predates the mesh partition; pair-local semantics never gated as such: converge within pair, no cross-pair propagation, clean pair restart) — (`lmxopcua master @ 1bd79c8e`. **PASS 4/4**, `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`: pair convergence <=7 s; intra-pair delete <1 s; clean pair restart; **no cross-pair propagation proven with seeded isolation controls on the other pairs**, not by mere absence. Supersedes the 2026-07-18 six-node gate as evidence. Rig fix landed with it: `Secrets__SqlitePath` moved onto each node's per-node `/app/data` volume. Stale appsettings `_comment` claiming the feature is non-functional corrected in `lmxopcua master @ 01256344`.) - [x] Rotation: document per-node `rewrap-all` fan-out under both topologies (re-wraps deliberately do not replicate) — (`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`, Production-enablement section. OtOpcUa = both nodes of every pair. ScadaBridge = **both central nodes** plus both nodes of every site pair, because each central node has its own independent store — see the residual below.) ## Side benefit (not in scope) A ciphertext-only central->site secrets channel is what the permanently-unreplicated `notification_lists`/`smtp_configurations` tables have been waiting for: an SMTP password stored as a `${secret:}` ref rides this channel as ciphertext, removing the plaintext-password objection that blocked replicating that config. --- **Status 2026-08-07: all work items done; closing.** One residual is carried forward as [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) — ScadaBridge's central pair does not converge with itself and the site client dials a single endpoint, which are one design question. `Secrets:Replication:Enabled` remains **default-off in both products**; production enablement is a separate operator action, and for ScadaBridge it should wait on #4.
dohertj2 commented 2026-08-07 08:58:54 -04:00
Author
Owner
Copy Link
Copy Source

Closing — all four work items delivered and live-gated

Package. ZB.MOM.WW.Secrets.Replicator.Grpc shipped in family 0.4.0 (scadaproj main @ 321ab22), then 0.4.1 (c86cead) adding a rate-limited denial Warning that the ScadaBridge live gate asked for. 6 packages on the dohertj2-gitea feed, restore-verified. ZB.MOM.WW.Secrets/CHANGELOG.md created. Grpc package suite 166 tests; solution 444 pass / 0 fail / 15 skipped.

Decision 4's pull-only invariant is enforced at the contract, not by convention: secrets_hub.v1 declares exactly two RPCs — GetManifest and GetSecrets — and there is no write RPC to call. Pinned by a descriptor-reflection test in the package and a route pin on the ScadaBridge side. Auth is fail-closed constant-time bearer with uniform denial (no oracle), a single public constructor so the auth-less shape is unconstructable, and a GetSecrets batch cap answering ResourceExhausted. The site sweep is never-throw with per-row parse isolation, so one malformed row cannot poison a sweep and offline stays last-known-good.

ScadaBridge — main @ f6c3f7c5, gate PASS 4/4 (docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md). Convergence 5 s, byte-identical and decrypt-verified on both site-a nodes; cold boot with central down serves last-known-good and resumes (14 s / 43 s); tombstone ≤9 s with no resurrection; auth negatives uniform, the denial Warning fires immediately, the rate limit proven to defer, not drop, and zero credential material in any log — including the rejected tokens themselves. Wiring detail worth recording: the node role is a method parameter (SecretsNodeRole.Central|Site), not a config key, so no configuration change can flip a site node into hosting the hub; the mode key Secrets:Replication:Mode defaults to SqlServer (legacy-equivalent); the hub is mapped on central's existing h2c control-plane listener (CentralGrpcPort, default 8083) rather than a new port; startup fails closed on a missing bearer token or endpoint. 31 DI/endpoint pin tests.

OtOpcUa — master @ 1bd79c8e, gate PASS 4/4 (docs/plans/2026-08-07-secrets-pair-local-live-gate.md). Pair convergence ≤7 s; intra-pair delete <1 s; clean pair restart; and no cross-pair propagation proven with seeded isolation controls on the other pairs, rather than inferred from a row's absence. This supersedes the 2026-07-18 six-node gate as evidence — that gate predates the mesh Phase 6 partition, and Phase 6's own secrets leg was inference-only, with no secret ever seeded. A rig defect landed with it: Secrets__SqlitePath moved onto each node's per-node /app/data volume. Separately, the OtOpcUa appsettings.json Secrets:Replication block still warned that the feature was "KNOWN NON-FUNCTIONAL … do not enable until the library is fixed" — false since 0.2.2 — corrected in lmxopcua master @ 01256344.

Docs. Runbook ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md (Production enablement + rotation fan-out), components/secrets/GAPS.md, and the scadaproj index row all updated.

Both products remain default-off. Production enablement is a separate operator action; nothing here turns replication on anywhere but the dev rigs.

Residual → scadaproj#4. ScadaBridge's central pair does not converge with itself — each central node has an independent local store, and central-b was live-observed answering an authenticated GetManifest with an empty manifest for the whole gate run while central-a held every secret. Combined with the client dialing a single Endpoint, that makes central-node authority and endpoint failover one design question rather than two: failover to the second central would succeed against an empty hub — not data loss, since pull-only never deletes, but a silent convergence stop. ScadaBridge production enablement should wait on #4. Two minor notes are folded into the same issue: delete-while-follower-offline is untested, and alerting on the denial Warning must sum per-window deltas because the warning is deliberately rate-limited.

Closing.

## Closing — all four work items delivered and live-gated **Package.** `ZB.MOM.WW.Secrets.Replicator.Grpc` shipped in family `0.4.0` (scadaproj `main @ 321ab22`), then `0.4.1` (`c86cead`) adding a rate-limited denial Warning that the ScadaBridge live gate asked for. 6 packages on the `dohertj2-gitea` feed, restore-verified. `ZB.MOM.WW.Secrets/CHANGELOG.md` created. Grpc package suite 166 tests; solution 444 pass / 0 fail / 15 skipped. Decision 4's pull-only invariant is enforced **at the contract, not by convention**: `secrets_hub.v1` declares exactly two RPCs — `GetManifest` and `GetSecrets` — and there is no write RPC to call. Pinned by a descriptor-reflection test in the package and a route pin on the ScadaBridge side. Auth is fail-closed constant-time bearer with uniform denial (no oracle), a single public constructor so the auth-less shape is unconstructable, and a `GetSecrets` batch cap answering `ResourceExhausted`. The site sweep is never-throw with per-row parse isolation, so one malformed row cannot poison a sweep and offline stays last-known-good. **ScadaBridge — `main @ f6c3f7c5`, gate PASS 4/4** (`docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`). Convergence 5 s, byte-identical and decrypt-verified on both site-a nodes; cold boot with central down serves last-known-good and resumes (14 s / 43 s); tombstone ≤9 s with no resurrection; auth negatives uniform, the denial Warning fires immediately, the rate limit proven to **defer, not drop**, and zero credential material in any log — including the rejected tokens themselves. Wiring detail worth recording: the node role is a **method parameter** (`SecretsNodeRole.Central|Site`), not a config key, so no configuration change can flip a site node into hosting the hub; the mode key `Secrets:Replication:Mode` defaults to `SqlServer` (legacy-equivalent); the hub is mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083) rather than a new port; startup fails closed on a missing bearer token or endpoint. 31 DI/endpoint pin tests. **OtOpcUa — `master @ 1bd79c8e`, gate PASS 4/4** (`docs/plans/2026-08-07-secrets-pair-local-live-gate.md`). Pair convergence ≤7 s; intra-pair delete <1 s; clean pair restart; and **no cross-pair propagation proven with seeded isolation controls on the other pairs**, rather than inferred from a row's absence. This supersedes the 2026-07-18 six-node gate as evidence — that gate predates the mesh Phase 6 partition, and Phase 6's own secrets leg was inference-only, with no secret ever seeded. A rig defect landed with it: `Secrets__SqlitePath` moved onto each node's per-node `/app/data` volume. Separately, the OtOpcUa `appsettings.json` `Secrets:Replication` block still warned that the feature was "KNOWN NON-FUNCTIONAL … do not enable until the library is fixed" — false since `0.2.2` — corrected in `lmxopcua master @ 01256344`. **Docs.** Runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md` (Production enablement + rotation fan-out), `components/secrets/GAPS.md`, and the scadaproj index row all updated. **Both products remain default-off.** Production enablement is a separate operator action; nothing here turns replication on anywhere but the dev rigs. **Residual → [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4).** ScadaBridge's central pair does not converge with itself — each central node has an independent local store, and `central-b` was live-observed answering an *authenticated* `GetManifest` with an **empty manifest** for the whole gate run while `central-a` held every secret. Combined with the client dialing a single `Endpoint`, that makes central-node authority and endpoint failover one design question rather than two: failover to the second central would *succeed* against an empty hub — not data loss, since pull-only never deletes, but a silent convergence stop. **ScadaBridge production enablement should wait on #4.** Two minor notes are folded into the same issue: delete-while-follower-offline is untested, and alerting on the denial Warning must sum per-window deltas because the warning is deliberately rate-limited. Closing.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/scadaproj#3
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?