docs(secrets): production replication topologies decided — OtOpcUa Akka pair-local, ScadaBridge central gRPC hub (pull-only); scadaproj#3
Design doc + runbook Production-enablement section + GAPS/index updates. The previously-untracked production-KEK decision is now recorded: one fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects the SqlServer hub on topology grounds (sites talk to central, never the DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package — pull-only, updates originate at central only, by design. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
@@ -187,6 +187,30 @@ Start at the left and move right only when a stated requirement forces it.
|
||||
|
||||
---
|
||||
|
||||
## Production enablement (decided 2026-08-07)
|
||||
|
||||
Which topology each app runs in production is decided — tracked as
|
||||
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design at
|
||||
`docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md` (scadaproj):
|
||||
|
||||
- **OtOpcUa: Akka peer-to-peer, pair-local.** Each application cluster is its own 2-node Akka
|
||||
mesh (mesh Phase 6), so replication converges within a pair and nowhere else — seed each pair
|
||||
independently (central via `/admin/secrets`; site pairs via the `secret` CLI or ciphertext-only
|
||||
bundle import).
|
||||
- **ScadaBridge: central gRPC hub (§2's SQL-hub mode is NOT used).** Site servers never talk to
|
||||
the DB directly — they talk to central for everything — so the SqlServer hub's site-side
|
||||
connection string violates the topology rule. A `ZB.MOM.WW.Secrets.Replicator.Grpc` package
|
||||
(central-hosted `secrets_hub.v1`, **pull-only: updates originate at central only, by design**)
|
||||
replaces it; until that ships, ScadaBridge replication stays off.
|
||||
- **KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override,
|
||||
never committed.** The committed dev KEK is dev-only. Verify per pair with the §0 digest
|
||||
compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
|
||||
- **Rotation fan-out:** re-wraps do not replicate, so `secret rewrap-all` runs against each
|
||||
node's store individually (both nodes of every pair; on ScadaBridge also central). The KEK
|
||||
doctor is the recovery path for a missed node.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`SecretDecryptionException` / `kek_id` mismatch after enabling replication.**
|
||||
|
||||
@@ -168,9 +168,19 @@ project 38/38 green.
|
||||
page owning the route with `@rendermode InteractiveServer` + policy/route-parity pin tests
|
||||
(`SecretsPageWiringTests`); Playwright-verified live on the rig (circuit + working Add-secret).
|
||||
The wrapper is the reference pattern for any static-router consumer of Secrets.Ui.
|
||||
**Production/wonder enablement is deliberately NOT done** — needs its own decision + real KEK
|
||||
distribution (never the committed dev key) per the clustered-secrets runbook.
|
||||
- ✅ ScadaBridge's hub topology **is** live-validated and safe to enable (config + a shared KEK).
|
||||
**Production/wonder enablement is deliberately NOT done** — ~~needs its own decision + real KEK
|
||||
distribution (never the committed dev key) per the clustered-secrets runbook~~ **the decision is
|
||||
now made and tracked (2026-08-07,
|
||||
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3); design
|
||||
`docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka
|
||||
pair-local with one fleet-wide appsettings-delivered KEK; ScadaBridge = a new central gRPC hub
|
||||
replicator (`ZB.MOM.WW.Secrets.Replicator.Grpc`, pull-only — updates originate at central only),
|
||||
because the SqlServer hub's site-side connection string violates ScadaBridge's
|
||||
sites-talk-to-central-not-the-DB rule. Enablement waits on the hub package (ScadaBridge) and the
|
||||
pair-local re-gate on the post-Phase-6 rig (OtOpcUa).
|
||||
- ✅ ScadaBridge's hub topology **is** live-validated and technically safe to enable — but the
|
||||
2026-08-07 decision rejects it for production on topology grounds (site nodes must not hold a
|
||||
central SQL connection string); superseded by the gRPC hub above.
|
||||
|
||||
Harness: `scratchpad/livegate/` (two scenarios, `akka` / `sql`; not committed).
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# Secrets: production replication topologies + the central gRPC hub replicator
|
||||
|
||||
**Status:** Design — decisions owner-approved 2026-08-07; implementation NOT started.
|
||||
**Tracking:** [scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3)
|
||||
**Supersedes nothing** — this closes the previously-untracked production-enablement gap recorded in
|
||||
`components/secrets/GAPS.md` and the CLAUDE.md Secrets row ("production enablement would need real
|
||||
KEK distribution — own decision/issue"). This is that decision and that issue.
|
||||
|
||||
---
|
||||
|
||||
## 1. What this decides
|
||||
|
||||
`ZB.MOM.WW.Secrets` clustered replication (lib 0.2.0+) is built, published, wired into the apps
|
||||
behind `Secrets:Replication:Enabled` (default **false**), and live-validated only on dev rigs with
|
||||
the committed dev KEK. Nothing decided how a production deployment would enable it. Four decisions,
|
||||
all owner-approved 2026-08-07:
|
||||
|
||||
### D1 — OtOpcUa production topology: Akka peer-to-peer, pair-local
|
||||
|
||||
Since mesh Phase 6, each OtOpcUa application cluster is its own 2-node Akka mesh and sites have no
|
||||
SQL — so the Akka replicator is the only topology sites can run, and its scope is **pair-local by
|
||||
construction**: a secret seeded on one node of a pair converges to its partner and nowhere else.
|
||||
That is accepted, not worked around. Each pair is seeded independently:
|
||||
|
||||
- **central pair** — via the `/admin/secrets` page (AdminUI runs only on central);
|
||||
- **site pairs** — via the `secret` interactive CLI on-box, or the 0.3.0 console's
|
||||
ciphertext-only bundle export/import (which requires D2's shared KEK).
|
||||
|
||||
### D2 — One fleet-wide production KEK, delivered per node via appsettings/env, never committed
|
||||
|
||||
Per-pair KEKs would work (secrets never cross pairs) but a single fleet KEK keeps bundle-based
|
||||
seeding possible between pairs and makes rotation one procedure instead of N. Delivery is the
|
||||
node's appsettings/environment override — the same never-committed channel the rigs use for their
|
||||
observability overrides. Verification is the runbook §0 digest compare, per pair. The committed
|
||||
dev KEK remains dev-only.
|
||||
|
||||
### D3 — ScadaBridge does NOT use the SqlServer hub replicator
|
||||
|
||||
**The shipped hub-replication mode has each site node open a connection string to central SQL and
|
||||
sweep against it directly. That violates ScadaBridge's topology rule: site servers never talk to
|
||||
the DB — for templates, config, and deploys they talk to central, and central talks to its DB.**
|
||||
Secrets must follow the same path. (The same rule is why OtOpcUa's mesh Phase 6 dropped SQL-hub
|
||||
for its sites.) The SqlServer package stays on the shelf — it remains correct for a topology whose
|
||||
nodes legitimately share a database — but ScadaBridge will not enable it.
|
||||
|
||||
### D4 — Build `ZB.MOM.WW.Secrets.Replicator.Grpc`: a central-hosted secrets hub, pull-only
|
||||
|
||||
New shared package, third sibling to the SqlServer and AkkaDotNet replicators. Central hosts a
|
||||
small `secrets_hub.v1` gRPC service backed by **its own** store (central SQL, which only central
|
||||
touches); site nodes run the existing reconciler sweep against that endpoint instead of a SQL
|
||||
connection.
|
||||
|
||||
**Pull-only is a design invariant, not a simplification: updates originate at central only.**
|
||||
That is how every other site-bound configuration flows in ScadaBridge, and it is owner-confirmed
|
||||
for secrets. Sites never push; the hub service exposes no write RPC at all — the narrower contract
|
||||
is the enforcement.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why the seams make D4 cheap
|
||||
|
||||
The SqlServer package already splits transport from mechanics:
|
||||
|
||||
- `ISecretReplicationHub` — the transport seam the sweep talks to;
|
||||
- `SecretReplicationReconciler` (core) — bidirectional anti-entropy with `SecretLastWriterWins`
|
||||
and tombstone-based delete propagation;
|
||||
- `SqlServerSecretSyncService` — the `BackgroundService` sweep pattern: `SyncOnStartup`, a
|
||||
`PeriodicTimer` interval (default 30 s), and the documented guarantee that **a hub outage
|
||||
degrades the node to serving its last-known-good local secrets rather than crashing the host**
|
||||
(the sweep never throws out of `ExecuteAsync`; failures log and retry next tick).
|
||||
|
||||
The gRPC package reuses all of that. What is new:
|
||||
|
||||
1. **`secrets_hub.v1` proto** — read-only surface mirroring the pull half of
|
||||
`ISecretReplicationHub`: enumerate rows with versions/timestamps (tombstones included), fetch
|
||||
row content. Ciphertext only — plaintext never crosses the wire, same trust boundary as every
|
||||
other topology.
|
||||
2. **Central-side service** — hosted on ScadaBridge central's existing gRPC listener, adapting the
|
||||
central store to the proto. Bearer auth on the established central↔site pattern, fail-closed:
|
||||
unauthenticated or unauthorized ⇒ deny; a misconfigured hub serves nothing rather than
|
||||
everything. Exactly one public constructor on the interceptor (the Grpc.AspNetCore
|
||||
multiple-ctor trap). No key material in logs or error messages.
|
||||
3. **Site-side sweep client** — the `SqlServerSecretSyncService` shape with a gRPC-backed hub and
|
||||
the reconciler run **pull-direction only**. Same options surface (`SyncInterval`,
|
||||
`SyncOnStartup`, command timeout equivalent as a gRPC deadline).
|
||||
4. **Bootstrap simplification** — the SqlServer mode's "the hub connstr cannot come from the hub"
|
||||
seeding constraint disappears: sites already hold the central endpoint and its credentials.
|
||||
|
||||
Offline posture, restated because it is the point: a site cluster booting with no route to central
|
||||
resolves every `${secret:}` ref from its last-synced local store, and converges automatically on
|
||||
the first successful sweep — the same last-known-good stance as the #485 config guarantee and
|
||||
LocalDb FetchAndCache.
|
||||
|
||||
## 3. Out of scope, recorded
|
||||
|
||||
- **Bidirectional sync / site-side authoring.** Excluded by D4's invariant. If a genuine need ever
|
||||
appears it is a new decision, not a flag.
|
||||
- **OtOpcUa adoption of the gRPC hub.** D1 stands (Akka pair-local). The hub would slot onto
|
||||
OtOpcUa's existing central planes if central-authored secret flow is ever wanted there; noted,
|
||||
not planned.
|
||||
- **`notification_lists`/`smtp_configurations` replication.** Those tables stayed permanently
|
||||
unreplicated because the channel would carry plaintext SMTP passwords. An SMTP password stored
|
||||
as a `${secret:}` ref rides this channel as ciphertext, removing that objection — a follow-up
|
||||
candidate once the hub ships, not part of this work.
|
||||
|
||||
## 4. Live gates (each blocks its half's "done")
|
||||
|
||||
**ScadaBridge gRPC hub gate (docker cluster):**
|
||||
1. Secret added centrally converges to both nodes of every site pair ≤ one sweep interval,
|
||||
byte-identical ciphertext, decrypt-verified.
|
||||
2. Site pair cold-boots with central down ⇒ serves last-known-good secrets; converges after
|
||||
central returns.
|
||||
3. Central delete (tombstone) propagates; the row does not resurrect on site restart.
|
||||
4. Auth negatives fail closed: bad/absent bearer ⇒ denied + one warning; no key material in any
|
||||
log line.
|
||||
|
||||
**OtOpcUa pair-local re-gate (three-mesh rig):** the 2026-07-18 six-node convergence gate ran on
|
||||
the pre-Phase-6 fleet-wide mesh and is obsolete as evidence. Re-gate pair-local semantics as such:
|
||||
seed on one node of a pair ⇒ partner converges ≤ 60 s; **no cross-pair propagation** (now the
|
||||
correct behavior, not a failure); delete propagates; clean pair restart.
|
||||
|
||||
## 5. Rotation under these topologies
|
||||
|
||||
Re-wraps deliberately do not replicate, so a KEK rotation is `secret rewrap-all` run against
|
||||
**each node's store individually** — both nodes of every pair (OtOpcUa), and central + both nodes
|
||||
of every site pair (ScadaBridge) — with the KEK doctor as recovery if a node is missed. The
|
||||
kek-rotation runbook covers mechanics; the clustered-secrets runbook's Production enablement
|
||||
section covers the fan-out.
|
||||
Reference in New Issue
Block a user