scadaproj/docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md

Secrets: production replication topologies + the central gRPC hub replicator

Status: Design — decisions owner-approved 2026-08-07; implementation NOT started. Tracking: scadaproj#3 Supersedes nothing — this closes the previously-untracked production-enablement gap recorded in components/secrets/GAPS.md and the CLAUDE.md Secrets row ("production enablement would need real KEK distribution — own decision/issue"). This is that decision and that issue.

---

1. What this decides

ZB.MOM.WW.Secrets clustered replication (lib 0.2.0+) is built, published, wired into the apps behind Secrets:Replication:Enabled (default false), and live-validated only on dev rigs with the committed dev KEK. Nothing decided how a production deployment would enable it. Four decisions, all owner-approved 2026-08-07:

D1 — OtOpcUa production topology: Akka peer-to-peer, pair-local

Since mesh Phase 6, each OtOpcUa application cluster is its own 2-node Akka mesh and sites have no SQL — so the Akka replicator is the only topology sites can run, and its scope is pair-local by construction: a secret seeded on one node of a pair converges to its partner and nowhere else. That is accepted, not worked around. Each pair is seeded independently:

• central pair — via the /admin/secrets page (AdminUI runs only on central);
• site pairs — via the secret interactive CLI on-box, or the 0.3.0 console's ciphertext-only bundle export/import (which requires D2's shared KEK).

D2 — One fleet-wide production KEK, delivered per node via appsettings/env, never committed

Per-pair KEKs would work (secrets never cross pairs) but a single fleet KEK keeps bundle-based seeding possible between pairs and makes rotation one procedure instead of N. Delivery is the node's appsettings/environment override — the same never-committed channel the rigs use for their observability overrides. Verification is the runbook §0 digest compare, per pair. The committed dev KEK remains dev-only.

D3 — ScadaBridge does NOT use the SqlServer hub replicator

The shipped hub-replication mode has each site node open a connection string to central SQL and sweep against it directly. That violates ScadaBridge's topology rule: site servers never talk to the DB — for templates, config, and deploys they talk to central, and central talks to its DB. Secrets must follow the same path. (The same rule is why OtOpcUa's mesh Phase 6 dropped SQL-hub for its sites.) The SqlServer package stays on the shelf — it remains correct for a topology whose nodes legitimately share a database — but ScadaBridge will not enable it.

D4 — Build ZB.MOM.WW.Secrets.Replicator.Grpc: a central-hosted secrets hub, pull-only

New shared package, third sibling to the SqlServer and AkkaDotNet replicators. Central hosts a small secrets_hub.v1 gRPC service backed by its own store (central SQL, which only central touches); site nodes run the existing reconciler sweep against that endpoint instead of a SQL connection.

Pull-only is a design invariant, not a simplification: updates originate at central only. That is how every other site-bound configuration flows in ScadaBridge, and it is owner-confirmed for secrets. Sites never push; the hub service exposes no write RPC at all — the narrower contract is the enforcement.

---

2. Why the seams make D4 cheap

The SqlServer package already splits transport from mechanics:

• ISecretReplicationHub — the transport seam the sweep talks to;
• SecretReplicationReconciler (core) — bidirectional anti-entropy with SecretLastWriterWins and tombstone-based delete propagation;
• SqlServerSecretSyncService — the BackgroundService sweep pattern: SyncOnStartup, a PeriodicTimer interval (default 30 s), and the documented guarantee that a hub outage degrades the node to serving its last-known-good local secrets rather than crashing the host (the sweep never throws out of ExecuteAsync; failures log and retry next tick).

The gRPC package reuses all of that. What is new:

1. secrets_hub.v1 proto — read-only surface mirroring the pull half of ISecretReplicationHub: enumerate rows with versions/timestamps (tombstones included), fetch row content. Ciphertext only — plaintext never crosses the wire, same trust boundary as every other topology.
2. Central-side service — hosted on ScadaBridge central's existing gRPC listener, adapting the central store to the proto. Bearer auth on the established central↔site pattern, fail-closed: unauthenticated or unauthorized ⇒ deny; a misconfigured hub serves nothing rather than everything. Exactly one public constructor on the interceptor (the Grpc.AspNetCore multiple-ctor trap). No key material in logs or error messages.
3. Site-side sweep client — the SqlServerSecretSyncService shape with a gRPC-backed hub and the reconciler run pull-direction only. Same options surface (SyncInterval, SyncOnStartup, command timeout equivalent as a gRPC deadline).
4. Bootstrap simplification — the SqlServer mode's "the hub connstr cannot come from the hub" seeding constraint disappears: sites already hold the central endpoint and its credentials.

Offline posture, restated because it is the point: a site cluster booting with no route to central resolves every ${secret:} ref from its last-synced local store, and converges automatically on the first successful sweep — the same last-known-good stance as the #485 config guarantee and LocalDb FetchAndCache.

3. Out of scope, recorded

• Bidirectional sync / site-side authoring. Excluded by D4's invariant. If a genuine need ever appears it is a new decision, not a flag.
• OtOpcUa adoption of the gRPC hub. D1 stands (Akka pair-local). The hub would slot onto OtOpcUa's existing central planes if central-authored secret flow is ever wanted there; noted, not planned.
• notification_lists/smtp_configurations replication. Those tables stayed permanently unreplicated because the channel would carry plaintext SMTP passwords. An SMTP password stored as a ${secret:} ref rides this channel as ciphertext, removing that objection — a follow-up candidate once the hub ships, not part of this work.

4. Live gates (each blocks its half's "done")

ScadaBridge gRPC hub gate (docker cluster):

1. Secret added centrally converges to both nodes of every site pair ≤ one sweep interval, byte-identical ciphertext, decrypt-verified.
2. Site pair cold-boots with central down ⇒ serves last-known-good secrets; converges after central returns.
3. Central delete (tombstone) propagates; the row does not resurrect on site restart.
4. Auth negatives fail closed: bad/absent bearer ⇒ denied + one warning; no key material in any log line.

OtOpcUa pair-local re-gate (three-mesh rig): the 2026-07-18 six-node convergence gate ran on the pre-Phase-6 fleet-wide mesh and is obsolete as evidence. Re-gate pair-local semantics as such: seed on one node of a pair ⇒ partner converges ≤ 60 s; no cross-pair propagation (now the correct behavior, not a failure); delete propagates; clean pair restart.

5. Rotation under these topologies

Re-wraps deliberately do not replicate, so a KEK rotation is secret rewrap-all run against each node's store individually — both nodes of every pair (OtOpcUa), and central + both nodes of every site pair (ScadaBridge) — with the KEK doctor as recovery if a node is missed. The kek-rotation runbook covers mechanics; the clustered-secrets runbook's Production enablement section covers the fan-out.