scadaproj/docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md

G-7 — Clustered secret replication: design & fork resolution

Status: design-only (G-7 in components/secrets/GAPS.md). Companion executable plan: 2026-07-17-secrets-g7-sqlserver-store.md. Prerequisite G-2…G-6 have landed for both clustered apps (ScadaBridge, OtOpcUa); G-8 (KEK rotation) is built. This resolves the SPEC's "shared SQL store vs Akka replicator" fork and specifies the recommended build.

Problem

ZB.MOM.WW.Secrets stores secrets in a local SQLite file by default. The two Akka-clustered apps run multiple nodes (ScadaBridge: central pair + N site pairs; OtOpcUa: admin/driver/dev roles). A secret written on one node must be resolvable on every node that needs it, without exposing plaintext or the KEK over the wire, and without each node holding a divergent copy.

The library was built anticipating this: the schema already carries revision / updated_utc / is_deleted; ISecretStore exposes GetManifestAsync + ApplyReplicatedAsync (last-writer-wins); ISecretReplicator is a no-op seam; SecretManifestEntry supports anti-entropy. Nothing needs a migration to turn replication on.

The fork (from SPEC §"Clustered pairs")

Two ways to make one logical secret set visible cluster-wide. Both require a shared KEK — every node must resolve the same master key (shared mounted key file or shared env), because a row whose kek_id doesn't match the local provider fails closed on resolve. Ciphertext can cross the wire / sit in a shared DB safely because the KEK never does.

Option A — Shared SQL-Server ISecretStore (one source of truth)

Point every node's ISecretStore at one shared SQL-Server database instead of a per-node SQLite file. There is exactly one copy of each row; no replication, no reconciliation. The store still holds ciphertext only (the KEK stays per-node/out-of-DB). Build a SqlServerSecretStore : ISecretStore behind the existing seam (the SPEC already scoped "SQL-Server ISecretStore provider construction — build with G-7").

• Pro: zero distributed-systems code or failure modes (no split-brain, no anti-entropy lag, no tombstone GC, no LWW clock-skew). ~1 new file mirroring SqliteSecretStore's SQL + a migrator.
• Pro: both apps already run shared SQL Server (ScadaBridge ConfigDb + central dbo.*; OtOpcUa central config DB) and already share their Data-Protection key ring through SQL — a shared secret store is operationally identical to what ops already run and back up.
• Pro: rotation (G-8 rewrap-all) runs once against the one store.
• Con: availability is coupled to the shared DB. A brief outage is bridged by the resolver's in-memory TTL cache; but a node that must keep resolving secrets while partitioned from the shared DB is not served by this option (see Option B).

Option B — Akka replicator ZB.MOM.WW.Secrets.Akka (each node keeps a local store)

Each node keeps its local SQLite store; a real ISecretReplicator broadcasts each newly-written encrypted row to peers, and a cluster anti-entropy actor periodically exchanges manifests (GetManifestAsync) and pulls missing/newer rows (ApplyReplicatedAsync, LWW). Tombstones propagate deletes.

• Pro: each node resolves from local state, so a node survives partition from its peers / from any central DB — the right answer for air-gapped or intermittently-connected sites.
• Con: a whole new package + actor lifecycle (a cluster-singleton reconciler or per-node gossip), message serialization for StoredSecret, LWW edge cases (clock skew, tombstone retention), and — critically — it can only be proven against a live multi-node cluster, a G-2-class live-validation effort per app.

Decision

Build Option A (shared SQL-Server ISecretStore) as G-7. Specify Option B as a deferred phase-2, to be built only when an app declares a concrete requirement to resolve secrets while partitioned from the shared store (an availability SLA neither app states today).

Rationale: Option A delivers cluster-wide secrets for both apps with the least new code and no new failure modes, reusing the shared-SQL operational posture both apps already depend on for config, audit, and the DP key ring. The clustered-secrets requirement today is "every node sees the same secrets," which a shared store satisfies exactly. Partition-tolerance (Option B's only unique benefit) is a stronger, unstated requirement; paying its distributed-systems complexity now would be speculative (YAGNI). The ISecretReplicator seam stays in place, so Option B remains a drop-in later with no change to consumers or the wire/store contract.

Option A — build shape (detail in the executable plan)

1. SqlServerSecretStore : ISecretStore — mirrors SqliteSecretStore in T-SQL: GetAsync, UpsertAsync (revision bump via MERGE/UPDATE), DeleteAsync (tombstone), ListAsync (metadata projection — never ciphertext), GetManifestAsync, ApplyReplicatedAsync (LWW under a serializable transaction), and ApplyRewrapAsync (G-8; UPDATE the 4 wrap columns only). Microsoft.Data.SqlClient, fully parameterized.
2. SqlServerSecretsStoreMigrator — schema-versioned, idempotent (IF NOT EXISTS), same column set/semantics as SQLite; varbinary(max) for the crypto BLOBs, datetimeoffset (or ISO-8601 text, to match the SQLite round-trip exactly) for timestamps.
3. DI selection — add SecretsOptions.Store (Sqlite default | SqlServer) + SqlServerConnectionString; AddZbSecrets binds the store + migrator from it. SQLite stays the default so single-process consumers (HistorianGateway, mxaccessgw) are unaffected.
4. Adoption (ScadaBridge, OtOpcUa) — set Secrets:Store = SqlServer + a connection string (delivered via ${secret:} / a secret: ref like every other connstr), and ensure the same KEK on every node (shared key file or shared env). Run G-8 rewrap-all once against the shared store when the KEK rotates.

Hard constraints (both options)

• Same KEK on every node. Non-negotiable — a mismatched kek_id fails closed on resolve.
• Ciphertext only crosses trust boundaries. The store/wire never carries plaintext or the KEK.
• Rotation is per independent store. G-8 rewrap-all: once for the shared SQL store (Option A); once per node for per-node SQLite (Option B).

Out of scope

• Akka remoting auth/TLS hardening (a separate concern from secret storage).
• Option B's actual construction — captured as the deferred phase-2 in the executable plan.