Joseph Doherty
dd0a846b64
Secrets were per-node SQLite, so a secret written on one node was invisible to the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka replicator" fork to build only the former; both are built here so the choice is a deployment decision (availability vs partition tolerance) rather than a library limitation. Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over distributed pub/sub). Core gains ISecretsStoreMigrator, one shared SecretLastWriterWins predicate so no two stores can disagree on a tie, the transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was inert and local writes would not have propagated at all. Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural divergence between the stores fails) and a 9-test in-process 2-node Akka cluster over real remoting. A post-build review caught six defects, all fixed and now covered: both replication modes could not resolve from the container (no test had built one), an unbounded fetch that broke past SQL Server's 2100-parameter cap, a poison row that aborted the rest of its batch forever, Enum.Parse on peer input that could restart the actor in a loop, null crypto blobs crossing the trust boundary, and a silently dropped pull-read failure. Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
110 lines
7.1 KiB
Markdown
110 lines
7.1 KiB
Markdown
# G-7 — Clustered secret replication: design & fork resolution
|
|
|
|
> **✅ SUPERSEDED BY THE BUILD, 2026-07-18.** This document resolved the fork to Option A and
|
|
> deferred Option B. On the user's instruction **both were built**, as
|
|
> `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared-store *and* hub modes) and
|
|
> `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`. The fork analysis below is still the right way to
|
|
> *choose* between them — it is now a deployment decision, not a build one. The "Decision" section's
|
|
> YAGNI reasoning for deferring Option B no longer applies.
|
|
|
|
> **Status:** design-only (G-7 in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md)).
|
|
> Companion executable plan: [`2026-07-17-secrets-g7-sqlserver-store.md`](2026-07-17-secrets-g7-sqlserver-store.md).
|
|
> Prerequisite G-2…G-6 have landed for both clustered apps (ScadaBridge, OtOpcUa); G-8 (KEK
|
|
> rotation) is built. This resolves the SPEC's "shared SQL store **vs** Akka replicator" fork and
|
|
> specifies the recommended build.
|
|
|
|
## Problem
|
|
|
|
`ZB.MOM.WW.Secrets` stores secrets in a **local SQLite** file by default. The two Akka-clustered
|
|
apps run **multiple nodes** (ScadaBridge: central pair + N site pairs; OtOpcUa: admin/driver/dev
|
|
roles). A secret written on one node must be resolvable on every node that needs it, **without**
|
|
exposing plaintext or the KEK over the wire, and **without** each node holding a divergent copy.
|
|
|
|
The library was built anticipating this: the schema already carries `revision` / `updated_utc` /
|
|
`is_deleted`; `ISecretStore` exposes `GetManifestAsync` + `ApplyReplicatedAsync` (last-writer-wins);
|
|
`ISecretReplicator` is a no-op seam; `SecretManifestEntry` supports anti-entropy. Nothing needs a
|
|
migration to turn replication on.
|
|
|
|
## The fork (from SPEC §"Clustered pairs")
|
|
|
|
Two ways to make one logical secret set visible cluster-wide. **Both require a shared KEK** — every
|
|
node must resolve the *same* master key (shared mounted key file or shared env), because a row whose
|
|
`kek_id` doesn't match the local provider fails closed on resolve. Ciphertext can cross the wire /
|
|
sit in a shared DB safely because the KEK never does.
|
|
|
|
### Option A — Shared SQL-Server `ISecretStore` (one source of truth)
|
|
|
|
Point every node's `ISecretStore` at **one shared SQL-Server database** instead of a per-node
|
|
SQLite file. There is exactly one copy of each row; no replication, no reconciliation. The store
|
|
still holds **ciphertext only** (the KEK stays per-node/out-of-DB). Build a
|
|
`SqlServerSecretStore : ISecretStore` behind the existing seam (the SPEC already scoped "SQL-Server
|
|
`ISecretStore` provider construction — build with G-7").
|
|
|
|
- **Pro:** zero distributed-systems code or failure modes (no split-brain, no anti-entropy lag, no
|
|
tombstone GC, no LWW clock-skew). ~1 new file mirroring `SqliteSecretStore`'s SQL + a migrator.
|
|
- **Pro:** both apps **already run shared SQL Server** (ScadaBridge ConfigDb + central `dbo.*`;
|
|
OtOpcUa central config DB) and already share their Data-Protection key ring through SQL — a shared
|
|
secret store is operationally identical to what ops already run and back up.
|
|
- **Pro:** rotation (G-8 `rewrap-all`) runs **once** against the one store.
|
|
- **Con:** availability is coupled to the shared DB. A brief outage is bridged by the resolver's
|
|
in-memory TTL cache; but a node that must keep resolving secrets while *partitioned from* the
|
|
shared DB is not served by this option (see Option B).
|
|
|
|
### Option B — Akka replicator `ZB.MOM.WW.Secrets.Akka` (each node keeps a local store)
|
|
|
|
Each node keeps its **local SQLite** store; a real `ISecretReplicator` broadcasts each newly-written
|
|
encrypted row to peers, and a cluster anti-entropy actor periodically exchanges manifests
|
|
(`GetManifestAsync`) and pulls missing/newer rows (`ApplyReplicatedAsync`, LWW). Tombstones
|
|
propagate deletes.
|
|
|
|
- **Pro:** each node resolves from local state, so a node **survives partition** from its peers /
|
|
from any central DB — the right answer for air-gapped or intermittently-connected sites.
|
|
- **Con:** a whole new package + actor lifecycle (a cluster-singleton reconciler or per-node gossip),
|
|
message serialization for `StoredSecret`, LWW edge cases (clock skew, tombstone retention), and —
|
|
critically — it can only be *proven* against a live multi-node cluster, a G-2-class live-validation
|
|
effort per app.
|
|
|
|
## Decision
|
|
|
|
**Build Option A (shared SQL-Server `ISecretStore`) as G-7.** Specify Option B as a **deferred
|
|
phase-2**, to be built only when an app declares a concrete requirement to resolve secrets while
|
|
partitioned from the shared store (an availability SLA neither app states today).
|
|
|
|
Rationale: Option A delivers cluster-wide secrets for both apps with the least new code and **no new
|
|
failure modes**, reusing the shared-SQL operational posture both apps already depend on for config,
|
|
audit, and the DP key ring. The clustered-secrets requirement today is "every node sees the same
|
|
secrets," which a shared store satisfies exactly. Partition-tolerance (Option B's only unique
|
|
benefit) is a stronger, unstated requirement; paying its distributed-systems complexity now would be
|
|
speculative (YAGNI). The `ISecretReplicator` seam stays in place, so Option B remains a drop-in later
|
|
with **no change** to consumers or the wire/store contract.
|
|
|
|
## Option A — build shape (detail in the executable plan)
|
|
|
|
1. **`SqlServerSecretStore : ISecretStore`** — mirrors `SqliteSecretStore` in T-SQL: `GetAsync`,
|
|
`UpsertAsync` (revision bump via `MERGE`/`UPDATE`), `DeleteAsync` (tombstone), `ListAsync`
|
|
(metadata projection — never ciphertext), `GetManifestAsync`, `ApplyReplicatedAsync` (LWW under a
|
|
serializable transaction), and **`ApplyRewrapAsync`** (G-8; UPDATE the 4 wrap columns only).
|
|
`Microsoft.Data.SqlClient`, fully parameterized.
|
|
2. **`SqlServerSecretsStoreMigrator`** — schema-versioned, idempotent (`IF NOT EXISTS`), same column
|
|
set/semantics as SQLite; `varbinary(max)` for the crypto BLOBs, `datetimeoffset` (or ISO-8601
|
|
text, to match the SQLite round-trip exactly) for timestamps.
|
|
3. **DI selection** — add `SecretsOptions.Store` (`Sqlite` default | `SqlServer`) +
|
|
`SqlServerConnectionString`; `AddZbSecrets` binds the store + migrator from it. SQLite stays the
|
|
default so single-process consumers (HistorianGateway, mxaccessgw) are unaffected.
|
|
4. **Adoption (ScadaBridge, OtOpcUa)** — set `Secrets:Store = SqlServer` + a connection string
|
|
(delivered via `${secret:}` / a `secret:` ref like every other connstr), and ensure **the same
|
|
KEK** on every node (shared key file or shared env). Run G-8 `rewrap-all` once against the shared
|
|
store when the KEK rotates.
|
|
|
|
## Hard constraints (both options)
|
|
|
|
- **Same KEK on every node.** Non-negotiable — a mismatched `kek_id` fails closed on resolve.
|
|
- **Ciphertext only crosses trust boundaries.** The store/wire never carries plaintext or the KEK.
|
|
- **Rotation is per independent store.** G-8 `rewrap-all`: once for the shared SQL store (Option A);
|
|
once per node for per-node SQLite (Option B).
|
|
|
|
## Out of scope
|
|
|
|
- Akka remoting auth/TLS hardening (a separate concern from secret storage).
|
|
- Option B's actual construction — captured as the deferred phase-2 in the executable plan.
|