feat(secrets): build G-8 KEK rotation (RewrapAll); design+plan G-7 clustered replication

G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
  (bodies never re-encrypted; revision/timestamps preserved -> invisible to
  cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
  wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
  updates only the 4 wrap columns + kek_id, compare-and-swap on the current
  wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
  review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
  tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
  fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
  JSON counts report; README section + operator runbook.

Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).

G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
  replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
  executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.

Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
This commit is contained in:
Joseph Doherty
2026-07-17 02:55:43 -04:00
parent b009507e10
commit d82d3451e7
24 changed files with 1280 additions and 12 deletions
@@ -0,0 +1,102 @@
# G-7 — Clustered secret replication: design & fork resolution
> **Status:** design-only (G-7 in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md)).
> Companion executable plan: [`2026-07-17-secrets-g7-sqlserver-store.md`](2026-07-17-secrets-g7-sqlserver-store.md).
> Prerequisite G-2…G-6 have landed for both clustered apps (ScadaBridge, OtOpcUa); G-8 (KEK
> rotation) is built. This resolves the SPEC's "shared SQL store **vs** Akka replicator" fork and
> specifies the recommended build.
## Problem
`ZB.MOM.WW.Secrets` stores secrets in a **local SQLite** file by default. The two Akka-clustered
apps run **multiple nodes** (ScadaBridge: central pair + N site pairs; OtOpcUa: admin/driver/dev
roles). A secret written on one node must be resolvable on every node that needs it, **without**
exposing plaintext or the KEK over the wire, and **without** each node holding a divergent copy.
The library was built anticipating this: the schema already carries `revision` / `updated_utc` /
`is_deleted`; `ISecretStore` exposes `GetManifestAsync` + `ApplyReplicatedAsync` (last-writer-wins);
`ISecretReplicator` is a no-op seam; `SecretManifestEntry` supports anti-entropy. Nothing needs a
migration to turn replication on.
## The fork (from SPEC §"Clustered pairs")
Two ways to make one logical secret set visible cluster-wide. **Both require a shared KEK** — every
node must resolve the *same* master key (shared mounted key file or shared env), because a row whose
`kek_id` doesn't match the local provider fails closed on resolve. Ciphertext can cross the wire /
sit in a shared DB safely because the KEK never does.
### Option A — Shared SQL-Server `ISecretStore` (one source of truth)
Point every node's `ISecretStore` at **one shared SQL-Server database** instead of a per-node
SQLite file. There is exactly one copy of each row; no replication, no reconciliation. The store
still holds **ciphertext only** (the KEK stays per-node/out-of-DB). Build a
`SqlServerSecretStore : ISecretStore` behind the existing seam (the SPEC already scoped "SQL-Server
`ISecretStore` provider construction — build with G-7").
- **Pro:** zero distributed-systems code or failure modes (no split-brain, no anti-entropy lag, no
tombstone GC, no LWW clock-skew). ~1 new file mirroring `SqliteSecretStore`'s SQL + a migrator.
- **Pro:** both apps **already run shared SQL Server** (ScadaBridge ConfigDb + central `dbo.*`;
OtOpcUa central config DB) and already share their Data-Protection key ring through SQL — a shared
secret store is operationally identical to what ops already run and back up.
- **Pro:** rotation (G-8 `rewrap-all`) runs **once** against the one store.
- **Con:** availability is coupled to the shared DB. A brief outage is bridged by the resolver's
in-memory TTL cache; but a node that must keep resolving secrets while *partitioned from* the
shared DB is not served by this option (see Option B).
### Option B — Akka replicator `ZB.MOM.WW.Secrets.Akka` (each node keeps a local store)
Each node keeps its **local SQLite** store; a real `ISecretReplicator` broadcasts each newly-written
encrypted row to peers, and a cluster anti-entropy actor periodically exchanges manifests
(`GetManifestAsync`) and pulls missing/newer rows (`ApplyReplicatedAsync`, LWW). Tombstones
propagate deletes.
- **Pro:** each node resolves from local state, so a node **survives partition** from its peers /
from any central DB — the right answer for air-gapped or intermittently-connected sites.
- **Con:** a whole new package + actor lifecycle (a cluster-singleton reconciler or per-node gossip),
message serialization for `StoredSecret`, LWW edge cases (clock skew, tombstone retention), and —
critically — it can only be *proven* against a live multi-node cluster, a G-2-class live-validation
effort per app.
## Decision
**Build Option A (shared SQL-Server `ISecretStore`) as G-7.** Specify Option B as a **deferred
phase-2**, to be built only when an app declares a concrete requirement to resolve secrets while
partitioned from the shared store (an availability SLA neither app states today).
Rationale: Option A delivers cluster-wide secrets for both apps with the least new code and **no new
failure modes**, reusing the shared-SQL operational posture both apps already depend on for config,
audit, and the DP key ring. The clustered-secrets requirement today is "every node sees the same
secrets," which a shared store satisfies exactly. Partition-tolerance (Option B's only unique
benefit) is a stronger, unstated requirement; paying its distributed-systems complexity now would be
speculative (YAGNI). The `ISecretReplicator` seam stays in place, so Option B remains a drop-in later
with **no change** to consumers or the wire/store contract.
## Option A — build shape (detail in the executable plan)
1. **`SqlServerSecretStore : ISecretStore`** — mirrors `SqliteSecretStore` in T-SQL: `GetAsync`,
`UpsertAsync` (revision bump via `MERGE`/`UPDATE`), `DeleteAsync` (tombstone), `ListAsync`
(metadata projection — never ciphertext), `GetManifestAsync`, `ApplyReplicatedAsync` (LWW under a
serializable transaction), and **`ApplyRewrapAsync`** (G-8; UPDATE the 4 wrap columns only).
`Microsoft.Data.SqlClient`, fully parameterized.
2. **`SqlServerSecretsStoreMigrator`** — schema-versioned, idempotent (`IF NOT EXISTS`), same column
set/semantics as SQLite; `varbinary(max)` for the crypto BLOBs, `datetimeoffset` (or ISO-8601
text, to match the SQLite round-trip exactly) for timestamps.
3. **DI selection** — add `SecretsOptions.Store` (`Sqlite` default | `SqlServer`) +
`SqlServerConnectionString`; `AddZbSecrets` binds the store + migrator from it. SQLite stays the
default so single-process consumers (HistorianGateway, mxaccessgw) are unaffected.
4. **Adoption (ScadaBridge, OtOpcUa)** — set `Secrets:Store = SqlServer` + a connection string
(delivered via `${secret:}` / a `secret:` ref like every other connstr), and ensure **the same
KEK** on every node (shared key file or shared env). Run G-8 `rewrap-all` once against the shared
store when the KEK rotates.
## Hard constraints (both options)
- **Same KEK on every node.** Non-negotiable — a mismatched `kek_id` fails closed on resolve.
- **Ciphertext only crosses trust boundaries.** The store/wire never carries plaintext or the KEK.
- **Rotation is per independent store.** G-8 `rewrap-all`: once for the shared SQL store (Option A);
once per node for per-node SQLite (Option B).
## Out of scope
- Akka remoting auth/TLS hardening (a separate concern from secret storage).
- Option B's actual construction — captured as the deferred phase-2 in the executable plan.
@@ -0,0 +1,122 @@
# G-7 (Option A) — Shared SQL-Server `ISecretStore` Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or
> subagent-driven-development) to implement this plan task-by-task.
**Goal:** Add a shared SQL-Server `ISecretStore` provider to `ZB.MOM.WW.Secrets` so the Akka-clustered
apps (ScadaBridge, OtOpcUa) can back their secrets with one shared, ciphertext-only database — every
node sees the same secrets, no replication code.
**Architecture:** Mirror the existing `SqliteSecretStore` + `SqliteSecretsStoreMigrator` in T-SQL
behind the unchanged `ISecretStore` seam. Select the store in `AddZbSecrets` from a new
`SecretsOptions.Store` enum (SQLite stays the default). The KEK stays per-node/out-of-DB, so the DB
holds only ciphertext. See the design + fork rationale in
[`2026-07-17-secrets-g7-clustered-replication-design.md`](2026-07-17-secrets-g7-clustered-replication-design.md).
**Tech Stack:** .NET 10, `Microsoft.Data.SqlClient`, xUnit + Shouldly, env-gated live SQL tests
(LocalDB or a `mcr.microsoft.com/mssql/server` container), matching the app live-test idiom.
**Precondition:** G-8 is merged — `ISecretStore.ApplyRewrapAsync` exists and the SQL-Server store
must implement it. `rewrap-all` then runs once against the shared store.
---
### Task 1: SQL-Server schema + migrator
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (Task 2 depends on it)
**Files:**
- Create: `src/ZB.MOM.WW.Secrets/SqlServer/SqlServerSecretsSchema.cs`
- Create: `src/ZB.MOM.WW.Secrets/SqlServer/SqlServerSecretsStoreMigrator.cs`
- Create: `src/ZB.MOM.WW.Secrets/SqlServer/SecretsSqlServerConnectionFactory.cs`
- Test: `tests/ZB.MOM.WW.Secrets.Tests/SqlServer/SqlServerSecretsStoreMigratorTests.cs` (env-gated)
**Notes:**
- Mirror `SqliteSecretsSchema` / `SqliteSecretsStoreMigrator` semantics: `schema_version` single-row
table + `secret` table, `CurrentVersion = 1`, idempotent `IF NOT EXISTS` DDL, refuse a newer
on-disk version. Column set identical to SQLite; use `varbinary(max)` for the 6 crypto BLOBs,
`nvarchar` for text, and store timestamps as **ISO-8601 (`"O"`) text** so `StoredSecret`
round-trips byte-identically to the SQLite path (avoids `datetimeoffset` precision drift in tests).
- Add `Microsoft.Data.SqlClient` to `Directory.Packages.props` + the core csproj.
- Gate live tests on an env var (e.g. `SECRETS_SQLSERVER_CONNSTR`); skip cleanly when unset.
### Task 2: `SqlServerSecretStore : ISecretStore`
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none (depends on Task 1)
**Files:**
- Create: `src/ZB.MOM.WW.Secrets/SqlServer/SqlServerSecretStore.cs`
- Test: `tests/ZB.MOM.WW.Secrets.Tests/SqlServer/SqlServerSecretStoreTests.cs` (env-gated; port the
`SqliteSecretStoreTests` cases 1:1, incl. the G-8 `ApplyRewrap*` cases)
**Notes:**
- Implement all 7 members: `GetAsync`, `UpsertAsync` (revision bump — `MERGE` or
insert/`ON CONFLICT`-equivalent via `UPDATE`+`INSERT` under a transaction), `DeleteAsync`
(tombstone + revision bump), `ListAsync` (metadata projection — **never** the crypto columns),
`GetManifestAsync`, `ApplyReplicatedAsync` (LWW under `IsolationLevel.Serializable`, applied
verbatim, no revision bump), `ApplyRewrapAsync` (UPDATE the 4 wrap columns + `kek_id` only; no
revision/updated_utc change).
- Fully parameterized commands; same "created_utc/created_by preserved on overwrite" semantics as
SQLite. This is high-risk because the LWW + rewrap semantics MUST match SQLite exactly (a cluster
reconciles across both if a node ever migrates).
### Task 3: DI store-selection wiring
**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** none (depends on Task 2)
**Files:**
- Modify: `src/ZB.MOM.WW.Secrets/SecretsOptions.cs` (add `SecretsStoreKind Store` = `Sqlite` default;
add `string? SqlServerConnectionString`)
- Create: `src/ZB.MOM.WW.Secrets/SecretsStoreKind.cs` (enum `Sqlite | SqlServer`)
- Modify: `src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsServiceCollectionExtensions.cs`
(branch the `ISecretStore` + migrator registration on `Store`; keep SQLite the default so
HistorianGateway/mxaccessgw are unaffected)
- Modify: `src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsMigrationHostedService.cs` if it hard-refs
the SQLite migrator type (abstract behind a shared migrator seam or a switch)
- Test: `tests/ZB.MOM.WW.Secrets.Tests/DependencyInjection/AddZbSecretsTests.cs` (assert the correct
store type is resolved for each `Store` value)
**Notes:**
- Introduce a tiny `ISecretsStoreMigrator` seam (both migrators implement it) so the hosted service +
CLI stay store-agnostic, OR switch on `Store` at registration. Prefer the seam — cleaner.
- The connection string should itself be deliverable via `${secret:}` / a `secret:` ref (consumers
already do this for other connstrings).
### Task 4: Docs + adoption notes
**Classification:** small
**Estimated implement time:** ~3 min
**Parallelizable with:** Task 3
**Files:**
- Modify: `README.md` (Clustered deployments: document `Secrets:Store = SqlServer` + shared-KEK
requirement + "run `rewrap-all` once against the shared store")
- Create: `docs/operations/shared-sql-store.md` (operator setup: provision the DB, grant, set the
connstr via a secret ref, confirm the same KEK on every node)
- Modify: `components/secrets/GAPS.md` (G-7 → "Option A built"; note Option B deferred)
**Notes:** No app changes here — adoption in ScadaBridge/OtOpcUa is a follow-on (flip `Secrets:Store`,
supply the connstr + shared KEK). Capture that as a checklist, not code.
---
## Deferred phase-2 (NOT built here): Option B — `ZB.MOM.WW.Secrets.Akka`
Build only when an app declares it must resolve secrets while **partitioned** from the shared store.
Shape (for when it's warranted):
- New package `ZB.MOM.WW.Secrets.Akka` (net10; refs Akka.Cluster + Abstractions).
- `AkkaSecretReplicator : ISecretReplicator``PublishAsync` broadcasts the encrypted `StoredSecret`
to peers (a cluster-aware router / distributed pub-sub topic).
- An anti-entropy actor (cluster singleton or per-node) that periodically exchanges
`GetManifestAsync` digests and pulls newer/missing rows via `ApplyReplicatedAsync` (LWW);
tombstones propagate deletes; a retention window bounds tombstone GC.
- `StoredSecret` serialization (ciphertext only — never the KEK) + an `ISecretActorAccessor` wiring.
- **Validation:** a live 2-node ScadaBridge/OtOpcUa rig proving write-on-A → resolve-on-B, delete
propagation, and partition-heal resync — a G-2-class live gate.
- KEK constraint identical to Option A: same master KEK on every node; `rewrap-all` per node.
@@ -0,0 +1,11 @@
{
"planPath": "docs/plans/2026-07-17-secrets-g7-sqlserver-store.md",
"tasks": [
{"id": 0, "subject": "Task 1: SQL-Server schema + migrator", "status": "pending"},
{"id": 1, "subject": "Task 2: SqlServerSecretStore : ISecretStore", "status": "pending", "blockedBy": [0]},
{"id": 2, "subject": "Task 3: DI store-selection wiring", "status": "pending", "blockedBy": [1]},
{"id": 3, "subject": "Task 4: Docs + adoption notes", "status": "pending", "blockedBy": [1]}
],
"deferredPhase2": "Option B — ZB.MOM.WW.Secrets.Akka replicator (build only on a stated partition-tolerance requirement); see plan doc.",
"lastUpdated": "2026-07-17"
}