feat(secrets): build G-8 KEK rotation (RewrapAll); design+plan G-7 clustered replication

G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3: - ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only (bodies never re-encrypted; revision/timestamps preserved -> invisible to cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed wraps; DEK zeroed on all paths. - ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek): updates only the 4 wrap columns + kek_id, compare-and-swap on the current wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a review-caught TOCTOU). - KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry, fail-closed on unknown/identical KEK. - `secret rewrap-all` CLI verb: key material only via env-var name / file path, JSON counts report; README section + operator runbook. Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed). G-7 (clustered replication) — designed + planned, no code: - Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc + executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*. Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
2026-07-17 02:55:43 -04:00
parent b009507e10
commit d82d3451e7
24 changed files with 1280 additions and 12 deletions
@@ -143,7 +143,7 @@ each project's **code-verified current state**, and the **gaps** between. See
 | Config + validation (options / startup validation) | Adopted (lib `0.1.0`; all 3 apps, local) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
 | Audit (event model + writer seam) | Adopted (lib `0.1.0`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
 | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | Built (lib `0.1.0`, **published to the Gitea feed**; consumed by HistorianGateway as a feed `PackageReference`) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
-| Secrets (encrypted store + `${secret:}` resolution) | Built (libs `0.1.2`, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted) | Shared `ZB.MOM.WW.Secrets` lib (3 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
+| Secrets (encrypted store + `${secret:}` resolution) | Built (libs `0.1.2`, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap); republish to feed is opt-in. **G-7 clustered replication DESIGNED + PLANNED** — fork resolved to **build the shared SQL-Server `ISecretStore`** (Akka replicator deferred phase-2), executable plan `docs/plans/2026-07-17-secrets-g7-*`. Both not yet committed/pushed (awaiting go-ahead) | Shared `ZB.MOM.WW.Secrets` lib (3 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
 The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
 proposed [`shared-contract`](components/auth/shared-contract/ZB.MOM.WW.Auth.md), three
@@ -5,7 +5,7 @@
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <LangVersion>latest</LangVersion>
-    <Version>0.1.2</Version>
+    <Version>0.1.3</Version>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <PackageReadmeFile>README.md</PackageReadmeFile>
  </PropertyGroup>
@@ -12,7 +12,7 @@ plaintext back on demand — from application code, from configuration, or from
 | `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
 | `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
-A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate`) ships in the repo (not packed).
+A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).
 ## How it protects secrets
@@ -50,6 +50,24 @@ await expander.ExpandConfigurationAsync((IConfigurationRoot)builder.Configuratio
 The master key is 32 bytes, provided base64 in `ZB_SECRETS_MASTER_KEY` (Environment source).
 A missing/invalid key fails closed at startup (`MasterKeyUnavailableException`).
 ## Rotating the master KEK
 Because each row is a body sealed under a per-secret DEK that is *wrapped* by the KEK, rotating
 the master key only re-wraps DEKs — bodies are never re-encrypted and no value history changes.
 The `secret rewrap-all` CLI verb (backed by `KekRotationService`) migrates every row from the
 old KEK to the new one; it is idempotent and safe to re-run:
 ```bash
 # New KEK defaults to the configured Secrets:MasterKey; supply the OLD key by env-var name or path.
 ZB_SECRETS_MASTER_KEY=<new-base64>  ZB_SECRETS_OLD_KEY=<old-base64> \
  secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
 # → {"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}
 ```
 Run it with resolve traffic quiesced and once per independent store (once for a shared
 SQL-Server store; once per node for per-node SQLite on a shared KEK). See the operator runbook:
 [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md).
 ## Runtime + human access
 - **App code:** inject `ISecretResolver` and call `GetAsync(name, ct)`. Every resolve is
@@ -0,0 +1,120 @@
 # Operator runbook — rotating the master KEK (`ZB.MOM.WW.Secrets`)
 **Audience:** operators rotating the key-encryption key (KEK) that protects a secrets store.
 **Primitive:** `secret rewrap-all` (CLI) → `KekRotationService.RewrapAllAsync` → per-row
 `ISecretCipher.Rewrap` + `ISecretStore.ApplyRewrapAsync`.
 **Status:** G-8 (KEK-rotation `RewrapAll` + runbook). Companion to the deferred G-7 clustered
 replication (`ZB.MOM.WW.Secrets.Akka`).
 ---
 ## What rotation does (and does not) touch
 Each secret is stored as a body sealed under a fresh per-secret **data-encryption key (DEK)**;
 the DEK is then **wrapped** by the master **KEK**. Rotation **re-wraps the DEK only**:
 - **Changed:** `wrapped_dek`, `wrap_nonce`, `wrap_tag`, `kek_id` — the KEK-wrap envelope.
 - **Untouched:** the sealed body (`ciphertext`/`nonce`/`tag`), `revision`, `updated_utc`,
  `created_*`/`updated_by`, the tombstone flag. Plaintext is never decrypted to disk or logged.
 Because `revision`/`updated_utc` are preserved, a re-wrap is **invisible to cluster
 last-writer-wins** anti-entropy — so it never churns replication, and it must be run **per
 independent store** (see step 5).
 ## Properties you can rely on
 - **Idempotent / resumable.** A row already on the new KEK is skipped (`alreadyCurrent`). If a
  run is interrupted, just run it again — it resumes from where it stopped.
 - **Fail-closed.** A row wrapped by a KEK that is *neither* the old nor the new one **aborts**
  the pass (rows already re-wrapped stay persisted). A wrong old-key or tampered wrap aborts with
  `SecretDecryptionException`. An identical old/new KEK id is rejected up front.
 - **No plaintext exposure.** The report is counts only: `{total, rewrapped, alreadyCurrent}`.
 ## KEK identity — the one thing to get right
 The old KEK you supply **must resolve to the same `kek_id` that is stored on the rows.** The
 provider derives `kek_id` as `sha256:<12 hex>` of the key bytes **unless** an explicit
 `Secrets:MasterKey:KekId` was configured.
 - **Derived id (default):** supplying the same old key **bytes** reproduces the same id — nothing
  extra to pass.
 - **Explicit id:** the app set `Secrets:MasterKey:KekId`. Pass the matching id with
  `--old-key-id <id>` (and `--new-key-id <id>` if the new KEK also uses an explicit id).
 A mismatch fails closed (`Cannot rewrap … not the supplied old KEK`) — it never corrupts a row.
 > **The new KEK id MUST differ from the old one.** With derived ids this is automatic (new key
 > bytes → new `sha256:` id). With **explicit** ids you must assign a new `--new-key-id`; reusing the
 > old id for new key material is rejected up front (`old and new KEK identifiers are identical;
 > there is nothing to rotate`) because `kek_id` is the only per-row rotation discriminator.
 ## Key material handling
 - Key **values are never passed as command-line arguments** (they would be echoed/logged). Supply
  the old (and optionally new) key by **env-var NAME** (`--old-key-env VAR`) or **file PATH**
  (`--old-key-file /path`). The value lives only in that env var / file.
 - Never print `ZB_SECRETS_MASTER_KEY` or the old key to a terminal, log, or ticket.
 - The new KEK **defaults to the app's configured `Secrets:MasterKey` provider**, so you usually
  only supply the *old* key.
 ---
 ## Procedure
 Rotation is an offline, store-local operation. Do it once per store.
 **1. Back up the store.** Copy the SQLite file (or snapshot the SQL-Server DB). This is the
 undo — if anything is wrong, restore and retry.
 **2. Quiesce resolve traffic.** Stop (or drain) the app(s) reading this store. Rotation preserves
 `updated_utc`/`revision`, so a running node on the *old* KEK would keep working until you flip its
 key — but running rotation against a store that other writers are mutating risks racing a fresh
 write onto the old KEK. Simplest correct posture: app down.
 **3. Stage the new KEK.** Generate a fresh 32-byte key, base64-encode it, and place it where the
 CLI (and, after cutover, the app) will read it — e.g. `ZB_SECRETS_MASTER_KEY` for the Environment
 provider, or the mounted key file for the File provider.
 **4. Run `rewrap-all`.** Point the CLI at the store's `appsettings.json` (so `Secrets:SqlitePath`
 + the new `Secrets:MasterKey` resolve), and supply the old key:
 ```bash
 # Old key delivered via a throwaway env var (name only on the command line):
 ZB_SECRETS_OLD_KEY='<old-base64>' \
  secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
 # Explicit-id deployment:
 #   secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --old-key-id sha256:abc… --new-key-id sha256:def…
 # Override the new key explicitly instead of the configured provider:
 #   secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --new-key-file /run/secrets/new_kek
 ```
 Expected: `{"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}` and exit 0. On a
 re-run everything reports `alreadyCurrent` and `rewrapped:0`.
 **5. For clustered / multi-store deployments, repeat per store.**
 - **Shared SQL-Server store (the G-7 primary path):** one store → run rotation **once**.
 - **Per-node SQLite on a shared KEK:** each node has its own store file → run rotation **on each
  node** (all with the same old→new keys). Re-wrap does not replicate (revision preserved), so
  every store must be migrated independently.
 **6. Cut the app over to the new KEK.** Ensure `Secrets:MasterKey` (and every node's KEK source)
 now resolves the **new** key, then start the app(s). Confirm a resolve succeeds
 (e.g. `secret get <known-name>` or an app readiness probe that exercises a `${secret:}` value).
 **7. Destroy the old key.** Remove the old key material from env/files/secret managers once every
 store is confirmed on the new KEK and resolves are healthy.
 ## Rollback
 If step 6/7 surfaces a problem, restore the pre-rotation store backup (step 1) and revert the app
 to the old KEK. Because bodies were never re-encrypted, the backup is a complete, consistent undo.
 ## Troubleshooting
 | Symptom | Cause | Action |
 |---|---|---|
 | `error: … old and new KEK identifiers are identical` | New KEK resolves to the same id as the old | You supplied the wrong new key, or forgot to stage it. Verify `Secrets:MasterKey`. |
 | `error: Secret '…' is wrapped by KEK 'X', which is neither the old nor the new KEK` | A row is on a third KEK (a prior partial/aborted rotation, or a mis-set id) | Investigate that row (`secret list` shows each row's `kekId`); re-point old/new keys or fix the row, then re-run (completed rows are skipped). |
 | `error: … DEK could not be unwrapped under the supplied old KEK` | Right `kek_id` but wrong key **bytes**, or a tampered row | Confirm you staged the correct old key bytes. If the row is genuinely tampered, restore from backup. |
 | Post-cutover resolves fail closed (`Row wrapped by unknown KEK`) | App still on the old KEK, or a store/node was missed in step 5 | Ensure every node's `Secrets:MasterKey` resolves the new key and every store was rewrapped. |
@@ -32,4 +32,34 @@ public interface ISecretCipher
    /// The authentication tag fails to verify, or the row references an unknown or unavailable KEK.
    /// </exception>
    string Decrypt(StoredSecret secret);
    /// <summary>
    /// Re-wraps the data-encryption key (DEK) of an existing row from <paramref name="oldKek"/> to
    /// <paramref name="newKek"/> — the KEK-rotation primitive. The DEK is unwrapped under the old
    /// master key and re-wrapped under the new one; the sealed body
    /// (<see cref="StoredSecret.Ciphertext"/>/<see cref="StoredSecret.Nonce"/>/<see cref="StoredSecret.Tag"/>)
    /// is <b>never</b> re-encrypted, and <see cref="StoredSecret.Revision"/>, the timestamps, the
    /// tombstone flag, and every other field are preserved verbatim. Only
    /// <see cref="StoredSecret.WrappedDek"/>/<see cref="StoredSecret.WrapNonce"/>/<see cref="StoredSecret.WrapTag"/>
    /// and <see cref="StoredSecret.KekId"/> change. Plaintext is never exposed.
    /// </summary>
    /// <remarks>
    /// This operation is <i>provider-explicit</i>: it uses only the two supplied providers and does
    /// not consult whatever KEK the cipher instance was constructed with, because a rotation spans
    /// two keys. The caller must have already established that <paramref name="secret"/> is wrapped
    /// by <paramref name="oldKek"/> (the row's <see cref="StoredSecret.KekId"/> equals
    /// <see cref="IMasterKeyProvider.KekId"/> of <paramref name="oldKek"/>); a mismatch fails closed.
    /// </remarks>
    /// <param name="secret">The stored row whose DEK should be re-wrapped.</param>
    /// <param name="oldKek">The provider for the KEK that currently wraps the row's DEK.</param>
    /// <param name="newKek">The provider for the KEK the DEK should be re-wrapped under.</param>
    /// <returns>
    /// A copy of <paramref name="secret"/> with the DEK re-wrapped under <paramref name="newKek"/>
    /// and <see cref="StoredSecret.KekId"/> set to the new KEK id; all other fields unchanged.
    /// </returns>
    /// <exception cref="SecretDecryptionException">
    /// The row is not wrapped by <paramref name="oldKek"/>, or the DEK fails to unwrap (wrong old
    /// key material or a tampered wrap).
    /// </exception>
    StoredSecret Rewrap(StoredSecret secret, IMasterKeyProvider oldKek, IMasterKeyProvider newKek);
 }
@@ -63,4 +63,42 @@ public interface ISecretStore
    /// <param name="ct">A token to cancel the operation.</param>
    /// <returns>A task that completes when the row has been applied or ignored.</returns>
    Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct);
    /// <summary>
    /// Applies a KEK re-wrap in place: overwrites <b>only</b> the KEK-wrap envelope columns
    /// (<see cref="StoredSecret.WrappedDek"/>/<see cref="StoredSecret.WrapNonce"/>/<see cref="StoredSecret.WrapTag"/>)
    /// and <see cref="StoredSecret.KekId"/> for the row named by <paramref name="rewrappedRow"/>,
    /// leaving the sealed body, revision, timestamps, tombstone flag, and audit stamps untouched.
    /// </summary>
    /// <remarks>
    /// <para>
    /// This is the persistence half of KEK rotation. It deliberately does <b>not</b> bump the
    /// revision or refresh <c>updated_utc</c>: a re-wrap does not change the secret's plaintext, so
    /// it must stay invisible to the (updated_utc, revision) last-writer-wins ordering used by
    /// cluster anti-entropy — otherwise every node's independent re-wrap would churn replication.
    /// </para>
    /// <para>
    /// The update is a <b>compare-and-swap</b> on <paramref name="expectedCurrentWrappedDek"/>: the
    /// row is updated only if its current <see cref="StoredSecret.WrappedDek"/> still equals the DEK
    /// wrap that was unwrapped to produce <paramref name="rewrappedRow"/>. A concurrent write to the
    /// same secret (a <c>set</c>/<c>rotate</c> that generated a new DEK) changes that wrap, so the
    /// CAS matches 0 rows and the row is left untouched — rather than corrupting it by pairing a
    /// stale re-wrap with a newer body. The caller re-processes a 0-row result (fetch → re-wrap).
    /// </para>
    /// </remarks>
    /// <param name="rewrappedRow">
    /// The row carrying the new wrap envelope + <see cref="StoredSecret.KekId"/>; matched by
    /// <see cref="StoredSecret.Name"/>. Only the four wrap-related columns are read from it.
    /// </param>
    /// <param name="expectedCurrentWrappedDek">
    /// The <see cref="StoredSecret.WrappedDek"/> the row must still carry for the swap to apply — the
    /// wrap that was unwrapped to produce <paramref name="rewrappedRow"/>.
    /// </param>
    /// <param name="ct">A token to cancel the operation.</param>
    /// <returns>
    /// <c>true</c> if the row was updated; <c>false</c> if no row matched (absent, or its wrapped DEK
    /// changed under a concurrent write).
    /// </returns>
    Task<bool> ApplyRewrapAsync(
        StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct);
 }
@@ -1,9 +1,11 @@
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Hosting;
 using Microsoft.Extensions.DependencyInjection;
 using ZB.MOM.WW.Secrets.Abstractions;
 using ZB.MOM.WW.Secrets.Cli;
 using ZB.MOM.WW.Secrets.DependencyInjection;
 using ZB.MOM.WW.Secrets.MasterKey;
 using ZB.MOM.WW.Secrets.Sqlite;
 // Headless `secret` CLI: set / get / list / rm / rotate over the ZB.MOM.WW secrets store.
@@ -77,6 +79,41 @@ try
            return await commands.RemoveAsync(args[1], actor, ct);
        }
        case "rewrap-all":
        {
            // Rotate the master KEK by re-wrapping every row's DEK. Key material is supplied ONLY by
            // env-var NAME or file PATH — never as a literal argument (which would be echoed/logged).
            string? oldKind = null, oldValue = null, oldKekId = null;
            string? newKind = null, newValue = null, newKekId = null;
            for (int i = 1; i < args.Length; i++)
            {
                switch (args[i])
                {
                    case "--old-key-env": (oldKind, oldValue) = ("env", RequireValue(args, ref i)); break;
                    case "--old-key-file": (oldKind, oldValue) = ("file", RequireValue(args, ref i)); break;
                    case "--old-key-id": oldKekId = RequireValue(args, ref i); break;
                    case "--new-key-env": (newKind, newValue) = ("env", RequireValue(args, ref i)); break;
                    case "--new-key-file": (newKind, newValue) = ("file", RequireValue(args, ref i)); break;
                    case "--new-key-id": newKekId = RequireValue(args, ref i); break;
                    default: return Usage($"Unknown option '{args[i]}'.");
                }
            }
            if (oldKind is null)
                return Usage("rewrap-all requires --old-key-env <VAR> or --old-key-file <path>.");
            if (newKind is null && newKekId is not null)
                return Usage("--new-key-id requires --new-key-env <VAR> or --new-key-file <path>.");
            IMasterKeyProvider oldKek = BuildProvider(oldKind, oldValue!, oldKekId);
            // Default the new KEK to the app's configured provider (Secrets:MasterKey); override with flags.
            IMasterKeyProvider newKek = newKind is null
                ? host.Services.GetRequiredService<IMasterKeyProvider>()
                : BuildProvider(newKind, newValue!, newKekId);
            return await commands.RewrapAllAsync(oldKek, newKek, ct);
        }
        default:
            return Usage($"Unknown command '{args[0]}'.");
    }
@@ -132,6 +169,26 @@ static SecretContentType ParseContentType(string value) => value switch
        $"Unknown --content-type '{value}'; expected text|connection-string|json|binary-base64."),
 };
 // Reads the value that must follow an option flag at args[i], advancing i past it.
 static string RequireValue(string[] args, ref int i)
 {
    if (i + 1 >= args.Length)
        throw new ArgumentException($"{args[i]} requires a value.");
    return args[++i];
 }
 // Builds a master-key provider from a source kind ("env" | "file") + its env-var name or file path,
 // with an optional explicit KEK id (needed when the app configured Secrets:MasterKey:KekId rather
 // than the derived sha256: id). The key VALUE itself is only ever read from the env var / file.
 static IMasterKeyProvider BuildProvider(string kind, string value, string? kekId) => kind switch
 {
    "env" => MasterKeyProviderFactory.Create(
        new MasterKeyOptions { Source = MasterKeySource.Environment, EnvVarName = value, KekId = kekId }),
    "file" => MasterKeyProviderFactory.Create(
        new MasterKeyOptions { Source = MasterKeySource.File, FilePath = value, KekId = kekId }),
    _ => throw new ArgumentException($"Unknown key source '{kind}'."),
 };
 // Prints usage (optionally preceded by an error line) and returns the standard usage exit code.
 int Usage(string? error = null)
 {
@@ -147,6 +204,14 @@ int Usage(string? error = null)
          get <name>
          list [--include-deleted]
          rm <name>
          rewrap-all --old-key-env <VAR>|--old-key-file <path> [--old-key-id <id>]
                     [--new-key-env <VAR>|--new-key-file <path>] [--new-key-id <id>]
        rewrap-all rotates the master KEK: it re-wraps every stored secret's data key from the old
        KEK to the new one (bodies are never re-encrypted, no plaintext is exposed). The new KEK
        defaults to the configured Secrets:MasterKey provider. Supply key material only by env-var
        NAME or file PATH — never as a literal. Run it with resolve traffic quiesced; it is
        idempotent and safe to re-run.
        """);
    return 2;
 }
@@ -1,5 +1,6 @@
 using System.Text.Json;
 using ZB.MOM.WW.Secrets.Abstractions;
 using ZB.MOM.WW.Secrets.Rotation;
 namespace ZB.MOM.WW.Secrets.Cli;
@@ -156,6 +157,47 @@ public sealed class SecretCommands
        return removed ? 0 : 1;
    }
    /// <summary>
    /// rewrap-all: rotates the master KEK by re-wrapping every row's DEK from <paramref name="oldKek"/>
    /// onto <paramref name="newKek"/> — bodies are never re-encrypted and no plaintext is exposed.
    /// Prints a JSON report of the counts (total / rewrapped / already-current) and NEVER any key
    /// material. Idempotent + resumable; fails closed (non-zero) on an identical-KEK mistake or a row
    /// wrapped by an unrecognized KEK.
    /// </summary>
    /// <param name="oldKek">The provider for the KEK rows are currently wrapped under.</param>
    /// <param name="newKek">The provider for the KEK rows should be re-wrapped onto.</param>
    /// <param name="ct">A token to cancel the operation.</param>
    /// <returns>0 on success; non-zero if the rotation was rejected or aborted.</returns>
    public async Task<int> RewrapAllAsync(
        IMasterKeyProvider oldKek, IMasterKeyProvider newKek, CancellationToken ct)
    {
        var service = new KekRotationService(_store, _cipher);
        try
        {
            RewrapReport report = await service.RewrapAllAsync(oldKek, newKek, ct).ConfigureAwait(false);
            WriteJson(new
            {
                action = "rewrap-all",
                total = report.Total,
                rewrapped = report.Rewrapped,
                alreadyCurrent = report.AlreadyCurrent,
            });
            return 0;
        }
        catch (ArgumentException ex)
        {
            // Identical old/new KEK ids — nothing to rotate.
            WriteJson(new { action = "rewrap-all", error = ex.Message });
            return 1;
        }
        catch (SecretDecryptionException ex)
        {
            // A row on an unrecognized KEK aborted the pass, or a DEK failed to unwrap.
            WriteJson(new { action = "rewrap-all", error = ex.Message });
            return 1;
        }
    }
    /// <summary>Seals a value under the cipher and upserts it, carrying description + actor stamps.</summary>
    private async Task UpsertSealedAsync(
        SecretName name, string value, SecretContentType contentType, string? description, string? actor, CancellationToken ct)
@@ -153,4 +153,75 @@ public sealed class AesGcmEnvelopeCipher : ISecretCipher
            CryptographicOperations.ZeroMemory(dek);
        }
    }
    /// <inheritdoc />
    public StoredSecret Rewrap(StoredSecret secret, IMasterKeyProvider oldKek, IMasterKeyProvider newKek)
    {
        ArgumentNullException.ThrowIfNull(secret);
        ArgumentNullException.ThrowIfNull(oldKek);
        ArgumentNullException.ThrowIfNull(newKek);
        // Fail closed if the row is not actually wrapped by the supplied old KEK — a caller must
        // not silently "rewrap" a row that is already on a different key.
        if (!string.Equals(secret.KekId, oldKek.KekId, StringComparison.Ordinal))
        {
            throw new SecretDecryptionException(
                $"Cannot rewrap secret '{secret.Name.Value}': row is wrapped by KEK '{secret.KekId}', " +
                $"not the supplied old KEK '{oldKek.KekId}'.");
        }
        ReadOnlySpan<byte> oldMasterKey = oldKek.GetMasterKey().Span;
        ReadOnlySpan<byte> newMasterKey = newKek.GetMasterKey().Span;
        string newKekId = newKek.KekId;
        byte[] dek = new byte[KeySizeBytes];
        try
        {
            // Unwrap the DEK under the OLD master key, verifying the old KEK-id AAD binding. A wrong
            // old key or tampered wrap surfaces as SecretDecryptionException (never a raw crypto leak).
            byte[] oldKekAad = Encoding.UTF8.GetBytes(secret.KekId);
            try
            {
                using var unwrap = new AesGcm(oldMasterKey, TagSizeBytes);
                unwrap.Decrypt(secret.WrapNonce, secret.WrappedDek, secret.WrapTag, dek, oldKekAad);
            }
            // CryptographicException = wrong key / tampered wrap; ArgumentException = a malformed
            // (wrong-length) wrap nonce/tag on a corrupt row. Both fail closed as a decryption error
            // rather than leaking a raw crypto/arg exception (which the caller would misclassify).
            catch (Exception ex) when (ex is CryptographicException or ArgumentException)
            {
                throw new SecretDecryptionException(
                    $"Failed to rewrap secret '{secret.Name.Value}': the DEK could not be unwrapped " +
                    "under the supplied old KEK.", ex);
            }
            // Re-wrap the SAME DEK under the NEW master key with a fresh nonce, binding the new KEK id
            // as AAD. The body ciphertext is left untouched — rotation re-wraps DEKs only.
            byte[] wrapNonce = new byte[NonceSizeBytes];
            byte[] wrapTag = new byte[TagSizeBytes];
            byte[] wrappedDek = new byte[KeySizeBytes];
            byte[] newKekAad = Encoding.UTF8.GetBytes(newKekId);
            RandomNumberGenerator.Fill(wrapNonce);
            using (var wrapGcm = new AesGcm(newMasterKey, TagSizeBytes))
            {
                wrapGcm.Encrypt(wrapNonce, dek, wrappedDek, wrapTag, newKekAad);
            }
            // Only the KEK-wrap envelope + kek_id change; body, revision, timestamps, tombstone,
            // content-type, and every other field are preserved so a rewrap is invisible to the
            // (updated_utc, revision) last-writer-wins ordering used by cluster anti-entropy.
            return secret with
            {
                WrappedDek = wrappedDek,
                WrapNonce = wrapNonce,
                WrapTag = wrapTag,
                KekId = newKekId,
            };
        }
        finally
        {
            CryptographicOperations.ZeroMemory(dek);
        }
    }
 }
@@ -0,0 +1,155 @@
 using ZB.MOM.WW.Secrets.Abstractions;
 namespace ZB.MOM.WW.Secrets.Rotation;
 /// <summary>
 /// Rotates the master key-encryption key (KEK) across every stored secret by re-wrapping each
 /// row's data-encryption key (DEK) from the old KEK to the new one. Because the envelope model
 /// wraps a per-secret DEK under the KEK, rotation never re-encrypts secret bodies — only the DEK
 /// wrap changes — so plaintext is never exposed and no value history is touched.
 /// </summary>
 /// <remarks>
 /// <para>
 /// The pass is <b>idempotent and resumable</b>: a row already wrapped by the new KEK is skipped
 /// (counted in <see cref="RewrapReport.AlreadyCurrent"/>), so a run interrupted part-way can simply
 /// be re-run to finish. It is <b>fail-closed</b>: a row wrapped by neither the old nor the new KEK
 /// aborts the pass (rather than silently leaving an un-migratable row that would become
 /// undecryptable once the old KEK is retired); rows already re-wrapped in the aborted run stay
 /// persisted, so a re-run resumes cleanly after the anomaly is investigated.
 /// </para>
 /// <para>
 /// Rotation is a per-store operation and must be run once per independent store (once for a shared
 /// SQL-Server store; once per node for a per-node SQLite store on a shared KEK). It should be run
 /// with resolve traffic quiesced: the persistence path deliberately does not bump revision or
 /// <c>updated_utc</c>, so a re-wrap is invisible to cluster last-writer-wins reconciliation.
 /// </para>
 /// </remarks>
 public sealed class KekRotationService
 {
    private readonly ISecretStore _store;
    private readonly ISecretCipher _cipher;
    /// <summary>Creates the rotation service over the encrypted store and envelope cipher.</summary>
    /// <param name="store">The store to enumerate and re-wrap in place.</param>
    /// <param name="cipher">The envelope cipher providing the DEK re-wrap primitive.</param>
    public KekRotationService(ISecretStore store, ISecretCipher cipher)
    {
        _store = store ?? throw new ArgumentNullException(nameof(store));
        _cipher = cipher ?? throw new ArgumentNullException(nameof(cipher));
    }
    /// <summary>
    /// Re-wraps every row from <paramref name="oldKek"/> onto <paramref name="newKek"/>.
    /// </summary>
    /// <param name="oldKek">The provider for the KEK rows are currently wrapped under.</param>
    /// <param name="newKek">The provider for the KEK rows should be re-wrapped onto.</param>
    /// <param name="ct">A token to cancel the pass (already-persisted re-wraps are retained).</param>
    /// <returns>A <see cref="RewrapReport"/> summarizing the pass (no secret material).</returns>
    /// <exception cref="ArgumentException">
    /// The old and new KEK identifiers are the same — there is nothing to rotate.
    /// </exception>
    /// <exception cref="SecretDecryptionException">
    /// A row is wrapped by a KEK that is neither the old nor the new one (the pass aborts), or a
    /// row's DEK fails to unwrap under the supplied old KEK.
    /// </exception>
    public async Task<RewrapReport> RewrapAllAsync(
        IMasterKeyProvider oldKek, IMasterKeyProvider newKek, CancellationToken ct)
    {
        ArgumentNullException.ThrowIfNull(oldKek);
        ArgumentNullException.ThrowIfNull(newKek);
        if (string.Equals(oldKek.KekId, newKek.KekId, StringComparison.Ordinal))
        {
            throw new ArgumentException(
                $"The old and new KEK identifiers are identical ('{oldKek.KekId}'); there is nothing to rotate.",
                nameof(newKek));
        }
        // Enumerate metadata (never ciphertext) for ALL rows including tombstones — a tombstoned row
        // still carries a KEK-wrapped DEK and would become un-migratable if left on a retired KEK.
        IReadOnlyList<SecretMetadata> all =
            await _store.ListAsync(includeDeleted: true, ct).ConfigureAwait(false);
        int rewrapped = 0;
        int alreadyCurrent = 0;
        foreach (SecretMetadata meta in all)
        {
            ct.ThrowIfCancellationRequested();
            // Fast-path skip on the metadata kek_id (no ciphertext fetch): a row already on the new
            // KEK is a completed migration — this is what makes the pass idempotent / resumable.
            if (string.Equals(meta.KekId, newKek.KekId, StringComparison.Ordinal))
            {
                alreadyCurrent++;
                continue;
            }
            switch (await RewrapOneAsync(meta.Name, oldKek, newKek, rewrapped, ct).ConfigureAwait(false))
            {
                case RowOutcome.Rewrapped:
                    rewrapped++;
                    break;
                case RowOutcome.AlreadyCurrent:
                    alreadyCurrent++;
                    break;
                case RowOutcome.Vanished:
                    break;
            }
        }
        return new RewrapReport { Total = all.Count, Rewrapped = rewrapped, AlreadyCurrent = alreadyCurrent };
    }
    private enum RowOutcome { Rewrapped, AlreadyCurrent, Vanished }
    // Re-wraps a single row with a bounded fetch→rewrap→compare-and-swap retry so a rare concurrent
    // write (the pass should run with writes quiesced) is retried rather than corrupting the row.
    private async Task<RowOutcome> RewrapOneAsync(
        SecretName name, IMasterKeyProvider oldKek, IMasterKeyProvider newKek, int rewrappedSoFar, CancellationToken ct)
    {
        const int maxAttempts = 5;
        for (int attempt = 1; ; attempt++)
        {
            ct.ThrowIfCancellationRequested();
            StoredSecret? row = await _store.GetAsync(name, ct).ConfigureAwait(false);
            if (row is null)
            {
                // Vanished between the list and the fetch (a concurrent hard-absence).
                return RowOutcome.Vanished;
            }
            // Re-evaluate against the FRESH row: a concurrent write may have already moved it.
            if (string.Equals(row.KekId, newKek.KekId, StringComparison.Ordinal))
            {
                return RowOutcome.AlreadyCurrent;
            }
            if (!string.Equals(row.KekId, oldKek.KekId, StringComparison.Ordinal))
            {
                throw new SecretDecryptionException(
                    $"Secret '{name.Value}' is wrapped by KEK '{row.KekId}', which is neither the old " +
                    $"('{oldKek.KekId}') nor the new ('{newKek.KekId}') KEK. Rotation aborted after re-wrapping " +
                    $"{rewrappedSoFar} row(s); investigate the anomalous row and re-run to resume.");
            }
            // Rewrap re-verifies row.KekId == oldKek.KekId and fails closed on a wrong old key.
            StoredSecret rewrappedRow = _cipher.Rewrap(row, oldKek, newKek);
            // Compare-and-swap on the wrap we just unwrapped: a concurrent set/rotate changed it →
            // 0 rows → retry (fetch the newer row and re-evaluate).
            if (await _store.ApplyRewrapAsync(rewrappedRow, row.WrappedDek, ct).ConfigureAwait(false))
            {
                return RowOutcome.Rewrapped;
            }
            if (attempt >= maxAttempts)
            {
                throw new SecretDecryptionException(
                    $"Secret '{name.Value}' is being modified concurrently; rotation could not converge after " +
                    $"{maxAttempts} attempts. Re-run with secret writes quiesced.");
            }
        }
    }
 }
@@ -0,0 +1,21 @@
 namespace ZB.MOM.WW.Secrets.Rotation;
 /// <summary>
 /// The outcome of a <see cref="KekRotationService.RewrapAllAsync"/> pass: how many rows were
 /// re-wrapped onto the new KEK, how many were already on it (skipped), and the total scanned.
 /// Carries no secret material.
 /// </summary>
 public sealed record RewrapReport
 {
    /// <summary>Total number of rows scanned (includes tombstoned rows).</summary>
    public required int Total { get; init; }
    /// <summary>Number of rows re-wrapped from the old KEK onto the new KEK in this pass.</summary>
    public required int Rewrapped { get; init; }
    /// <summary>
    /// Number of rows already wrapped by the new KEK and therefore skipped — the count that makes a
    /// re-run idempotent (a completed rotation reports <see cref="Rewrapped"/> 0 and all rows here).
    /// </summary>
    public required int AlreadyCurrent { get; init; }
 }
@@ -260,6 +260,43 @@ public sealed class SqliteSecretStore(SecretsSqliteConnectionFactory connectionF
        await transaction.CommitAsync(ct).ConfigureAwait(false);
    }
    /// <inheritdoc />
    public async Task<bool> ApplyRewrapAsync(
        StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
    {
        ArgumentNullException.ThrowIfNull(rewrappedRow);
        ArgumentNullException.ThrowIfNull(expectedCurrentWrappedDek);
        await using SqliteConnection connection =
            await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
        // Overwrite ONLY the KEK-wrap envelope + kek_id. Revision, updated_utc/by, the sealed body,
        // and the tombstone state are deliberately untouched — a re-wrap is not a logical change, so
        // it must not bump the revision or updated timestamp (see ISecretStore.ApplyRewrapAsync).
        //
        // Compare-and-swap on wrapped_dek: a concurrent set/rotate generates a fresh (random) DEK
        // wrap, so this matches 0 rows instead of pairing this stale re-wrap with a newer body —
        // which would leave the row permanently undecryptable. The caller re-processes a 0-row miss.
        await using SqliteCommand command = connection.CreateCommand();
        command.CommandText = """
            UPDATE secret SET
                wrapped_dek = $wrapped_dek,
                wrap_nonce  = $wrap_nonce,
                wrap_tag    = $wrap_tag,
                kek_id      = $kek_id
            WHERE name = $name AND wrapped_dek = $expected_wrapped_dek;
            """;
        command.Parameters.AddWithValue("$wrapped_dek", rewrappedRow.WrappedDek);
        command.Parameters.AddWithValue("$wrap_nonce", rewrappedRow.WrapNonce);
        command.Parameters.AddWithValue("$wrap_tag", rewrappedRow.WrapTag);
        command.Parameters.AddWithValue("$kek_id", rewrappedRow.KekId);
        command.Parameters.AddWithValue("$name", rewrappedRow.Name.Value);
        command.Parameters.AddWithValue("$expected_wrapped_dek", expectedCurrentWrappedDek);
        int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
        return rowsAffected > 0;
    }
    // Binds the identity, description, content-type, KEK id, and all six crypto BLOB columns
    // shared by every insert path.
    private static void BindCryptoColumns(SqliteCommand command, StoredSecret row)
@@ -19,6 +19,7 @@ public sealed class SecretCommandsTests : IAsyncLifetime, IDisposable
    private readonly SecretsSqliteConnectionFactory _factory;
    private readonly SqliteSecretStore _store;
    private readonly FakeMasterKeyProvider _oldKek = new("kek-cli");
    private readonly AesGcmEnvelopeCipher _cipher;
    private readonly DefaultSecretResolver _resolver;
@@ -26,7 +27,7 @@ public sealed class SecretCommandsTests : IAsyncLifetime, IDisposable
    {
        _factory = new SecretsSqliteConnectionFactory(_dbPath);
        _store = new SqliteSecretStore(_factory);
-        _cipher = new AesGcmEnvelopeCipher(new FakeMasterKeyProvider("kek-cli"));
+        _cipher = new AesGcmEnvelopeCipher(_oldKek);
        // A near-zero TTL keeps the resolver from serving a stale plaintext across a rotate/remove.
        _resolver = new DefaultSecretResolver(
            _store, _cipher, NoOpAuditWriter.Instance, TimeSpan.Zero);
@@ -53,6 +54,45 @@ public sealed class SecretCommandsTests : IAsyncLifetime, IDisposable
    private static CancellationToken Ct => CancellationToken.None;
    [Fact]
    public async Task RewrapAll_MigratesRowsToNewKek_ReportsCountsWithoutLeakingValues()
    {
        var (seed, _) = NewSut();
        await seed.SetAsync("sql/a", "value-a", SecretContentType.Text, null, "alice", Ct);
        await seed.SetAsync("sql/b", "value-b", SecretContentType.Text, null, "alice", Ct);
        var newKek = new FakeMasterKeyProvider("kek-new");
        var (cmd, output) = NewSut();
        int rc = await cmd.RewrapAllAsync(_oldKek, newKek, Ct);
        Assert.Equal(0, rc);
        string outStr = output.ToString();
        Assert.Contains("\"action\":\"rewrap-all\"", outStr);
        Assert.Contains("\"total\":2", outStr);
        Assert.Contains("\"rewrapped\":2", outStr);
        Assert.Contains("\"alreadyCurrent\":0", outStr);
        // The report carries counts only — never a secret value.
        Assert.DoesNotContain("value-a", outStr);
        Assert.DoesNotContain("value-b", outStr);
        // The rows now decrypt under the NEW KEK (proof the rewrap actually re-keyed them).
        var newCipher = new AesGcmEnvelopeCipher(newKek);
        Assert.Equal("value-a", newCipher.Decrypt((await _store.GetAsync(new SecretName("sql/a"), Ct))!));
    }
    [Fact]
    public async Task RewrapAll_IdenticalKek_ReturnsErrorExitCode()
    {
        var (seed, _) = NewSut();
        await seed.SetAsync("sql/a", "value-a", SecretContentType.Text, null, "alice", Ct);
        var (cmd, output) = NewSut();
        int rc = await cmd.RewrapAllAsync(_oldKek, new FakeMasterKeyProvider("kek-cli"), Ct);
        Assert.Equal(1, rc);
        Assert.Contains("\"error\"", output.ToString());
    }
    [Fact]
    public async Task Set_Then_Get_RoundTrips()
    {
@@ -95,6 +95,97 @@ public class AesGcmEnvelopeCipherTests
        Assert.Contains("unknown KEK", ex.Message, StringComparison.Ordinal);
    }
    [Fact]
    public void Rewrap_RewrapsDek_BodyStillDecryptsUnderNewKek()
    {
        var oldKek = new FakeMasterKeyProvider("kek-old");
        var newKek = new FakeMasterKeyProvider("kek-new");
        var oldCipher = new AesGcmEnvelopeCipher(oldKek);
        var name = new SecretName("sql/foo");
        StoredSecret row = oldCipher.Encrypt(name, "hunter2", SecretContentType.Text);
        Assert.Equal("kek-old", row.KekId);
        // Rewrap is provider-explicit — it can be driven through any cipher instance.
        StoredSecret rewrapped = oldCipher.Rewrap(row, oldKek, newKek);
        Assert.Equal("kek-new", rewrapped.KekId);
        // The DEK-sealed body is untouched; only the KEK-wrap envelope changed.
        Assert.True(rewrapped.Ciphertext.AsSpan().SequenceEqual(row.Ciphertext));
        Assert.True(rewrapped.Nonce.AsSpan().SequenceEqual(row.Nonce));
        Assert.True(rewrapped.Tag.AsSpan().SequenceEqual(row.Tag));
        Assert.False(rewrapped.WrappedDek.AsSpan().SequenceEqual(row.WrappedDek));
        // The rewrapped row decrypts under the NEW KEK...
        var newCipher = new AesGcmEnvelopeCipher(newKek);
        Assert.Equal("hunter2", newCipher.Decrypt(rewrapped));
        // ...and no longer under the OLD KEK (kek_id now names the new KEK; old cipher fails closed).
        Assert.Throws<SecretDecryptionException>(() => oldCipher.Decrypt(rewrapped));
    }
    [Fact]
    public void Rewrap_PreservesRevisionTimestampsTombstoneAndMetadata()
    {
        var oldKek = new FakeMasterKeyProvider("kek-old");
        var newKek = new FakeMasterKeyProvider("kek-new");
        var cipher = new AesGcmEnvelopeCipher(oldKek);
        StoredSecret row = cipher.Encrypt(new SecretName("sql/foo"), "hunter2", SecretContentType.Text) with
        {
            Revision = 7,
            IsDeleted = true,
            DeletedUtc = DateTimeOffset.UnixEpoch,
            CreatedUtc = DateTimeOffset.UnixEpoch,
            UpdatedUtc = DateTimeOffset.UnixEpoch.AddMinutes(5),
            CreatedBy = "alice",
            UpdatedBy = "bob",
            Description = "the foo password",
        };
        StoredSecret rewrapped = cipher.Rewrap(row, oldKek, newKek);
        // Only the wrap envelope + kek_id change — a rewrap must be invisible to (updated_utc, revision) LWW.
        Assert.Equal(7, rewrapped.Revision);
        Assert.True(rewrapped.IsDeleted);
        Assert.Equal(row.DeletedUtc, rewrapped.DeletedUtc);
        Assert.Equal(row.CreatedUtc, rewrapped.CreatedUtc);
        Assert.Equal(row.UpdatedUtc, rewrapped.UpdatedUtc);
        Assert.Equal("alice", rewrapped.CreatedBy);
        Assert.Equal("bob", rewrapped.UpdatedBy);
        Assert.Equal("the foo password", rewrapped.Description);
        Assert.Equal(SecretContentType.Text, rewrapped.ContentType);
    }
    [Fact]
    public void Rewrap_WrongOldKekId_FailsClosed()
    {
        var actualKek = new FakeMasterKeyProvider("kek-old");
        var wrongOld = new FakeMasterKeyProvider("kek-somethingelse");
        var newKek = new FakeMasterKeyProvider("kek-new");
        var cipher = new AesGcmEnvelopeCipher(actualKek);
        StoredSecret row = cipher.Encrypt(new SecretName("sql/foo"), "hunter2", SecretContentType.Text);
        // Row is wrapped by "kek-old" but the caller claims the old KEK is "kek-somethingelse".
        var ex = Assert.Throws<SecretDecryptionException>(() => cipher.Rewrap(row, wrongOld, newKek));
        Assert.Contains("kek-old", ex.Message, StringComparison.Ordinal);
    }
    [Fact]
    public void Rewrap_CorrectKekIdButWrongKeyMaterial_FailsClosed()
    {
        // Same KEK-id string, different key bytes → the unwrap must fail closed (tag mismatch),
        // never emitting a garbage DEK.
        var realOld = new FakeMasterKeyProvider("kek-old");
        var imposterOld = new FakeMasterKeyProvider("kek-old"); // same id, fresh random bytes
        var newKek = new FakeMasterKeyProvider("kek-new");
        var cipher = new AesGcmEnvelopeCipher(realOld);
        StoredSecret row = cipher.Encrypt(new SecretName("sql/foo"), "hunter2", SecretContentType.Text);
        Assert.Throws<SecretDecryptionException>(() => cipher.Rewrap(row, imposterOld, newKek));
    }
    private static bool Contains(byte[] haystack, byte[] needle)
    {
        if (needle.Length == 0 || haystack.Length < needle.Length)
@@ -48,4 +48,9 @@ public sealed class CountingSecretStore : ISecretStore
    /// <inheritdoc />
    public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) =>
        throw new NotSupportedException();
    /// <inheritdoc />
    public Task<bool> ApplyRewrapAsync(
        StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
        throw new NotSupportedException();
 }
@@ -0,0 +1,163 @@
 using Microsoft.Data.Sqlite;
 using ZB.MOM.WW.Secrets.Abstractions;
 using ZB.MOM.WW.Secrets.Crypto;
 using ZB.MOM.WW.Secrets.Rotation;
 using ZB.MOM.WW.Secrets.Sqlite;
 using ZB.MOM.WW.Secrets.Tests.Fakes;
 namespace ZB.MOM.WW.Secrets.Tests.Rotation;
 /// <summary>
 /// End-to-end KEK-rotation tests against the REAL SQLite store and REAL AES-256-GCM cipher — the
 /// combination that actually proves a re-wrapped row decrypts under the new KEK (and no longer
 /// under the old one).
 /// </summary>
 public sealed class KekRotationServiceTests : IAsyncLifetime, IDisposable
 {
    private readonly string _dbPath =
        Path.Combine(Path.GetTempPath(), $"zb-secrets-rotate-{Guid.NewGuid():N}.db");
    private readonly SecretsSqliteConnectionFactory _factory;
    private readonly SqliteSecretStore _store;
    private readonly FakeMasterKeyProvider _oldKek = new("kek-old");
    private readonly FakeMasterKeyProvider _newKek = new("kek-new");
    public KekRotationServiceTests()
    {
        _factory = new SecretsSqliteConnectionFactory(_dbPath);
        _store = new SqliteSecretStore(_factory);
    }
    public async Task InitializeAsync() =>
        await new SqliteSecretsStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
    public Task DisposeAsync() => Task.CompletedTask;
    // Seeds a secret sealed under a given KEK provider (defaults to the old KEK).
    private async Task SeedAsync(string name, string value, IMasterKeyProvider? kek = null)
    {
        var cipher = new AesGcmEnvelopeCipher(kek ?? _oldKek);
        StoredSecret row = cipher.Encrypt(new SecretName(name), value, SecretContentType.Text);
        await _store.UpsertAsync(row, CancellationToken.None);
    }
    private KekRotationService Service() => new(_store, new AesGcmEnvelopeCipher(_oldKek));
    [Fact]
    public async Task RewrapAll_MigratesEveryRow_DecryptsUnderNewKekOnly()
    {
        await SeedAsync("sql/a", "value-a");
        await SeedAsync("sql/b", "value-b");
        await SeedAsync("ldap/c", "value-c");
        RewrapReport report = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
        Assert.Equal(3, report.Total);
        Assert.Equal(3, report.Rewrapped);
        Assert.Equal(0, report.AlreadyCurrent);
        var newCipher = new AesGcmEnvelopeCipher(_newKek);
        var oldCipher = new AesGcmEnvelopeCipher(_oldKek);
        foreach ((string name, string value) in new[] { ("sql/a", "value-a"), ("sql/b", "value-b"), ("ldap/c", "value-c") })
        {
            StoredSecret row = (await _store.GetAsync(new SecretName(name), CancellationToken.None))!;
            Assert.Equal("kek-new", row.KekId);
            // Decrypts cleanly under the NEW KEK...
            Assert.Equal(value, newCipher.Decrypt(row));
            // ...and fails closed under the OLD KEK (kek_id no longer matches).
            Assert.Throws<SecretDecryptionException>(() => oldCipher.Decrypt(row));
        }
    }
    [Fact]
    public async Task RewrapAll_IsIdempotent_SecondRunSkipsEverything()
    {
        await SeedAsync("sql/a", "value-a");
        await SeedAsync("sql/b", "value-b");
        RewrapReport first = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
        Assert.Equal(2, first.Rewrapped);
        RewrapReport second = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
        Assert.Equal(2, second.Total);
        Assert.Equal(0, second.Rewrapped);
        Assert.Equal(2, second.AlreadyCurrent);
    }
    [Fact]
    public async Task RewrapAll_RewrapsTombstonedRows()
    {
        await SeedAsync("sql/live", "live");
        await SeedAsync("sql/dead", "dead");
        await _store.DeleteAsync(new SecretName("sql/dead"), "carol", CancellationToken.None);
        RewrapReport report = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
        Assert.Equal(2, report.Rewrapped);
        StoredSecret dead = (await _store.GetAsync(new SecretName("sql/dead"), CancellationToken.None))!;
        Assert.Equal("kek-new", dead.KekId);
        Assert.True(dead.IsDeleted);
    }
    [Fact]
    public async Task RewrapAll_DoesNotBumpRevisionOrUpdatedUtc()
    {
        await SeedAsync("sql/a", "value-a");
        StoredSecret before = (await _store.GetAsync(new SecretName("sql/a"), CancellationToken.None))!;
        await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
        StoredSecret after = (await _store.GetAsync(new SecretName("sql/a"), CancellationToken.None))!;
        Assert.Equal(before.Revision, after.Revision);
        Assert.Equal(before.UpdatedUtc, after.UpdatedUtc);
    }
    [Fact]
    public async Task RewrapAll_IdenticalKekIds_Throws()
    {
        await SeedAsync("sql/a", "value-a");
        var same = new FakeMasterKeyProvider("kek-old");
        await Assert.ThrowsAsync<ArgumentException>(
            () => Service().RewrapAllAsync(_oldKek, same, CancellationToken.None));
    }
    [Fact]
    public async Task RewrapAll_RowOnUnknownKek_Aborts_ButKeepsPriorProgress()
    {
        // Names are scanned in ORDER BY name: "a-old" and "b-old" migrate, then "z-third" (a KEK that
        // is neither old nor new) aborts the pass.
        await SeedAsync("a-old", "va");
        await SeedAsync("b-old", "vb");
        await SeedAsync("z-third", "vz", new FakeMasterKeyProvider("kek-third"));
        await Assert.ThrowsAsync<SecretDecryptionException>(
            () => Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None));
        // The two old-KEK rows scanned before the anomaly are already persisted on the new KEK,
        // so a re-run (after removing the anomaly) resumes cleanly.
        Assert.Equal("kek-new", (await _store.GetAsync(new SecretName("a-old"), CancellationToken.None))!.KekId);
        Assert.Equal("kek-new", (await _store.GetAsync(new SecretName("b-old"), CancellationToken.None))!.KekId);
        Assert.Equal("kek-third", (await _store.GetAsync(new SecretName("z-third"), CancellationToken.None))!.KekId);
    }
    public void Dispose()
    {
        SqliteConnection.ClearAllPools();
        foreach (string path in new[] { _dbPath, _dbPath + "-wal", _dbPath + "-shm" })
        {
            try
            {
                if (File.Exists(path))
                {
                    File.Delete(path);
                }
            }
            catch (IOException)
            {
                // Best-effort temp cleanup.
            }
        }
    }
 }
@@ -204,6 +204,101 @@ public sealed class SqliteSecretStoreTests : IAsyncLifetime, IDisposable
        Assert.Equal(new byte[] { 6, 6, 6 }, afterStale.Ciphertext);
    }
    [Fact]
    public async Task ApplyRewrap_ChangesOnlyWrapEnvelopeAndKekId_PreservingRevisionBodyAndTimestamps()
    {
        DateTimeOffset created = new(2026, 1, 1, 0, 0, 0, TimeSpan.Zero);
        StoredSecret seed = MakeSecret("app/rewrap", ciphertext: [1, 2, 3], createdUtc: created);
        await _store.UpsertAsync(seed, CancellationToken.None);
        // Bump the revision once so we can prove ApplyRewrap does NOT change it.
        await _store.UpsertAsync(MakeSecret("app/rewrap", ciphertext: [1, 2, 3]), CancellationToken.None);
        StoredSecret before = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!;
        Assert.Equal(1, before.Revision);
        // Re-wrap: only the four wrap-related fields differ; everything else is carried from `before`.
        StoredSecret rewrapped = before with
        {
            WrappedDek = [90, 91, 92],
            WrapNonce = [93, 94, 95],
            WrapTag = [96, 97, 98],
            KekId = "kek-2",
        };
        bool applied = await _store.ApplyRewrapAsync(rewrapped, before.WrappedDek, CancellationToken.None);
        Assert.True(applied);
        StoredSecret after = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!;
        // Wrap envelope + kek_id changed...
        Assert.Equal(new byte[] { 90, 91, 92 }, after.WrappedDek);
        Assert.Equal(new byte[] { 93, 94, 95 }, after.WrapNonce);
        Assert.Equal(new byte[] { 96, 97, 98 }, after.WrapTag);
        Assert.Equal("kek-2", after.KekId);
        // ...but revision, updated timestamp, the sealed body, and audit stamps are untouched.
        Assert.Equal(before.Revision, after.Revision);
        Assert.Equal(before.UpdatedUtc, after.UpdatedUtc);
        Assert.Equal(before.CreatedUtc, after.CreatedUtc);
        Assert.Equal(before.UpdatedBy, after.UpdatedBy);
        Assert.Equal(new byte[] { 1, 2, 3 }, after.Ciphertext);
        Assert.Equal(before.Nonce, after.Nonce);
        Assert.Equal(before.Tag, after.Tag);
    }
    [Fact]
    public async Task ApplyRewrap_RewrapsTombstonedRows()
    {
        await _store.UpsertAsync(MakeSecret("app/dead"), CancellationToken.None);
        await _store.DeleteAsync(new SecretName("app/dead"), "carol", CancellationToken.None);
        StoredSecret tomb = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!;
        Assert.True(tomb.IsDeleted);
        bool applied = await _store.ApplyRewrapAsync(
            tomb with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" },
            tomb.WrappedDek,
            CancellationToken.None);
        Assert.True(applied);
        StoredSecret after = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!;
        Assert.Equal("kek-2", after.KekId);
        // The tombstone survives the rewrap.
        Assert.True(after.IsDeleted);
        Assert.Equal(tomb.Revision, after.Revision);
    }
    [Fact]
    public async Task ApplyRewrap_ReturnsFalse_WhenRowAbsent()
    {
        StoredSecret ghost = MakeSecret("never-existed") with { KekId = "kek-2" };
        bool applied = await _store.ApplyRewrapAsync(ghost, ghost.WrappedDek, CancellationToken.None);
        Assert.False(applied);
    }
    [Fact]
    public async Task ApplyRewrap_ReturnsFalse_WhenCurrentWrapChanged_UnderConcurrentWrite()
    {
        StoredSecret original = MakeSecret("app/cas", ciphertext: [1, 2, 3]);
        await _store.UpsertAsync(original, CancellationToken.None);
        StoredSecret before = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!;
        // Simulate a concurrent set/rotate: a fresh write changes the wrapped DEK out from under us.
        await _store.UpsertAsync(
            MakeSecret("app/cas", ciphertext: [9, 9, 9]) with { WrappedDek = [77, 78, 79] },
            CancellationToken.None);
        // A rewrap that still expects the ORIGINAL wrap must NOT apply (0 rows) — it would otherwise
        // pair a stale re-wrap with the newer body and corrupt the row.
        bool applied = await _store.ApplyRewrapAsync(
            before with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" },
            before.WrappedDek,
            CancellationToken.None);
        Assert.False(applied);
        StoredSecret after = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!;
        // The concurrent write's wrap survived untouched; no stale re-wrap was applied.
        Assert.Equal(new byte[] { 77, 78, 79 }, after.WrappedDek);
        Assert.Equal("kek-1", after.KekId);
    }
    public void Dispose()
    {
        // Drop pooled connections so the WAL/-shm/-wal sidecars release before we delete.
@@ -42,4 +42,8 @@ public sealed class FakeCipher : ISecretCipher
        return Encoding.UTF8.GetString(secret.Ciphertext);
    }
    /// <inheritdoc />
    public StoredSecret Rewrap(StoredSecret secret, IMasterKeyProvider oldKek, IMasterKeyProvider newKek)
        => secret with { KekId = newKek.KekId };
 }
@@ -88,4 +88,9 @@ public sealed class RecordingSecretStore : ISecretStore
    /// <inheritdoc />
    public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
        => throw new NotSupportedException();
    /// <inheritdoc />
    public Task<bool> ApplyRewrapAsync(
        StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
        => throw new NotSupportedException();
 }
@@ -124,5 +124,9 @@ public class SecretsListTests : TestContext
        public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
            => throw new NotSupportedException();
        public Task<bool> ApplyRewrapAsync(
            StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
            => throw new NotSupportedException();
    }
 }
@@ -39,6 +39,14 @@ construction).
  parallelization for Host.Tests).
 - **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression** — `AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class` → `sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver` → `GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
  **Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
 - **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
  G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
  `ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
  operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
  crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
  fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
  with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
  entries below. Not yet committed/pushed (awaiting go-ahead).
 ## Design + implementation plans (2026-07-16)
@@ -55,7 +63,7 @@ code-verified anchors, co-located `.tasks.json`):
 - [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
  — **G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).
-G-7/G-8 remain design-only/deferred (below).
+**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).
 ## Cross-cutting observations
@@ -109,14 +117,35 @@ mounted key)** for clustered pairs (hard requirement — same KEK on every node)
 Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
 and map each app's admin role onto `secrets:manage` / `secrets:reveal`.
-### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — build `ZB.MOM.WW.Secrets.Akka`
+### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)
-Deferred until G-2…G-6 land for a clustered app. Options per SPEC: shared SQL-Server
+The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A —
-`ISecretStore` (simplest — mirrors the shared Data-Protection key ring) **or** the Akka
+a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared
-replicator (LWW, anti-entropy resync, tombstones). Requires a shared KEK.
+Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure
 modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred
 phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the
 `ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable
 plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
 + [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.**
-### G-8 — KEK-rotation runbook + `RewrapAll`
+### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
-The `RewrapAll(oldKek, newKek)` admin primitive + an operator runbook. Deferred (design-only
+The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
-in this cut).
+`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
 DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
 LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
 on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
 `ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
 columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
 corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
 (enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
 fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
 env-var-name/file-path — never a literal), README section, and the operator runbook
 [`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
 **Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
 (migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
 Version bumped `0.1.2`→`0.1.3` (additive interface members); **republish to the Gitea feed is a
 separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
 by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
 write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.
 ## Out of scope (this component)
@@ -0,0 +1,102 @@
 # G-7 — Clustered secret replication: design & fork resolution
 > **Status:** design-only (G-7 in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md)).
 > Companion executable plan: [`2026-07-17-secrets-g7-sqlserver-store.md`](2026-07-17-secrets-g7-sqlserver-store.md).
 > Prerequisite G-2…G-6 have landed for both clustered apps (ScadaBridge, OtOpcUa); G-8 (KEK
 > rotation) is built. This resolves the SPEC's "shared SQL store **vs** Akka replicator" fork and
 > specifies the recommended build.
 ## Problem
 `ZB.MOM.WW.Secrets` stores secrets in a **local SQLite** file by default. The two Akka-clustered
 apps run **multiple nodes** (ScadaBridge: central pair + N site pairs; OtOpcUa: admin/driver/dev
 roles). A secret written on one node must be resolvable on every node that needs it, **without**
 exposing plaintext or the KEK over the wire, and **without** each node holding a divergent copy.
 The library was built anticipating this: the schema already carries `revision` / `updated_utc` /
 `is_deleted`; `ISecretStore` exposes `GetManifestAsync` + `ApplyReplicatedAsync` (last-writer-wins);
 `ISecretReplicator` is a no-op seam; `SecretManifestEntry` supports anti-entropy. Nothing needs a
 migration to turn replication on.
 ## The fork (from SPEC §"Clustered pairs")
 Two ways to make one logical secret set visible cluster-wide. **Both require a shared KEK** — every
 node must resolve the *same* master key (shared mounted key file or shared env), because a row whose
 `kek_id` doesn't match the local provider fails closed on resolve. Ciphertext can cross the wire /
 sit in a shared DB safely because the KEK never does.
 ### Option A — Shared SQL-Server `ISecretStore` (one source of truth)
 Point every node's `ISecretStore` at **one shared SQL-Server database** instead of a per-node
 SQLite file. There is exactly one copy of each row; no replication, no reconciliation. The store
 still holds **ciphertext only** (the KEK stays per-node/out-of-DB). Build a
 `SqlServerSecretStore : ISecretStore` behind the existing seam (the SPEC already scoped "SQL-Server
 `ISecretStore` provider construction — build with G-7").
 - **Pro:** zero distributed-systems code or failure modes (no split-brain, no anti-entropy lag, no
  tombstone GC, no LWW clock-skew). ~1 new file mirroring `SqliteSecretStore`'s SQL + a migrator.
 - **Pro:** both apps **already run shared SQL Server** (ScadaBridge ConfigDb + central `dbo.*`;
  OtOpcUa central config DB) and already share their Data-Protection key ring through SQL — a shared
  secret store is operationally identical to what ops already run and back up.
 - **Pro:** rotation (G-8 `rewrap-all`) runs **once** against the one store.
 - **Con:** availability is coupled to the shared DB. A brief outage is bridged by the resolver's
  in-memory TTL cache; but a node that must keep resolving secrets while *partitioned from* the
  shared DB is not served by this option (see Option B).
 ### Option B — Akka replicator `ZB.MOM.WW.Secrets.Akka` (each node keeps a local store)
 Each node keeps its **local SQLite** store; a real `ISecretReplicator` broadcasts each newly-written
 encrypted row to peers, and a cluster anti-entropy actor periodically exchanges manifests
 (`GetManifestAsync`) and pulls missing/newer rows (`ApplyReplicatedAsync`, LWW). Tombstones
 propagate deletes.
 - **Pro:** each node resolves from local state, so a node **survives partition** from its peers /
  from any central DB — the right answer for air-gapped or intermittently-connected sites.
 - **Con:** a whole new package + actor lifecycle (a cluster-singleton reconciler or per-node gossip),
  message serialization for `StoredSecret`, LWW edge cases (clock skew, tombstone retention), and —
  critically — it can only be *proven* against a live multi-node cluster, a G-2-class live-validation
  effort per app.
 ## Decision
 **Build Option A (shared SQL-Server `ISecretStore`) as G-7.** Specify Option B as a **deferred
 phase-2**, to be built only when an app declares a concrete requirement to resolve secrets while
 partitioned from the shared store (an availability SLA neither app states today).
 Rationale: Option A delivers cluster-wide secrets for both apps with the least new code and **no new
 failure modes**, reusing the shared-SQL operational posture both apps already depend on for config,
 audit, and the DP key ring. The clustered-secrets requirement today is "every node sees the same
 secrets," which a shared store satisfies exactly. Partition-tolerance (Option B's only unique
 benefit) is a stronger, unstated requirement; paying its distributed-systems complexity now would be
 speculative (YAGNI). The `ISecretReplicator` seam stays in place, so Option B remains a drop-in later
 with **no change** to consumers or the wire/store contract.
 ## Option A — build shape (detail in the executable plan)
 1. **`SqlServerSecretStore : ISecretStore`** — mirrors `SqliteSecretStore` in T-SQL: `GetAsync`,
   `UpsertAsync` (revision bump via `MERGE`/`UPDATE`), `DeleteAsync` (tombstone), `ListAsync`
   (metadata projection — never ciphertext), `GetManifestAsync`, `ApplyReplicatedAsync` (LWW under a
   serializable transaction), and **`ApplyRewrapAsync`** (G-8; UPDATE the 4 wrap columns only).
   `Microsoft.Data.SqlClient`, fully parameterized.
 2. **`SqlServerSecretsStoreMigrator`** — schema-versioned, idempotent (`IF NOT EXISTS`), same column
   set/semantics as SQLite; `varbinary(max)` for the crypto BLOBs, `datetimeoffset` (or ISO-8601
   text, to match the SQLite round-trip exactly) for timestamps.
 3. **DI selection** — add `SecretsOptions.Store` (`Sqlite` default | `SqlServer`) +
   `SqlServerConnectionString`; `AddZbSecrets` binds the store + migrator from it. SQLite stays the
   default so single-process consumers (HistorianGateway, mxaccessgw) are unaffected.
 4. **Adoption (ScadaBridge, OtOpcUa)** — set `Secrets:Store = SqlServer` + a connection string
   (delivered via `${secret:}` / a `secret:` ref like every other connstr), and ensure **the same
   KEK** on every node (shared key file or shared env). Run G-8 `rewrap-all` once against the shared
   store when the KEK rotates.
 ## Hard constraints (both options)
 - **Same KEK on every node.** Non-negotiable — a mismatched `kek_id` fails closed on resolve.
 - **Ciphertext only crosses trust boundaries.** The store/wire never carries plaintext or the KEK.
 - **Rotation is per independent store.** G-8 `rewrap-all`: once for the shared SQL store (Option A);
  once per node for per-node SQLite (Option B).
 ## Out of scope
 - Akka remoting auth/TLS hardening (a separate concern from secret storage).
 - Option B's actual construction — captured as the deferred phase-2 in the executable plan.
@@ -0,0 +1,122 @@
 # G-7 (Option A) — Shared SQL-Server `ISecretStore` Implementation Plan
 > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or
 > subagent-driven-development) to implement this plan task-by-task.
 **Goal:** Add a shared SQL-Server `ISecretStore` provider to `ZB.MOM.WW.Secrets` so the Akka-clustered
 apps (ScadaBridge, OtOpcUa) can back their secrets with one shared, ciphertext-only database — every
 node sees the same secrets, no replication code.
 **Architecture:** Mirror the existing `SqliteSecretStore` + `SqliteSecretsStoreMigrator` in T-SQL
 behind the unchanged `ISecretStore` seam. Select the store in `AddZbSecrets` from a new
 `SecretsOptions.Store` enum (SQLite stays the default). The KEK stays per-node/out-of-DB, so the DB
 holds only ciphertext. See the design + fork rationale in
 [`2026-07-17-secrets-g7-clustered-replication-design.md`](2026-07-17-secrets-g7-clustered-replication-design.md).
 **Tech Stack:** .NET 10, `Microsoft.Data.SqlClient`, xUnit + Shouldly, env-gated live SQL tests
 (LocalDB or a `mcr.microsoft.com/mssql/server` container), matching the app live-test idiom.
 **Precondition:** G-8 is merged — `ISecretStore.ApplyRewrapAsync` exists and the SQL-Server store
 must implement it. `rewrap-all` then runs once against the shared store.
 ---
 ### Task 1: SQL-Server schema + migrator
 **Classification:** standard
 **Estimated implement time:** ~5 min
 **Parallelizable with:** none (Task 2 depends on it)
 **Files:**
 - Create: `src/ZB.MOM.WW.Secrets/SqlServer/SqlServerSecretsSchema.cs`
 - Create: `src/ZB.MOM.WW.Secrets/SqlServer/SqlServerSecretsStoreMigrator.cs`
 - Create: `src/ZB.MOM.WW.Secrets/SqlServer/SecretsSqlServerConnectionFactory.cs`
 - Test: `tests/ZB.MOM.WW.Secrets.Tests/SqlServer/SqlServerSecretsStoreMigratorTests.cs` (env-gated)
 **Notes:**
 - Mirror `SqliteSecretsSchema` / `SqliteSecretsStoreMigrator` semantics: `schema_version` single-row
  table + `secret` table, `CurrentVersion = 1`, idempotent `IF NOT EXISTS` DDL, refuse a newer
  on-disk version. Column set identical to SQLite; use `varbinary(max)` for the 6 crypto BLOBs,
  `nvarchar` for text, and store timestamps as **ISO-8601 (`"O"`) text** so `StoredSecret`
  round-trips byte-identically to the SQLite path (avoids `datetimeoffset` precision drift in tests).
 - Add `Microsoft.Data.SqlClient` to `Directory.Packages.props` + the core csproj.
 - Gate live tests on an env var (e.g. `SECRETS_SQLSERVER_CONNSTR`); skip cleanly when unset.
 ### Task 2: `SqlServerSecretStore : ISecretStore`
 **Classification:** high-risk
 **Estimated implement time:** ~5 min
 **Parallelizable with:** none (depends on Task 1)
 **Files:**
 - Create: `src/ZB.MOM.WW.Secrets/SqlServer/SqlServerSecretStore.cs`
 - Test: `tests/ZB.MOM.WW.Secrets.Tests/SqlServer/SqlServerSecretStoreTests.cs` (env-gated; port the
  `SqliteSecretStoreTests` cases 1:1, incl. the G-8 `ApplyRewrap*` cases)
 **Notes:**
 - Implement all 7 members: `GetAsync`, `UpsertAsync` (revision bump — `MERGE` or
  insert/`ON CONFLICT`-equivalent via `UPDATE`+`INSERT` under a transaction), `DeleteAsync`
  (tombstone + revision bump), `ListAsync` (metadata projection — **never** the crypto columns),
  `GetManifestAsync`, `ApplyReplicatedAsync` (LWW under `IsolationLevel.Serializable`, applied
  verbatim, no revision bump), `ApplyRewrapAsync` (UPDATE the 4 wrap columns + `kek_id` only; no
  revision/updated_utc change).
 - Fully parameterized commands; same "created_utc/created_by preserved on overwrite" semantics as
  SQLite. This is high-risk because the LWW + rewrap semantics MUST match SQLite exactly (a cluster
  reconciles across both if a node ever migrates).
 ### Task 3: DI store-selection wiring
 **Classification:** standard
 **Estimated implement time:** ~4 min
 **Parallelizable with:** none (depends on Task 2)
 **Files:**
 - Modify: `src/ZB.MOM.WW.Secrets/SecretsOptions.cs` (add `SecretsStoreKind Store` = `Sqlite` default;
  add `string? SqlServerConnectionString`)
 - Create: `src/ZB.MOM.WW.Secrets/SecretsStoreKind.cs` (enum `Sqlite | SqlServer`)
 - Modify: `src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsServiceCollectionExtensions.cs`
  (branch the `ISecretStore` + migrator registration on `Store`; keep SQLite the default so
  HistorianGateway/mxaccessgw are unaffected)
 - Modify: `src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsMigrationHostedService.cs` if it hard-refs
  the SQLite migrator type (abstract behind a shared migrator seam or a switch)
 - Test: `tests/ZB.MOM.WW.Secrets.Tests/DependencyInjection/AddZbSecretsTests.cs` (assert the correct
  store type is resolved for each `Store` value)
 **Notes:**
 - Introduce a tiny `ISecretsStoreMigrator` seam (both migrators implement it) so the hosted service +
  CLI stay store-agnostic, OR switch on `Store` at registration. Prefer the seam — cleaner.
 - The connection string should itself be deliverable via `${secret:}` / a `secret:` ref (consumers
  already do this for other connstrings).
 ### Task 4: Docs + adoption notes
 **Classification:** small
 **Estimated implement time:** ~3 min
 **Parallelizable with:** Task 3
 **Files:**
 - Modify: `README.md` (Clustered deployments: document `Secrets:Store = SqlServer` + shared-KEK
  requirement + "run `rewrap-all` once against the shared store")
 - Create: `docs/operations/shared-sql-store.md` (operator setup: provision the DB, grant, set the
  connstr via a secret ref, confirm the same KEK on every node)
 - Modify: `components/secrets/GAPS.md` (G-7 → "Option A built"; note Option B deferred)
 **Notes:** No app changes here — adoption in ScadaBridge/OtOpcUa is a follow-on (flip `Secrets:Store`,
 supply the connstr + shared KEK). Capture that as a checklist, not code.
 ---
 ## Deferred phase-2 (NOT built here): Option B — `ZB.MOM.WW.Secrets.Akka`
 Build only when an app declares it must resolve secrets while **partitioned** from the shared store.
 Shape (for when it's warranted):
 - New package `ZB.MOM.WW.Secrets.Akka` (net10; refs Akka.Cluster + Abstractions).
 - `AkkaSecretReplicator : ISecretReplicator` — `PublishAsync` broadcasts the encrypted `StoredSecret`
  to peers (a cluster-aware router / distributed pub-sub topic).
 - An anti-entropy actor (cluster singleton or per-node) that periodically exchanges
  `GetManifestAsync` digests and pulls newer/missing rows via `ApplyReplicatedAsync` (LWW);
  tombstones propagate deletes; a retention window bounds tombstone GC.
 - `StoredSecret` serialization (ciphertext only — never the KEK) + an `ISecretActorAccessor` wiring.
 - **Validation:** a live 2-node ScadaBridge/OtOpcUa rig proving write-on-A → resolve-on-B, delete
  propagation, and partition-heal resync — a G-2-class live gate.
 - KEK constraint identical to Option A: same master KEK on every node; `rewrap-all` per node.
@@ -0,0 +1,11 @@
 {
  "planPath": "docs/plans/2026-07-17-secrets-g7-sqlserver-store.md",
  "tasks": [
    {"id": 0, "subject": "Task 1: SQL-Server schema + migrator", "status": "pending"},
    {"id": 1, "subject": "Task 2: SqlServerSecretStore : ISecretStore", "status": "pending", "blockedBy": [0]},
    {"id": 2, "subject": "Task 3: DI store-selection wiring", "status": "pending", "blockedBy": [1]},
    {"id": 3, "subject": "Task 4: Docs + adoption notes", "status": "pending", "blockedBy": [1]}
  ],
  "deferredPhase2": "Option B — ZB.MOM.WW.Secrets.Akka replicator (build only on a stated partition-tolerance requirement); see plan doc.",
  "lastUpdated": "2026-07-17"
 }