feat(secrets): build G-8 KEK rotation (RewrapAll); design+plan G-7 clustered replication
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3: - ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only (bodies never re-encrypted; revision/timestamps preserved -> invisible to cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed wraps; DEK zeroed on all paths. - ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek): updates only the 4 wrap columns + kek_id, compare-and-swap on the current wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a review-caught TOCTOU). - KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry, fail-closed on unknown/identical KEK. - `secret rewrap-all` CLI verb: key material only via env-var name / file path, JSON counts report; README section + operator runbook. Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed). G-7 (clustered replication) — designed + planned, no code: - Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc + executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*. Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
This commit is contained in:
Joseph Doherty
@@ -5,7 +5,7 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
<Version>0.1.2</Version>
|
||||
<Version>0.1.3</Version>
|
||||
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
|
||||
<PackageReadmeFile>README.md</PackageReadmeFile>
|
||||
</PropertyGroup>
|
||||
|
||||
@@ -12,7 +12,7 @@ plaintext back on demand — from application code, from configuration, or from
|
||||
| `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
|
||||
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
|
||||
|
||||
A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate`) ships in the repo (not packed).
|
||||
A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).
|
||||
|
||||
## How it protects secrets
|
||||
|
||||
@@ -50,6 +50,24 @@ await expander.ExpandConfigurationAsync((IConfigurationRoot)builder.Configuratio
|
||||
The master key is 32 bytes, provided base64 in `ZB_SECRETS_MASTER_KEY` (Environment source).
|
||||
A missing/invalid key fails closed at startup (`MasterKeyUnavailableException`).
|
||||
|
||||
## Rotating the master KEK
|
||||
|
||||
Because each row is a body sealed under a per-secret DEK that is *wrapped* by the KEK, rotating
|
||||
the master key only re-wraps DEKs — bodies are never re-encrypted and no value history changes.
|
||||
The `secret rewrap-all` CLI verb (backed by `KekRotationService`) migrates every row from the
|
||||
old KEK to the new one; it is idempotent and safe to re-run:
|
||||
|
||||
```bash
|
||||
# New KEK defaults to the configured Secrets:MasterKey; supply the OLD key by env-var name or path.
|
||||
ZB_SECRETS_MASTER_KEY=<new-base64> ZB_SECRETS_OLD_KEY=<old-base64> \
|
||||
secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
|
||||
# → {"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}
|
||||
```
|
||||
|
||||
Run it with resolve traffic quiesced and once per independent store (once for a shared
|
||||
SQL-Server store; once per node for per-node SQLite on a shared KEK). See the operator runbook:
|
||||
[`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md).
|
||||
|
||||
## Runtime + human access
|
||||
|
||||
- **App code:** inject `ISecretResolver` and call `GetAsync(name, ct)`. Every resolve is
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# Operator runbook — rotating the master KEK (`ZB.MOM.WW.Secrets`)
|
||||
|
||||
**Audience:** operators rotating the key-encryption key (KEK) that protects a secrets store.
|
||||
**Primitive:** `secret rewrap-all` (CLI) → `KekRotationService.RewrapAllAsync` → per-row
|
||||
`ISecretCipher.Rewrap` + `ISecretStore.ApplyRewrapAsync`.
|
||||
**Status:** G-8 (KEK-rotation `RewrapAll` + runbook). Companion to the deferred G-7 clustered
|
||||
replication (`ZB.MOM.WW.Secrets.Akka`).
|
||||
|
||||
---
|
||||
|
||||
## What rotation does (and does not) touch
|
||||
|
||||
Each secret is stored as a body sealed under a fresh per-secret **data-encryption key (DEK)**;
|
||||
the DEK is then **wrapped** by the master **KEK**. Rotation **re-wraps the DEK only**:
|
||||
|
||||
- **Changed:** `wrapped_dek`, `wrap_nonce`, `wrap_tag`, `kek_id` — the KEK-wrap envelope.
|
||||
- **Untouched:** the sealed body (`ciphertext`/`nonce`/`tag`), `revision`, `updated_utc`,
|
||||
`created_*`/`updated_by`, the tombstone flag. Plaintext is never decrypted to disk or logged.
|
||||
|
||||
Because `revision`/`updated_utc` are preserved, a re-wrap is **invisible to cluster
|
||||
last-writer-wins** anti-entropy — so it never churns replication, and it must be run **per
|
||||
independent store** (see step 5).
|
||||
|
||||
## Properties you can rely on
|
||||
|
||||
- **Idempotent / resumable.** A row already on the new KEK is skipped (`alreadyCurrent`). If a
|
||||
run is interrupted, just run it again — it resumes from where it stopped.
|
||||
- **Fail-closed.** A row wrapped by a KEK that is *neither* the old nor the new one **aborts**
|
||||
the pass (rows already re-wrapped stay persisted). A wrong old-key or tampered wrap aborts with
|
||||
`SecretDecryptionException`. An identical old/new KEK id is rejected up front.
|
||||
- **No plaintext exposure.** The report is counts only: `{total, rewrapped, alreadyCurrent}`.
|
||||
|
||||
## KEK identity — the one thing to get right
|
||||
|
||||
The old KEK you supply **must resolve to the same `kek_id` that is stored on the rows.** The
|
||||
provider derives `kek_id` as `sha256:<12 hex>` of the key bytes **unless** an explicit
|
||||
`Secrets:MasterKey:KekId` was configured.
|
||||
|
||||
- **Derived id (default):** supplying the same old key **bytes** reproduces the same id — nothing
|
||||
extra to pass.
|
||||
- **Explicit id:** the app set `Secrets:MasterKey:KekId`. Pass the matching id with
|
||||
`--old-key-id <id>` (and `--new-key-id <id>` if the new KEK also uses an explicit id).
|
||||
|
||||
A mismatch fails closed (`Cannot rewrap … not the supplied old KEK`) — it never corrupts a row.
|
||||
|
||||
> **The new KEK id MUST differ from the old one.** With derived ids this is automatic (new key
|
||||
> bytes → new `sha256:` id). With **explicit** ids you must assign a new `--new-key-id`; reusing the
|
||||
> old id for new key material is rejected up front (`old and new KEK identifiers are identical;
|
||||
> there is nothing to rotate`) because `kek_id` is the only per-row rotation discriminator.
|
||||
|
||||
## Key material handling
|
||||
|
||||
- Key **values are never passed as command-line arguments** (they would be echoed/logged). Supply
|
||||
the old (and optionally new) key by **env-var NAME** (`--old-key-env VAR`) or **file PATH**
|
||||
(`--old-key-file /path`). The value lives only in that env var / file.
|
||||
- Never print `ZB_SECRETS_MASTER_KEY` or the old key to a terminal, log, or ticket.
|
||||
- The new KEK **defaults to the app's configured `Secrets:MasterKey` provider**, so you usually
|
||||
only supply the *old* key.
|
||||
|
||||
---
|
||||
|
||||
## Procedure
|
||||
|
||||
Rotation is an offline, store-local operation. Do it once per store.
|
||||
|
||||
**1. Back up the store.** Copy the SQLite file (or snapshot the SQL-Server DB). This is the
|
||||
undo — if anything is wrong, restore and retry.
|
||||
|
||||
**2. Quiesce resolve traffic.** Stop (or drain) the app(s) reading this store. Rotation preserves
|
||||
`updated_utc`/`revision`, so a running node on the *old* KEK would keep working until you flip its
|
||||
key — but running rotation against a store that other writers are mutating risks racing a fresh
|
||||
write onto the old KEK. Simplest correct posture: app down.
|
||||
|
||||
**3. Stage the new KEK.** Generate a fresh 32-byte key, base64-encode it, and place it where the
|
||||
CLI (and, after cutover, the app) will read it — e.g. `ZB_SECRETS_MASTER_KEY` for the Environment
|
||||
provider, or the mounted key file for the File provider.
|
||||
|
||||
**4. Run `rewrap-all`.** Point the CLI at the store's `appsettings.json` (so `Secrets:SqlitePath`
|
||||
+ the new `Secrets:MasterKey` resolve), and supply the old key:
|
||||
|
||||
```bash
|
||||
# Old key delivered via a throwaway env var (name only on the command line):
|
||||
ZB_SECRETS_OLD_KEY='<old-base64>' \
|
||||
secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
|
||||
# Explicit-id deployment:
|
||||
# secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --old-key-id sha256:abc… --new-key-id sha256:def…
|
||||
# Override the new key explicitly instead of the configured provider:
|
||||
# secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --new-key-file /run/secrets/new_kek
|
||||
```
|
||||
|
||||
Expected: `{"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}` and exit 0. On a
|
||||
re-run everything reports `alreadyCurrent` and `rewrapped:0`.
|
||||
|
||||
**5. For clustered / multi-store deployments, repeat per store.**
|
||||
|
||||
- **Shared SQL-Server store (the G-7 primary path):** one store → run rotation **once**.
|
||||
- **Per-node SQLite on a shared KEK:** each node has its own store file → run rotation **on each
|
||||
node** (all with the same old→new keys). Re-wrap does not replicate (revision preserved), so
|
||||
every store must be migrated independently.
|
||||
|
||||
**6. Cut the app over to the new KEK.** Ensure `Secrets:MasterKey` (and every node's KEK source)
|
||||
now resolves the **new** key, then start the app(s). Confirm a resolve succeeds
|
||||
(e.g. `secret get <known-name>` or an app readiness probe that exercises a `${secret:}` value).
|
||||
|
||||
**7. Destroy the old key.** Remove the old key material from env/files/secret managers once every
|
||||
store is confirmed on the new KEK and resolves are healthy.
|
||||
|
||||
## Rollback
|
||||
|
||||
If step 6/7 surfaces a problem, restore the pre-rotation store backup (step 1) and revert the app
|
||||
to the old KEK. Because bodies were never re-encrypted, the backup is a complete, consistent undo.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| `error: … old and new KEK identifiers are identical` | New KEK resolves to the same id as the old | You supplied the wrong new key, or forgot to stage it. Verify `Secrets:MasterKey`. |
|
||||
| `error: Secret '…' is wrapped by KEK 'X', which is neither the old nor the new KEK` | A row is on a third KEK (a prior partial/aborted rotation, or a mis-set id) | Investigate that row (`secret list` shows each row's `kekId`); re-point old/new keys or fix the row, then re-run (completed rows are skipped). |
|
||||
| `error: … DEK could not be unwrapped under the supplied old KEK` | Right `kek_id` but wrong key **bytes**, or a tampered row | Confirm you staged the correct old key bytes. If the row is genuinely tampered, restore from backup. |
|
||||
| Post-cutover resolves fail closed (`Row wrapped by unknown KEK`) | App still on the old KEK, or a store/node was missed in step 5 | Ensure every node's `Secrets:MasterKey` resolves the new key and every store was rewrapped. |
|
||||
@@ -32,4 +32,34 @@ public interface ISecretCipher
|
||||
/// The authentication tag fails to verify, or the row references an unknown or unavailable KEK.
|
||||
/// </exception>
|
||||
string Decrypt(StoredSecret secret);
|
||||
|
||||
/// <summary>
|
||||
/// Re-wraps the data-encryption key (DEK) of an existing row from <paramref name="oldKek"/> to
|
||||
/// <paramref name="newKek"/> — the KEK-rotation primitive. The DEK is unwrapped under the old
|
||||
/// master key and re-wrapped under the new one; the sealed body
|
||||
/// (<see cref="StoredSecret.Ciphertext"/>/<see cref="StoredSecret.Nonce"/>/<see cref="StoredSecret.Tag"/>)
|
||||
/// is <b>never</b> re-encrypted, and <see cref="StoredSecret.Revision"/>, the timestamps, the
|
||||
/// tombstone flag, and every other field are preserved verbatim. Only
|
||||
/// <see cref="StoredSecret.WrappedDek"/>/<see cref="StoredSecret.WrapNonce"/>/<see cref="StoredSecret.WrapTag"/>
|
||||
/// and <see cref="StoredSecret.KekId"/> change. Plaintext is never exposed.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This operation is <i>provider-explicit</i>: it uses only the two supplied providers and does
|
||||
/// not consult whatever KEK the cipher instance was constructed with, because a rotation spans
|
||||
/// two keys. The caller must have already established that <paramref name="secret"/> is wrapped
|
||||
/// by <paramref name="oldKek"/> (the row's <see cref="StoredSecret.KekId"/> equals
|
||||
/// <see cref="IMasterKeyProvider.KekId"/> of <paramref name="oldKek"/>); a mismatch fails closed.
|
||||
/// </remarks>
|
||||
/// <param name="secret">The stored row whose DEK should be re-wrapped.</param>
|
||||
/// <param name="oldKek">The provider for the KEK that currently wraps the row's DEK.</param>
|
||||
/// <param name="newKek">The provider for the KEK the DEK should be re-wrapped under.</param>
|
||||
/// <returns>
|
||||
/// A copy of <paramref name="secret"/> with the DEK re-wrapped under <paramref name="newKek"/>
|
||||
/// and <see cref="StoredSecret.KekId"/> set to the new KEK id; all other fields unchanged.
|
||||
/// </returns>
|
||||
/// <exception cref="SecretDecryptionException">
|
||||
/// The row is not wrapped by <paramref name="oldKek"/>, or the DEK fails to unwrap (wrong old
|
||||
/// key material or a tampered wrap).
|
||||
/// </exception>
|
||||
StoredSecret Rewrap(StoredSecret secret, IMasterKeyProvider oldKek, IMasterKeyProvider newKek);
|
||||
}
|
||||
|
||||
@@ -63,4 +63,42 @@ public interface ISecretStore
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>A task that completes when the row has been applied or ignored.</returns>
|
||||
Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct);
|
||||
|
||||
/// <summary>
|
||||
/// Applies a KEK re-wrap in place: overwrites <b>only</b> the KEK-wrap envelope columns
|
||||
/// (<see cref="StoredSecret.WrappedDek"/>/<see cref="StoredSecret.WrapNonce"/>/<see cref="StoredSecret.WrapTag"/>)
|
||||
/// and <see cref="StoredSecret.KekId"/> for the row named by <paramref name="rewrappedRow"/>,
|
||||
/// leaving the sealed body, revision, timestamps, tombstone flag, and audit stamps untouched.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is the persistence half of KEK rotation. It deliberately does <b>not</b> bump the
|
||||
/// revision or refresh <c>updated_utc</c>: a re-wrap does not change the secret's plaintext, so
|
||||
/// it must stay invisible to the (updated_utc, revision) last-writer-wins ordering used by
|
||||
/// cluster anti-entropy — otherwise every node's independent re-wrap would churn replication.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The update is a <b>compare-and-swap</b> on <paramref name="expectedCurrentWrappedDek"/>: the
|
||||
/// row is updated only if its current <see cref="StoredSecret.WrappedDek"/> still equals the DEK
|
||||
/// wrap that was unwrapped to produce <paramref name="rewrappedRow"/>. A concurrent write to the
|
||||
/// same secret (a <c>set</c>/<c>rotate</c> that generated a new DEK) changes that wrap, so the
|
||||
/// CAS matches 0 rows and the row is left untouched — rather than corrupting it by pairing a
|
||||
/// stale re-wrap with a newer body. The caller re-processes a 0-row result (fetch → re-wrap).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="rewrappedRow">
|
||||
/// The row carrying the new wrap envelope + <see cref="StoredSecret.KekId"/>; matched by
|
||||
/// <see cref="StoredSecret.Name"/>. Only the four wrap-related columns are read from it.
|
||||
/// </param>
|
||||
/// <param name="expectedCurrentWrappedDek">
|
||||
/// The <see cref="StoredSecret.WrappedDek"/> the row must still carry for the swap to apply — the
|
||||
/// wrap that was unwrapped to produce <paramref name="rewrappedRow"/>.
|
||||
/// </param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>
|
||||
/// <c>true</c> if the row was updated; <c>false</c> if no row matched (absent, or its wrapped DEK
|
||||
/// changed under a concurrent write).
|
||||
/// </returns>
|
||||
Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct);
|
||||
}
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Cli;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.MasterKey;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
// Headless `secret` CLI: set / get / list / rm / rotate over the ZB.MOM.WW secrets store.
|
||||
@@ -77,6 +79,41 @@ try
|
||||
return await commands.RemoveAsync(args[1], actor, ct);
|
||||
}
|
||||
|
||||
case "rewrap-all":
|
||||
{
|
||||
// Rotate the master KEK by re-wrapping every row's DEK. Key material is supplied ONLY by
|
||||
// env-var NAME or file PATH — never as a literal argument (which would be echoed/logged).
|
||||
string? oldKind = null, oldValue = null, oldKekId = null;
|
||||
string? newKind = null, newValue = null, newKekId = null;
|
||||
|
||||
for (int i = 1; i < args.Length; i++)
|
||||
{
|
||||
switch (args[i])
|
||||
{
|
||||
case "--old-key-env": (oldKind, oldValue) = ("env", RequireValue(args, ref i)); break;
|
||||
case "--old-key-file": (oldKind, oldValue) = ("file", RequireValue(args, ref i)); break;
|
||||
case "--old-key-id": oldKekId = RequireValue(args, ref i); break;
|
||||
case "--new-key-env": (newKind, newValue) = ("env", RequireValue(args, ref i)); break;
|
||||
case "--new-key-file": (newKind, newValue) = ("file", RequireValue(args, ref i)); break;
|
||||
case "--new-key-id": newKekId = RequireValue(args, ref i); break;
|
||||
default: return Usage($"Unknown option '{args[i]}'.");
|
||||
}
|
||||
}
|
||||
|
||||
if (oldKind is null)
|
||||
return Usage("rewrap-all requires --old-key-env <VAR> or --old-key-file <path>.");
|
||||
if (newKind is null && newKekId is not null)
|
||||
return Usage("--new-key-id requires --new-key-env <VAR> or --new-key-file <path>.");
|
||||
|
||||
IMasterKeyProvider oldKek = BuildProvider(oldKind, oldValue!, oldKekId);
|
||||
// Default the new KEK to the app's configured provider (Secrets:MasterKey); override with flags.
|
||||
IMasterKeyProvider newKek = newKind is null
|
||||
? host.Services.GetRequiredService<IMasterKeyProvider>()
|
||||
: BuildProvider(newKind, newValue!, newKekId);
|
||||
|
||||
return await commands.RewrapAllAsync(oldKek, newKek, ct);
|
||||
}
|
||||
|
||||
default:
|
||||
return Usage($"Unknown command '{args[0]}'.");
|
||||
}
|
||||
@@ -132,6 +169,26 @@ static SecretContentType ParseContentType(string value) => value switch
|
||||
$"Unknown --content-type '{value}'; expected text|connection-string|json|binary-base64."),
|
||||
};
|
||||
|
||||
// Reads the value that must follow an option flag at args[i], advancing i past it.
|
||||
static string RequireValue(string[] args, ref int i)
|
||||
{
|
||||
if (i + 1 >= args.Length)
|
||||
throw new ArgumentException($"{args[i]} requires a value.");
|
||||
return args[++i];
|
||||
}
|
||||
|
||||
// Builds a master-key provider from a source kind ("env" | "file") + its env-var name or file path,
|
||||
// with an optional explicit KEK id (needed when the app configured Secrets:MasterKey:KekId rather
|
||||
// than the derived sha256: id). The key VALUE itself is only ever read from the env var / file.
|
||||
static IMasterKeyProvider BuildProvider(string kind, string value, string? kekId) => kind switch
|
||||
{
|
||||
"env" => MasterKeyProviderFactory.Create(
|
||||
new MasterKeyOptions { Source = MasterKeySource.Environment, EnvVarName = value, KekId = kekId }),
|
||||
"file" => MasterKeyProviderFactory.Create(
|
||||
new MasterKeyOptions { Source = MasterKeySource.File, FilePath = value, KekId = kekId }),
|
||||
_ => throw new ArgumentException($"Unknown key source '{kind}'."),
|
||||
};
|
||||
|
||||
// Prints usage (optionally preceded by an error line) and returns the standard usage exit code.
|
||||
int Usage(string? error = null)
|
||||
{
|
||||
@@ -147,6 +204,14 @@ int Usage(string? error = null)
|
||||
get <name>
|
||||
list [--include-deleted]
|
||||
rm <name>
|
||||
rewrap-all --old-key-env <VAR>|--old-key-file <path> [--old-key-id <id>]
|
||||
[--new-key-env <VAR>|--new-key-file <path>] [--new-key-id <id>]
|
||||
|
||||
rewrap-all rotates the master KEK: it re-wraps every stored secret's data key from the old
|
||||
KEK to the new one (bodies are never re-encrypted, no plaintext is exposed). The new KEK
|
||||
defaults to the configured Secrets:MasterKey provider. Supply key material only by env-var
|
||||
NAME or file PATH — never as a literal. Run it with resolve traffic quiesced; it is
|
||||
idempotent and safe to re-run.
|
||||
""");
|
||||
return 2;
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using System.Text.Json;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Rotation;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Cli;
|
||||
|
||||
@@ -156,6 +157,47 @@ public sealed class SecretCommands
|
||||
return removed ? 0 : 1;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// rewrap-all: rotates the master KEK by re-wrapping every row's DEK from <paramref name="oldKek"/>
|
||||
/// onto <paramref name="newKek"/> — bodies are never re-encrypted and no plaintext is exposed.
|
||||
/// Prints a JSON report of the counts (total / rewrapped / already-current) and NEVER any key
|
||||
/// material. Idempotent + resumable; fails closed (non-zero) on an identical-KEK mistake or a row
|
||||
/// wrapped by an unrecognized KEK.
|
||||
/// </summary>
|
||||
/// <param name="oldKek">The provider for the KEK rows are currently wrapped under.</param>
|
||||
/// <param name="newKek">The provider for the KEK rows should be re-wrapped onto.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>0 on success; non-zero if the rotation was rejected or aborted.</returns>
|
||||
public async Task<int> RewrapAllAsync(
|
||||
IMasterKeyProvider oldKek, IMasterKeyProvider newKek, CancellationToken ct)
|
||||
{
|
||||
var service = new KekRotationService(_store, _cipher);
|
||||
try
|
||||
{
|
||||
RewrapReport report = await service.RewrapAllAsync(oldKek, newKek, ct).ConfigureAwait(false);
|
||||
WriteJson(new
|
||||
{
|
||||
action = "rewrap-all",
|
||||
total = report.Total,
|
||||
rewrapped = report.Rewrapped,
|
||||
alreadyCurrent = report.AlreadyCurrent,
|
||||
});
|
||||
return 0;
|
||||
}
|
||||
catch (ArgumentException ex)
|
||||
{
|
||||
// Identical old/new KEK ids — nothing to rotate.
|
||||
WriteJson(new { action = "rewrap-all", error = ex.Message });
|
||||
return 1;
|
||||
}
|
||||
catch (SecretDecryptionException ex)
|
||||
{
|
||||
// A row on an unrecognized KEK aborted the pass, or a DEK failed to unwrap.
|
||||
WriteJson(new { action = "rewrap-all", error = ex.Message });
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Seals a value under the cipher and upserts it, carrying description + actor stamps.</summary>
|
||||
private async Task UpsertSealedAsync(
|
||||
SecretName name, string value, SecretContentType contentType, string? description, string? actor, CancellationToken ct)
|
||||
|
||||
@@ -153,4 +153,75 @@ public sealed class AesGcmEnvelopeCipher : ISecretCipher
|
||||
CryptographicOperations.ZeroMemory(dek);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public StoredSecret Rewrap(StoredSecret secret, IMasterKeyProvider oldKek, IMasterKeyProvider newKek)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(secret);
|
||||
ArgumentNullException.ThrowIfNull(oldKek);
|
||||
ArgumentNullException.ThrowIfNull(newKek);
|
||||
|
||||
// Fail closed if the row is not actually wrapped by the supplied old KEK — a caller must
|
||||
// not silently "rewrap" a row that is already on a different key.
|
||||
if (!string.Equals(secret.KekId, oldKek.KekId, StringComparison.Ordinal))
|
||||
{
|
||||
throw new SecretDecryptionException(
|
||||
$"Cannot rewrap secret '{secret.Name.Value}': row is wrapped by KEK '{secret.KekId}', " +
|
||||
$"not the supplied old KEK '{oldKek.KekId}'.");
|
||||
}
|
||||
|
||||
ReadOnlySpan<byte> oldMasterKey = oldKek.GetMasterKey().Span;
|
||||
ReadOnlySpan<byte> newMasterKey = newKek.GetMasterKey().Span;
|
||||
string newKekId = newKek.KekId;
|
||||
|
||||
byte[] dek = new byte[KeySizeBytes];
|
||||
try
|
||||
{
|
||||
// Unwrap the DEK under the OLD master key, verifying the old KEK-id AAD binding. A wrong
|
||||
// old key or tampered wrap surfaces as SecretDecryptionException (never a raw crypto leak).
|
||||
byte[] oldKekAad = Encoding.UTF8.GetBytes(secret.KekId);
|
||||
try
|
||||
{
|
||||
using var unwrap = new AesGcm(oldMasterKey, TagSizeBytes);
|
||||
unwrap.Decrypt(secret.WrapNonce, secret.WrappedDek, secret.WrapTag, dek, oldKekAad);
|
||||
}
|
||||
// CryptographicException = wrong key / tampered wrap; ArgumentException = a malformed
|
||||
// (wrong-length) wrap nonce/tag on a corrupt row. Both fail closed as a decryption error
|
||||
// rather than leaking a raw crypto/arg exception (which the caller would misclassify).
|
||||
catch (Exception ex) when (ex is CryptographicException or ArgumentException)
|
||||
{
|
||||
throw new SecretDecryptionException(
|
||||
$"Failed to rewrap secret '{secret.Name.Value}': the DEK could not be unwrapped " +
|
||||
"under the supplied old KEK.", ex);
|
||||
}
|
||||
|
||||
// Re-wrap the SAME DEK under the NEW master key with a fresh nonce, binding the new KEK id
|
||||
// as AAD. The body ciphertext is left untouched — rotation re-wraps DEKs only.
|
||||
byte[] wrapNonce = new byte[NonceSizeBytes];
|
||||
byte[] wrapTag = new byte[TagSizeBytes];
|
||||
byte[] wrappedDek = new byte[KeySizeBytes];
|
||||
byte[] newKekAad = Encoding.UTF8.GetBytes(newKekId);
|
||||
|
||||
RandomNumberGenerator.Fill(wrapNonce);
|
||||
using (var wrapGcm = new AesGcm(newMasterKey, TagSizeBytes))
|
||||
{
|
||||
wrapGcm.Encrypt(wrapNonce, dek, wrappedDek, wrapTag, newKekAad);
|
||||
}
|
||||
|
||||
// Only the KEK-wrap envelope + kek_id change; body, revision, timestamps, tombstone,
|
||||
// content-type, and every other field are preserved so a rewrap is invisible to the
|
||||
// (updated_utc, revision) last-writer-wins ordering used by cluster anti-entropy.
|
||||
return secret with
|
||||
{
|
||||
WrappedDek = wrappedDek,
|
||||
WrapNonce = wrapNonce,
|
||||
WrapTag = wrapTag,
|
||||
KekId = newKekId,
|
||||
};
|
||||
}
|
||||
finally
|
||||
{
|
||||
CryptographicOperations.ZeroMemory(dek);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Rotation;
|
||||
|
||||
/// <summary>
|
||||
/// Rotates the master key-encryption key (KEK) across every stored secret by re-wrapping each
|
||||
/// row's data-encryption key (DEK) from the old KEK to the new one. Because the envelope model
|
||||
/// wraps a per-secret DEK under the KEK, rotation never re-encrypts secret bodies — only the DEK
|
||||
/// wrap changes — so plaintext is never exposed and no value history is touched.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The pass is <b>idempotent and resumable</b>: a row already wrapped by the new KEK is skipped
|
||||
/// (counted in <see cref="RewrapReport.AlreadyCurrent"/>), so a run interrupted part-way can simply
|
||||
/// be re-run to finish. It is <b>fail-closed</b>: a row wrapped by neither the old nor the new KEK
|
||||
/// aborts the pass (rather than silently leaving an un-migratable row that would become
|
||||
/// undecryptable once the old KEK is retired); rows already re-wrapped in the aborted run stay
|
||||
/// persisted, so a re-run resumes cleanly after the anomaly is investigated.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Rotation is a per-store operation and must be run once per independent store (once for a shared
|
||||
/// SQL-Server store; once per node for a per-node SQLite store on a shared KEK). It should be run
|
||||
/// with resolve traffic quiesced: the persistence path deliberately does not bump revision or
|
||||
/// <c>updated_utc</c>, so a re-wrap is invisible to cluster last-writer-wins reconciliation.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class KekRotationService
|
||||
{
|
||||
private readonly ISecretStore _store;
|
||||
private readonly ISecretCipher _cipher;
|
||||
|
||||
/// <summary>Creates the rotation service over the encrypted store and envelope cipher.</summary>
|
||||
/// <param name="store">The store to enumerate and re-wrap in place.</param>
|
||||
/// <param name="cipher">The envelope cipher providing the DEK re-wrap primitive.</param>
|
||||
public KekRotationService(ISecretStore store, ISecretCipher cipher)
|
||||
{
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_cipher = cipher ?? throw new ArgumentNullException(nameof(cipher));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Re-wraps every row from <paramref name="oldKek"/> onto <paramref name="newKek"/>.
|
||||
/// </summary>
|
||||
/// <param name="oldKek">The provider for the KEK rows are currently wrapped under.</param>
|
||||
/// <param name="newKek">The provider for the KEK rows should be re-wrapped onto.</param>
|
||||
/// <param name="ct">A token to cancel the pass (already-persisted re-wraps are retained).</param>
|
||||
/// <returns>A <see cref="RewrapReport"/> summarizing the pass (no secret material).</returns>
|
||||
/// <exception cref="ArgumentException">
|
||||
/// The old and new KEK identifiers are the same — there is nothing to rotate.
|
||||
/// </exception>
|
||||
/// <exception cref="SecretDecryptionException">
|
||||
/// A row is wrapped by a KEK that is neither the old nor the new one (the pass aborts), or a
|
||||
/// row's DEK fails to unwrap under the supplied old KEK.
|
||||
/// </exception>
|
||||
public async Task<RewrapReport> RewrapAllAsync(
|
||||
IMasterKeyProvider oldKek, IMasterKeyProvider newKek, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(oldKek);
|
||||
ArgumentNullException.ThrowIfNull(newKek);
|
||||
|
||||
if (string.Equals(oldKek.KekId, newKek.KekId, StringComparison.Ordinal))
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"The old and new KEK identifiers are identical ('{oldKek.KekId}'); there is nothing to rotate.",
|
||||
nameof(newKek));
|
||||
}
|
||||
|
||||
// Enumerate metadata (never ciphertext) for ALL rows including tombstones — a tombstoned row
|
||||
// still carries a KEK-wrapped DEK and would become un-migratable if left on a retired KEK.
|
||||
IReadOnlyList<SecretMetadata> all =
|
||||
await _store.ListAsync(includeDeleted: true, ct).ConfigureAwait(false);
|
||||
|
||||
int rewrapped = 0;
|
||||
int alreadyCurrent = 0;
|
||||
|
||||
foreach (SecretMetadata meta in all)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
|
||||
// Fast-path skip on the metadata kek_id (no ciphertext fetch): a row already on the new
|
||||
// KEK is a completed migration — this is what makes the pass idempotent / resumable.
|
||||
if (string.Equals(meta.KekId, newKek.KekId, StringComparison.Ordinal))
|
||||
{
|
||||
alreadyCurrent++;
|
||||
continue;
|
||||
}
|
||||
|
||||
switch (await RewrapOneAsync(meta.Name, oldKek, newKek, rewrapped, ct).ConfigureAwait(false))
|
||||
{
|
||||
case RowOutcome.Rewrapped:
|
||||
rewrapped++;
|
||||
break;
|
||||
case RowOutcome.AlreadyCurrent:
|
||||
alreadyCurrent++;
|
||||
break;
|
||||
case RowOutcome.Vanished:
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return new RewrapReport { Total = all.Count, Rewrapped = rewrapped, AlreadyCurrent = alreadyCurrent };
|
||||
}
|
||||
|
||||
private enum RowOutcome { Rewrapped, AlreadyCurrent, Vanished }
|
||||
|
||||
// Re-wraps a single row with a bounded fetch→rewrap→compare-and-swap retry so a rare concurrent
|
||||
// write (the pass should run with writes quiesced) is retried rather than corrupting the row.
|
||||
private async Task<RowOutcome> RewrapOneAsync(
|
||||
SecretName name, IMasterKeyProvider oldKek, IMasterKeyProvider newKek, int rewrappedSoFar, CancellationToken ct)
|
||||
{
|
||||
const int maxAttempts = 5;
|
||||
for (int attempt = 1; ; attempt++)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
|
||||
StoredSecret? row = await _store.GetAsync(name, ct).ConfigureAwait(false);
|
||||
if (row is null)
|
||||
{
|
||||
// Vanished between the list and the fetch (a concurrent hard-absence).
|
||||
return RowOutcome.Vanished;
|
||||
}
|
||||
|
||||
// Re-evaluate against the FRESH row: a concurrent write may have already moved it.
|
||||
if (string.Equals(row.KekId, newKek.KekId, StringComparison.Ordinal))
|
||||
{
|
||||
return RowOutcome.AlreadyCurrent;
|
||||
}
|
||||
|
||||
if (!string.Equals(row.KekId, oldKek.KekId, StringComparison.Ordinal))
|
||||
{
|
||||
throw new SecretDecryptionException(
|
||||
$"Secret '{name.Value}' is wrapped by KEK '{row.KekId}', which is neither the old " +
|
||||
$"('{oldKek.KekId}') nor the new ('{newKek.KekId}') KEK. Rotation aborted after re-wrapping " +
|
||||
$"{rewrappedSoFar} row(s); investigate the anomalous row and re-run to resume.");
|
||||
}
|
||||
|
||||
// Rewrap re-verifies row.KekId == oldKek.KekId and fails closed on a wrong old key.
|
||||
StoredSecret rewrappedRow = _cipher.Rewrap(row, oldKek, newKek);
|
||||
|
||||
// Compare-and-swap on the wrap we just unwrapped: a concurrent set/rotate changed it →
|
||||
// 0 rows → retry (fetch the newer row and re-evaluate).
|
||||
if (await _store.ApplyRewrapAsync(rewrappedRow, row.WrappedDek, ct).ConfigureAwait(false))
|
||||
{
|
||||
return RowOutcome.Rewrapped;
|
||||
}
|
||||
|
||||
if (attempt >= maxAttempts)
|
||||
{
|
||||
throw new SecretDecryptionException(
|
||||
$"Secret '{name.Value}' is being modified concurrently; rotation could not converge after " +
|
||||
$"{maxAttempts} attempts. Re-run with secret writes quiesced.");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
namespace ZB.MOM.WW.Secrets.Rotation;
|
||||
|
||||
/// <summary>
|
||||
/// The outcome of a <see cref="KekRotationService.RewrapAllAsync"/> pass: how many rows were
|
||||
/// re-wrapped onto the new KEK, how many were already on it (skipped), and the total scanned.
|
||||
/// Carries no secret material.
|
||||
/// </summary>
|
||||
public sealed record RewrapReport
|
||||
{
|
||||
/// <summary>Total number of rows scanned (includes tombstoned rows).</summary>
|
||||
public required int Total { get; init; }
|
||||
|
||||
/// <summary>Number of rows re-wrapped from the old KEK onto the new KEK in this pass.</summary>
|
||||
public required int Rewrapped { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Number of rows already wrapped by the new KEK and therefore skipped — the count that makes a
|
||||
/// re-run idempotent (a completed rotation reports <see cref="Rewrapped"/> 0 and all rows here).
|
||||
/// </summary>
|
||||
public required int AlreadyCurrent { get; init; }
|
||||
}
|
||||
@@ -260,6 +260,43 @@ public sealed class SqliteSecretStore(SecretsSqliteConnectionFactory connectionF
|
||||
await transaction.CommitAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(rewrappedRow);
|
||||
ArgumentNullException.ThrowIfNull(expectedCurrentWrappedDek);
|
||||
|
||||
await using SqliteConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Overwrite ONLY the KEK-wrap envelope + kek_id. Revision, updated_utc/by, the sealed body,
|
||||
// and the tombstone state are deliberately untouched — a re-wrap is not a logical change, so
|
||||
// it must not bump the revision or updated timestamp (see ISecretStore.ApplyRewrapAsync).
|
||||
//
|
||||
// Compare-and-swap on wrapped_dek: a concurrent set/rotate generates a fresh (random) DEK
|
||||
// wrap, so this matches 0 rows instead of pairing this stale re-wrap with a newer body —
|
||||
// which would leave the row permanently undecryptable. The caller re-processes a 0-row miss.
|
||||
await using SqliteCommand command = connection.CreateCommand();
|
||||
command.CommandText = """
|
||||
UPDATE secret SET
|
||||
wrapped_dek = $wrapped_dek,
|
||||
wrap_nonce = $wrap_nonce,
|
||||
wrap_tag = $wrap_tag,
|
||||
kek_id = $kek_id
|
||||
WHERE name = $name AND wrapped_dek = $expected_wrapped_dek;
|
||||
""";
|
||||
command.Parameters.AddWithValue("$wrapped_dek", rewrappedRow.WrappedDek);
|
||||
command.Parameters.AddWithValue("$wrap_nonce", rewrappedRow.WrapNonce);
|
||||
command.Parameters.AddWithValue("$wrap_tag", rewrappedRow.WrapTag);
|
||||
command.Parameters.AddWithValue("$kek_id", rewrappedRow.KekId);
|
||||
command.Parameters.AddWithValue("$name", rewrappedRow.Name.Value);
|
||||
command.Parameters.AddWithValue("$expected_wrapped_dek", expectedCurrentWrappedDek);
|
||||
|
||||
int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
return rowsAffected > 0;
|
||||
}
|
||||
|
||||
// Binds the identity, description, content-type, KEK id, and all six crypto BLOB columns
|
||||
// shared by every insert path.
|
||||
private static void BindCryptoColumns(SqliteCommand command, StoredSecret row)
|
||||
|
||||
@@ -19,6 +19,7 @@ public sealed class SecretCommandsTests : IAsyncLifetime, IDisposable
|
||||
|
||||
private readonly SecretsSqliteConnectionFactory _factory;
|
||||
private readonly SqliteSecretStore _store;
|
||||
private readonly FakeMasterKeyProvider _oldKek = new("kek-cli");
|
||||
private readonly AesGcmEnvelopeCipher _cipher;
|
||||
private readonly DefaultSecretResolver _resolver;
|
||||
|
||||
@@ -26,7 +27,7 @@ public sealed class SecretCommandsTests : IAsyncLifetime, IDisposable
|
||||
{
|
||||
_factory = new SecretsSqliteConnectionFactory(_dbPath);
|
||||
_store = new SqliteSecretStore(_factory);
|
||||
_cipher = new AesGcmEnvelopeCipher(new FakeMasterKeyProvider("kek-cli"));
|
||||
_cipher = new AesGcmEnvelopeCipher(_oldKek);
|
||||
// A near-zero TTL keeps the resolver from serving a stale plaintext across a rotate/remove.
|
||||
_resolver = new DefaultSecretResolver(
|
||||
_store, _cipher, NoOpAuditWriter.Instance, TimeSpan.Zero);
|
||||
@@ -53,6 +54,45 @@ public sealed class SecretCommandsTests : IAsyncLifetime, IDisposable
|
||||
|
||||
private static CancellationToken Ct => CancellationToken.None;
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_MigratesRowsToNewKek_ReportsCountsWithoutLeakingValues()
|
||||
{
|
||||
var (seed, _) = NewSut();
|
||||
await seed.SetAsync("sql/a", "value-a", SecretContentType.Text, null, "alice", Ct);
|
||||
await seed.SetAsync("sql/b", "value-b", SecretContentType.Text, null, "alice", Ct);
|
||||
|
||||
var newKek = new FakeMasterKeyProvider("kek-new");
|
||||
var (cmd, output) = NewSut();
|
||||
int rc = await cmd.RewrapAllAsync(_oldKek, newKek, Ct);
|
||||
|
||||
Assert.Equal(0, rc);
|
||||
string outStr = output.ToString();
|
||||
Assert.Contains("\"action\":\"rewrap-all\"", outStr);
|
||||
Assert.Contains("\"total\":2", outStr);
|
||||
Assert.Contains("\"rewrapped\":2", outStr);
|
||||
Assert.Contains("\"alreadyCurrent\":0", outStr);
|
||||
// The report carries counts only — never a secret value.
|
||||
Assert.DoesNotContain("value-a", outStr);
|
||||
Assert.DoesNotContain("value-b", outStr);
|
||||
|
||||
// The rows now decrypt under the NEW KEK (proof the rewrap actually re-keyed them).
|
||||
var newCipher = new AesGcmEnvelopeCipher(newKek);
|
||||
Assert.Equal("value-a", newCipher.Decrypt((await _store.GetAsync(new SecretName("sql/a"), Ct))!));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_IdenticalKek_ReturnsErrorExitCode()
|
||||
{
|
||||
var (seed, _) = NewSut();
|
||||
await seed.SetAsync("sql/a", "value-a", SecretContentType.Text, null, "alice", Ct);
|
||||
|
||||
var (cmd, output) = NewSut();
|
||||
int rc = await cmd.RewrapAllAsync(_oldKek, new FakeMasterKeyProvider("kek-cli"), Ct);
|
||||
|
||||
Assert.Equal(1, rc);
|
||||
Assert.Contains("\"error\"", output.ToString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Set_Then_Get_RoundTrips()
|
||||
{
|
||||
|
||||
@@ -95,6 +95,97 @@ public class AesGcmEnvelopeCipherTests
|
||||
Assert.Contains("unknown KEK", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Rewrap_RewrapsDek_BodyStillDecryptsUnderNewKek()
|
||||
{
|
||||
var oldKek = new FakeMasterKeyProvider("kek-old");
|
||||
var newKek = new FakeMasterKeyProvider("kek-new");
|
||||
var oldCipher = new AesGcmEnvelopeCipher(oldKek);
|
||||
var name = new SecretName("sql/foo");
|
||||
|
||||
StoredSecret row = oldCipher.Encrypt(name, "hunter2", SecretContentType.Text);
|
||||
Assert.Equal("kek-old", row.KekId);
|
||||
|
||||
// Rewrap is provider-explicit — it can be driven through any cipher instance.
|
||||
StoredSecret rewrapped = oldCipher.Rewrap(row, oldKek, newKek);
|
||||
|
||||
Assert.Equal("kek-new", rewrapped.KekId);
|
||||
// The DEK-sealed body is untouched; only the KEK-wrap envelope changed.
|
||||
Assert.True(rewrapped.Ciphertext.AsSpan().SequenceEqual(row.Ciphertext));
|
||||
Assert.True(rewrapped.Nonce.AsSpan().SequenceEqual(row.Nonce));
|
||||
Assert.True(rewrapped.Tag.AsSpan().SequenceEqual(row.Tag));
|
||||
Assert.False(rewrapped.WrappedDek.AsSpan().SequenceEqual(row.WrappedDek));
|
||||
|
||||
// The rewrapped row decrypts under the NEW KEK...
|
||||
var newCipher = new AesGcmEnvelopeCipher(newKek);
|
||||
Assert.Equal("hunter2", newCipher.Decrypt(rewrapped));
|
||||
// ...and no longer under the OLD KEK (kek_id now names the new KEK; old cipher fails closed).
|
||||
Assert.Throws<SecretDecryptionException>(() => oldCipher.Decrypt(rewrapped));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Rewrap_PreservesRevisionTimestampsTombstoneAndMetadata()
|
||||
{
|
||||
var oldKek = new FakeMasterKeyProvider("kek-old");
|
||||
var newKek = new FakeMasterKeyProvider("kek-new");
|
||||
var cipher = new AesGcmEnvelopeCipher(oldKek);
|
||||
|
||||
StoredSecret row = cipher.Encrypt(new SecretName("sql/foo"), "hunter2", SecretContentType.Text) with
|
||||
{
|
||||
Revision = 7,
|
||||
IsDeleted = true,
|
||||
DeletedUtc = DateTimeOffset.UnixEpoch,
|
||||
CreatedUtc = DateTimeOffset.UnixEpoch,
|
||||
UpdatedUtc = DateTimeOffset.UnixEpoch.AddMinutes(5),
|
||||
CreatedBy = "alice",
|
||||
UpdatedBy = "bob",
|
||||
Description = "the foo password",
|
||||
};
|
||||
|
||||
StoredSecret rewrapped = cipher.Rewrap(row, oldKek, newKek);
|
||||
|
||||
// Only the wrap envelope + kek_id change — a rewrap must be invisible to (updated_utc, revision) LWW.
|
||||
Assert.Equal(7, rewrapped.Revision);
|
||||
Assert.True(rewrapped.IsDeleted);
|
||||
Assert.Equal(row.DeletedUtc, rewrapped.DeletedUtc);
|
||||
Assert.Equal(row.CreatedUtc, rewrapped.CreatedUtc);
|
||||
Assert.Equal(row.UpdatedUtc, rewrapped.UpdatedUtc);
|
||||
Assert.Equal("alice", rewrapped.CreatedBy);
|
||||
Assert.Equal("bob", rewrapped.UpdatedBy);
|
||||
Assert.Equal("the foo password", rewrapped.Description);
|
||||
Assert.Equal(SecretContentType.Text, rewrapped.ContentType);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Rewrap_WrongOldKekId_FailsClosed()
|
||||
{
|
||||
var actualKek = new FakeMasterKeyProvider("kek-old");
|
||||
var wrongOld = new FakeMasterKeyProvider("kek-somethingelse");
|
||||
var newKek = new FakeMasterKeyProvider("kek-new");
|
||||
var cipher = new AesGcmEnvelopeCipher(actualKek);
|
||||
|
||||
StoredSecret row = cipher.Encrypt(new SecretName("sql/foo"), "hunter2", SecretContentType.Text);
|
||||
|
||||
// Row is wrapped by "kek-old" but the caller claims the old KEK is "kek-somethingelse".
|
||||
var ex = Assert.Throws<SecretDecryptionException>(() => cipher.Rewrap(row, wrongOld, newKek));
|
||||
Assert.Contains("kek-old", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Rewrap_CorrectKekIdButWrongKeyMaterial_FailsClosed()
|
||||
{
|
||||
// Same KEK-id string, different key bytes → the unwrap must fail closed (tag mismatch),
|
||||
// never emitting a garbage DEK.
|
||||
var realOld = new FakeMasterKeyProvider("kek-old");
|
||||
var imposterOld = new FakeMasterKeyProvider("kek-old"); // same id, fresh random bytes
|
||||
var newKek = new FakeMasterKeyProvider("kek-new");
|
||||
var cipher = new AesGcmEnvelopeCipher(realOld);
|
||||
|
||||
StoredSecret row = cipher.Encrypt(new SecretName("sql/foo"), "hunter2", SecretContentType.Text);
|
||||
|
||||
Assert.Throws<SecretDecryptionException>(() => cipher.Rewrap(row, imposterOld, newKek));
|
||||
}
|
||||
|
||||
private static bool Contains(byte[] haystack, byte[] needle)
|
||||
{
|
||||
if (needle.Length == 0 || haystack.Length < needle.Length)
|
||||
|
||||
@@ -48,4 +48,9 @@ public sealed class CountingSecretStore : ISecretStore
|
||||
/// <inheritdoc />
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
|
||||
throw new NotSupportedException();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Crypto;
|
||||
using ZB.MOM.WW.Secrets.Rotation;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
using ZB.MOM.WW.Secrets.Tests.Fakes;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Tests.Rotation;
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end KEK-rotation tests against the REAL SQLite store and REAL AES-256-GCM cipher — the
|
||||
/// combination that actually proves a re-wrapped row decrypts under the new KEK (and no longer
|
||||
/// under the old one).
|
||||
/// </summary>
|
||||
public sealed class KekRotationServiceTests : IAsyncLifetime, IDisposable
|
||||
{
|
||||
private readonly string _dbPath =
|
||||
Path.Combine(Path.GetTempPath(), $"zb-secrets-rotate-{Guid.NewGuid():N}.db");
|
||||
|
||||
private readonly SecretsSqliteConnectionFactory _factory;
|
||||
private readonly SqliteSecretStore _store;
|
||||
|
||||
private readonly FakeMasterKeyProvider _oldKek = new("kek-old");
|
||||
private readonly FakeMasterKeyProvider _newKek = new("kek-new");
|
||||
|
||||
public KekRotationServiceTests()
|
||||
{
|
||||
_factory = new SecretsSqliteConnectionFactory(_dbPath);
|
||||
_store = new SqliteSecretStore(_factory);
|
||||
}
|
||||
|
||||
public async Task InitializeAsync() =>
|
||||
await new SqliteSecretsStoreMigrator(_factory).MigrateAsync(CancellationToken.None);
|
||||
|
||||
public Task DisposeAsync() => Task.CompletedTask;
|
||||
|
||||
// Seeds a secret sealed under a given KEK provider (defaults to the old KEK).
|
||||
private async Task SeedAsync(string name, string value, IMasterKeyProvider? kek = null)
|
||||
{
|
||||
var cipher = new AesGcmEnvelopeCipher(kek ?? _oldKek);
|
||||
StoredSecret row = cipher.Encrypt(new SecretName(name), value, SecretContentType.Text);
|
||||
await _store.UpsertAsync(row, CancellationToken.None);
|
||||
}
|
||||
|
||||
private KekRotationService Service() => new(_store, new AesGcmEnvelopeCipher(_oldKek));
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_MigratesEveryRow_DecryptsUnderNewKekOnly()
|
||||
{
|
||||
await SeedAsync("sql/a", "value-a");
|
||||
await SeedAsync("sql/b", "value-b");
|
||||
await SeedAsync("ldap/c", "value-c");
|
||||
|
||||
RewrapReport report = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
|
||||
|
||||
Assert.Equal(3, report.Total);
|
||||
Assert.Equal(3, report.Rewrapped);
|
||||
Assert.Equal(0, report.AlreadyCurrent);
|
||||
|
||||
var newCipher = new AesGcmEnvelopeCipher(_newKek);
|
||||
var oldCipher = new AesGcmEnvelopeCipher(_oldKek);
|
||||
foreach ((string name, string value) in new[] { ("sql/a", "value-a"), ("sql/b", "value-b"), ("ldap/c", "value-c") })
|
||||
{
|
||||
StoredSecret row = (await _store.GetAsync(new SecretName(name), CancellationToken.None))!;
|
||||
Assert.Equal("kek-new", row.KekId);
|
||||
// Decrypts cleanly under the NEW KEK...
|
||||
Assert.Equal(value, newCipher.Decrypt(row));
|
||||
// ...and fails closed under the OLD KEK (kek_id no longer matches).
|
||||
Assert.Throws<SecretDecryptionException>(() => oldCipher.Decrypt(row));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_IsIdempotent_SecondRunSkipsEverything()
|
||||
{
|
||||
await SeedAsync("sql/a", "value-a");
|
||||
await SeedAsync("sql/b", "value-b");
|
||||
|
||||
RewrapReport first = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
|
||||
Assert.Equal(2, first.Rewrapped);
|
||||
|
||||
RewrapReport second = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
|
||||
Assert.Equal(2, second.Total);
|
||||
Assert.Equal(0, second.Rewrapped);
|
||||
Assert.Equal(2, second.AlreadyCurrent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_RewrapsTombstonedRows()
|
||||
{
|
||||
await SeedAsync("sql/live", "live");
|
||||
await SeedAsync("sql/dead", "dead");
|
||||
await _store.DeleteAsync(new SecretName("sql/dead"), "carol", CancellationToken.None);
|
||||
|
||||
RewrapReport report = await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
|
||||
|
||||
Assert.Equal(2, report.Rewrapped);
|
||||
StoredSecret dead = (await _store.GetAsync(new SecretName("sql/dead"), CancellationToken.None))!;
|
||||
Assert.Equal("kek-new", dead.KekId);
|
||||
Assert.True(dead.IsDeleted);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_DoesNotBumpRevisionOrUpdatedUtc()
|
||||
{
|
||||
await SeedAsync("sql/a", "value-a");
|
||||
StoredSecret before = (await _store.GetAsync(new SecretName("sql/a"), CancellationToken.None))!;
|
||||
|
||||
await Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None);
|
||||
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("sql/a"), CancellationToken.None))!;
|
||||
Assert.Equal(before.Revision, after.Revision);
|
||||
Assert.Equal(before.UpdatedUtc, after.UpdatedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_IdenticalKekIds_Throws()
|
||||
{
|
||||
await SeedAsync("sql/a", "value-a");
|
||||
var same = new FakeMasterKeyProvider("kek-old");
|
||||
|
||||
await Assert.ThrowsAsync<ArgumentException>(
|
||||
() => Service().RewrapAllAsync(_oldKek, same, CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RewrapAll_RowOnUnknownKek_Aborts_ButKeepsPriorProgress()
|
||||
{
|
||||
// Names are scanned in ORDER BY name: "a-old" and "b-old" migrate, then "z-third" (a KEK that
|
||||
// is neither old nor new) aborts the pass.
|
||||
await SeedAsync("a-old", "va");
|
||||
await SeedAsync("b-old", "vb");
|
||||
await SeedAsync("z-third", "vz", new FakeMasterKeyProvider("kek-third"));
|
||||
|
||||
await Assert.ThrowsAsync<SecretDecryptionException>(
|
||||
() => Service().RewrapAllAsync(_oldKek, _newKek, CancellationToken.None));
|
||||
|
||||
// The two old-KEK rows scanned before the anomaly are already persisted on the new KEK,
|
||||
// so a re-run (after removing the anomaly) resumes cleanly.
|
||||
Assert.Equal("kek-new", (await _store.GetAsync(new SecretName("a-old"), CancellationToken.None))!.KekId);
|
||||
Assert.Equal("kek-new", (await _store.GetAsync(new SecretName("b-old"), CancellationToken.None))!.KekId);
|
||||
Assert.Equal("kek-third", (await _store.GetAsync(new SecretName("z-third"), CancellationToken.None))!.KekId);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
SqliteConnection.ClearAllPools();
|
||||
foreach (string path in new[] { _dbPath, _dbPath + "-wal", _dbPath + "-shm" })
|
||||
{
|
||||
try
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
// Best-effort temp cleanup.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -204,6 +204,101 @@ public sealed class SqliteSecretStoreTests : IAsyncLifetime, IDisposable
|
||||
Assert.Equal(new byte[] { 6, 6, 6 }, afterStale.Ciphertext);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyRewrap_ChangesOnlyWrapEnvelopeAndKekId_PreservingRevisionBodyAndTimestamps()
|
||||
{
|
||||
DateTimeOffset created = new(2026, 1, 1, 0, 0, 0, TimeSpan.Zero);
|
||||
StoredSecret seed = MakeSecret("app/rewrap", ciphertext: [1, 2, 3], createdUtc: created);
|
||||
await _store.UpsertAsync(seed, CancellationToken.None);
|
||||
// Bump the revision once so we can prove ApplyRewrap does NOT change it.
|
||||
await _store.UpsertAsync(MakeSecret("app/rewrap", ciphertext: [1, 2, 3]), CancellationToken.None);
|
||||
|
||||
StoredSecret before = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!;
|
||||
Assert.Equal(1, before.Revision);
|
||||
|
||||
// Re-wrap: only the four wrap-related fields differ; everything else is carried from `before`.
|
||||
StoredSecret rewrapped = before with
|
||||
{
|
||||
WrappedDek = [90, 91, 92],
|
||||
WrapNonce = [93, 94, 95],
|
||||
WrapTag = [96, 97, 98],
|
||||
KekId = "kek-2",
|
||||
};
|
||||
bool applied = await _store.ApplyRewrapAsync(rewrapped, before.WrappedDek, CancellationToken.None);
|
||||
Assert.True(applied);
|
||||
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!;
|
||||
// Wrap envelope + kek_id changed...
|
||||
Assert.Equal(new byte[] { 90, 91, 92 }, after.WrappedDek);
|
||||
Assert.Equal(new byte[] { 93, 94, 95 }, after.WrapNonce);
|
||||
Assert.Equal(new byte[] { 96, 97, 98 }, after.WrapTag);
|
||||
Assert.Equal("kek-2", after.KekId);
|
||||
// ...but revision, updated timestamp, the sealed body, and audit stamps are untouched.
|
||||
Assert.Equal(before.Revision, after.Revision);
|
||||
Assert.Equal(before.UpdatedUtc, after.UpdatedUtc);
|
||||
Assert.Equal(before.CreatedUtc, after.CreatedUtc);
|
||||
Assert.Equal(before.UpdatedBy, after.UpdatedBy);
|
||||
Assert.Equal(new byte[] { 1, 2, 3 }, after.Ciphertext);
|
||||
Assert.Equal(before.Nonce, after.Nonce);
|
||||
Assert.Equal(before.Tag, after.Tag);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyRewrap_RewrapsTombstonedRows()
|
||||
{
|
||||
await _store.UpsertAsync(MakeSecret("app/dead"), CancellationToken.None);
|
||||
await _store.DeleteAsync(new SecretName("app/dead"), "carol", CancellationToken.None);
|
||||
|
||||
StoredSecret tomb = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!;
|
||||
Assert.True(tomb.IsDeleted);
|
||||
|
||||
bool applied = await _store.ApplyRewrapAsync(
|
||||
tomb with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" },
|
||||
tomb.WrappedDek,
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.True(applied);
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!;
|
||||
Assert.Equal("kek-2", after.KekId);
|
||||
// The tombstone survives the rewrap.
|
||||
Assert.True(after.IsDeleted);
|
||||
Assert.Equal(tomb.Revision, after.Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyRewrap_ReturnsFalse_WhenRowAbsent()
|
||||
{
|
||||
StoredSecret ghost = MakeSecret("never-existed") with { KekId = "kek-2" };
|
||||
bool applied = await _store.ApplyRewrapAsync(ghost, ghost.WrappedDek, CancellationToken.None);
|
||||
Assert.False(applied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyRewrap_ReturnsFalse_WhenCurrentWrapChanged_UnderConcurrentWrite()
|
||||
{
|
||||
StoredSecret original = MakeSecret("app/cas", ciphertext: [1, 2, 3]);
|
||||
await _store.UpsertAsync(original, CancellationToken.None);
|
||||
StoredSecret before = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!;
|
||||
|
||||
// Simulate a concurrent set/rotate: a fresh write changes the wrapped DEK out from under us.
|
||||
await _store.UpsertAsync(
|
||||
MakeSecret("app/cas", ciphertext: [9, 9, 9]) with { WrappedDek = [77, 78, 79] },
|
||||
CancellationToken.None);
|
||||
|
||||
// A rewrap that still expects the ORIGINAL wrap must NOT apply (0 rows) — it would otherwise
|
||||
// pair a stale re-wrap with the newer body and corrupt the row.
|
||||
bool applied = await _store.ApplyRewrapAsync(
|
||||
before with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" },
|
||||
before.WrappedDek,
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.False(applied);
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!;
|
||||
// The concurrent write's wrap survived untouched; no stale re-wrap was applied.
|
||||
Assert.Equal(new byte[] { 77, 78, 79 }, after.WrappedDek);
|
||||
Assert.Equal("kek-1", after.KekId);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
// Drop pooled connections so the WAL/-shm/-wal sidecars release before we delete.
|
||||
|
||||
@@ -42,4 +42,8 @@ public sealed class FakeCipher : ISecretCipher
|
||||
|
||||
return Encoding.UTF8.GetString(secret.Ciphertext);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public StoredSecret Rewrap(StoredSecret secret, IMasterKeyProvider oldKek, IMasterKeyProvider newKek)
|
||||
=> secret with { KekId = newKek.KekId };
|
||||
}
|
||||
|
||||
@@ -88,4 +88,9 @@ public sealed class RecordingSecretStore : ISecretStore
|
||||
/// <inheritdoc />
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
|
||||
=> throw new NotSupportedException();
|
||||
}
|
||||
|
||||
@@ -124,5 +124,9 @@ public class SecretsListTests : TestContext
|
||||
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
|
||||
=> throw new NotSupportedException();
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user