feat(secrets): build G-8 KEK rotation (RewrapAll); design+plan G-7 clustered replication
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3: - ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only (bodies never re-encrypted; revision/timestamps preserved -> invisible to cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed wraps; DEK zeroed on all paths. - ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek): updates only the 4 wrap columns + kek_id, compare-and-swap on the current wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a review-caught TOCTOU). - KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry, fail-closed on unknown/identical KEK. - `secret rewrap-all` CLI verb: key material only via env-var name / file path, JSON counts report; README section + operator runbook. Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed). G-7 (clustered replication) — designed + planned, no code: - Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc + executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*. Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,120 @@
|
||||
# Operator runbook — rotating the master KEK (`ZB.MOM.WW.Secrets`)
|
||||
|
||||
**Audience:** operators rotating the key-encryption key (KEK) that protects a secrets store.
|
||||
**Primitive:** `secret rewrap-all` (CLI) → `KekRotationService.RewrapAllAsync` → per-row
|
||||
`ISecretCipher.Rewrap` + `ISecretStore.ApplyRewrapAsync`.
|
||||
**Status:** G-8 (KEK-rotation `RewrapAll` + runbook). Companion to the deferred G-7 clustered
|
||||
replication (`ZB.MOM.WW.Secrets.Akka`).
|
||||
|
||||
---
|
||||
|
||||
## What rotation does (and does not) touch
|
||||
|
||||
Each secret is stored as a body sealed under a fresh per-secret **data-encryption key (DEK)**;
|
||||
the DEK is then **wrapped** by the master **KEK**. Rotation **re-wraps the DEK only**:
|
||||
|
||||
- **Changed:** `wrapped_dek`, `wrap_nonce`, `wrap_tag`, `kek_id` — the KEK-wrap envelope.
|
||||
- **Untouched:** the sealed body (`ciphertext`/`nonce`/`tag`), `revision`, `updated_utc`,
|
||||
`created_*`/`updated_by`, the tombstone flag. Plaintext is never decrypted to disk or logged.
|
||||
|
||||
Because `revision`/`updated_utc` are preserved, a re-wrap is **invisible to cluster
|
||||
last-writer-wins** anti-entropy — so it never churns replication, and it must be run **per
|
||||
independent store** (see step 5).
|
||||
|
||||
## Properties you can rely on
|
||||
|
||||
- **Idempotent / resumable.** A row already on the new KEK is skipped (`alreadyCurrent`). If a
|
||||
run is interrupted, just run it again — it resumes from where it stopped.
|
||||
- **Fail-closed.** A row wrapped by a KEK that is *neither* the old nor the new one **aborts**
|
||||
the pass (rows already re-wrapped stay persisted). A wrong old-key or tampered wrap aborts with
|
||||
`SecretDecryptionException`. An identical old/new KEK id is rejected up front.
|
||||
- **No plaintext exposure.** The report is counts only: `{total, rewrapped, alreadyCurrent}`.
|
||||
|
||||
## KEK identity — the one thing to get right
|
||||
|
||||
The old KEK you supply **must resolve to the same `kek_id` that is stored on the rows.** The
|
||||
provider derives `kek_id` as `sha256:<12 hex>` of the key bytes **unless** an explicit
|
||||
`Secrets:MasterKey:KekId` was configured.
|
||||
|
||||
- **Derived id (default):** supplying the same old key **bytes** reproduces the same id — nothing
|
||||
extra to pass.
|
||||
- **Explicit id:** the app set `Secrets:MasterKey:KekId`. Pass the matching id with
|
||||
`--old-key-id <id>` (and `--new-key-id <id>` if the new KEK also uses an explicit id).
|
||||
|
||||
A mismatch fails closed (`Cannot rewrap … not the supplied old KEK`) — it never corrupts a row.
|
||||
|
||||
> **The new KEK id MUST differ from the old one.** With derived ids this is automatic (new key
|
||||
> bytes → new `sha256:` id). With **explicit** ids you must assign a new `--new-key-id`; reusing the
|
||||
> old id for new key material is rejected up front (`old and new KEK identifiers are identical;
|
||||
> there is nothing to rotate`) because `kek_id` is the only per-row rotation discriminator.
|
||||
|
||||
## Key material handling
|
||||
|
||||
- Key **values are never passed as command-line arguments** (they would be echoed/logged). Supply
|
||||
the old (and optionally new) key by **env-var NAME** (`--old-key-env VAR`) or **file PATH**
|
||||
(`--old-key-file /path`). The value lives only in that env var / file.
|
||||
- Never print `ZB_SECRETS_MASTER_KEY` or the old key to a terminal, log, or ticket.
|
||||
- The new KEK **defaults to the app's configured `Secrets:MasterKey` provider**, so you usually
|
||||
only supply the *old* key.
|
||||
|
||||
---
|
||||
|
||||
## Procedure
|
||||
|
||||
Rotation is an offline, store-local operation. Do it once per store.
|
||||
|
||||
**1. Back up the store.** Copy the SQLite file (or snapshot the SQL-Server DB). This is the
|
||||
undo — if anything is wrong, restore and retry.
|
||||
|
||||
**2. Quiesce resolve traffic.** Stop (or drain) the app(s) reading this store. Rotation preserves
|
||||
`updated_utc`/`revision`, so a running node on the *old* KEK would keep working until you flip its
|
||||
key — but running rotation against a store that other writers are mutating risks racing a fresh
|
||||
write onto the old KEK. Simplest correct posture: app down.
|
||||
|
||||
**3. Stage the new KEK.** Generate a fresh 32-byte key, base64-encode it, and place it where the
|
||||
CLI (and, after cutover, the app) will read it — e.g. `ZB_SECRETS_MASTER_KEY` for the Environment
|
||||
provider, or the mounted key file for the File provider.
|
||||
|
||||
**4. Run `rewrap-all`.** Point the CLI at the store's `appsettings.json` (so `Secrets:SqlitePath`
|
||||
+ the new `Secrets:MasterKey` resolve), and supply the old key:
|
||||
|
||||
```bash
|
||||
# Old key delivered via a throwaway env var (name only on the command line):
|
||||
ZB_SECRETS_OLD_KEY='<old-base64>' \
|
||||
secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
|
||||
# Explicit-id deployment:
|
||||
# secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --old-key-id sha256:abc… --new-key-id sha256:def…
|
||||
# Override the new key explicitly instead of the configured provider:
|
||||
# secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --new-key-file /run/secrets/new_kek
|
||||
```
|
||||
|
||||
Expected: `{"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}` and exit 0. On a
|
||||
re-run everything reports `alreadyCurrent` and `rewrapped:0`.
|
||||
|
||||
**5. For clustered / multi-store deployments, repeat per store.**
|
||||
|
||||
- **Shared SQL-Server store (the G-7 primary path):** one store → run rotation **once**.
|
||||
- **Per-node SQLite on a shared KEK:** each node has its own store file → run rotation **on each
|
||||
node** (all with the same old→new keys). Re-wrap does not replicate (revision preserved), so
|
||||
every store must be migrated independently.
|
||||
|
||||
**6. Cut the app over to the new KEK.** Ensure `Secrets:MasterKey` (and every node's KEK source)
|
||||
now resolves the **new** key, then start the app(s). Confirm a resolve succeeds
|
||||
(e.g. `secret get <known-name>` or an app readiness probe that exercises a `${secret:}` value).
|
||||
|
||||
**7. Destroy the old key.** Remove the old key material from env/files/secret managers once every
|
||||
store is confirmed on the new KEK and resolves are healthy.
|
||||
|
||||
## Rollback
|
||||
|
||||
If step 6/7 surfaces a problem, restore the pre-rotation store backup (step 1) and revert the app
|
||||
to the old KEK. Because bodies were never re-encrypted, the backup is a complete, consistent undo.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Action |
|
||||
|---|---|---|
|
||||
| `error: … old and new KEK identifiers are identical` | New KEK resolves to the same id as the old | You supplied the wrong new key, or forgot to stage it. Verify `Secrets:MasterKey`. |
|
||||
| `error: Secret '…' is wrapped by KEK 'X', which is neither the old nor the new KEK` | A row is on a third KEK (a prior partial/aborted rotation, or a mis-set id) | Investigate that row (`secret list` shows each row's `kekId`); re-point old/new keys or fix the row, then re-run (completed rows are skipped). |
|
||||
| `error: … DEK could not be unwrapped under the supplied old KEK` | Right `kek_id` but wrong key **bytes**, or a tampered row | Confirm you staged the correct old key bytes. If the row is genuinely tampered, restore from backup. |
|
||||
| Post-cutover resolves fail closed (`Row wrapped by unknown KEK`) | App still on the old KEK, or a store/node was missed in step 5 | Ensure every node's `Secrets:MasterKey` resolves the new key and every store was rewrapped. |
|
||||
Reference in New Issue
Block a user