Joseph Doherty
123 lines
7.0 KiB
Markdown
123 lines
7.0 KiB
Markdown
# Operator runbook — rotating the master KEK (`ZB.MOM.WW.Secrets`)
|
|
|
|
**Audience:** operators rotating the key-encryption key (KEK) that protects a secrets store.
|
|
**Primitive:** `secret rewrap-all` (CLI) → `KekRotationService.RewrapAllAsync` → per-row
|
|
`ISecretCipher.Rewrap` + `ISecretStore.ApplyRewrapAsync`.
|
|
**Status:** G-8 (KEK-rotation `RewrapAll` + runbook). Companion to the G-7 clustered replication,
|
|
which was **BUILT 2026-07-18** (lib `0.2.0`, both options: `ZB.MOM.WW.Secrets.Replicator.SqlServer`
|
|
and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`; see `clustered-secrets.md`) — the earlier "deferred"
|
|
wording here predated that build.
|
|
|
|
---
|
|
|
|
## What rotation does (and does not) touch
|
|
|
|
Each secret is stored as a body sealed under a fresh per-secret **data-encryption key (DEK)**;
|
|
the DEK is then **wrapped** by the master **KEK**. Rotation **re-wraps the DEK only**:
|
|
|
|
- **Changed:** `wrapped_dek`, `wrap_nonce`, `wrap_tag`, `kek_id` — the KEK-wrap envelope.
|
|
- **Untouched:** the sealed body (`ciphertext`/`nonce`/`tag`), `revision`, `updated_utc`,
|
|
`created_*`/`updated_by`, the tombstone flag. Plaintext is never decrypted to disk or logged.
|
|
|
|
Because `revision`/`updated_utc` are preserved, a re-wrap is **invisible to cluster
|
|
last-writer-wins** anti-entropy — so it never churns replication, and it must be run **per
|
|
independent store** (see step 5).
|
|
|
|
## Properties you can rely on
|
|
|
|
- **Idempotent / resumable.** A row already on the new KEK is skipped (`alreadyCurrent`). If a
|
|
run is interrupted, just run it again — it resumes from where it stopped.
|
|
- **Fail-closed.** A row wrapped by a KEK that is *neither* the old nor the new one **aborts**
|
|
the pass (rows already re-wrapped stay persisted). A wrong old-key or tampered wrap aborts with
|
|
`SecretDecryptionException`. An identical old/new KEK id is rejected up front.
|
|
- **No plaintext exposure.** The report is counts only: `{total, rewrapped, alreadyCurrent}`.
|
|
|
|
## KEK identity — the one thing to get right
|
|
|
|
The old KEK you supply **must resolve to the same `kek_id` that is stored on the rows.** The
|
|
provider derives `kek_id` as `sha256:<12 hex>` of the key bytes **unless** an explicit
|
|
`Secrets:MasterKey:KekId` was configured.
|
|
|
|
- **Derived id (default):** supplying the same old key **bytes** reproduces the same id — nothing
|
|
extra to pass.
|
|
- **Explicit id:** the app set `Secrets:MasterKey:KekId`. Pass the matching id with
|
|
`--old-key-id <id>` (and `--new-key-id <id>` if the new KEK also uses an explicit id).
|
|
|
|
A mismatch fails closed (`Cannot rewrap … not the supplied old KEK`) — it never corrupts a row.
|
|
|
|
> **The new KEK id MUST differ from the old one.** With derived ids this is automatic (new key
|
|
> bytes → new `sha256:` id). With **explicit** ids you must assign a new `--new-key-id`; reusing the
|
|
> old id for new key material is rejected up front (`old and new KEK identifiers are identical;
|
|
> there is nothing to rotate`) because `kek_id` is the only per-row rotation discriminator.
|
|
|
|
## Key material handling
|
|
|
|
- Key **values are never passed as command-line arguments** (they would be echoed/logged). Supply
|
|
the old (and optionally new) key by **env-var NAME** (`--old-key-env VAR`) or **file PATH**
|
|
(`--old-key-file /path`). The value lives only in that env var / file.
|
|
- Never print `ZB_SECRETS_MASTER_KEY` or the old key to a terminal, log, or ticket.
|
|
- The new KEK **defaults to the app's configured `Secrets:MasterKey` provider**, so you usually
|
|
only supply the *old* key.
|
|
|
|
---
|
|
|
|
## Procedure
|
|
|
|
Rotation is an offline, store-local operation. Do it once per store.
|
|
|
|
**1. Back up the store.** Copy the SQLite file (or snapshot the SQL-Server DB). This is the
|
|
undo — if anything is wrong, restore and retry.
|
|
|
|
**2. Quiesce resolve traffic.** Stop (or drain) the app(s) reading this store. Rotation preserves
|
|
`updated_utc`/`revision`, so a running node on the *old* KEK would keep working until you flip its
|
|
key — but running rotation against a store that other writers are mutating risks racing a fresh
|
|
write onto the old KEK. Simplest correct posture: app down.
|
|
|
|
**3. Stage the new KEK.** Generate a fresh 32-byte key, base64-encode it, and place it where the
|
|
CLI (and, after cutover, the app) will read it — e.g. `ZB_SECRETS_MASTER_KEY` for the Environment
|
|
provider, or the mounted key file for the File provider.
|
|
|
|
**4. Run `rewrap-all`.** Point the CLI at the store's `appsettings.json` (so `Secrets:SqlitePath`
|
|
+ the new `Secrets:MasterKey` resolve), and supply the old key:
|
|
|
|
```bash
|
|
# Old key delivered via a throwaway env var (name only on the command line):
|
|
ZB_SECRETS_OLD_KEY='<old-base64>' \
|
|
secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
|
|
# Explicit-id deployment:
|
|
# secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --old-key-id sha256:abc… --new-key-id sha256:def…
|
|
# Override the new key explicitly instead of the configured provider:
|
|
# secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --new-key-file /run/secrets/new_kek
|
|
```
|
|
|
|
Expected: `{"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}` and exit 0. On a
|
|
re-run everything reports `alreadyCurrent` and `rewrapped:0`.
|
|
|
|
**5. For clustered / multi-store deployments, repeat per store.**
|
|
|
|
- **Shared SQL-Server store (the G-7 primary path):** one store → run rotation **once**.
|
|
- **Per-node SQLite on a shared KEK:** each node has its own store file → run rotation **on each
|
|
node** (all with the same old→new keys). Re-wrap does not replicate (revision preserved), so
|
|
every store must be migrated independently.
|
|
|
|
**6. Cut the app over to the new KEK.** Ensure `Secrets:MasterKey` (and every node's KEK source)
|
|
now resolves the **new** key, then start the app(s). Confirm a resolve succeeds
|
|
(e.g. `secret get <known-name>` or an app readiness probe that exercises a `${secret:}` value).
|
|
|
|
**7. Destroy the old key.** Remove the old key material from env/files/secret managers once every
|
|
store is confirmed on the new KEK and resolves are healthy.
|
|
|
|
## Rollback
|
|
|
|
If step 6/7 surfaces a problem, restore the pre-rotation store backup (step 1) and revert the app
|
|
to the old KEK. Because bodies were never re-encrypted, the backup is a complete, consistent undo.
|
|
|
|
## Troubleshooting
|
|
|
|
| Symptom | Cause | Action |
|
|
|---|---|---|
|
|
| `error: … old and new KEK identifiers are identical` | New KEK resolves to the same id as the old | You supplied the wrong new key, or forgot to stage it. Verify `Secrets:MasterKey`. |
|
|
| `error: Secret '…' is wrapped by KEK 'X', which is neither the old nor the new KEK` | A row is on a third KEK (a prior partial/aborted rotation, or a mis-set id) | Investigate that row (`secret list` shows each row's `kekId`); re-point old/new keys or fix the row, then re-run (completed rows are skipped). |
|
|
| `error: … DEK could not be unwrapped under the supplied old KEK` | Right `kek_id` but wrong key **bytes**, or a tampered row | Confirm you staged the correct old key bytes. If the row is genuinely tampered, restore from backup. |
|
|
| Post-cutover resolves fail closed (`Row wrapped by unknown KEK`) | App still on the old KEK, or a store/node was missed in step 5 | Ensure every node's `Secrets:MasterKey` resolves the new key and every store was rewrapped. |
|