# Operator runbook — rotating the master KEK (`ZB.MOM.WW.Secrets`)

**Audience:** operators rotating the key-encryption key (KEK) that protects a secrets store.
**Primitive:** `secret rewrap-all` (CLI) → `KekRotationService.RewrapAllAsync` → per-row
`ISecretCipher.Rewrap` + `ISecretStore.ApplyRewrapAsync`.
**Status:** G-8 (KEK-rotation `RewrapAll` + runbook). Companion to the deferred G-7 clustered
replication (`ZB.MOM.WW.Secrets.Akka`).

---

## What rotation does (and does not) touch

Each secret is stored as a body sealed under a fresh per-secret **data-encryption key (DEK)**;
the DEK is then **wrapped** by the master **KEK**. Rotation **re-wraps the DEK only**:

- **Changed:** `wrapped_dek`, `wrap_nonce`, `wrap_tag`, `kek_id` — the KEK-wrap envelope.
- **Untouched:** the sealed body (`ciphertext`/`nonce`/`tag`), `revision`, `updated_utc`,
  `created_*`/`updated_by`, the tombstone flag. Plaintext is never decrypted to disk or logged.

Because `revision`/`updated_utc` are preserved, a re-wrap is **invisible to cluster
last-writer-wins** anti-entropy — so it never churns replication, and it must be run **per
independent store** (see step 5).

## Properties you can rely on

- **Idempotent / resumable.** A row already on the new KEK is skipped (`alreadyCurrent`). If a
  run is interrupted, just run it again — it resumes from where it stopped.
- **Fail-closed.** A row wrapped by a KEK that is *neither* the old nor the new one **aborts**
  the pass (rows already re-wrapped stay persisted). A wrong old-key or tampered wrap aborts with
  `SecretDecryptionException`. An identical old/new KEK id is rejected up front.
- **No plaintext exposure.** The report is counts only: `{total, rewrapped, alreadyCurrent}`.

## KEK identity — the one thing to get right

The old KEK you supply **must resolve to the same `kek_id` that is stored on the rows.** The
provider derives `kek_id` as `sha256:<12 hex>` of the key bytes **unless** an explicit
`Secrets:MasterKey:KekId` was configured.

- **Derived id (default):** supplying the same old key **bytes** reproduces the same id — nothing
  extra to pass.
- **Explicit id:** the app set `Secrets:MasterKey:KekId`. Pass the matching id with
  `--old-key-id <id>` (and `--new-key-id <id>` if the new KEK also uses an explicit id).

A mismatch fails closed (`Cannot rewrap … not the supplied old KEK`) — it never corrupts a row.

> **The new KEK id MUST differ from the old one.** With derived ids this is automatic (new key
> bytes → new `sha256:` id). With **explicit** ids you must assign a new `--new-key-id`; reusing the
> old id for new key material is rejected up front (`old and new KEK identifiers are identical;
> there is nothing to rotate`) because `kek_id` is the only per-row rotation discriminator.

## Key material handling

- Key **values are never passed as command-line arguments** (they would be echoed/logged). Supply
  the old (and optionally new) key by **env-var NAME** (`--old-key-env VAR`) or **file PATH**
  (`--old-key-file /path`). The value lives only in that env var / file.
- Never print `ZB_SECRETS_MASTER_KEY` or the old key to a terminal, log, or ticket.
- The new KEK **defaults to the app's configured `Secrets:MasterKey` provider**, so you usually
  only supply the *old* key.

---

## Procedure

Rotation is an offline, store-local operation. Do it once per store.

**1. Back up the store.** Copy the SQLite file (or snapshot the SQL-Server DB). This is the
undo — if anything is wrong, restore and retry.

**2. Quiesce resolve traffic.** Stop (or drain) the app(s) reading this store. Rotation preserves
`updated_utc`/`revision`, so a running node on the *old* KEK would keep working until you flip its
key — but running rotation against a store that other writers are mutating risks racing a fresh
write onto the old KEK. Simplest correct posture: app down.

**3. Stage the new KEK.** Generate a fresh 32-byte key, base64-encode it, and place it where the
CLI (and, after cutover, the app) will read it — e.g. `ZB_SECRETS_MASTER_KEY` for the Environment
provider, or the mounted key file for the File provider.

**4. Run `rewrap-all`.** Point the CLI at the store's `appsettings.json` (so `Secrets:SqlitePath`
+ the new `Secrets:MasterKey` resolve), and supply the old key:

```bash
# Old key delivered via a throwaway env var (name only on the command line):
ZB_SECRETS_OLD_KEY='<old-base64>' \
  secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
# Explicit-id deployment:
#   secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --old-key-id sha256:abc… --new-key-id sha256:def…
# Override the new key explicitly instead of the configured provider:
#   secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --new-key-file /run/secrets/new_kek
```

Expected: `{"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}` and exit 0. On a
re-run everything reports `alreadyCurrent` and `rewrapped:0`.

**5. For clustered / multi-store deployments, repeat per store.**

- **Shared SQL-Server store (the G-7 primary path):** one store → run rotation **once**.
- **Per-node SQLite on a shared KEK:** each node has its own store file → run rotation **on each
  node** (all with the same old→new keys). Re-wrap does not replicate (revision preserved), so
  every store must be migrated independently.

**6. Cut the app over to the new KEK.** Ensure `Secrets:MasterKey` (and every node's KEK source)
now resolves the **new** key, then start the app(s). Confirm a resolve succeeds
(e.g. `secret get <known-name>` or an app readiness probe that exercises a `${secret:}` value).

**7. Destroy the old key.** Remove the old key material from env/files/secret managers once every
store is confirmed on the new KEK and resolves are healthy.

## Rollback

If step 6/7 surfaces a problem, restore the pre-rotation store backup (step 1) and revert the app
to the old KEK. Because bodies were never re-encrypted, the backup is a complete, consistent undo.

## Troubleshooting

| Symptom | Cause | Action |
|---|---|---|
| `error: … old and new KEK identifiers are identical` | New KEK resolves to the same id as the old | You supplied the wrong new key, or forgot to stage it. Verify `Secrets:MasterKey`. |
| `error: Secret '…' is wrapped by KEK 'X', which is neither the old nor the new KEK` | A row is on a third KEK (a prior partial/aborted rotation, or a mis-set id) | Investigate that row (`secret list` shows each row's `kekId`); re-point old/new keys or fix the row, then re-run (completed rows are skipped). |
| `error: … DEK could not be unwrapped under the supplied old KEK` | Right `kek_id` but wrong key **bytes**, or a tampered row | Confirm you staged the correct old key bytes. If the row is genuinely tampered, restore from backup. |
| Post-cutover resolves fail closed (`Row wrapped by unknown KEK`) | App still on the old KEK, or a store/node was missed in step 5 | Ensure every node's `Secrets:MasterKey` resolves the new key and every store was rewrapped. |