scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md

Operator runbook — rotating the master KEK (ZB.MOM.WW.Secrets)

Audience: operators rotating the key-encryption key (KEK) that protects a secrets store. Primitive: secret rewrap-all (CLI) → KekRotationService.RewrapAllAsync → per-row ISecretCipher.Rewrap + ISecretStore.ApplyRewrapAsync. Status: G-8 (KEK-rotation RewrapAll + runbook). Companion to the deferred G-7 clustered replication (ZB.MOM.WW.Secrets.Akka).

---

What rotation does (and does not) touch

Each secret is stored as a body sealed under a fresh per-secret data-encryption key (DEK); the DEK is then wrapped by the master KEK. Rotation re-wraps the DEK only:

• Changed: wrapped_dek, wrap_nonce, wrap_tag, kek_id — the KEK-wrap envelope.
• Untouched: the sealed body (ciphertext/nonce/tag), revision, updated_utc, created_*/updated_by, the tombstone flag. Plaintext is never decrypted to disk or logged.

Because revision/updated_utc are preserved, a re-wrap is invisible to cluster last-writer-wins anti-entropy — so it never churns replication, and it must be run per independent store (see step 5).

Properties you can rely on

• Idempotent / resumable. A row already on the new KEK is skipped (alreadyCurrent). If a run is interrupted, just run it again — it resumes from where it stopped.
• Fail-closed. A row wrapped by a KEK that is neither the old nor the new one aborts the pass (rows already re-wrapped stay persisted). A wrong old-key or tampered wrap aborts with SecretDecryptionException. An identical old/new KEK id is rejected up front.
• No plaintext exposure. The report is counts only: {total, rewrapped, alreadyCurrent}.

KEK identity — the one thing to get right

The old KEK you supply must resolve to the same kek_id that is stored on the rows. The provider derives kek_id as sha256:<12 hex> of the key bytes unless an explicit Secrets:MasterKey:KekId was configured.

• Derived id (default): supplying the same old key bytes reproduces the same id — nothing extra to pass.
• Explicit id: the app set Secrets:MasterKey:KekId. Pass the matching id with --old-key-id <id> (and --new-key-id <id> if the new KEK also uses an explicit id).

A mismatch fails closed (Cannot rewrap … not the supplied old KEK) — it never corrupts a row.

The new KEK id MUST differ from the old one. With derived ids this is automatic (new key bytes → new sha256: id). With explicit ids you must assign a new --new-key-id; reusing the old id for new key material is rejected up front (old and new KEK identifiers are identical; there is nothing to rotate) because kek_id is the only per-row rotation discriminator.

Key material handling

• Key values are never passed as command-line arguments (they would be echoed/logged). Supply the old (and optionally new) key by env-var NAME (--old-key-env VAR) or file PATH (--old-key-file /path). The value lives only in that env var / file.
• Never print ZB_SECRETS_MASTER_KEY or the old key to a terminal, log, or ticket.
• The new KEK defaults to the app's configured Secrets:MasterKey provider, so you usually only supply the old key.

---

Procedure

Rotation is an offline, store-local operation. Do it once per store.

1. Back up the store. Copy the SQLite file (or snapshot the SQL-Server DB). This is the undo — if anything is wrong, restore and retry.

2. Quiesce resolve traffic. Stop (or drain) the app(s) reading this store. Rotation preserves updated_utc/revision, so a running node on the old KEK would keep working until you flip its key — but running rotation against a store that other writers are mutating risks racing a fresh write onto the old KEK. Simplest correct posture: app down.

3. Stage the new KEK. Generate a fresh 32-byte key, base64-encode it, and place it where the CLI (and, after cutover, the app) will read it — e.g. ZB_SECRETS_MASTER_KEY for the Environment provider, or the mounted key file for the File provider.

4. Run rewrap-all. Point the CLI at the store's appsettings.json (so Secrets:SqlitePath

• the new Secrets:MasterKey resolve), and supply the old key:

# Old key delivered via a throwaway env var (name only on the command line):
ZB_SECRETS_OLD_KEY='<old-base64>' \
  secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
# Explicit-id deployment:
#   secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --old-key-id sha256:abc… --new-key-id sha256:def…
# Override the new key explicitly instead of the configured provider:
#   secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY --new-key-file /run/secrets/new_kek

Expected: {"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0} and exit 0. On a re-run everything reports alreadyCurrent and rewrapped:0.

5. For clustered / multi-store deployments, repeat per store.

• Shared SQL-Server store (the G-7 primary path): one store → run rotation once.
• Per-node SQLite on a shared KEK: each node has its own store file → run rotation on each node (all with the same old→new keys). Re-wrap does not replicate (revision preserved), so every store must be migrated independently.

6. Cut the app over to the new KEK. Ensure Secrets:MasterKey (and every node's KEK source) now resolves the new key, then start the app(s). Confirm a resolve succeeds (e.g. secret get <known-name> or an app readiness probe that exercises a ${secret:} value).

7. Destroy the old key. Remove the old key material from env/files/secret managers once every store is confirmed on the new KEK and resolves are healthy.

Rollback

If step 6/7 surfaces a problem, restore the pre-rotation store backup (step 1) and revert the app to the old KEK. Because bodies were never re-encrypted, the backup is a complete, consistent undo.

Troubleshooting

Symptom	Cause	Action
error: … old and new KEK identifiers are identical	New KEK resolves to the same id as the old	You supplied the wrong new key, or forgot to stage it. Verify Secrets:MasterKey.
error: Secret '…' is wrapped by KEK 'X', which is neither the old nor the new KEK	A row is on a third KEK (a prior partial/aborted rotation, or a mis-set id)	Investigate that row (secret list shows each row's kekId); re-point old/new keys or fix the row, then re-run (completed rows are skipped).
error: … DEK could not be unwrapped under the supplied old KEK	Right kek_id but wrong key bytes, or a tampered row	Confirm you staged the correct old key bytes. If the row is genuinely tampered, restore from backup.
Post-cutover resolves fail closed (Row wrapped by unknown KEK)	App still on the old KEK, or a store/node was missed in step 5	Ensure every node's Secrets:MasterKey resolves the new key and every store was rewrapped.