# Operator runbook — the `secret` interactive console (`ZB.MOM.WW.Secrets`)

**Audience:** operators running the `secret` CLI at a terminal against a deployment's store.
**Primitive:** a bare `secret` invocation (no verb) on a real TTY → `InteractiveEntry` →
`InteractiveShell` → pluggable `IInteractiveFlow`s over an opened `SecretsSession`.
**Status:** ships alongside the headless verbs (`set` / `get` / `list` / `rm` / `rotate` /
`rewrap-all`), which are unaffected. Companion to
[`kek-rotation.md`](kek-rotation.md) (the `rewrap-all` primitive the KEK-doctor remedy uses
under the hood) and [`clustered-secrets.md`](clustered-secrets.md) (replication topologies the
console does not manage).

---

## What the console is (and is not)

Every flow in the console is a thin UI wrapper over the same store/cipher primitives the
headless CLI and the app itself use — `ISecretStore`, `ISecretCipher`, `KekRotationService`,
`SecretReferenceExpander`'s token pattern. Nothing about the store, the wrap scheme, or the
crypto changes. What the console adds is:

- **Target discovery** — pick a deployment's store the same way its app would, instead of
  hand-assembling a connection string.
- **A guided menu** in place of memorized verb syntax, with confirm-gates and masked prompts
  where a headless verb would take a positional argument.
- **Two flows with no headless equivalent**: the reference audit & seed (deployment setup) and
  the KEK doctor (lockout triage + guided remedy).

It is **not** a login surface, a replication controller, or a replacement for `rewrap-all` —
it drives that primitive for one convenience case (rewrap onto the *session's* KEK) and defers
to the headless verb (and [`kek-rotation.md`](kek-rotation.md)) for a general old→new rotation.

## Launching

```bash
secret
```

- **Real terminal, no args:** opens the console. The check is exact —
  `args.Length == 0 && !Console.IsInputRedirected && !Console.IsOutputRedirected` — so any
  redirection at all (a pipe, `secret | cat`, a CI runner, a script capturing output) falls
  through to the ordinary headless path instead of hanging on a prompt.
- **Any verb, or a redirected stream:** unchanged headless behavior — `set` / `get` / `list` /
  `rm` / `rotate` / `rewrap-all` dispatch exactly as before; a bare `secret` with a redirected
  stream (or an unrecognized verb) prints usage and exits **2**, same as pre-console builds.
- **Exit codes:** the console itself returns **0** on a clean quit (menu "Quit", or Ctrl-C at
  the target picker or the action menu) and never returns a non-zero code on its own — a
  failed action is caught, rendered as an error panel, and returns you to the menu rather than
  exiting the process. Ctrl-C mid-flow abandons just that action.
- **Ctrl-C behavior:** cooperative cancellation is checked between actions and at the target
  picker; Spectre's blocking prompts cannot observe the token mid-prompt, but the surrounding
  loop always resolves to a clean exit rather than a stack trace.

## Targeting a deployment

The first screen is always the target picker:

1. **Recent targets** — up to 10 entries from `~/.zb-secrets/recent-targets.json`, newest
   first, labeled `<name> — <appsettings path>`. Selecting one re-reads that path (config may
   have changed since last time) and re-stamps it as most-recent.
2. **Enter an appsettings.json path…** — point at any app's base `appsettings.json`.
3. **Manual entry (SQLite db + KEK env var)…** — a raw SQLite file path plus a master-key
   environment variable name (default `ZB_SECRETS_MASTER_KEY`), for a store with no
   accompanying appsettings file. Manual targets are never added to recents (there is no
   appsettings path to remember).

**The honest-store guarantee.** Option 2 composes configuration *exactly the way the target
app would at startup*: `appsettings.json` as the base, an optional `appsettings.<env>.json`
overlay, then environment variables — layered in that order via the same
`ConfigurationBuilder` pattern the app's own `Host.CreateApplicationBuilder` uses — and binds
the `Secrets` section (and, if present, `Secrets:SqlServer`) out of the result. There is no
separate "CLI view" of the config that could drift from what the app actually reads: if the app
resolves its SQLite path or its SQL-Server hub connection string a particular way, the console
resolves it the same way, against the same files, from the same working directory. This is
also why `list`/`get`/`set` here can be trusted as evidence of what the running app sees.

**Store selection mirrors the app's DI wiring**: a configured `Secrets:SqlServer` connection
string wins over local SQLite — *unless* that connection string is itself an unresolved
`${secret:...}` reference (the app resolves that from its own bootstrapped store at startup;
the console has no such bootstrap step), in which case the console falls back to the local
SQLite store and prints a warning. In hub-replication mode, editing the SQL-Server hub through
the console is the correct target — the change converges to every node's local store on the
next sync sweep, rather than racing one node's SQLite against the hub.

**Recents contain no secret material** — only a display name and the appsettings path, ever.
See the security notes below.

## Degraded sessions and the KEK prompt

Opening a session never fails outright on a bad or missing KEK. If the configured master key
cannot be resolved — unset/empty env var, missing key file, a malformed `MasterKey:Source`,
DPAPI off Windows — the session opens anyway, **degraded**: metadata-only operations (List,
Delete — tombstoning touches no ciphertext) work immediately, and the header banner shows
`KEK UNAVAILABLE (degraded)` in red instead of `KEK OK` in green.

The moment the operator picks a flow that needs plaintext access (Set/rotate, Get, reference
audit & seed, KEK doctor, bundle export/import — anything whose `RequiresKek` is true), the
shell intercepts the dispatch and prompts:

- **Paste base64 key** — a masked `TextPrompt` (nothing echoed) fed straight to an in-memory
  provider; nothing is written to disk.
- **Key file path** — points at a file the configured `File` provider can read.

A successful upgrade re-opens the session with that key and continues into the flow the
operator originally chose. A key that fails to resolve, or resolves but still doesn't unlock
the session (wrong key), reports the failure and returns to the menu — the degraded session is
never silently discarded.

## Deployment seeding walk-through: Reference audit & seed

This is the flow to run **the first time you point the console at a fresh deployment**, or
whenever an app fails to start because a `${secret:NAME}` reference can't resolve.

1. The auditor walks every value in the target's *composed* configuration (same honest-store
   composition as targeting, above) with the identical `\$\{secret:([^}]+)\}` token pattern
   `SecretReferenceExpander` uses at app startup — so what you see here is exactly what the app
   would try to resolve, including config keys documented via a `"_comment": "...${secret:x}..."`
   convention, which are skipped the same way the expander skips them.
2. Each distinct referenced name is looked up against the store and classified:
   - **Ok** — exists and decrypts under the session KEK. Nothing to do.
   - **Missing** — no row at all. A fail-closed expansion would throw at app startup.
   - **Tombstoned** — a soft-deleted row. Also unresolvable to the app.
   - **Undecryptable** — the row exists but does not decrypt under the session KEK (wrong or
     rotated key, or damage). *This is often the first sign a KEK doctor pass is needed.*
   - **InvalidName** — the token text isn't a legal secret name (illegal characters, an
     unfilled placeholder like `REPLACE_ME`). Can't be looked up or seeded — fix the token in
     the config file itself.
3. A color-coded table renders every finding with the config key paths that reference it (so
   you know exactly which setting(s) are affected), then the flow walks each seedable gap
   (Missing / Tombstoned / Undecryptable) in turn, asking to seed it now. Accepting prompts for
   a masked value, a content type, and an optional description, then seals a fresh row under
   the session KEK — **overwriting** an Undecryptable row and **reviving** (un-tombstoning) a
   Tombstoned one.
4. A closing re-audit prints a one-line count-by-status summary so you can confirm the gap
   closed without re-reading the whole table.

Run this again any time after adding a new `${secret:}` reference to a config file, or after
deploying to a fresh environment — it is the fast path from "app won't start, references don't
resolve" to "every reference is Ok."

## Lockout playbook: KEK doctor

Run this when an app (or the console itself) reports decrypt failures it didn't before —
typically after a KEK rotation was only partially applied, a config value pointing at the
wrong key source, or a store copied from another environment.

**Diagnosis** probes *every* row (including tombstones, since they still carry a KEK-wrapped
DEK and block a clean `rewrap-all`) against the session's KEK:

- **Ok** — the row's `kek_id` matches the session KEK and it decrypts cleanly.
- **WrongKek** — the row's `kek_id` differs from the session KEK. Not decrypted (no point —
  the wrap alone tells you it won't work); reported with the row's *own* `kek_id` so you know
  which key to go find. The summary line also lists the distinct **foreign KEK ids** seen
  across all wrong-KEK rows — usually there is exactly one (the store's actual current key),
  which tells you the session simply has the wrong key configured.
- **Corrupt** — the `kek_id` matches (so the wrap envelope claims to be under the right key),
  but the decrypt still fails closed. This means the wrap envelope or the sealed body itself is
  damaged — a **different problem than WrongKek**, and *not* fixable by rewrapping: rewrapping
  only re-wraps a DEK it can first unwrap, and a Corrupt row can't be unwrapped at all.

If every row reports Ok, the doctor says so and there is nothing to remedy — the lockout is
elsewhere (check the app's actual configured KEK source against what you used to open this
session).

**Two remedies**, offered only when something is unhealthy:

- **Rewrap from old KEK** — the non-destructive path, and the *only* correct remedy for
  WrongKek rows. You supply the old KEK (env var, key file, or a masked paste) the affected
  rows are actually wrapped under; the flow shows exactly how many rows will move and from
  which KEK id, then asks for an explicit confirmation (default **No** — nothing happens on
  Enter) before calling the same `KekRotationService.RewrapAllAsync` primitive
  [`kek-rotation.md`](kek-rotation.md) documents for the headless `rewrap-all` verb. Secret
  bodies are never touched — only the DEK wrap moves onto the session KEK. A wrong old key
  surfaces as a decryption failure and aborts (rows already rewrapped before the anomaly stay
  persisted, so a retry after fixing the key resumes cleanly).
- **Old KEK is lost — re-set affected secrets** — the destructive last resort, and the *only*
  remedy for a Corrupt row (which cannot be rewrapped because it can't be unwrapped in the
  first place). For every WrongKek or Corrupt row still affected, the flow offers a masked
  re-set: type a fresh value, and it's sealed fresh under the session KEK, overwriting the
  unrecoverable row. Each row is confirmed individually (default **No**) so you can skip rows
  you'd rather chase down the old key for instead.

A closing re-diagnosis prints a one-line count-by-status summary so you can see the remedy's
effect immediately, in the same session, without re-running the flow.

## Bundles: export / import

Bundles move secret rows **between deployment stores** — cloning a deployment, staging a
recovery, seeding a new environment from a known-good one — without ever touching plaintext.

**Export bundle (ciphertext-only)** writes every live row (or, if you opt in, tombstones too)
to a JSON file: for each row, the six crypto fields (`ciphertext`/`nonce`/`tag` for the body,
`wrapped_dek`/`wrap_nonce`/`wrap_tag` for the DEK wrap) ride as base64, plus the row's
`kek_id`, revision, and timestamps. The bundle also records the exporting session's KEK id
(`SourceKekId`) and a `FormatVersion`. A default filename
(`secrets-bundle-<target>-<yyyyMMdd>.json`) is offered but can be overridden. The bundle is
useless without the source KEK — there is no plaintext anywhere in it — so it is safe to copy,
attach to a ticket, or commit to a secured artifact store, but treat it with the same care as
any other encrypted-at-rest artifact (it does reveal *which* names and content types exist).

**Import bundle** reads and parses the file *before* touching the store or prompting for
anything, so a malformed file or unsupported `FormatVersion` (this build supports **version
1** only) is caught up front. Then, per row:

- **Same KEK as the session** — imported directly.
- **Different KEK (`SourceKekId` ≠ the session's)** — the flow tells you both KEK ids and asks
  for the bundle's source key (paste / env var / key file) so it can re-wrap each row onto the
  session KEK. The supplied key's own derived id is checked against the bundle's declared
  `SourceKekId` *before* anything is imported — a valid-but-wrong key is rejected with both ids
  shown, rather than silently landing every row in the "skipped foreign KEK" bucket. Declining
  to supply a key aborts the whole import (no key means no row can be re-wrapped).
- **Name collision (a row with that name already exists locally)** — resolved by **last-writer-
  wins** (the shared `SecretLastWriterWins` tie-break used by every replication topology:
  later `updated_utc`, then higher revision, wins) unless you opt into **per-conflict
  prompting**, in which case each collision shows both rows' timestamp and revision and asks
  whether to take the bundle's row (default **No** — keep local). A winning row lands via the
  store's replicated-apply path so its own revision/timestamps are preserved verbatim (the same
  path cluster anti-entropy uses); a row you force *against* LWW ordering is instead written as
  a fresh local upsert so the override actually sticks rather than being reverted by the next
  reconciliation.

A closing tally table (Imported / Skipped-older / Skipped-foreign-KEK / Conflicts) reports the
outcome; a non-zero "skipped foreign KEK" count is annotated with the bundle's source KEK id so
a wrong-key paste is diagnosable from the report alone without re-running the import.

## Security notes

- **No login.** The console is a local admin tool: whoever can run it already has direct
  filesystem/DB access to the target store (they could `sqlite3` the file open themselves), so
  the console adds no new trust boundary — it does not check identity, does not audit its own
  actions distinctly from the store-level audit the underlying `ISecretResolver`/store calls
  already perform, and should be restricted the same way you'd restrict shell access to the
  host or the deployment's config files.
- **Values are never echoed.** Every prompt for a secret value or key material uses Spectre's
  masked `TextPrompt(...).Secret()` — nothing typed is drawn to the terminal, and nothing typed
  is written to a log.
- **Values are never cached.** Get (reveal) decrypts and writes the plaintext exactly once, with
  no resolver, no in-memory cache, and no re-use across the session — asking again re-decrypts
  from the row.
- **Recents contain no key material.** `~/.zb-secrets/recent-targets.json` stores only a
  display name, an appsettings path, and a timestamp — never a KEK, a secret value, or a
  connection string. It is safe to inspect or delete.
- **Bundles are ciphertext-only and safe at rest.** A bundle file contains no plaintext under
  any circumstance — export cannot produce one and import cannot be pointed at one that
  "leaks" plaintext through the format; the only way to recover a value from a bundle is to
  import it into a session holding the matching KEK and then explicitly reveal it.

## Troubleshooting

| Symptom | Cause | Action |
|---|---|---|
| Bare `secret` prints usage instead of opening the console | stdin or stdout is redirected (a pipe, a script, CI) | Expected — run it directly at an interactive terminal with no redirection. |
| Header shows `KEK UNAVAILABLE (degraded)` | The target's configured KEK source didn't resolve in this shell (unset env var, missing file, DPAPI off Windows) | List/Delete still work. Pick a KEK-requiring flow to get the upgrade prompt, or fix the env var/file and re-select the target. |
| Reference audit shows `Undecryptable` for a name that used to resolve | The row is wrapped under a KEK the session doesn't have — often a partially-applied rotation | Run the KEK doctor; if it reports `WrongKek`, rewrap from the old KEK. |
| KEK doctor reports `Corrupt`, not `WrongKek` | `kek_id` matches but the wrap/body itself is damaged | Rewrap cannot fix this (nothing to unwrap first) — use "Old KEK is lost" to re-set the row, or restore the store from a backup. |
| Bundle import reports a large `Skipped (foreign KEK)` count | Declined (or gave the wrong) source key when prompted | Re-run the import and supply the bundle's actual source key — the report shows the expected `SourceKekId`. |
| `secret rewrap-all`-style behavior needed across a whole fleet, not just this session's KEK | The KEK doctor's rewrap remedy always targets the *session's* KEK, for lockout recovery on one store | For a general old→new rotation across every store, use the headless `secret rewrap-all` verb per [`kek-rotation.md`](kek-rotation.md). |