scadaproj/ZB.MOM.WW.Secrets/docs/operations/interactive-console.md

Operator runbook — the secret interactive console (ZB.MOM.WW.Secrets)

Audience: operators running the secret CLI at a terminal against a deployment's store. Primitive: a bare secret invocation (no verb) on a real TTY → InteractiveEntry → InteractiveShell → pluggable IInteractiveFlows over an opened SecretsSession. Status: ships alongside the headless verbs (set / get / list / rm / rotate / rewrap-all), which are unaffected. Companion to kek-rotation.md (the rewrap-all primitive the KEK-doctor remedy uses under the hood) and clustered-secrets.md (replication topologies the console does not manage). For a task-first walkthrough of the two headline jobs (pre-startup seeding of a new deployment; a 2am KEK/decrypt lockout) with illustrative menu transcripts, start with the quickstart; this document is the full reference.

---

What the console is (and is not)

Every flow in the console is a thin UI wrapper over the same store/cipher primitives the headless CLI and the app itself use — ISecretStore, ISecretCipher, KekRotationService, SecretReferenceExpander's token pattern. Nothing about the store, the wrap scheme, or the crypto changes. What the console adds is:

• Target discovery — pick a deployment's store the same way its app would, instead of hand-assembling a connection string.
• A guided menu in place of memorized verb syntax, with confirm-gates and masked prompts where a headless verb would take a positional argument.
• Two flows with no headless equivalent: the reference audit & seed (deployment setup) and the KEK doctor (lockout triage + guided remedy).

It is not a login surface, a replication controller, or a replacement for rewrap-all — it drives that primitive for one convenience case (rewrap onto the session's KEK) and defers to the headless verb (and kek-rotation.md) for a general old→new rotation.

Launching

secret

• Real terminal, no args: opens the console. The check is exact — args.Length == 0 && !Console.IsInputRedirected && !Console.IsOutputRedirected — so any redirection at all (a pipe, secret | cat, a CI runner, a script capturing output) falls through to the ordinary headless path instead of hanging on a prompt.
• Any verb, or a redirected stream: unchanged headless behavior — set / get / list / rm / rotate / rewrap-all dispatch exactly as before; a bare secret with a redirected stream (or an unrecognized verb) prints usage and exits 2, same as pre-console builds.
• Exit codes: the console itself returns 0 on a clean quit (menu "Quit", or Ctrl-C at the target picker or the action menu) and never returns a non-zero code on its own — a failed action is caught, rendered as an error panel, and returns you to the menu rather than exiting the process. Ctrl-C mid-flow abandons just that action.
• Ctrl-C behavior: cooperative cancellation is checked between actions and at the target picker; Spectre's blocking prompts cannot observe the token mid-prompt, but the surrounding loop always resolves to a clean exit rather than a stack trace.

Targeting a deployment

The first screen is always the target picker:

1. Recent targets — up to 10 entries from ~/.zb-secrets/recent-targets.json, newest first, labeled <name> — <appsettings path>. Selecting one re-reads that path (config may have changed since last time) and re-stamps it as most-recent.
2. Enter an appsettings.json path… — point at any app's base appsettings.json.
3. Manual entry (SQLite db + KEK env var)… — a raw SQLite file path plus a master-key environment variable name (default ZB_SECRETS_MASTER_KEY), for a store with no accompanying appsettings file. Manual targets are never added to recents (there is no appsettings path to remember).

The honest-store guarantee. Option 2 composes configuration exactly the way the target app would at startup: appsettings.json as the base, an optional appsettings.<env>.json overlay, then environment variables — layered in that order via the same ConfigurationBuilder pattern the app's own Host.CreateApplicationBuilder uses — and binds the Secrets section (and, if present, Secrets:SqlServer) out of the result. There is no separate "CLI view" of the config that could drift from what the app actually reads: if the app resolves its SQLite path or its SQL-Server hub connection string a particular way, the console resolves it the same way, against the same files, from the same working directory. This is also why list/get/set here can be trusted as evidence of what the running app sees.

Store selection mirrors the app's DI wiring: a configured Secrets:SqlServer connection string wins over local SQLite — unless that connection string is itself an unresolved ${secret:...} reference (the app resolves that from its own bootstrapped store at startup; the console has no such bootstrap step), in which case the console falls back to the local SQLite store and prints a warning. In hub-replication mode, editing the SQL-Server hub through the console is the correct target — the change converges to every node's local store on the next sync sweep, rather than racing one node's SQLite against the hub.

Recents contain no secret material — only a display name and the appsettings path, ever. See the security notes below.

Degraded sessions and the KEK prompt

Opening a session never fails outright on a bad or missing KEK. If the configured master key cannot be resolved — unset/empty env var, missing key file, a malformed MasterKey:Source, DPAPI off Windows — the session opens anyway, degraded: metadata-only operations (List, Delete — tombstoning touches no ciphertext) work immediately, and the header banner shows KEK UNAVAILABLE (degraded) in red instead of KEK OK in green.

The moment the operator picks a flow that needs plaintext access (Set/rotate, Get, reference audit & seed, KEK doctor, bundle export/import — anything whose RequiresKek is true), the shell intercepts the dispatch and prompts:

• Paste base64 key — a masked TextPrompt (nothing echoed) fed straight to an in-memory provider; nothing is written to disk.
• Key file path — points at a file the configured File provider can read.

A successful upgrade re-opens the session with that key and continues into the flow the operator originally chose. A key that fails to resolve, or resolves but still doesn't unlock the session (wrong key), reports the failure and returns to the menu — the degraded session is never silently discarded.

Deployment seeding walk-through: Reference audit & seed

This is the flow to run the first time you point the console at a fresh deployment, or whenever an app fails to start because a ${secret:NAME} reference can't resolve.

1. The auditor walks every value in the target's composed configuration (same honest-store composition as targeting, above) with the identical \$\{secret:([^}]+)\} token pattern SecretReferenceExpander uses at app startup — so what you see here is exactly what the app would try to resolve, including config keys documented via a "_comment": "...${secret:x}..." convention, which are skipped the same way the expander skips them.
2. Each distinct referenced name is looked up against the store and classified into one of the six ReferenceStatus values:
   • Ok — exists and decrypts under the session KEK. Nothing to do.
   • Missing — no row at all. A fail-closed expansion would throw at app startup.
   • Tombstoned — a soft-deleted row. Also unresolvable to the app.
   • Undecryptable — the row exists but does not decrypt under the session KEK (wrong or rotated key, or damage). This is often the first sign a KEK doctor pass is needed.
   • InvalidName — the token text isn't a legal secret name (illegal characters, an unfilled placeholder like REPLACE_ME). Can't be looked up or seeded — fix the token in the config file itself.
   • PresentUnverified — the row exists but the session is degraded (no KEK), so presence is all that could be established — decryptability wasn't checked. You will not see this status in the seed flow, which requires a KEK (the shell upgrades a degraded session before dispatch); it only surfaces if the auditor is run without a key. In the seed flow an existing row therefore always resolves to Ok or Undecryptable, never PresentUnverified.
3. A color-coded table renders every finding with the config key paths that reference it (so you know exactly which setting(s) are affected), then the flow walks each seedable gap (Missing / Tombstoned / Undecryptable) in turn, asking to seed it now. Accepting prompts for a masked value, a content type, and an optional description, then seals a fresh row under the session KEK — overwriting an Undecryptable row and reviving (un-tombstoning) a Tombstoned one.
4. A closing re-audit prints a one-line count-by-status summary so you can confirm the gap closed without re-reading the whole table.

Run this again any time after adding a new ${secret:} reference to a config file, or after deploying to a fresh environment — it is the fast path from "app won't start, references don't resolve" to "every reference is Ok."

Lockout playbook: KEK doctor

Run this when an app (or the console itself) reports decrypt failures it didn't before — typically after a KEK rotation was only partially applied, a config value pointing at the wrong key source, or a store copied from another environment.

Diagnosis probes every row (including tombstones, since they still carry a KEK-wrapped DEK and block a clean rewrap-all) against the session's KEK:

• Ok — the row's kek_id matches the session KEK and it decrypts cleanly.
• WrongKek — the row's kek_id differs from the session KEK. Not decrypted (no point — the wrap alone tells you it won't work); reported with the row's own kek_id so you know which key to go find. The summary line also lists the distinct foreign KEK ids seen across all wrong-KEK rows — usually there is exactly one (the store's actual current key), which tells you the session simply has the wrong key configured.
• Corrupt — the kek_id matches (so the wrap envelope claims to be under the right key), but the decrypt still fails closed. This means the wrap envelope or the sealed body itself is damaged — a different problem than WrongKek, and not fixable by rewrapping: rewrapping only re-wraps a DEK it can first unwrap, and a Corrupt row can't be unwrapped at all.

If every row reports Ok, the doctor says so and there is nothing to remedy — the lockout is elsewhere (check the app's actual configured KEK source against what you used to open this session).

Two remedies, offered only when something is unhealthy:

• Rewrap from old KEK — the non-destructive path, and the only correct remedy for WrongKek rows. You supply the old KEK (env var, key file, or a masked paste) the affected rows are actually wrapped under; the flow shows exactly how many rows will move and from which KEK id, then asks for an explicit confirmation (default No — nothing happens on Enter) before calling the same KekRotationService.RewrapAllAsync primitive kek-rotation.md documents for the headless rewrap-all verb. Secret bodies are never touched — only the DEK wrap moves onto the session KEK. A wrong old key surfaces as a decryption failure and aborts (rows already rewrapped before the anomaly stay persisted, so a retry after fixing the key resumes cleanly).
• Old KEK is lost — re-set affected secrets — the destructive last resort, and the only remedy for a Corrupt row (which cannot be rewrapped because it can't be unwrapped in the first place). For every WrongKek or Corrupt row still affected, the flow offers a masked re-set: type a fresh value, and it's sealed fresh under the session KEK, overwriting the unrecoverable row. Each row is confirmed individually (default No) so you can skip rows you'd rather chase down the old key for instead.

A closing re-diagnosis prints a one-line count-by-status summary so you can see the remedy's effect immediately, in the same session, without re-running the flow.

Bundles: export / import

Bundles move secret rows between deployment stores — cloning a deployment, staging a recovery, seeding a new environment from a known-good one — without ever touching plaintext.

Export bundle (ciphertext-only) writes every live row (or, if you opt in, tombstones too) to a JSON file: for each row, the six crypto fields (ciphertext/nonce/tag for the body, wrapped_dek/wrap_nonce/wrap_tag for the DEK wrap) ride as base64, plus the row's kek_id, revision, and timestamps. The bundle also records the exporting session's KEK id (SourceKekId) and a FormatVersion. A default filename (secrets-bundle-<target>-<yyyyMMdd>.json) is offered but can be overridden. The bundle is useless without the source KEK — there is no plaintext anywhere in it — so it is safe to copy, attach to a ticket, or commit to a secured artifact store, but treat it with the same care as any other encrypted-at-rest artifact (it does reveal which names and content types exist).

Import bundle reads and parses the file before touching the store or prompting for anything, so a malformed file or unsupported FormatVersion (this build supports version 1 only) is caught up front. Then, per row:

• Same KEK as the session — imported directly.
• Different KEK (SourceKekId ≠ the session's) — the flow tells you both KEK ids and asks for the bundle's source key (paste / env var / key file) so it can re-wrap each row onto the session KEK. The supplied key's own derived id is checked against the bundle's declared SourceKekId before anything is imported — a valid-but-wrong key is rejected with both ids shown, rather than silently landing every row in the "skipped foreign KEK" bucket. Declining to supply a key aborts the whole import (no key means no row can be re-wrapped).
• Name collision (a row with that name already exists locally) — resolved by last-writer- wins (the shared SecretLastWriterWins tie-break used by every replication topology: later updated_utc, then higher revision, wins) unless you opt into per-conflict prompting, in which case each collision shows both rows' timestamp and revision and asks whether to take the bundle's row (default No — keep local). A winning row lands via the store's replicated-apply path so its own revision/timestamps are preserved verbatim (the same path cluster anti-entropy uses); a row you force against LWW ordering is instead written as a fresh local upsert so the override actually sticks rather than being reverted by the next reconciliation.

A closing tally table (Imported / Skipped-older / Skipped-foreign-KEK / Conflicts) reports the outcome; a non-zero "skipped foreign KEK" count is annotated with the bundle's source KEK id so a wrong-key paste is diagnosable from the report alone without re-running the import.

Security notes

• No login. The console is a local admin tool: whoever can run it already has direct filesystem/DB access to the target store (they could sqlite3 the file open themselves), so the console adds no new trust boundary — it does not check identity, does not audit its own actions distinctly from the store-level audit the underlying ISecretResolver/store calls already perform, and should be restricted the same way you'd restrict shell access to the host or the deployment's config files.
• Values are never echoed. Every prompt for a secret value or key material uses Spectre's masked TextPrompt(...).Secret() — nothing typed is drawn to the terminal, and nothing typed is written to a log.
• Values are never cached. Get (reveal) decrypts and writes the plaintext exactly once, with no resolver, no in-memory cache, and no re-use across the session — asking again re-decrypts from the row.
• Recents contain no key material. ~/.zb-secrets/recent-targets.json stores only a display name, an appsettings path, and a timestamp — never a KEK, a secret value, or a connection string. It is safe to inspect or delete.
• Bundles are ciphertext-only and safe at rest. A bundle file contains no plaintext under any circumstance — export cannot produce one and import cannot be pointed at one that "leaks" plaintext through the format; the only way to recover a value from a bundle is to import it into a session holding the matching KEK and then explicitly reveal it.

Troubleshooting

Symptom	Cause	Action
Bare secret prints usage instead of opening the console	stdin or stdout is redirected (a pipe, a script, CI)	Expected — run it directly at an interactive terminal with no redirection.
Header shows KEK UNAVAILABLE (degraded)	The target's configured KEK source didn't resolve in this shell (unset env var, missing file, DPAPI off Windows)	List/Delete still work. Pick a KEK-requiring flow to get the upgrade prompt, or fix the env var/file and re-select the target.
Reference audit shows Undecryptable for a name that used to resolve	The row is wrapped under a KEK the session doesn't have — often a partially-applied rotation	Run the KEK doctor; if it reports WrongKek, rewrap from the old KEK.
KEK doctor reports Corrupt, not WrongKek	kek_id matches but the wrap/body itself is damaged	Rewrap cannot fix this (nothing to unwrap first) — use "Old KEK is lost" to re-set the row, or restore the store from a backup.
Bundle import reports a large Skipped (foreign KEK) count	Declined (or gave the wrong) source key when prompted	Re-run the import and supply the bundle's actual source key — the report shows the expected SourceKekId.
secret rewrap-all-style behavior needed across a whole fleet, not just this session's KEK	The KEK doctor's rewrap remedy always targets the session's KEK, for lockout recovery on one store	For a general old→new rotation across every store, use the headless secret rewrap-all verb per kek-rotation.md.