scadaproj/ZB.MOM.WW.Secrets/README.md

# ZB.MOM.WW.Secrets

A reusable, envelope-encrypted **secrets manager** for the `ZB.MOM.WW.*` SCADA family:
store secrets (SQL passwords, API tokens, connection strings) encrypted at rest and get the
plaintext back on demand — from application code, from configuration, or from an operator UI.

## Packages

| Package | Purpose |
|---|---|
| `ZB.MOM.WW.Secrets.Abstractions` | Contracts only (`ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, `ISecretCacheInvalidator`, `ISecretActorAccessor`), value types, exceptions. Dependency-light. |
| `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | Cluster-wide secrets via SQL Server — as one shared store, or as a hub each node syncs against. |
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | Peer-to-peer replication over an Akka.NET cluster — partition-tolerant, no shared database. |

A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).

### Interactive console

Run `secret` with **no arguments** in a real terminal and it opens an interactive console
instead of printing usage — a menu-driven session for an operator sitting at a deployment,
rather than a scripted one-shot verb. The launch condition is exact: no verb **and** neither
stdin nor stdout is redirected. Any redirection (a pipe, `| cat`, CI, a script) falls straight
through to the headless usage/exit-2 behavior above, so nothing scripted ever hangs on a
prompt.

The first screen picks a **store target** — recent targets (persisted to
`~/.zb-secrets/recent-targets.json`, names + paths only, never key material), an app's
`appsettings.json` (the console composes base + `appsettings.<env>.json` + environment
variables and binds `Secrets` exactly like the app itself, so every operation hits precisely
the store the app reads), or manual entry (a raw SQLite path + a master-key environment
variable name). If the configured KEK cannot be resolved in the operator's shell, the session
still opens — **degraded**, metadata-only — and any flow that needs the key prompts for one
(masked paste or a key file) to upgrade it in place.

Once a target is open, the menu offers:

- **List secrets** — metadata table (name, content type, KEK id, revision, updated); no KEK needed.
- **Set / rotate a secret** — masked value prompt, seals a fresh row under the session KEK.
- **Get (reveal) a secret** — confirm-gated; prints the plaintext once, never cached or logged.
- **Delete a secret** — confirm-gated tombstone; no KEK needed.
- **Reference audit & seed** — scans the target's composed configuration for `${secret:NAME}`
  tokens, classifies each Ok / Missing / Tombstoned / Undecryptable / InvalidName, and walks the
  gaps through a seed prompt. This is *the* deployment-setup flow.
- **KEK doctor (lockout recovery)** — per-row diagnosis (Ok / WrongKek / Corrupt, with the
  foreign KEK id) plus a guided remedy: rewrap every row from an old KEK, or re-set individual
  rows when the old KEK is unrecoverable.
- **Export bundle (ciphertext-only)** / **Import bundle** — move rows between stores. Bundles
  carry name + metadata + wrapped DEK + ciphertext, never plaintext; importing across KEKs
  verifies the pasted source key against the bundle's KEK id before re-wrapping, conflicts
  resolve last-writer-wins (with an optional per-row prompt), and the format is versioned.

There is no login — the console runs locally with direct file/DB access, the same trust model
as any other admin CLI. See the operator runbook for the full walkthrough:
[`docs/operations/interactive-console.md`](docs/operations/interactive-console.md).

## How it protects secrets

Each value is encrypted with a fresh **data key (DEK)** using AES-256-GCM; the DEK is then
**wrapped** by a **master KEK** that never touches the database. The AAD binds the ciphertext
to the secret's name and the wrapped DEK to the KEK id, so rows can't be swapped and a
tampered/mis-keyed row **fails closed** (`SecretDecryptionException`) rather than returning a
wrong value. The master KEK is resolved through a pluggable `IMasterKeyProvider`
(environment variable, key file, or Windows DPAPI).

## Wiring it up

```csharp
builder.Services.AddZbSecrets(builder.Configuration, "Secrets");

// After config load, BEFORE options validation — expand ${secret:...} references:
var expander = new SecretReferenceExpander(app.Services.GetRequiredService<ISecretResolver>());
await expander.ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, ct);
```

```jsonc
// appsettings.json
"Secrets": {
  "SqlitePath": "secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true,
  "ResolveCacheTtl": "00:00:30"
},
// Keep plaintext out of config — reference a stored secret instead:
"ConnectionStrings": {
  "Historian": "Server=...;User Id=sa;Password=${secret:sql/historian-password}"
}
```

The master key is 32 bytes, provided base64 in `ZB_SECRETS_MASTER_KEY` (Environment source).
A missing/invalid key fails closed at startup (`MasterKeyUnavailableException`).

## Rotating the master KEK

Because each row is a body sealed under a per-secret DEK that is *wrapped* by the KEK, rotating
the master key only re-wraps DEKs — bodies are never re-encrypted and no value history changes.
The `secret rewrap-all` CLI verb (backed by `KekRotationService`) migrates every row from the
old KEK to the new one; it is idempotent and safe to re-run:

```bash
# New KEK defaults to the configured Secrets:MasterKey; supply the OLD key by env-var name or path.
ZB_SECRETS_MASTER_KEY=<new-base64>  ZB_SECRETS_OLD_KEY=<old-base64> \
  secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
# → {"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}
```

Run it with resolve traffic quiesced and once per independent store (once for a shared
SQL-Server store; once per node for per-node SQLite on a shared KEK). See the operator runbook:
[`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md).

## Runtime + human access

- **App code:** inject `ISecretResolver` and call `GetAsync(name, ct)`. Every resolve is
  audited via `ZB.MOM.WW.Audit` (name + outcome only — the value is never logged).
- **Operators:** mount the `ZB.MOM.WW.Secrets.Ui` `/admin/secrets` page. The list shows
  **metadata only**; revealing a value requires the `secrets:reveal` authorization policy and
  is audited. Add/rotate/delete require `secrets:manage`.

## Clustered deployments

Storage is local SQLite by default, which is correct for a single-process app and wrong the
moment a secret written on one node has to be readable on another. Three topologies are
supported; pick the least complicated one that meets the requirement.

> **Every topology requires the same master KEK on every node.** The database and the wire carry
> ciphertext only, so a node with a different key cannot decrypt what another wrote — it fails
> closed on a `kek_id` mismatch, which looks like corruption but is a deployment mistake. Mount the
> same key file, or supply the same key environment variable, everywhere.

### 1. Shared SQL-Server store — *the default choice*

Every node points at one database. There is exactly one copy of each row, so there is no
replication, no reconciliation, no split-brain, and no clock skew to reason about.

```csharp
builder.Services.AddZbSecretsSqlServerStore(builder.Configuration);
```

```jsonc
"Secrets": {
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "SqlServer": {
    "ConnectionString": "Server=sql01;Database=ZbSecrets;...",
    "SchemaName": "zbsecrets"
  }
}
```

**Trade-off:** availability is coupled to that database. A node cut off from it falls back only to
the resolver's short in-memory cache. Rotation (`rewrap-all`) runs **once**, against the one store.

### 2. SQL-Server hub replication

Each node keeps its local store and stays converged with a shared hub: writes publish immediately,
and a background sweep reconciles **both directions** on an interval (default 30s).

```csharp
builder.Services.AddZbSecretsSqlServerReplication(builder.Configuration);
```

Choose this over (1) when a node must keep **resolving** secrets while cut off from the hub — reads
never leave the node, so a hub outage degrades propagation rather than availability.

### 3. Akka cluster peer-to-peer

No shared database at all. Nodes broadcast writes over distributed pub/sub and repair divergence
with a periodic manifest exchange.

```csharp
builder.Services.AddZbSecretsAkkaReplication(builder.Configuration);
// Recommended — pins the serializer that carries ciphertext:
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
```

Requires the application to register its `ActorSystem` in DI. The local store is SQLite. This is
the right answer for air-gapped or intermittently-connected sites; it is the wrong answer if every
node can always reach a shared database, because you pay real distributed-systems behaviour for it.

### What replication costs you (2 and 3)

- **Eventual consistency.** Rows converge within the sweep interval, not instantly.
- **Last-writer-wins.** Two nodes writing the same secret concurrently: the later `updated_utc`
  wins and the other write is discarded. There is no merge and no conflict report.
- **Publishing is best-effort.** A failed broadcast is logged, not thrown — the local write is
  already durable and anti-entropy repairs it. Writes never fail because a peer was unreachable.
- **KEK rotation does not replicate.** A re-wrap deliberately leaves the revision untouched so it
  stays invisible to last-writer-wins. Run `rewrap-all` against **each** independent store.

See [`docs/operations/clustered-secrets.md`](docs/operations/clustered-secrets.md) for operator
setup, [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md) for rotation, and
[`docs/operations/interactive-console.md`](docs/operations/interactive-console.md) for the
`secret` interactive console (deployment seeding, lockout recovery, bundles).