scadaproj/docs/plans/2026-07-15-secrets-manager-design.md

# ZB.MOM.WW.Secrets — reusable encrypted secrets manager (design)

**Date:** 2026-07-15
**Status:** Design approved (brainstorming), ready for implementation planning
**Author:** Joseph Doherty + Claude

## 1. Purpose

A new shared library in the `ZB.MOM.WW.*` family that stores secrets **encrypted at
rest** and returns plaintext **on demand**, plus a Blazor UI to manage them. It normalizes
the ad-hoc secret handling scattered across the family today: ScadaBridge's
Data-Protection-encrypted connection strings, peppers/passwords riding in environment
variables, and LDAP passwords sitting in `appsettings`.

Canonical use case: *"add a SQL login password, then get the value later, but keep it
encrypted for storage."*

Lives as **plain files under `scadaproj/`** (like every other `ZB.MOM.WW.*` lib) — it is
**not** a nested git repo. (`CLAUDE.md` mis-states the nested-repo convention; do not
`git init` it.)

## 2. Decisions (from brainstorming)

| Axis | Decision |
|---|---|
| Crypto | **AES-256-GCM envelope encryption** — per-secret DEK wrapped by a master KEK |
| Master key | **Pluggable `IMasterKeyProvider` seam** with built-in env / file / DPAPI providers |
| Storage | **SQLite default behind an `ISecretStore` seam** (SQL Server swappable later) |
| UI | **`ZB.MOM.WW.Secrets.Ui` RCL** on the Theme kit, embedded in each app's dashboard |
| Consumption | **Both** — `ISecretResolver` for app runtime **and** audited human reveal; `${secret:name}` config expander |
| Versioning | **Overwrite-in-place** — no value history (tombstones for deletes only, see §8) |
| Scope | Build lib + UI + CLI, wire **HistorianGateway** as the reference consumer |
| Cluster sync | **Seam + schema now, Akka replicator deferred** until a clustered app adopts |

## 3. Package decomposition (mirrors `ZB.MOM.WW.Auth`'s shape)

| Package | Role | Depends on |
|---|---|---|
| `ZB.MOM.WW.Secrets.Abstractions` | Contracts only: `ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, records / enums / exceptions | BCL + `DI.Abstractions` |
| `ZB.MOM.WW.Secrets` | Impl: `AesGcmEnvelopeCipher`, `SqliteSecretStore` + migrator, `DefaultSecretResolver`, master-key providers, `${secret:…}` config expander, `AddZbSecrets(…)` | Abstractions, `Microsoft.Data.Sqlite`, `ZB.MOM.WW.Audit` |
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete / gated reveal | Abstractions, `ZB.MOM.WW.Theme`, `Auth.AspNetCore` |
| `ZB.MOM.WW.Secrets.Cli` *(tool, not packed)* | `secret set / get / list / rm / rotate`, like the `apikey` CLI | Secrets |
| `ZB.MOM.WW.Secrets.Akka` *(deferred)* | Cluster replication of encrypted rows — **design only in this cut**, built when a clustered app adopts | Secrets, Akka.Cluster |

## 4. Crypto — envelope encryption

- **Master KEK** (32 bytes, AES-256) resolved from `IMasterKeyProvider`; **never** persisted
  to the secrets DB.
- Every write generates a fresh **per-write DEK** (32 bytes). Plaintext → AES-256-GCM under
  the DEK → `(ciphertext, nonce, tag)`. The DEK is then **wrapped** by the KEK
  (AES-256-GCM) → `(wrapped_dek, wrap_nonce, wrap_tag)`.
- **AAD binding**: the secret body is authenticated against `name` (the AAD) so a ciphertext
  row cannot be swapped under a different secret name.
- **KEK rotation is cheap**: re-wrap each DEK, never re-encrypt bodies. A `kek_id` column
  records which master key wrapped each row; an admin op `RewrapAll(oldKek, newKek)` walks
  the table. A row whose `kek_id` is unknown to the current provider fails closed on resolve.
- All primitives from `System.Security.Cryptography.AesGcm` and `RandomNumberGenerator`
  (BCL, cross-platform, no native deps). 12-byte random nonces, 16-byte tags.

## 5. Master-key providers (`IMasterKeyProvider` seam)

```csharp
public interface IMasterKeyProvider
{
    // 32-byte AES-256 key material; null/throw => fail closed at construction.
    ReadOnlyMemory<byte> GetMasterKey();
    string KekId { get; } // stable identifier of the currently-active KEK
}
```

Built-in implementations, selected by config (`Secrets:MasterKey:Source`):

- `EnvironmentMasterKeyProvider` — base64 32 bytes from an env var (default
  `ZB_SECRETS_MASTER_KEY`). Container/CI friendly; matches how peppers already ride in env.
- `FileMasterKeyProvider` — reads a key file (path from config), mountable as a Docker/K8s
  secret. **This is the provider used to satisfy the shared-KEK requirement across a
  clustered pair** (same file mounted on both nodes).
- `DpapiMasterKeyProvider` — Windows DPAPI-protected key. **Windows-only**, guarded behind an
  OS check; skipped on macOS/Docker Linux.

Mirrors the family's existing `IApiKeyPepperProvider` / `IGroupRoleMapper<TRole>` seam
pattern. Fail-closed: a missing/invalid key throws `MasterKeyUnavailableException` at
construction so the app won't boot half-blind (same posture as `ApiKeyVerifier` on a missing
pepper).

## 6. Storage (`ISecretStore` seam, SQLite default)

```csharp
public interface ISecretStore
{
    Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct);
    Task UpsertAsync(StoredSecret row, CancellationToken ct);       // overwrite-in-place
    Task<bool> DeleteAsync(SecretName name, CancellationToken ct);  // soft delete (tombstone)
    Task<IReadOnlyList<SecretMetadata>> ListAsync(CancellationToken ct); // metadata only, no ciphertext
    // Replication support (see §8):
    Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct);
    Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct);   // LWW upsert
}
```

`SqliteSecretStore` with a schema-versioned, idempotent migrator (exactly like
`SqliteAuthStoreMigrator` / `SqliteAuthSchema`), schema **v1**.

**Table `secret`** (one row per secret — overwrite-in-place, no value-history table):

| Column | Notes |
|---|---|
| `name` | TEXT PK, canonical namespaced key e.g. `sql/historiangw/historian-password` |
| `description` | TEXT |
| `content_type` | TEXT — `text` \| `connection-string` \| `json` \| `binary-base64` |
| `ciphertext`, `nonce`, `tag` | BLOB — AES-GCM under the DEK |
| `wrapped_dek`, `wrap_nonce`, `wrap_tag` | BLOB — DEK wrapped by the KEK |
| `kek_id` | TEXT — which master key wrapped this row |
| `revision` | INTEGER — per-row monotonic counter (bumped every write); conflict resolution |
| `is_deleted`, `deleted_utc` | tombstone (soft delete) — propagates deletes across a cluster |
| `created_utc`, `updated_utc`, `created_by`, `updated_by` | audit columns |

`schema_version` table/pragma drives the idempotent migrator.

> The `revision` / `updated_utc` / `is_deleted` columns exist **now** even though the Akka
> replicator is deferred — they're the cheap future-proofing so cluster sync needs no schema
> migration later (§8). They are **not** value-history: values are still overwritten in
> place; only a delete-*marker* survives.

## 7. Consumption

### 7a. App runtime — `ISecretResolver`

```csharp
public interface ISecretResolver
{
    Task<string?> GetAsync(SecretName name, CancellationToken ct);
    Task<bool> TryGetAsync(SecretName name, out string value, CancellationToken ct);
}
```

`DefaultSecretResolver` decrypts via the cipher + KEK, with a small in-memory cache
invalidated on rotate/delete. **Every resolve writes a `ZB.MOM.WW.Audit` event** (actor +
secret name + outcome — **never the value**).

### 7b. Config integration — `${secret:name}` expander

A `SecretReferenceExpander` runs **after** configuration load and **before** options
validation, expanding `${secret:name}` tokens in bound config:

```json
"ConnectionStrings:Historian": "Server=wonder-sql-vd03,32565;...;Password=${secret:sql/historiangw/historian-password}"
```

Keeps plaintext out of `appsettings`. **Fail-closed**: a referenced-but-missing secret stops
startup with a clear error naming the token.

### 7c. Human reveal

The UI "reveal" action returns the current plaintext, **gated behind a dedicated
`secrets:reveal` authz policy** (strictly more privileged than `secrets:manage`), and writes
an audit event with the actor. Reveal is **opt-in per deployment**.

## 8. Clustered pairs — syncing local secrets (design; impl deferred)

Applies to the Akka-clustered apps (**ScadaBridge**, **OtOpcUa**) — **not** the reference
consumer HistorianGateway (single-process sidecar). With per-node local SQLite, a secret
added/rotated on node A leaves node B stale.

**Enabler:** because each row is ciphertext + a KEK-wrapped DEK, whole rows can be shipped
over the cluster wire **without exposing plaintext or the KEK** — the transport need not be
trusted with secrets.

**Hard operational constraint:** every node in a pair MUST resolve the **same master KEK**
(same env var / same mounted key file). A replicated row whose `kek_id` doesn't match the
local provider fails closed on resolve ("row wrapped by unknown KEK") rather than returning
garbage. Documented as a deployment requirement for clustered adopters; `FileMasterKeyProvider`
with a shared mounted key is the intended path.

**Mechanism** — optional `ZB.MOM.WW.Secrets.Akka` package (core stays Akka-free, exactly
like `ZB.MOM.WW.Health.Akka`), driving an `ISecretReplicator` seam defined in Abstractions
(no-op default in core):

- **On write** (any node): bump `revision` + `updated_utc`, broadcast a
  `SecretReplicationEnvelope` (the **encrypted** row only) over Akka `DistributedPubSub`;
  peers `ApplyReplicatedAsync`.
- **Conflict resolution:** last-writer-wins by `(updated_utc, revision, node-id tiebreak)` —
  deterministic on every node. No singleton writer bottleneck.
- **Anti-entropy resync** on node join / rejoin / cold start: exchange a compact
  `(name → revision)` manifest with a peer, pull rows where the peer is ahead, push where
  local is ahead. Heals partitions and freshly-recreated nodes.
- **Deletes = tombstones.** A hard delete would be resurrected from a peer on next resync, so
  deletes are soft (`is_deleted` + `deleted_utc`), replicated, and reaped after a TTL.

**Simpler alternative available via the seam:** a clustered app may instead point
`ISecretStore` at a **shared SQL Server store** — one source of truth, no replication (the
shared DB still holds only ciphertext, since the KEK stays per-node). This is the usual
best-practice answer for clustered state; app-level replication is for when a shared
datastore isn't available.

**First-cut scope:** ship the `revision` / `updated_utc` / `is_deleted` columns +
`ISecretReplicator` no-op seam + manifest/apply store methods **now**; build the real
`ZB.MOM.WW.Secrets.Akka` replicator when a clustered app adopts (it's dead code for
HistorianGateway).

## 9. UI (RCL on Theme)

Routable `/admin/secrets` page the host mounts into its dashboard, styled with Theme tokens /
side-rail. Features:

- **List** — name, description, content-type, `kek_id`, `created/updated_utc/_by`.
  **Metadata only; never the value.**
- **Add** — name + description + content-type + value (masked input).
- **Rotate** — enter a new value → overwrite-in-place (bumps `revision`).
- **Delete** — in-page **Theme modal** confirm (no JS `confirm`/`alert` — browser-dialog
  hazard).
- **Reveal** — opt-in, `secrets:reveal`-gated, audited.

Two permissions wired through `Auth.AspNetCore` claims: `secrets:manage` (CRUD) vs
`secrets:reveal` (see plaintext).

## 10. Error handling — fail closed everywhere

- Missing/invalid master key → `MasterKeyUnavailableException` at construction (app won't
  boot half-blind).
- GCM tag mismatch on decrypt → `SecretDecryptionException` (tamper / wrong KEK) — never
  returns garbage.
- Unknown secret on resolve → `SecretNotFoundException` / `TryGet` false; `${secret:}`
  expansion fails startup naming the token.
- Row wrapped by an unknown `kek_id` → fail-closed `SecretDecryptionException`.
- Nothing logs plaintext; failures audit as `Outcome=Failure`.

## 11. Testing

- **Crypto/unit (xUnit):** cipher round-trip; tamper detection (flip a byte → throws); AAD
  name-binding (swap rows → throws); KEK re-wrap; provider selection (env/file, DPAPI
  Windows-guarded); resolver cache invalidation on rotate/delete; `${secret:}` expansion +
  fail-closed.
- **Store:** SQLite migrator idempotency; schema v1 CRUD; soft-delete tombstone; manifest /
  `ApplyReplicated` LWW semantics.
- **UI (bUnit):** list renders metadata-only (asserts value never appears in markup); reveal
  gated by policy; add / rotate / delete flows.
- All offline / cross-platform (macOS / Windows / Docker) — `AesGcm` + Sqlite are portable;
  DPAPI provider guarded behind an OS check.

## 12. Reference consumer — HistorianGateway

Wire HistorianGateway to source its two SQL passwords (historian `wonder-sql-vd03` + Galaxy
SQL) via `${secret:…}`, seeded through the UI/CLI — proving both the runtime-resolve path and
the UI end-to-end in one place. Retains a plain-text fallback path only for local dev where no
master key is provisioned (fail-closed otherwise).

## 13. Out of scope (this cut)

- The `ZB.MOM.WW.Secrets.Akka` cluster replicator implementation (design only; §8).
- SQL Server `ISecretStore` provider (seam supports it; build when a consumer needs it).
- Full `components/secrets/` normalization (spec / current-state ×3 / GAPS) and all-app
  adoption — a tracked follow-on, like Health/Telemetry were "Built" before "Adopted".
- The secure-by-default / KEK-rotation runbook automation beyond the `RewrapAll` primitive.

## 14. Component-normalization follow-on

After the lib + reference consumer land, add `components/secrets/` (spec, shared-contract,
current-state per app documenting today's ad-hoc handling, GAPS backlog) and adopt across
ScadaBridge / OtOpcUa / mxaccessgw — at which point the `ZB.MOM.WW.Secrets.Akka` replicator
gets built for the clustered adopters.