scadaproj/docs/plans/2026-07-15-secrets-manager-design.md

ZB.MOM.WW.Secrets — reusable encrypted secrets manager (design)

Date: 2026-07-15 Status: Design approved (brainstorming), ready for implementation planning Author: Joseph Doherty + Claude

1. Purpose

A new shared library in the ZB.MOM.WW.* family that stores secrets encrypted at rest and returns plaintext on demand, plus a Blazor UI to manage them. It normalizes the ad-hoc secret handling scattered across the family today: ScadaBridge's Data-Protection-encrypted connection strings, peppers/passwords riding in environment variables, and LDAP passwords sitting in appsettings.

Canonical use case: "add a SQL login password, then get the value later, but keep it encrypted for storage."

Lives as plain files under scadaproj/ (like every other ZB.MOM.WW.* lib) — it is not a nested git repo. (CLAUDE.md mis-states the nested-repo convention; do not git init it.)

2. Decisions (from brainstorming)

Axis	Decision
Crypto	AES-256-GCM envelope encryption — per-secret DEK wrapped by a master KEK
Master key	Pluggable IMasterKeyProvider seam with built-in env / file / DPAPI providers
Storage	SQLite default behind an ISecretStore seam (SQL Server swappable later)
UI	ZB.MOM.WW.Secrets.Ui RCL on the Theme kit, embedded in each app's dashboard
Consumption	Both — ISecretResolver for app runtime and audited human reveal; ${secret:name} config expander
Versioning	Overwrite-in-place — no value history (tombstones for deletes only, see §8)
Scope	Build lib + UI + CLI, wire HistorianGateway as the reference consumer
Cluster sync	Seam + schema now, Akka replicator deferred until a clustered app adopts

3. Package decomposition (mirrors ZB.MOM.WW.Auth's shape)

Package	Role	Depends on
ZB.MOM.WW.Secrets.Abstractions	Contracts only: ISecretStore, ISecretResolver, IMasterKeyProvider, ISecretCipher, ISecretReplicator, records / enums / exceptions	BCL + DI.Abstractions
ZB.MOM.WW.Secrets	Impl: AesGcmEnvelopeCipher, SqliteSecretStore + migrator, DefaultSecretResolver, master-key providers, ${secret:…} config expander, AddZbSecrets(…)	Abstractions, Microsoft.Data.Sqlite, ZB.MOM.WW.Audit
ZB.MOM.WW.Secrets.Ui	Blazor RCL on ZB.MOM.WW.Theme: list / add / rotate / delete / gated reveal	Abstractions, ZB.MOM.WW.Theme, Auth.AspNetCore
ZB.MOM.WW.Secrets.Cli (tool, not packed)	secret set / get / list / rm / rotate, like the apikey CLI	Secrets
ZB.MOM.WW.Secrets.Akka (deferred)	Cluster replication of encrypted rows — design only in this cut, built when a clustered app adopts	Secrets, Akka.Cluster

4. Crypto — envelope encryption

• Master KEK (32 bytes, AES-256) resolved from IMasterKeyProvider; never persisted to the secrets DB.
• Every write generates a fresh per-write DEK (32 bytes). Plaintext → AES-256-GCM under the DEK → (ciphertext, nonce, tag). The DEK is then wrapped by the KEK (AES-256-GCM) → (wrapped_dek, wrap_nonce, wrap_tag).
• AAD binding: the secret body is authenticated against name (the AAD) so a ciphertext row cannot be swapped under a different secret name.
• KEK rotation is cheap: re-wrap each DEK, never re-encrypt bodies. A kek_id column records which master key wrapped each row; an admin op RewrapAll(oldKek, newKek) walks the table. A row whose kek_id is unknown to the current provider fails closed on resolve.
• All primitives from System.Security.Cryptography.AesGcm and RandomNumberGenerator (BCL, cross-platform, no native deps). 12-byte random nonces, 16-byte tags.

5. Master-key providers (IMasterKeyProvider seam)

public interface IMasterKeyProvider
{
    // 32-byte AES-256 key material; null/throw => fail closed at construction.
    ReadOnlyMemory<byte> GetMasterKey();
    string KekId { get; } // stable identifier of the currently-active KEK
}

Built-in implementations, selected by config (Secrets:MasterKey:Source):

• EnvironmentMasterKeyProvider — base64 32 bytes from an env var (default ZB_SECRETS_MASTER_KEY). Container/CI friendly; matches how peppers already ride in env.
• FileMasterKeyProvider — reads a key file (path from config), mountable as a Docker/K8s secret. This is the provider used to satisfy the shared-KEK requirement across a clustered pair (same file mounted on both nodes).
• DpapiMasterKeyProvider — Windows DPAPI-protected key. Windows-only, guarded behind an OS check; skipped on macOS/Docker Linux.

Mirrors the family's existing IApiKeyPepperProvider / IGroupRoleMapper<TRole> seam pattern. Fail-closed: a missing/invalid key throws MasterKeyUnavailableException at construction so the app won't boot half-blind (same posture as ApiKeyVerifier on a missing pepper).

6. Storage (ISecretStore seam, SQLite default)

public interface ISecretStore
{
    Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct);
    Task UpsertAsync(StoredSecret row, CancellationToken ct);       // overwrite-in-place
    Task<bool> DeleteAsync(SecretName name, CancellationToken ct);  // soft delete (tombstone)
    Task<IReadOnlyList<SecretMetadata>> ListAsync(CancellationToken ct); // metadata only, no ciphertext
    // Replication support (see §8):
    Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct);
    Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct);   // LWW upsert
}

SqliteSecretStore with a schema-versioned, idempotent migrator (exactly like SqliteAuthStoreMigrator / SqliteAuthSchema), schema v1.

Table secret (one row per secret — overwrite-in-place, no value-history table):

Column	Notes
name	TEXT PK, canonical namespaced key e.g. sql/historiangw/historian-password
description	TEXT
content_type	TEXT — text | connection-string | json | binary-base64
ciphertext, nonce, tag	BLOB — AES-GCM under the DEK
wrapped_dek, wrap_nonce, wrap_tag	BLOB — DEK wrapped by the KEK
kek_id	TEXT — which master key wrapped this row
revision	INTEGER — per-row monotonic counter (bumped every write); conflict resolution
is_deleted, deleted_utc	tombstone (soft delete) — propagates deletes across a cluster
created_utc, updated_utc, created_by, updated_by	audit columns

schema_version table/pragma drives the idempotent migrator.

The revision / updated_utc / is_deleted columns exist now even though the Akka replicator is deferred — they're the cheap future-proofing so cluster sync needs no schema migration later (§8). They are not value-history: values are still overwritten in place; only a delete-marker survives.

7. Consumption

7a. App runtime — ISecretResolver

public interface ISecretResolver
{
    Task<string?> GetAsync(SecretName name, CancellationToken ct);
    Task<bool> TryGetAsync(SecretName name, out string value, CancellationToken ct);
}

DefaultSecretResolver decrypts via the cipher + KEK, with a small in-memory cache invalidated on rotate/delete. Every resolve writes a ZB.MOM.WW.Audit event (actor + secret name + outcome — never the value).

7b. Config integration — ${secret:name} expander

A SecretReferenceExpander runs after configuration load and before options validation, expanding ${secret:name} tokens in bound config:

"ConnectionStrings:Historian": "Server=wonder-sql-vd03,32565;...;Password=${secret:sql/historiangw/historian-password}"

Keeps plaintext out of appsettings. Fail-closed: a referenced-but-missing secret stops startup with a clear error naming the token.

7c. Human reveal

The UI "reveal" action returns the current plaintext, gated behind a dedicated secrets:reveal authz policy (strictly more privileged than secrets:manage), and writes an audit event with the actor. Reveal is opt-in per deployment.

8. Clustered pairs — syncing local secrets (design; impl deferred)

Applies to the Akka-clustered apps (ScadaBridge, OtOpcUa) — not the reference consumer HistorianGateway (single-process sidecar). With per-node local SQLite, a secret added/rotated on node A leaves node B stale.

Enabler: because each row is ciphertext + a KEK-wrapped DEK, whole rows can be shipped over the cluster wire without exposing plaintext or the KEK — the transport need not be trusted with secrets.

Hard operational constraint: every node in a pair MUST resolve the same master KEK (same env var / same mounted key file). A replicated row whose kek_id doesn't match the local provider fails closed on resolve ("row wrapped by unknown KEK") rather than returning garbage. Documented as a deployment requirement for clustered adopters; FileMasterKeyProvider with a shared mounted key is the intended path.

Mechanism — optional ZB.MOM.WW.Secrets.Akka package (core stays Akka-free, exactly like ZB.MOM.WW.Health.Akka), driving an ISecretReplicator seam defined in Abstractions (no-op default in core):

• On write (any node): bump revision + updated_utc, broadcast a SecretReplicationEnvelope (the encrypted row only) over Akka DistributedPubSub; peers ApplyReplicatedAsync.
• Conflict resolution: last-writer-wins by (updated_utc, revision, node-id tiebreak) — deterministic on every node. No singleton writer bottleneck.
• Anti-entropy resync on node join / rejoin / cold start: exchange a compact (name → revision) manifest with a peer, pull rows where the peer is ahead, push where local is ahead. Heals partitions and freshly-recreated nodes.
• Deletes = tombstones. A hard delete would be resurrected from a peer on next resync, so deletes are soft (is_deleted + deleted_utc), replicated, and reaped after a TTL.

Simpler alternative available via the seam: a clustered app may instead point ISecretStore at a shared SQL Server store — one source of truth, no replication (the shared DB still holds only ciphertext, since the KEK stays per-node). This is the usual best-practice answer for clustered state; app-level replication is for when a shared datastore isn't available.

First-cut scope: ship the revision / updated_utc / is_deleted columns + ISecretReplicator no-op seam + manifest/apply store methods now; build the real ZB.MOM.WW.Secrets.Akka replicator when a clustered app adopts (it's dead code for HistorianGateway).

9. UI (RCL on Theme)

Routable /admin/secrets page the host mounts into its dashboard, styled with Theme tokens / side-rail. Features:

• List — name, description, content-type, kek_id, created/updated_utc/_by. Metadata only; never the value.
• Add — name + description + content-type + value (masked input).
• Rotate — enter a new value → overwrite-in-place (bumps revision).
• Delete — in-page Theme modal confirm (no JS confirm/alert — browser-dialog hazard).
• Reveal — opt-in, secrets:reveal-gated, audited.

Two permissions wired through Auth.AspNetCore claims: secrets:manage (CRUD) vs secrets:reveal (see plaintext).

10. Error handling — fail closed everywhere

• Missing/invalid master key → MasterKeyUnavailableException at construction (app won't boot half-blind).
• GCM tag mismatch on decrypt → SecretDecryptionException (tamper / wrong KEK) — never returns garbage.
• Unknown secret on resolve → SecretNotFoundException / TryGet false; ${secret:} expansion fails startup naming the token.
• Row wrapped by an unknown kek_id → fail-closed SecretDecryptionException.
• Nothing logs plaintext; failures audit as Outcome=Failure.

11. Testing

• Crypto/unit (xUnit): cipher round-trip; tamper detection (flip a byte → throws); AAD name-binding (swap rows → throws); KEK re-wrap; provider selection (env/file, DPAPI Windows-guarded); resolver cache invalidation on rotate/delete; ${secret:} expansion + fail-closed.
• Store: SQLite migrator idempotency; schema v1 CRUD; soft-delete tombstone; manifest / ApplyReplicated LWW semantics.
• UI (bUnit): list renders metadata-only (asserts value never appears in markup); reveal gated by policy; add / rotate / delete flows.
• All offline / cross-platform (macOS / Windows / Docker) — AesGcm + Sqlite are portable; DPAPI provider guarded behind an OS check.

12. Reference consumer — HistorianGateway

Wire HistorianGateway to source its two SQL passwords (historian wonder-sql-vd03 + Galaxy SQL) via ${secret:…}, seeded through the UI/CLI — proving both the runtime-resolve path and the UI end-to-end in one place. Retains a plain-text fallback path only for local dev where no master key is provisioned (fail-closed otherwise).

13. Out of scope (this cut)

• The ZB.MOM.WW.Secrets.Akka cluster replicator implementation (design only; §8).
• SQL Server ISecretStore provider (seam supports it; build when a consumer needs it).
• Full components/secrets/ normalization (spec / current-state ×3 / GAPS) and all-app adoption — a tracked follow-on, like Health/Telemetry were "Built" before "Adopted".
• The secure-by-default / KEK-rotation runbook automation beyond the RewrapAll primitive.

14. Component-normalization follow-on

After the lib + reference consumer land, add components/secrets/ (spec, shared-contract, current-state per app documenting today's ad-hoc handling, GAPS backlog) and adopt across ScadaBridge / OtOpcUa / mxaccessgw — at which point the ZB.MOM.WW.Secrets.Akka replicator gets built for the clustered adopters.