Joseph Doherty
5e2d3dd1bc
Secrets-009: SecretReferenceExpander.ExpandAsync throws FormatException when a value holds a
${secret: marker that is not a complete reference with a non-blank name (${secret:}, an
unterminated ${secret:name, a nested marker), instead of passing the literal text through as the
credential. ExpandConfigurationAsync names the offending key (never the value) and writes nothing.
The README wiring sample drops the relative "SqlitePath": "secrets.db" (which failed validation)
for the per-user default and shows an absolute example; a test validates the sample.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
220 lines
12 KiB
Markdown
220 lines
12 KiB
Markdown
# ZB.MOM.WW.Secrets
|
|
|
|
A reusable, envelope-encrypted **secrets manager** for the `ZB.MOM.WW.*` SCADA family:
|
|
store secrets (SQL passwords, API tokens, connection strings) encrypted at rest and get the
|
|
plaintext back on demand — from application code, from configuration, or from an operator UI.
|
|
|
|
## Packages
|
|
|
|
| Package | Purpose |
|
|
|---|---|
|
|
| `ZB.MOM.WW.Secrets.Abstractions` | Contracts only (`ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, `ISecretCacheInvalidator`, `ISecretActorAccessor`), value types, exceptions. Dependency-light. |
|
|
| `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
|
|
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
|
|
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | Cluster-wide secrets via SQL Server — as one shared store, or as a hub each node syncs against. |
|
|
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | Peer-to-peer replication over an Akka.NET cluster — partition-tolerant, no shared database. |
|
|
| `ZB.MOM.WW.Secrets.Replicator.Grpc` | Central-hosted `secrets_hub.v1` gRPC hub, pull-only (sites sync from central; central never accepts a write over this channel) — for topologies where sites talk only to central, never a shared DB. |
|
|
|
|
A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).
|
|
`set` and `rotate` take the value on **stdin** (`printf '%s' "$PW" | secret set db/pw -`) or, at a
|
|
terminal, from a masked prompt (`secret set db/pw`). The old `secret set <name> <value>` form still
|
|
works but warns, because a value on the command line lands in shell history and the process list.
|
|
The CLI reads its configuration from `appsettings.json` and the environment only — never from argv.
|
|
|
|
### Interactive console
|
|
|
|
Run `secret` with **no arguments** in a real terminal and it opens an interactive console
|
|
instead of printing usage — a menu-driven session for an operator sitting at a deployment,
|
|
rather than a scripted one-shot verb. The launch condition is exact: no verb **and** neither
|
|
stdin nor stdout is redirected. Any redirection (a pipe, `| cat`, CI, a script) falls straight
|
|
through to the headless usage/exit-2 behavior above, so nothing scripted ever hangs on a
|
|
prompt.
|
|
|
|
The first screen picks a **store target** — recent targets (persisted to
|
|
`~/.zb-secrets/recent-targets.json`, names + paths only, never key material), an app's
|
|
`appsettings.json` (the console composes base + `appsettings.<env>.json` + environment
|
|
variables and binds `Secrets` exactly like the app itself, so every operation hits precisely
|
|
the store the app reads), or manual entry (a raw SQLite path + a master-key environment
|
|
variable name). If the configured KEK cannot be resolved in the operator's shell, the session
|
|
still opens — **degraded**, metadata-only — and any flow that needs the key prompts for one
|
|
(masked paste or a key file) to upgrade it in place.
|
|
|
|
Once a target is open, the menu offers:
|
|
|
|
- **List secrets** — metadata table (name, content type, KEK id, revision, updated); no KEK needed.
|
|
- **Set / rotate a secret** — masked value prompt, seals a fresh row under the session KEK.
|
|
- **Get (reveal) a secret** — confirm-gated; prints the plaintext once, never cached or logged.
|
|
- **Delete a secret** — confirm-gated tombstone; no KEK needed.
|
|
- **Reference audit & seed** — scans the target's composed configuration for `${secret:NAME}`
|
|
tokens, classifies each Ok / Missing / Tombstoned / Undecryptable / InvalidName, and walks the
|
|
gaps through a seed prompt. This is *the* deployment-setup flow.
|
|
- **KEK doctor (lockout recovery)** — per-row diagnosis (Ok / WrongKek / Corrupt, with the
|
|
foreign KEK id) plus a guided remedy: rewrap every row from an old KEK, or re-set individual
|
|
rows when the old KEK is unrecoverable.
|
|
- **Export bundle (ciphertext-only)** / **Import bundle** — move rows between stores. Bundles
|
|
carry name + metadata + wrapped DEK + ciphertext, never plaintext; importing across KEKs
|
|
verifies the pasted source key against the bundle's KEK id before re-wrapping, conflicts
|
|
resolve last-writer-wins (with an optional per-row prompt), and the format is versioned.
|
|
|
|
There is no login — the console runs locally with direct file/DB access, the same trust model
|
|
as any other admin CLI. New to it? Start with the task-first
|
|
[quickstart](docs/interactive-console-quickstart.md) (seed a new deployment; recover a KEK
|
|
lockout). For the full flow catalogue, exit-code/Ctrl-C semantics, and security model, see the
|
|
operator runbook: [`docs/operations/interactive-console.md`](docs/operations/interactive-console.md).
|
|
|
|
## How it protects secrets
|
|
|
|
Each value is encrypted with a fresh **data key (DEK)** using AES-256-GCM; the DEK is then
|
|
**wrapped** by a **master KEK** that never touches the database. The AAD binds the ciphertext
|
|
to the secret's name and the wrapped DEK to the KEK id, so rows can't be swapped and a
|
|
tampered/mis-keyed row **fails closed** (`SecretDecryptionException`) rather than returning a
|
|
wrong value. The master KEK is resolved through a pluggable `IMasterKeyProvider`
|
|
(environment variable, key file, or Windows DPAPI).
|
|
|
|
## Wiring it up
|
|
|
|
```csharp
|
|
builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
|
|
|
|
// After config load, BEFORE options validation — expand ${secret:...} references:
|
|
var expander = new SecretReferenceExpander(app.Services.GetRequiredService<ISecretResolver>());
|
|
await expander.ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, ct);
|
|
```
|
|
|
|
```jsonc
|
|
// appsettings.json
|
|
"Secrets": {
|
|
// SqlitePath omitted: the per-user default (<LocalApplicationData>/ZB.MOM.WW/secrets.db) applies.
|
|
// A service sets an ABSOLUTE path OUTSIDE its install directory instead, e.g.
|
|
// "SqlitePath": "E:\\ZB\\data\\secrets.db" — a relative path, or one under the content root,
|
|
// fails startup validation.
|
|
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
|
|
"RunMigrationsOnStartup": true,
|
|
"ResolveCacheTtl": "00:00:30"
|
|
},
|
|
// Keep plaintext out of config — reference a stored secret instead:
|
|
"ConnectionStrings": {
|
|
"Historian": "Server=...;User Id=sa;Password=${secret:sql/historian-password}"
|
|
}
|
|
```
|
|
|
|
**Reload.** Expansion is a startup snapshot that survives configuration reload (since `0.6.6`): pass
|
|
`builder.Configuration` (a `ConfigurationManager`) and the expanded values are pinned in an overlay
|
|
source that an `appsettings.json` reload cannot revert. A key that held a `${secret:}` reference
|
|
keeps its startup value until restart — editing it, or rotating the secret, takes effect on the next
|
|
start. A plain `ConfigurationBuilder().Build()` root gets the values re-applied after each reload,
|
|
which is weaker; build such a root without `reloadOnChange`.
|
|
|
|
The master key is 32 bytes, provided base64 in `ZB_SECRETS_MASTER_KEY` (Environment source).
|
|
A missing/invalid key fails closed at startup (`MasterKeyUnavailableException`).
|
|
|
|
## Rotating the master KEK
|
|
|
|
Because each row is a body sealed under a per-secret DEK that is *wrapped* by the KEK, rotating
|
|
the master key only re-wraps DEKs — bodies are never re-encrypted and no value history changes.
|
|
The `secret rewrap-all` CLI verb (backed by `KekRotationService`) migrates every row from the
|
|
old KEK to the new one; it is idempotent and safe to re-run:
|
|
|
|
```bash
|
|
# New KEK defaults to the configured Secrets:MasterKey; supply the OLD key by env-var name or path.
|
|
ZB_SECRETS_MASTER_KEY=<new-base64> ZB_SECRETS_OLD_KEY=<old-base64> \
|
|
secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
|
|
# → {"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}
|
|
```
|
|
|
|
Run it with resolve traffic quiesced and once per independent store (once for a shared
|
|
SQL-Server store; once per node for per-node SQLite on a shared KEK). See the operator runbook:
|
|
[`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md).
|
|
|
|
## Runtime + human access
|
|
|
|
- **App code:** inject `ISecretResolver` and call `GetAsync(name, ct)`. Every resolve is
|
|
audited via `ZB.MOM.WW.Audit` (name + outcome only — the value is never logged).
|
|
- **Operators:** mount the `ZB.MOM.WW.Secrets.Ui` `/admin/secrets` page. The list shows
|
|
**metadata only**; revealing a value requires the `secrets:reveal` authorization policy and
|
|
is audited. Add/rotate/delete require `secrets:manage`.
|
|
|
|
## Clustered deployments
|
|
|
|
Storage is local SQLite by default, which is correct for a single-process app and wrong the
|
|
moment a secret written on one node has to be readable on another. Three topologies are
|
|
supported; pick the least complicated one that meets the requirement.
|
|
|
|
> **Every topology requires the same master KEK on every node.** The database and the wire carry
|
|
> ciphertext only, so a node with a different key cannot decrypt what another wrote — it fails
|
|
> closed on a `kek_id` mismatch, which looks like corruption but is a deployment mistake. Mount the
|
|
> same key file, or supply the same key environment variable, everywhere.
|
|
|
|
### 1. Shared SQL-Server store — *the default choice*
|
|
|
|
Every node points at one database. There is exactly one copy of each row, so there is no
|
|
replication, no reconciliation, no split-brain, and no clock skew to reason about.
|
|
|
|
```csharp
|
|
builder.Services.AddZbSecretsSqlServerStore(builder.Configuration);
|
|
```
|
|
|
|
```jsonc
|
|
"Secrets": {
|
|
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
|
"SqlServer": {
|
|
"ConnectionString": "Server=sql01;Database=ZbSecrets;...",
|
|
"SchemaName": "zbsecrets"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Trade-off:** availability is coupled to that database. A node cut off from it falls back only to
|
|
the resolver's short in-memory cache. Rotation (`rewrap-all`) runs **once**, against the one store.
|
|
|
|
### 2. SQL-Server hub replication
|
|
|
|
Each node keeps its local store and stays converged with a shared hub: writes publish immediately,
|
|
and a background sweep reconciles **both directions** on an interval (default 30s).
|
|
|
|
```csharp
|
|
builder.Services.AddZbSecretsSqlServerReplication(builder.Configuration);
|
|
```
|
|
|
|
Choose this over (1) when a node must keep **resolving** secrets while cut off from the hub — reads
|
|
never leave the node, so a hub outage degrades propagation rather than availability.
|
|
|
|
### 3. Akka cluster peer-to-peer
|
|
|
|
No shared database at all. Nodes broadcast writes over distributed pub/sub and repair divergence
|
|
with a periodic manifest exchange.
|
|
|
|
```csharp
|
|
builder.Services.AddZbSecretsAkkaReplication(builder.Configuration);
|
|
// Recommended — pins the serializer that carries ciphertext:
|
|
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
|
|
```
|
|
|
|
Requires the application to register its `ActorSystem` in DI. Since `0.6.6` the replication actor
|
|
is created when the host has started, so a node replicates whether or not anything on it ever
|
|
resolves a secret. The local store is SQLite. This is
|
|
the right answer for air-gapped or intermittently-connected sites; it is the wrong answer if every
|
|
node can always reach a shared database, because you pay real distributed-systems behaviour for it.
|
|
|
|
Since `0.6.6` all three extensions are **order-independent** with respect to `AddZbSecrets`: call
|
|
it before or after them (or not at all — each calls it internally) and the same store, migrator and
|
|
replicator are bound. Each also takes an optional `contentRootPath`, forwarded to `AddZbSecrets`:
|
|
pass it from any **pre-host** composition (e.g. the container a `${secret:}` expander builds), just as
|
|
you would the 4-argument `AddZbSecrets`, so the store-path guard can reject a store inside the content
|
|
root. The last explicitly supplied root wins across calls.
|
|
|
|
### What replication costs you (2 and 3)
|
|
|
|
- **Eventual consistency.** Rows converge within the sweep interval, not instantly.
|
|
- **Last-writer-wins.** Two nodes writing the same secret concurrently: the later `updated_utc`
|
|
wins and the other write is discarded. There is no merge and no conflict report.
|
|
- **Publishing is best-effort.** A failed broadcast is logged, not thrown — the local write is
|
|
already durable and anti-entropy repairs it. Writes never fail because a peer was unreachable.
|
|
- **KEK rotation does not replicate.** A re-wrap deliberately leaves the revision untouched so it
|
|
stays invisible to last-writer-wins. Run `rewrap-all` against **each** independent store.
|
|
|
|
See [`docs/operations/clustered-secrets.md`](docs/operations/clustered-secrets.md) for operator
|
|
setup, [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md) for rotation, and
|
|
[`docs/operations/interactive-console.md`](docs/operations/interactive-console.md) for the
|
|
`secret` interactive console (deployment seeding, lockout recovery, bundles).
|