264 lines
16 KiB
Markdown
264 lines
16 KiB
Markdown
# ZB.MOM.WW.Secrets
|
||
|
||
A reusable, envelope-encrypted **secrets manager** for the `ZB.MOM.WW.*` SCADA family:
|
||
store secrets (SQL passwords, API tokens, connection strings) encrypted at rest and get the
|
||
plaintext back on demand — from application code, from configuration, or from an operator UI.
|
||
|
||
## Packages
|
||
|
||
| Package | Purpose |
|
||
|---|---|
|
||
| `ZB.MOM.WW.Secrets.Abstractions` | Contracts only (`ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, `ISecretCacheInvalidator`, `ISecretActorAccessor`), value types, exceptions. Dependency-light. |
|
||
| `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
|
||
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
|
||
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | Cluster-wide secrets via SQL Server — as one shared store, or as a hub each node syncs against. |
|
||
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | Peer-to-peer replication over an Akka.NET cluster — partition-tolerant, no shared database. |
|
||
| ~~`ZB.MOM.WW.Secrets.Replicator.Grpc`~~ | **Deprecated (scadaproj#8, 0.6.6).** The central-hosted, pull-only `secrets_hub.v1` gRPC hub. It has had no consumer since ScadaBridge retired the hub (2026-09-28: site secrets ride its artifact deploy), so it is no longer built, tested or packed with the family and its entry points are `[Obsolete]`. The last published version is `0.6.3`; source stays under `src/` for reference. Use the SqlServer or Akka replicator instead. |
|
||
|
||
A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).
|
||
`set` and `rotate` take the value on **stdin** (`printf '%s' "$PW" | secret set db/pw -`) or, at a
|
||
terminal, from a masked prompt (`secret set db/pw`). The old `secret set <name> <value>` form still
|
||
works but warns, because a value on the command line lands in shell history and the process list.
|
||
The CLI reads its configuration from `appsettings.json` and the environment only — never from argv.
|
||
|
||
### Interactive console
|
||
|
||
Run `secret` with **no arguments** in a real terminal and it opens an interactive console
|
||
instead of printing usage — a menu-driven session for an operator sitting at a deployment,
|
||
rather than a scripted one-shot verb. The launch condition is exact: no verb **and** neither
|
||
stdin nor stdout is redirected. Any redirection (a pipe, `| cat`, CI, a script) falls straight
|
||
through to the headless usage/exit-2 behavior above, so nothing scripted ever hangs on a
|
||
prompt.
|
||
|
||
The first screen picks a **store target** — recent targets (persisted to
|
||
`~/.zb-secrets/recent-targets.json`, names + paths only, never key material), an app's
|
||
`appsettings.json` (the console composes base + `appsettings.<env>.json` + environment
|
||
variables and binds `Secrets` exactly like the app itself, so every operation hits precisely
|
||
the store the app reads), or manual entry (a raw SQLite path + a master-key environment
|
||
variable name). If the configured KEK cannot be resolved in the operator's shell, the session
|
||
still opens — **degraded**, metadata-only — and any flow that needs the key prompts for one
|
||
(masked paste or a key file) to upgrade it in place.
|
||
|
||
Once a target is open, the menu offers:
|
||
|
||
- **List secrets** — metadata table (name, content type, KEK id, revision, updated); no KEK needed.
|
||
- **Set / rotate a secret** — masked value prompt, seals a fresh row under the session KEK.
|
||
- **Get (reveal) a secret** — confirm-gated; prints the plaintext once, never cached or logged.
|
||
- **Delete a secret** — confirm-gated tombstone; no KEK needed.
|
||
- **Reference audit & seed** — scans the target's composed configuration for `${secret:NAME}`
|
||
tokens, classifies each Ok / Missing / Tombstoned / Undecryptable / InvalidName, and walks the
|
||
gaps through a seed prompt. This is *the* deployment-setup flow.
|
||
- **KEK doctor (lockout recovery)** — per-row diagnosis (Ok / WrongKek / Corrupt, with the
|
||
foreign KEK id) plus a guided remedy: rewrap every row from an old KEK, or re-set individual
|
||
rows when the old KEK is unrecoverable.
|
||
- **Export bundle (ciphertext-only)** / **Import bundle** — move rows between stores. Bundles
|
||
carry name + metadata + wrapped DEK + ciphertext, never plaintext; importing across KEKs
|
||
verifies the pasted source key against the bundle's KEK id before re-wrapping, conflicts
|
||
resolve last-writer-wins (with an optional per-row prompt), and the format is versioned.
|
||
|
||
There is no login — the console runs locally with direct file/DB access, the same trust model
|
||
as any other admin CLI. New to it? Start with the task-first
|
||
[quickstart](docs/interactive-console-quickstart.md) (seed a new deployment; recover a KEK
|
||
lockout). For the full flow catalogue, exit-code/Ctrl-C semantics, and security model, see the
|
||
operator runbook: [`docs/operations/interactive-console.md`](docs/operations/interactive-console.md).
|
||
|
||
## How it protects secrets
|
||
|
||
Each value is encrypted with a fresh **data key (DEK)** using AES-256-GCM; the DEK is then
|
||
**wrapped** by a **master KEK** that never touches the database. The AAD binds the ciphertext
|
||
to the secret's name and the wrapped DEK to the KEK id, so rows can't be swapped and a
|
||
tampered/mis-keyed row **fails closed** (`SecretDecryptionException`) rather than returning a
|
||
wrong value. The master KEK is resolved through a pluggable `IMasterKeyProvider`
|
||
(environment variable, key file, or Windows DPAPI).
|
||
|
||
## Wiring it up
|
||
|
||
```csharp
|
||
builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
|
||
|
||
// After config load, BEFORE options validation — expand ${secret:...} references:
|
||
var expander = new SecretReferenceExpander(app.Services.GetRequiredService<ISecretResolver>());
|
||
await expander.ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, ct);
|
||
```
|
||
|
||
```jsonc
|
||
// appsettings.json
|
||
"Secrets": {
|
||
// SqlitePath omitted: the per-user default (<LocalApplicationData>/ZB.MOM.WW/secrets.db) applies.
|
||
// A service sets an ABSOLUTE path OUTSIDE its install directory instead, e.g.
|
||
// "SqlitePath": "E:\\ZB\\data\\secrets.db" — a relative path, or one under the content root,
|
||
// fails startup validation.
|
||
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
|
||
"RunMigrationsOnStartup": true,
|
||
"ResolveCacheTtl": "00:00:30"
|
||
},
|
||
// Keep plaintext out of config — reference a stored secret instead:
|
||
"ConnectionStrings": {
|
||
"Historian": "Server=...;User Id=sa;Password=${secret:sql/historian-password}"
|
||
}
|
||
```
|
||
|
||
**Reload.** Expansion is a startup snapshot that survives configuration reload (since `0.6.6`): pass
|
||
`builder.Configuration` (a `ConfigurationManager`) and the expanded values are pinned in an overlay
|
||
source that an `appsettings.json` reload cannot revert. A key that held a `${secret:}` reference
|
||
keeps its startup value until restart — editing it, or rotating the secret, takes effect on the next
|
||
start. A plain `ConfigurationBuilder().Build()` root gets the values re-applied after each reload,
|
||
which is weaker; build such a root without `reloadOnChange`.
|
||
|
||
*Known gap, not fixed (scadaproj#49):* a plain `IConfigurationRoot` has a fixed provider list, so
|
||
the overlay cannot be added to it; the library writes the values into its providers and re-applies
|
||
them from the root's reload token. Direct `IConfiguration` reads see the pinned value after a reload,
|
||
but an `IOptionsMonitor` bound to that root can rebind from the placeholder first — reload callbacks
|
||
run in an order the library does not control (the token's callbacks run last-registered-first, and
|
||
the expander registers early). Inserting a provider into a built root would mean mutating
|
||
`ConfigurationRoot`'s internal list, which this library deliberately does not do. Nobody in the
|
||
family is exposed today (ScadaBridge and the ignitionoee router build their plain root with
|
||
`reloadOnChange: false`; the other hosts pass a `ConfigurationManager`). If you must reload a plain
|
||
root, expand into a `ConfigurationManager` instead, or re-read the key per use rather than through
|
||
`IOptionsMonitor`.
|
||
|
||
The master key is 32 bytes, provided base64 in `ZB_SECRETS_MASTER_KEY` (Environment source).
|
||
A missing/invalid key fails closed at startup (`MasterKeyUnavailableException`).
|
||
|
||
## Rotating the master KEK
|
||
|
||
Because each row is a body sealed under a per-secret DEK that is *wrapped* by the KEK, rotating
|
||
the master key only re-wraps DEKs — bodies are never re-encrypted and no value history changes.
|
||
The `secret rewrap-all` CLI verb (backed by `KekRotationService`) migrates every row from the
|
||
old KEK to the new one; it is idempotent and safe to re-run:
|
||
|
||
```bash
|
||
# New KEK defaults to the configured Secrets:MasterKey; supply the OLD key by env-var name or path.
|
||
ZB_SECRETS_MASTER_KEY=<new-base64> ZB_SECRETS_OLD_KEY=<old-base64> \
|
||
secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY
|
||
# → {"action":"rewrap-all","total":N,"rewrapped":N,"alreadyCurrent":0}
|
||
```
|
||
|
||
Run it with resolve traffic quiesced and once per independent store (once for a shared
|
||
SQL-Server store; once per node for per-node SQLite on a shared KEK). See the operator runbook:
|
||
[`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md).
|
||
|
||
## Runtime + human access
|
||
|
||
- **App code:** inject `ISecretResolver` and call `GetAsync(name, ct)`. Every resolve is
|
||
audited via `ZB.MOM.WW.Audit` (name + outcome only — the value is never logged).
|
||
- **Operators:** mount the `ZB.MOM.WW.Secrets.Ui` `/admin/secrets` page. The list shows
|
||
**metadata only**; revealing a value requires the `secrets:reveal` authorization policy and
|
||
is audited. Add/rotate/delete require `secrets:manage`.
|
||
|
||
### UI 1.0.0 integration
|
||
|
||
The UI release is independent of the Core/store/replicator versions. Its reviewed dependency set is
|
||
`ZB.MOM.WW.Secrets.Ui` 1.0.0, `ZB.MOM.WW.Secrets.Abstractions` 0.6.6 and `ZB.MOM.WW.Theme` 1.0.1.
|
||
Keep an older host’s Core/backend versions separate; update an explicit Abstractions pin together
|
||
with the UI to avoid a dependency downgrade. The global source version stays 0.6.6; this UI release
|
||
does not publish Core/backend 0.6.6 set.
|
||
|
||
Load Bootstrap before `<ThemeHead />`, and serve Razor class library static assets. The UI supplies
|
||
its own namespaced confirmation styling and imports its dialog/reveal DOM adapters from
|
||
`_content/ZB.MOM.WW.Secrets.Ui/js/`. No host Bootstrap modal plugin or separate scoped-CSS bundle
|
||
is required for those adapters. Use a supported modern browser with native `dialog.showModal`.
|
||
|
||
A globally interactive host can route the RCL page. A host with static routing must instead own
|
||
one `/admin/secrets` wrapper carrying `@rendermode InteractiveServer`, restate the manage policy,
|
||
and render `<SecretsPage />` as its child. Registering that wrapper and the RCL’s routable assembly
|
||
together creates a duplicate route; do not add a nested render mode to the RCL page.
|
||
|
||
The name filter searches loaded metadata only. Values remain hidden until an authorized,
|
||
successfully audited Reveal; they stay visible until Hide, row removal/filtering, or navigation.
|
||
A failed reveal audit withholds the value. Add/Rotate drafts and pending callbacks belong to their
|
||
exact target. A committed mutation whose metadata refresh fails is reported as saved with Retry
|
||
refresh; retry never replays the credential change. Manage and reveal policies remain distinct.
|
||
|
||
Abstractions 0.6.6 adds the default `IMasterKeyProvider.GetSnapshot()` contract and a caller-owned,
|
||
zeroed-on-disposal `MasterKeySnapshot`. Legacy providers still implement the original key/id
|
||
members; providers with rotating sources should implement an atomic snapshot. `SecretName` now
|
||
normalizes and limits names to 450 characters. Check legacy names longer than 450, including
|
||
tombstones, before adoption; this UI work includes no schema or data migration. Exact binary
|
||
compatibility results, package provenance and host acceptance are recorded in
|
||
[UI consistency validation](docs/ui-consistency-validation.md).
|
||
|
||
## Clustered deployments
|
||
|
||
Storage is local SQLite by default, which is correct for a single-process app and wrong the
|
||
moment a secret written on one node has to be readable on another. Three topologies are
|
||
supported; pick the least complicated one that meets the requirement.
|
||
|
||
> **Every topology requires the same master KEK on every node.** The database and the wire carry
|
||
> ciphertext only, so a node with a different key cannot decrypt what another wrote — it fails
|
||
> closed on a `kek_id` mismatch, which looks like corruption but is a deployment mistake. Mount the
|
||
> same key file, or supply the same key environment variable, everywhere.
|
||
|
||
### 1. Shared SQL-Server store — *the default choice*
|
||
|
||
Every node points at one database. There is exactly one copy of each row, so there is no
|
||
replication, no reconciliation, no split-brain, and no clock skew to reason about.
|
||
|
||
```csharp
|
||
builder.Services.AddZbSecretsSqlServerStore(builder.Configuration);
|
||
```
|
||
|
||
```jsonc
|
||
"Secrets": {
|
||
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
||
"SqlServer": {
|
||
"ConnectionString": "Server=sql01;Database=ZbSecrets;...",
|
||
"SchemaName": "zbsecrets"
|
||
}
|
||
}
|
||
```
|
||
|
||
**Trade-off:** availability is coupled to that database. A node cut off from it falls back only to
|
||
the resolver's short in-memory cache. Rotation (`rewrap-all`) runs **once**, against the one store.
|
||
|
||
### 2. SQL-Server hub replication
|
||
|
||
Each node keeps its local store and stays converged with a shared hub: writes publish immediately,
|
||
and a background sweep reconciles **both directions** on an interval (default 30s).
|
||
|
||
```csharp
|
||
builder.Services.AddZbSecretsSqlServerReplication(builder.Configuration);
|
||
```
|
||
|
||
Choose this over (1) when a node must keep **resolving** secrets while cut off from the hub — reads
|
||
never leave the node, so a hub outage degrades propagation rather than availability.
|
||
|
||
### 3. Akka cluster peer-to-peer
|
||
|
||
No shared database at all. Nodes broadcast writes over distributed pub/sub and repair divergence
|
||
with a periodic manifest exchange.
|
||
|
||
```csharp
|
||
builder.Services.AddZbSecretsAkkaReplication(builder.Configuration);
|
||
// Recommended — pins the serializer that carries ciphertext:
|
||
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
|
||
```
|
||
|
||
Requires the application to register its `ActorSystem` in DI. Since `0.6.6` the replication actor
|
||
is created when the host has started, so a node replicates whether or not anything on it ever
|
||
resolves a secret. The local store is SQLite. This is
|
||
the right answer for air-gapped or intermittently-connected sites; it is the wrong answer if every
|
||
node can always reach a shared database, because you pay real distributed-systems behaviour for it.
|
||
|
||
Since `0.6.6` all three extensions are **order-independent** with respect to `AddZbSecrets`: call
|
||
it before or after them (or not at all — each calls it internally) and the same store, migrator and
|
||
replicator are bound. Each also takes an optional `contentRootPath`, forwarded to `AddZbSecrets`:
|
||
pass it from any **pre-host** composition (e.g. the container a `${secret:}` expander builds), just as
|
||
you would the 4-argument `AddZbSecrets`, so the store-path guard can reject a store inside the content
|
||
root. The last explicitly supplied root wins across calls.
|
||
|
||
### What replication costs you (2 and 3)
|
||
|
||
- **Eventual consistency.** Rows converge within the sweep interval, not instantly.
|
||
- **Last-writer-wins.** Two nodes writing the same secret concurrently: the later `updated_utc`
|
||
wins and the other write is discarded. There is no merge and no conflict report.
|
||
- **Publishing is best-effort.** A failed broadcast is logged, not thrown — the local write is
|
||
already durable and anti-entropy repairs it. Writes never fail because a peer was unreachable.
|
||
- **KEK rotation does not replicate.** A re-wrap deliberately leaves the revision untouched so it
|
||
stays invisible to last-writer-wins. Run `rewrap-all` against **each** independent store.
|
||
|
||
See [`docs/operations/clustered-secrets.md`](docs/operations/clustered-secrets.md) for operator
|
||
setup, [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md) for rotation, and
|
||
[`docs/operations/interactive-console.md`](docs/operations/interactive-console.md) for the
|
||
`secret` interactive console (deployment seeding, lockout recovery, bundles).
|