docs(secrets): G-5 clustered master-key posture runbook (Task 6)
OtOpcUa is Akka-clustered and AddZbSecrets is registered unconditionally, so every node (admin, driver, fused) resolves secrets and needs the SAME KEK + same store rows. Ship G-5 as a production-posture runbook rather than hardcoding Source=File + shared paths into the committed role-overlay appsettings (which dev + TwoNodeClusterHarness also consume — that would break every dev/CI boot). Base appsettings stays on Source=Environment. Documents the interim File-KEK + shared-SQLite posture and the G-7 hand-off (ConfigDb-backed ISecretStore mirroring the existing DP PersistKeysToDbContext<OtOpcUaConfigDbContext> key-ring sharing). Cross-linked from docs/security.md. Mirrors the ScadaBridge G-5 resolution.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,177 @@
|
||||
# Secrets: Clustered Master-Key Posture (All Roles)
|
||||
|
||||
## Purpose
|
||||
|
||||
`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
|
||||
pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node
|
||||
boot, before the host is built. It reads rows from an envelope-encrypted SQLite
|
||||
store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
|
||||
`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also
|
||||
registered unconditionally, independent of node role.
|
||||
|
||||
OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })`
|
||||
in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`):
|
||||
`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a
|
||||
fused admin+driver node, or a driver-only node). This runbook covers what a
|
||||
production deployment needs so that secret resolution behaves identically on
|
||||
**every node regardless of role** — not just admin nodes. That matters here more
|
||||
than it might elsewhere: the pre-host expander and the runtime resolver both run
|
||||
unconditionally on driver-only nodes too, and driver-only nodes are the ones that
|
||||
will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at
|
||||
runtime, not just at boot. It does **not** change any code; it is an
|
||||
operations/deployment posture, delivered out-of-band from the committed config.
|
||||
|
||||
## The two hard requirements
|
||||
|
||||
For the pre-host expander (and the runtime resolver) to resolve the same
|
||||
plaintext secret on every node, no matter its role:
|
||||
|
||||
1. **Identical KEK on every node.** All nodes — admin, driver, and any fused
|
||||
combination — must unwrap the store with the exact same master key. A
|
||||
per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
|
||||
decrypt every *other* node's ciphertext rows to garbage.
|
||||
2. **Identical store rows on every node.** All nodes must read the same SQLite
|
||||
database (same file, or a replicated/shared copy with the same rows) — not
|
||||
independently-seeded stores that happen to share a KEK.
|
||||
|
||||
`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
|
||||
— there is no built-in cross-node replication today. Meeting both requirements in
|
||||
production is a deployment concern, covered below.
|
||||
|
||||
## Recommended interim posture (G-5)
|
||||
|
||||
Until real replication exists (G-7, below), the recommended production posture is:
|
||||
|
||||
- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
|
||||
key file that is identical on every node of every role** — a base64-encoded
|
||||
32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
|
||||
to each node's filesystem/secret-mount by the deployment tooling, and **never
|
||||
committed** to the repo. Treat it with the same discipline as any other
|
||||
production secret (restrictive file ACLs, no logging, rotated via a future
|
||||
KEK-rotation runbook — not yet built).
|
||||
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
|
||||
that every node mounts (admin, driver, and dev/fused alike), so every node's
|
||||
migrator opens and reads the same rows at boot.
|
||||
|
||||
Writes to the store are rare and human-driven — an operator using the
|
||||
`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
|
||||
reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
|
||||
regardless of role. The access pattern is read-mostly / effectively
|
||||
single-writer, which is what makes a shared SQLite volume viable as an interim
|
||||
posture (see caveat below).
|
||||
|
||||
## How it's delivered (do NOT commit these values)
|
||||
|
||||
The File-KEK + shared-store posture is supplied per-node at deployment time —
|
||||
**never** by editing the committed `appsettings.json` or the role-overlay files
|
||||
(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`).
|
||||
Those committed overlays are also consumed by local dev and the
|
||||
`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path
|
||||
into them would break every dev/test/CI boot. Two acceptable delivery
|
||||
mechanisms instead:
|
||||
|
||||
**Option A — environment variable overrides** (Windows Service / NSSM env block,
|
||||
container `env_file`, etc.), applied identically on every node regardless of role:
|
||||
|
||||
```
|
||||
# production deployment — do not commit to the dev appsettings
|
||||
Secrets__MasterKey__Source=File
|
||||
Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
|
||||
Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db
|
||||
```
|
||||
|
||||
**Option B — a production-only config layer** that is *not* the committed dev
|
||||
base (e.g. an untracked `appsettings.Production.json` deployed alongside the
|
||||
binaries, or an orchestrator-injected config mount):
|
||||
|
||||
```jsonc
|
||||
// production deployment — do not commit to the dev appsettings
|
||||
{
|
||||
"Secrets": {
|
||||
"MasterKey": {
|
||||
"Source": "File",
|
||||
"FilePath": "/run/secrets/otopcua-master.key"
|
||||
},
|
||||
"SqlitePath": "/shared/secrets/otopcua-secrets.db"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Either way, the file/path referenced must exist and be identical on every node
|
||||
**before** that node boots — the pre-host expander runs unconditionally on every
|
||||
role and will throw (`SecretNotFoundException` / migration failure) if the store
|
||||
or key is missing.
|
||||
|
||||
## Caveat: SQLite over a shared volume is not real replication
|
||||
|
||||
SQLite's file-locking model does not tolerate concurrent multi-writer access well
|
||||
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
|
||||
block volume only one writer should be active at a time). The interim posture
|
||||
above is acceptable because:
|
||||
|
||||
- Reads dominate (every node's boot + cache-refresh cycle, across every role).
|
||||
- Writes are rare, human-initiated, and effectively single-writer in practice
|
||||
(an operator runs the CLI/UI against one admin node at a time).
|
||||
|
||||
It is **not** a substitute for real replication, and it is not safe if multiple
|
||||
nodes attempt concurrent writes. Do not build automation that writes secrets
|
||||
from more than one node simultaneously.
|
||||
|
||||
## Data Protection is independent — do not touch it here
|
||||
|
||||
OtOpcUa's cookie/session protection already has its own clustered-key story:
|
||||
`AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>()`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`),
|
||||
which shares the Data Protection key ring across every node via the existing
|
||||
ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption
|
||||
(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
|
||||
work — doing so risks invalidating active sessions for an unrelated reason.
|
||||
|
||||
It is, however, the model for where the *next* iteration of secret storage
|
||||
should go — see the G-7 hand-off below.
|
||||
|
||||
## The G-7 hand-off
|
||||
|
||||
The posture above is an interim, ops-only workaround. The long-term shape,
|
||||
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
|
||||
ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
|
||||
the Data Protection key ring:
|
||||
|
||||
```csharp
|
||||
services.AddDataProtection()
|
||||
.PersistKeysToDbContext<OtOpcUaConfigDbContext>()
|
||||
.SetApplicationName("OtOpcUa");
|
||||
```
|
||||
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
|
||||
`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
|
||||
alike — a single MS SQL-backed source of truth for the Data Protection key ring;
|
||||
the secret store is the natural next tenant of that same shared database instead
|
||||
of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
|
||||
code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
|
||||
overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
|
||||
built as part of this cut. This runbook's shared-SQLite-volume posture is the
|
||||
bridge until G-7 lands.
|
||||
|
||||
## Dev/test/default posture (unchanged)
|
||||
|
||||
The committed default in `appsettings.json` is:
|
||||
|
||||
```json
|
||||
"Secrets": {
|
||||
"SqlitePath": "otopcua-secrets.db",
|
||||
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
|
||||
"RunMigrationsOnStartup": true
|
||||
}
|
||||
```
|
||||
|
||||
This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
|
||||
path is relative to the working directory, so local dev, the role-overlay
|
||||
appsettings (`appsettings.admin.json`, `appsettings.driver.json`,
|
||||
`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration
|
||||
tests all boot cleanly with no external mount. The File-KEK + shared-volume
|
||||
posture in this runbook applies only to real clustered production deployments —
|
||||
it must never be baked into the committed dev/role-overlay base, because the
|
||||
expander runs unconditionally at every node boot (any role) and would break
|
||||
dev/CI if pointed at a nonexistent `/shared` mount.
|
||||
@@ -423,6 +423,10 @@ Only switch a value to a `${secret:}` token **after** the secret is seeded — a
|
||||
token fails the boot fail-closed. Do not commit the KEK (`ZB_SECRETS_MASTER_KEY`) or any
|
||||
secret value.
|
||||
|
||||
For the production posture needed so every clustered node (admin, driver, and any
|
||||
fused role) resolves the same secrets from the same store, see
|
||||
[`docs/operations/2026-07-16-secrets-clustered-master-key.md`](operations/2026-07-16-secrets-clustered-master-key.md).
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Reference in New Issue
Block a user