diff --git a/docs/operations/2026-07-16-secrets-clustered-master-key.md b/docs/operations/2026-07-16-secrets-clustered-master-key.md new file mode 100644 index 00000000..5b1d9e9f --- /dev/null +++ b/docs/operations/2026-07-16-secrets-clustered-master-key.md @@ -0,0 +1,177 @@ +# Secrets: Clustered Master-Key Posture (All Roles) + +## Purpose + +`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a +pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in +`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node +boot, before the host is built. It reads rows from an envelope-encrypted SQLite +store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per +`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also +registered unconditionally, independent of node role. + +OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })` +in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`): +`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a +fused admin+driver node, or a driver-only node). This runbook covers what a +production deployment needs so that secret resolution behaves identically on +**every node regardless of role** — not just admin nodes. That matters here more +than it might elsewhere: the pre-host expander and the runtime resolver both run +unconditionally on driver-only nodes too, and driver-only nodes are the ones that +will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at +runtime, not just at boot. It does **not** change any code; it is an +operations/deployment posture, delivered out-of-band from the committed config. + +## The two hard requirements + +For the pre-host expander (and the runtime resolver) to resolve the same +plaintext secret on every node, no matter its role: + +1. **Identical KEK on every node.** All nodes — admin, driver, and any fused + combination — must unwrap the store with the exact same master key. A + per-node KEK (e.g. a per-box DPAPI-protected key) would make each node + decrypt every *other* node's ciphertext rows to garbage. +2. **Identical store rows on every node.** All nodes must read the same SQLite + database (same file, or a replicated/shared copy with the same rows) — not + independently-seeded stores that happen to share a KEK. + +`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator` +— there is no built-in cross-node replication today. Meeting both requirements in +production is a deployment concern, covered below. + +## Recommended interim posture (G-5) + +Until real replication exists (G-7, below), the recommended production posture is: + +- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only + key file that is identical on every node of every role** — a base64-encoded + 32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed + to each node's filesystem/secret-mount by the deployment tooling, and **never + committed** to the repo. Treat it with the same discipline as any other + production secret (restrictive file ACLs, no logging, rotated via a future + KEK-rotation runbook — not yet built). +- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume** + that every node mounts (admin, driver, and dev/fused alike), so every node's + migrator opens and reads the same rows at boot. + +Writes to the store are rare and human-driven — an operator using the +`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while +reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle, +regardless of role. The access pattern is read-mostly / effectively +single-writer, which is what makes a shared SQLite volume viable as an interim +posture (see caveat below). + +## How it's delivered (do NOT commit these values) + +The File-KEK + shared-store posture is supplied per-node at deployment time — +**never** by editing the committed `appsettings.json` or the role-overlay files +(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`). +Those committed overlays are also consumed by local dev and the +`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path +into them would break every dev/test/CI boot. Two acceptable delivery +mechanisms instead: + +**Option A — environment variable overrides** (Windows Service / NSSM env block, +container `env_file`, etc.), applied identically on every node regardless of role: + +``` +# production deployment — do not commit to the dev appsettings +Secrets__MasterKey__Source=File +Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key +Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db +``` + +**Option B — a production-only config layer** that is *not* the committed dev +base (e.g. an untracked `appsettings.Production.json` deployed alongside the +binaries, or an orchestrator-injected config mount): + +```jsonc +// production deployment — do not commit to the dev appsettings +{ + "Secrets": { + "MasterKey": { + "Source": "File", + "FilePath": "/run/secrets/otopcua-master.key" + }, + "SqlitePath": "/shared/secrets/otopcua-secrets.db" + } +} +``` + +Either way, the file/path referenced must exist and be identical on every node +**before** that node boots — the pre-host expander runs unconditionally on every +role and will throw (`SecretNotFoundException` / migration failure) if the store +or key is missing. + +## Caveat: SQLite over a shared volume is not real replication + +SQLite's file-locking model does not tolerate concurrent multi-writer access well +over network filesystems (SMB/NFS locking is unreliable, and even on a clustered +block volume only one writer should be active at a time). The interim posture +above is acceptable because: + +- Reads dominate (every node's boot + cache-refresh cycle, across every role). +- Writes are rare, human-initiated, and effectively single-writer in practice + (an operator runs the CLI/UI against one admin node at a time). + +It is **not** a substitute for real replication, and it is not safe if multiple +nodes attempt concurrent writes. Do not build automation that writes secrets +from more than one node simultaneously. + +## Data Protection is independent — do not touch it here + +OtOpcUa's cookie/session protection already has its own clustered-key story: +`AddDataProtection().PersistKeysToDbContext()` +(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`), +which shares the Data Protection key ring across every node via the existing +ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption +(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption +work — doing so risks invalidating active sessions for an unrelated reason. + +It is, however, the model for where the *next* iteration of secret storage +should go — see the G-7 hand-off below. + +## The G-7 hand-off + +The posture above is an interim, ops-only workaround. The long-term shape, +tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a +ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for +the Data Protection key ring: + +```csharp +services.AddDataProtection() + .PersistKeysToDbContext() + .SetApplicationName("OtOpcUa"); +``` + +(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`). +`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused +alike — a single MS SQL-backed source of truth for the Data Protection key ring; +the secret store is the natural next tenant of that same shared database instead +of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library +code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet, +overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not +built as part of this cut. This runbook's shared-SQLite-volume posture is the +bridge until G-7 lands. + +## Dev/test/default posture (unchanged) + +The committed default in `appsettings.json` is: + +```json +"Secrets": { + "SqlitePath": "otopcua-secrets.db", + "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" }, + "RunMigrationsOnStartup": true +} +``` + +This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite +path is relative to the working directory, so local dev, the role-overlay +appsettings (`appsettings.admin.json`, `appsettings.driver.json`, +`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration +tests all boot cleanly with no external mount. The File-KEK + shared-volume +posture in this runbook applies only to real clustered production deployments — +it must never be baked into the committed dev/role-overlay base, because the +expander runs unconditionally at every node boot (any role) and would break +dev/CI if pointed at a nonexistent `/shared` mount. diff --git a/docs/security.md b/docs/security.md index a2a234c7..d0bd9f73 100644 --- a/docs/security.md +++ b/docs/security.md @@ -423,6 +423,10 @@ Only switch a value to a `${secret:}` token **after** the secret is seeded — a token fails the boot fail-closed. Do not commit the KEK (`ZB_SECRETS_MASTER_KEY`) or any secret value. +For the production posture needed so every clustered node (admin, driver, and any +fused role) resolves the same secrets from the same store, see +[`docs/operations/2026-07-16-secrets-clustered-master-key.md`](operations/2026-07-16-secrets-clustered-master-key.md). + --- ## Troubleshooting