From 56b428d2effd28f5e1742a71bb98eda07ad57550 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 16 Jul 2026 15:21:54 -0400 Subject: [PATCH] docs(secrets): clustered master-key posture runbook + Central pointer (ScadaBridge G-5 T8) --- ...2026-07-16-secrets-clustered-master-key.md | 156 ++++++++++++++++++ .../appsettings.Central.json | 1 + 2 files changed, 157 insertions(+) create mode 100644 docs/operations/2026-07-16-secrets-clustered-master-key.md diff --git a/docs/operations/2026-07-16-secrets-clustered-master-key.md b/docs/operations/2026-07-16-secrets-clustered-master-key.md new file mode 100644 index 00000000..376c3069 --- /dev/null +++ b/docs/operations/2026-07-16-secrets-clustered-master-key.md @@ -0,0 +1,156 @@ +# Secrets: Clustered Master-Key Posture (Central Pair) + +## Purpose + +`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a +pre-host expander that runs at every Central-role boot, before `StartupValidator` +(Host-003). It reads rows from an envelope-encrypted SQLite store +(`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per +`Secrets:MasterKey:Source`. + +ScadaBridge's central role is an Akka.NET-clustered **pair** (`central-a` / +`central-b`, see [failover-procedures.md](./failover-procedures.md)) — either node +can be active, and both boot independently. This runbook covers what a production +deployment of the central pair (and any site node that is ever given a `${secret:}` +token) needs so that secret resolution behaves identically no matter which node is +running. It does **not** change any code; it is an operations/deployment posture, +delivered out-of-band from the committed config. + +## The two hard requirements + +For the pre-host expander to resolve the same plaintext secret on every node: + +1. **Identical KEK on every node.** All central nodes must unwrap the store with + the exact same master key. A per-node KEK (e.g. two different DPAPI-protected + keys, one per Windows box) would make each node decrypt the *other* node's + ciphertext rows to garbage. +2. **Identical store rows on every node.** All central nodes must read the same + SQLite database (same file, or a replicated/shared copy with the same rows) — + not two independently-seeded stores that happen to use the same KEK. + +`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator` +— there is no built-in cross-node replication today. Meeting both requirements in +production is a deployment concern, covered below. + +## Recommended interim posture (G-5) + +Until real replication exists (G-7, below), the recommended production posture is: + +- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only + key file that is identical on every central node** — a base64-encoded 32-byte + key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed to each + node's filesystem/secret-mount by the deployment tooling, and **never committed** + to the repo. Treat it with the same discipline as any other production secret + (restrictive file ACLs, no logging, rotated via the KEK-rotation runbook — G-8, + not yet built). +- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume** + that both central nodes mount, so every node's `SqliteSecretsStoreMigrator` opens + and reads the same rows. + +Writes to the store are rare and human-driven — an operator using the +`/admin/secrets` UI (G-6) or the `ZB.MOM.WW.Secrets` CLI on one node — while reads +happen on every boot and on the `ResolveCacheTtl` refresh cycle on both nodes. The +access pattern is read-mostly / effectively single-writer, which is what makes a +shared SQLite volume viable as an interim posture (see caveat below). + +## How it's delivered (do NOT commit these values) + +The File-KEK + shared-store posture is supplied per-node at deployment time — +**never** by editing the committed `appsettings.json` or +`appsettings.Central.json`. Two acceptable delivery mechanisms: + +**Option A — environment variable overrides** (Windows Service / NSSM env block, +container `env_file`, etc.), applied identically on `central-a` and `central-b`: + +``` +# production deployment — do not commit to the dev appsettings +Secrets__MasterKey__Source=File +Secrets__MasterKey__FilePath=/run/secrets/scadabridge-master.key +Secrets__SqlitePath=/shared/secrets/scadabridge-secrets.db +``` + +**Option B — a production-only config layer** that is *not* the committed dev +base (e.g. an untracked `appsettings.Production.json` deployed alongside the +binaries, or an orchestrator-injected config mount): + +```jsonc +// production deployment — do not commit to the dev appsettings +{ + "Secrets": { + "MasterKey": { + "Source": "File", + "FilePath": "/run/secrets/scadabridge-master.key" + }, + "SqlitePath": "/shared/secrets/scadabridge-secrets.db" + } +} +``` + +Either way, the file/path referenced must exist and be identical on every central +node **before** that node boots — the expander runs unconditionally and will throw +(`SecretNotFoundException` / migration failure) if the store or key is missing. + +## Caveat: SQLite over a shared volume is not real replication + +SQLite's file-locking model does not tolerate concurrent multi-writer access well +over network filesystems (SMB/NFS locking is unreliable, and even on a clustered +block volume only one writer should be active at a time). The interim posture +above is acceptable because: + +- Reads dominate (every boot + cache-refresh cycle on both nodes). +- Writes are rare, human-initiated, and effectively single-writer in practice + (an operator runs the CLI/UI against one node at a time). + +It is **not** a substitute for real replication, and it is not safe if both nodes +attempt concurrent writes. Do not build automation that writes secrets from both +central nodes simultaneously. + +## Data Protection is independent — do not touch it here + +ScadaBridge's cookie/session and hub-token protection already has its own +clustered-key story: `AddDataProtection().PersistKeysToDbContext()` +(see `docs/components/Security.md`), which shares the Data Protection key ring +across both central nodes via the existing MS SQL `ConfigurationDb`. That mechanism +is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption (KEK + SQLite store) and +must **not** be reconfigured as part of secrets-adoption work — doing so risks +invalidating active sessions/cookies for an unrelated reason. + +## The G-7 hand-off + +The posture above is an interim, ops-only workaround. The long-term shape, +tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is one of: + +- A ConfigDb-backed `ISecretStore` — mirroring the pattern ScadaBridge already + uses for the Data Protection key ring (`PersistKeysToDbContext`), giving both + central nodes a single MS SQL-backed source of truth for secret rows instead of + a shared SQLite file; or +- The `ZB.MOM.WW.Secrets.Akka` replicator (LWW + anti-entropy resync + + tombstones) referenced in the library's SPEC. + +Both require new library code that does not exist yet. This runbook's posture is +the bridge until one of those lands. + +## Dev/test/default posture (unchanged) + +The committed default in `appsettings.json` is: + +```json +"Secrets": { + "SqlitePath": "scadabridge-secrets.db", + "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" }, + "RunMigrationsOnStartup": true, + "ResolveCacheTtl": "00:00:30" +} +``` + +This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite +path is relative to the working directory, so local dev and the +`WebApplicationFactory` `Host.Tests` boot cleanly with no external mount. +Local dev and docker-compose environments supply concrete secret values via the +**whole-key environment override** (e.g. `ScadaBridge__Database__ConfigurationDb`), +which bypasses `${secret:...}` resolution entirely, so the expander is effectively +a no-op there. The File-KEK + shared-volume posture in this runbook applies only +to real clustered production deployments of the central pair — it must never be +baked into the committed dev base, because the expander runs unconditionally at +every Central boot and would break dev/CI if pointed at a nonexistent `/shared` +mount. diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json index 001e51b4..e48e4586 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json +++ b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json @@ -1,5 +1,6 @@ { "ScadaBridge": { + "_secretsClusterPosture": "Central pair KEK/store posture (production): see docs/operations/2026-07-16-secrets-clustered-master-key.md. Committed default is Source=Environment + relative SqlitePath (dev-safe); production uses a File KEK + shared store volume delivered out-of-band. Do NOT hardcode /shared paths here — the pre-host expander migrates the store at every Central boot and would break dev/test.", "_nodeName": "Host-018: NodeName stamps SourceNode on AuditLog/Notifications/SiteCalls rows (CLAUDE.md 'Centralized Audit Log' decision) and backs IX_AuditLog_Node_Occurred. Convention: 'central-a'/'central-b' for central nodes, 'node-a'/'node-b' for site nodes. Override per-node in multi-node deployments (the docker per-node configs do this). When left at the default below, single-node dev rows are stamped with 'central-a'; an empty value normalises to a NULL SourceNode.", "Node": { "Role": "Central",