docs(secrets): clustered master-key posture runbook + Central pointer (ScadaBridge G-5 T8)

docs/operations/2026-07-16-secrets-clustered-master-key.md

@@ -0,0 +1,156 @@
+# Secrets: Clustered Master-Key Posture (Central Pair)
+
+## Purpose
+
+`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
+pre-host expander that runs at every Central-role boot, before `StartupValidator`
+(Host-003). It reads rows from an envelope-encrypted SQLite store
+(`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
+`Secrets:MasterKey:Source`.
+
+ScadaBridge's central role is an Akka.NET-clustered **pair** (`central-a` /
+`central-b`, see [failover-procedures.md](./failover-procedures.md)) — either node
+can be active, and both boot independently. This runbook covers what a production
+deployment of the central pair (and any site node that is ever given a `${secret:}`
+token) needs so that secret resolution behaves identically no matter which node is
+running. It does **not** change any code; it is an operations/deployment posture,
+delivered out-of-band from the committed config.
+
+## The two hard requirements
+
+For the pre-host expander to resolve the same plaintext secret on every node:
+
+1. **Identical KEK on every node.** All central nodes must unwrap the store with
+   the exact same master key. A per-node KEK (e.g. two different DPAPI-protected
+   keys, one per Windows box) would make each node decrypt the *other* node's
+   ciphertext rows to garbage.
+2. **Identical store rows on every node.** All central nodes must read the same
+   SQLite database (same file, or a replicated/shared copy with the same rows) —
+   not two independently-seeded stores that happen to use the same KEK.
+
+`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
+— there is no built-in cross-node replication today. Meeting both requirements in
+production is a deployment concern, covered below.
+
+## Recommended interim posture (G-5)
+
+Until real replication exists (G-7, below), the recommended production posture is:
+
+- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
+  key file that is identical on every central node** — a base64-encoded 32-byte
+  key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed to each
+  node's filesystem/secret-mount by the deployment tooling, and **never committed**
+  to the repo. Treat it with the same discipline as any other production secret
+  (restrictive file ACLs, no logging, rotated via the KEK-rotation runbook — G-8,
+  not yet built).
+- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
+  that both central nodes mount, so every node's `SqliteSecretsStoreMigrator` opens
+  and reads the same rows.
+
+Writes to the store are rare and human-driven — an operator using the
+`/admin/secrets` UI (G-6) or the `ZB.MOM.WW.Secrets` CLI on one node — while reads
+happen on every boot and on the `ResolveCacheTtl` refresh cycle on both nodes. The
+access pattern is read-mostly / effectively single-writer, which is what makes a
+shared SQLite volume viable as an interim posture (see caveat below).
+
+## How it's delivered (do NOT commit these values)
+
+The File-KEK + shared-store posture is supplied per-node at deployment time —
+**never** by editing the committed `appsettings.json` or
+`appsettings.Central.json`. Two acceptable delivery mechanisms:
+
+**Option A — environment variable overrides** (Windows Service / NSSM env block,
+container `env_file`, etc.), applied identically on `central-a` and `central-b`:
+
+```
+# production deployment — do not commit to the dev appsettings
+Secrets__MasterKey__Source=File
+Secrets__MasterKey__FilePath=/run/secrets/scadabridge-master.key
+Secrets__SqlitePath=/shared/secrets/scadabridge-secrets.db
+```
+
+**Option B — a production-only config layer** that is *not* the committed dev
+base (e.g. an untracked `appsettings.Production.json` deployed alongside the
+binaries, or an orchestrator-injected config mount):
+
+```jsonc
+// production deployment — do not commit to the dev appsettings
+{
+  "Secrets": {
+    "MasterKey": {
+      "Source": "File",
+      "FilePath": "/run/secrets/scadabridge-master.key"
+    },
+    "SqlitePath": "/shared/secrets/scadabridge-secrets.db"
+  }
+}
+```
+
+Either way, the file/path referenced must exist and be identical on every central
+node **before** that node boots — the expander runs unconditionally and will throw
+(`SecretNotFoundException` / migration failure) if the store or key is missing.
+
+## Caveat: SQLite over a shared volume is not real replication
+
+SQLite's file-locking model does not tolerate concurrent multi-writer access well
+over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
+block volume only one writer should be active at a time). The interim posture
+above is acceptable because:
+
+- Reads dominate (every boot + cache-refresh cycle on both nodes).
+- Writes are rare, human-initiated, and effectively single-writer in practice
+  (an operator runs the CLI/UI against one node at a time).
+
+It is **not** a substitute for real replication, and it is not safe if both nodes
+attempt concurrent writes. Do not build automation that writes secrets from both
+central nodes simultaneously.
+
+## Data Protection is independent — do not touch it here
+
+ScadaBridge's cookie/session and hub-token protection already has its own
+clustered-key story: `AddDataProtection().PersistKeysToDbContext<ScadaBridgeDbContext>()`
+(see `docs/components/Security.md`), which shares the Data Protection key ring
+across both central nodes via the existing MS SQL `ConfigurationDb`. That mechanism
+is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption (KEK + SQLite store) and
+must **not** be reconfigured as part of secrets-adoption work — doing so risks
+invalidating active sessions/cookies for an unrelated reason.
+
+## The G-7 hand-off
+
+The posture above is an interim, ops-only workaround. The long-term shape,
+tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is one of:
+
+- A ConfigDb-backed `ISecretStore` — mirroring the pattern ScadaBridge already
+  uses for the Data Protection key ring (`PersistKeysToDbContext`), giving both
+  central nodes a single MS SQL-backed source of truth for secret rows instead of
+  a shared SQLite file; or
+- The `ZB.MOM.WW.Secrets.Akka` replicator (LWW + anti-entropy resync +
+  tombstones) referenced in the library's SPEC.
+
+Both require new library code that does not exist yet. This runbook's posture is
+the bridge until one of those lands.
+
+## Dev/test/default posture (unchanged)
+
+The committed default in `appsettings.json` is:
+
+```json
+"Secrets": {
+  "SqlitePath": "scadabridge-secrets.db",
+  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
+  "RunMigrationsOnStartup": true,
+  "ResolveCacheTtl": "00:00:30"
+}
+```
+
+This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
+path is relative to the working directory, so local dev and the
+`WebApplicationFactory<Program>` `Host.Tests` boot cleanly with no external mount.
+Local dev and docker-compose environments supply concrete secret values via the
+**whole-key environment override** (e.g. `ScadaBridge__Database__ConfigurationDb`),
+which bypasses `${secret:...}` resolution entirely, so the expander is effectively
+a no-op there. The File-KEK + shared-volume posture in this runbook applies only
+to real clustered production deployments of the central pair — it must never be
+baked into the committed dev base, because the expander runs unconditionally at
+every Central boot and would break dev/CI if pointed at a nonexistent `/shared`
+mount.

src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json

@@ -1,5 +1,6 @@
 {
   "ScadaBridge": {
+    "_secretsClusterPosture": "Central pair KEK/store posture (production): see docs/operations/2026-07-16-secrets-clustered-master-key.md. Committed default is Source=Environment + relative SqlitePath (dev-safe); production uses a File KEK + shared store volume delivered out-of-band. Do NOT hardcode /shared paths here — the pre-host expander migrates the store at every Central boot and would break dev/test.",
     "_nodeName": "Host-018: NodeName stamps SourceNode on AuditLog/Notifications/SiteCalls rows (CLAUDE.md 'Centralized Audit Log' decision) and backs IX_AuditLog_Node_Occurred. Convention: 'central-a'/'central-b' for central nodes, 'node-a'/'node-b' for site nodes. Override per-node in multi-node deployments (the docker per-node configs do this). When left at the default below, single-node dev rows are stamped with 'central-a'; an empty value normalises to a NULL SourceNode.",
     "Node": {
       "Role": "Central",