docs(secrets): clustered master-key posture runbook + Central pointer (ScadaBridge G-5 T8)
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,156 @@
|
||||
# Secrets: Clustered Master-Key Posture (Central Pair)
|
||||
|
||||
## Purpose
|
||||
|
||||
`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
|
||||
pre-host expander that runs at every Central-role boot, before `StartupValidator`
|
||||
(Host-003). It reads rows from an envelope-encrypted SQLite store
|
||||
(`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
|
||||
`Secrets:MasterKey:Source`.
|
||||
|
||||
ScadaBridge's central role is an Akka.NET-clustered **pair** (`central-a` /
|
||||
`central-b`, see [failover-procedures.md](./failover-procedures.md)) — either node
|
||||
can be active, and both boot independently. This runbook covers what a production
|
||||
deployment of the central pair (and any site node that is ever given a `${secret:}`
|
||||
token) needs so that secret resolution behaves identically no matter which node is
|
||||
running. It does **not** change any code; it is an operations/deployment posture,
|
||||
delivered out-of-band from the committed config.
|
||||
|
||||
## The two hard requirements
|
||||
|
||||
For the pre-host expander to resolve the same plaintext secret on every node:
|
||||
|
||||
1. **Identical KEK on every node.** All central nodes must unwrap the store with
|
||||
the exact same master key. A per-node KEK (e.g. two different DPAPI-protected
|
||||
keys, one per Windows box) would make each node decrypt the *other* node's
|
||||
ciphertext rows to garbage.
|
||||
2. **Identical store rows on every node.** All central nodes must read the same
|
||||
SQLite database (same file, or a replicated/shared copy with the same rows) —
|
||||
not two independently-seeded stores that happen to use the same KEK.
|
||||
|
||||
`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
|
||||
— there is no built-in cross-node replication today. Meeting both requirements in
|
||||
production is a deployment concern, covered below.
|
||||
|
||||
## Recommended interim posture (G-5)
|
||||
|
||||
Until real replication exists (G-7, below), the recommended production posture is:
|
||||
|
||||
- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
|
||||
key file that is identical on every central node** — a base64-encoded 32-byte
|
||||
key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed to each
|
||||
node's filesystem/secret-mount by the deployment tooling, and **never committed**
|
||||
to the repo. Treat it with the same discipline as any other production secret
|
||||
(restrictive file ACLs, no logging, rotated via the KEK-rotation runbook — G-8,
|
||||
not yet built).
|
||||
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
|
||||
that both central nodes mount, so every node's `SqliteSecretsStoreMigrator` opens
|
||||
and reads the same rows.
|
||||
|
||||
Writes to the store are rare and human-driven — an operator using the
|
||||
`/admin/secrets` UI (G-6) or the `ZB.MOM.WW.Secrets` CLI on one node — while reads
|
||||
happen on every boot and on the `ResolveCacheTtl` refresh cycle on both nodes. The
|
||||
access pattern is read-mostly / effectively single-writer, which is what makes a
|
||||
shared SQLite volume viable as an interim posture (see caveat below).
|
||||
|
||||
## How it's delivered (do NOT commit these values)
|
||||
|
||||
The File-KEK + shared-store posture is supplied per-node at deployment time —
|
||||
**never** by editing the committed `appsettings.json` or
|
||||
`appsettings.Central.json`. Two acceptable delivery mechanisms:
|
||||
|
||||
**Option A — environment variable overrides** (Windows Service / NSSM env block,
|
||||
container `env_file`, etc.), applied identically on `central-a` and `central-b`:
|
||||
|
||||
```
|
||||
# production deployment — do not commit to the dev appsettings
|
||||
Secrets__MasterKey__Source=File
|
||||
Secrets__MasterKey__FilePath=/run/secrets/scadabridge-master.key
|
||||
Secrets__SqlitePath=/shared/secrets/scadabridge-secrets.db
|
||||
```
|
||||
|
||||
**Option B — a production-only config layer** that is *not* the committed dev
|
||||
base (e.g. an untracked `appsettings.Production.json` deployed alongside the
|
||||
binaries, or an orchestrator-injected config mount):
|
||||
|
||||
```jsonc
|
||||
// production deployment — do not commit to the dev appsettings
|
||||
{
|
||||
"Secrets": {
|
||||
"MasterKey": {
|
||||
"Source": "File",
|
||||
"FilePath": "/run/secrets/scadabridge-master.key"
|
||||
},
|
||||
"SqlitePath": "/shared/secrets/scadabridge-secrets.db"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Either way, the file/path referenced must exist and be identical on every central
|
||||
node **before** that node boots — the expander runs unconditionally and will throw
|
||||
(`SecretNotFoundException` / migration failure) if the store or key is missing.
|
||||
|
||||
## Caveat: SQLite over a shared volume is not real replication
|
||||
|
||||
SQLite's file-locking model does not tolerate concurrent multi-writer access well
|
||||
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
|
||||
block volume only one writer should be active at a time). The interim posture
|
||||
above is acceptable because:
|
||||
|
||||
- Reads dominate (every boot + cache-refresh cycle on both nodes).
|
||||
- Writes are rare, human-initiated, and effectively single-writer in practice
|
||||
(an operator runs the CLI/UI against one node at a time).
|
||||
|
||||
It is **not** a substitute for real replication, and it is not safe if both nodes
|
||||
attempt concurrent writes. Do not build automation that writes secrets from both
|
||||
central nodes simultaneously.
|
||||
|
||||
## Data Protection is independent — do not touch it here
|
||||
|
||||
ScadaBridge's cookie/session and hub-token protection already has its own
|
||||
clustered-key story: `AddDataProtection().PersistKeysToDbContext<ScadaBridgeDbContext>()`
|
||||
(see `docs/components/Security.md`), which shares the Data Protection key ring
|
||||
across both central nodes via the existing MS SQL `ConfigurationDb`. That mechanism
|
||||
is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption (KEK + SQLite store) and
|
||||
must **not** be reconfigured as part of secrets-adoption work — doing so risks
|
||||
invalidating active sessions/cookies for an unrelated reason.
|
||||
|
||||
## The G-7 hand-off
|
||||
|
||||
The posture above is an interim, ops-only workaround. The long-term shape,
|
||||
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is one of:
|
||||
|
||||
- A ConfigDb-backed `ISecretStore` — mirroring the pattern ScadaBridge already
|
||||
uses for the Data Protection key ring (`PersistKeysToDbContext`), giving both
|
||||
central nodes a single MS SQL-backed source of truth for secret rows instead of
|
||||
a shared SQLite file; or
|
||||
- The `ZB.MOM.WW.Secrets.Akka` replicator (LWW + anti-entropy resync +
|
||||
tombstones) referenced in the library's SPEC.
|
||||
|
||||
Both require new library code that does not exist yet. This runbook's posture is
|
||||
the bridge until one of those lands.
|
||||
|
||||
## Dev/test/default posture (unchanged)
|
||||
|
||||
The committed default in `appsettings.json` is:
|
||||
|
||||
```json
|
||||
"Secrets": {
|
||||
"SqlitePath": "scadabridge-secrets.db",
|
||||
"MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
|
||||
"RunMigrationsOnStartup": true,
|
||||
"ResolveCacheTtl": "00:00:30"
|
||||
}
|
||||
```
|
||||
|
||||
This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
|
||||
path is relative to the working directory, so local dev and the
|
||||
`WebApplicationFactory<Program>` `Host.Tests` boot cleanly with no external mount.
|
||||
Local dev and docker-compose environments supply concrete secret values via the
|
||||
**whole-key environment override** (e.g. `ScadaBridge__Database__ConfigurationDb`),
|
||||
which bypasses `${secret:...}` resolution entirely, so the expander is effectively
|
||||
a no-op there. The File-KEK + shared-volume posture in this runbook applies only
|
||||
to real clustered production deployments of the central pair — it must never be
|
||||
baked into the committed dev base, because the expander runs unconditionally at
|
||||
every Central boot and would break dev/CI if pointed at a nonexistent `/shared`
|
||||
mount.
|
||||
@@ -1,5 +1,6 @@
|
||||
{
|
||||
"ScadaBridge": {
|
||||
"_secretsClusterPosture": "Central pair KEK/store posture (production): see docs/operations/2026-07-16-secrets-clustered-master-key.md. Committed default is Source=Environment + relative SqlitePath (dev-safe); production uses a File KEK + shared store volume delivered out-of-band. Do NOT hardcode /shared paths here — the pre-host expander migrates the store at every Central boot and would break dev/test.",
|
||||
"_nodeName": "Host-018: NodeName stamps SourceNode on AuditLog/Notifications/SiteCalls rows (CLAUDE.md 'Centralized Audit Log' decision) and backs IX_AuditLog_Node_Occurred. Convention: 'central-a'/'central-b' for central nodes, 'node-a'/'node-b' for site nodes. Override per-node in multi-node deployments (the docker per-node configs do this). When left at the default below, single-node dev rows are stamped with 'central-a'; an empty value normalises to a NULL SourceNode.",
|
||||
"Node": {
|
||||
"Role": "Central",
|
||||
|
||||
Reference in New Issue
Block a user