Secrets: Clustered Master-Key Posture (Central Pair)

Purpose

ZB.MOM.WW.Secrets resolves ${secret:...} tokens in appsettings.*.json via a pre-host expander that runs at every Central-role boot, before StartupValidator (Host-003). It reads rows from an envelope-encrypted SQLite store (Secrets:SqlitePath) unwrapped with a key-encryption key (KEK) sourced per Secrets:MasterKey:Source.

ScadaBridge's central role is an Akka.NET-clustered pair (central-a / central-b, see failover-procedures.md) — either node can be active, and both boot independently. This runbook covers what a production deployment of the central pair (and any site node that is ever given a ${secret:} token) needs so that secret resolution behaves identically no matter which node is running. It does not change any code; it is an operations/deployment posture, delivered out-of-band from the committed config.

The two hard requirements

For the pre-host expander to resolve the same plaintext secret on every node:

Identical KEK on every node. All central nodes must unwrap the store with the exact same master key. A per-node KEK (e.g. two different DPAPI-protected keys, one per Windows box) would make each node decrypt the other node's ciphertext rows to garbage.
Identical store rows on every node. All central nodes must read the same SQLite database (same file, or a replicated/shared copy with the same rows) — not two independently-seeded stores that happen to use the same KEK.

ZB.MOM.WW.Secrets ships a SQLite-only ISecretStore with a NoOpSecretReplicator — there is no built-in cross-node replication today. Meeting both requirements in production is a deployment concern, covered below.

How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time — never by editing the committed appsettings.json or appsettings.Central.json. Two acceptable delivery mechanisms:

Option A — environment variable overrides (Windows Service / NSSM env block, container env_file, etc.), applied identically on central-a and central-b:

# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/scadabridge-master.key
Secrets__SqlitePath=/shared/secrets/scadabridge-secrets.db

Option B — a production-only config layer that is not the committed dev base (e.g. an untracked appsettings.Production.json deployed alongside the binaries, or an orchestrator-injected config mount):

// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/scadabridge-master.key"
    },
    "SqlitePath": "/shared/secrets/scadabridge-secrets.db"
  }
}

Either way, the file/path referenced must exist and be identical on every central node before that node boots — the expander runs unconditionally and will throw (SecretNotFoundException / migration failure) if the store or key is missing.

Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well over network filesystems (SMB/NFS locking is unreliable, and even on a clustered block volume only one writer should be active at a time). The interim posture above is acceptable because:

Reads dominate (every boot + cache-refresh cycle on both nodes).
Writes are rare, human-initiated, and effectively single-writer in practice (an operator runs the CLI/UI against one node at a time).

It is not a substitute for real replication, and it is not safe if both nodes attempt concurrent writes. Do not build automation that writes secrets from both central nodes simultaneously.

Data Protection is independent — do not touch it here

ScadaBridge's cookie/session and hub-token protection already has its own clustered-key story: AddDataProtection().PersistKeysToDbContext<ScadaBridgeDbContext>() (see docs/components/Security.md), which shares the Data Protection key ring across both central nodes via the existing MS SQL ConfigurationDb. That mechanism is unrelated to ZB.MOM.WW.Secrets' envelope encryption (KEK + SQLite store) and must not be reconfigured as part of secrets-adoption work — doing so risks invalidating active sessions/cookies for an unrelated reason.

The G-7 hand-off

The posture above is an interim, ops-only workaround. The long-term shape, tracked as G-7 in scadaproj/components/secrets/GAPS.md, is one of:

A ConfigDb-backed ISecretStore — mirroring the pattern ScadaBridge already uses for the Data Protection key ring (PersistKeysToDbContext), giving both central nodes a single MS SQL-backed source of truth for secret rows instead of a shared SQLite file; or
The ZB.MOM.WW.Secrets.Akka replicator (LWW + anti-entropy resync + tombstones) referenced in the library's SPEC.

Both require new library code that does not exist yet. This runbook's posture is the bridge until one of those lands.

Dev/test/default posture (unchanged)

The committed default in appsettings.json is:

"Secrets": {
  "SqlitePath": "scadabridge-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true,
  "ResolveCacheTtl": "00:00:30"
}

This is dev-safe: Source=Environment needs no filesystem key, and the SQLite path is relative to the working directory, so local dev and the WebApplicationFactory<Program> Host.Tests boot cleanly with no external mount. Local dev and docker-compose environments supply concrete secret values via the whole-key environment override (e.g. ScadaBridge__Database__ConfigurationDb), which bypasses ${secret:...} resolution entirely, so the expander is effectively a no-op there. The File-KEK + shared-volume posture in this runbook applies only to real clustered production deployments of the central pair — it must never be baked into the committed dev base, because the expander runs unconditionally at every Central boot and would break dev/CI if pointed at a nonexistent /shared mount.

7.3 KiB Raw Blame History