Files

T

Joseph Doherty fdc6b0c2bb chore(secrets): adopt ZB.MOM.WW.Secrets 0.6.2 and close the pre-host guard gap

0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.

The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.

The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.

Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.

2026-08-11 09:16:31 -04:00

9.2 KiB

Raw Blame History

Secrets: Clustered Master-Key Posture (Central Pair)

Update 2026-08-07 (truth sweep): two claims below are stale. (1) "there is no built-in cross-node replication today" — cross-node replication has since shipped: ScadaBridge adopted opt-in SQL-Server hub replication for the host secret store (commit 8e12f994, "feat(secrets): opt-in SQL-Server hub replication for the host secret store"; ZB.MOM.WW.Secrets 0.2.x Replicator packages), covered by the shared library's clustered-secrets runbook (scadaproj/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md). (2) the G-8 KEK-rotation runbook is no longer "not yet built" — it ships with the shared library at scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md (lib 0.1.3, Rewrap/rewrap-all). The interim shared-volume posture below remains valid but is no longer the only option.

Purpose

ZB.MOM.WW.Secrets resolves ${secret:...} tokens in appsettings.*.json via a pre-host expander that runs at every Central-role boot, before StartupValidator (Host-003). It reads rows from an envelope-encrypted SQLite store (Secrets:SqlitePath) unwrapped with a key-encryption key (KEK) sourced per Secrets:MasterKey:Source.

ScadaBridge's central role is an Akka.NET-clustered pair (central-a / central-b, see failover-procedures.md) — either node can be active, and both boot independently. This runbook covers what a production deployment of the central pair (and any site node that is ever given a ${secret:} token) needs so that secret resolution behaves identically no matter which node is running. It does not change any code; it is an operations/deployment posture, delivered out-of-band from the committed config.

The two hard requirements

For the pre-host expander to resolve the same plaintext secret on every node:

Identical KEK on every node. All central nodes must unwrap the store with the exact same master key. A per-node KEK (e.g. two different DPAPI-protected keys, one per Windows box) would make each node decrypt the other node's ciphertext rows to garbage.
Identical store rows on every node. All central nodes must read the same SQLite database (same file, or a replicated/shared copy with the same rows) — not two independently-seeded stores that happen to use the same KEK.

ZB.MOM.WW.Secrets ships a SQLite-only ISecretStore with a NoOpSecretReplicator — there is no built-in cross-node replication today. Meeting both requirements in production is a deployment concern, covered below.

How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time — never by editing the committed appsettings.json or appsettings.Central.json. Two acceptable delivery mechanisms:

Option A — environment variable overrides (Windows Service / NSSM env block, container env_file, etc.), applied identically on central-a and central-b:

# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/scadabridge-master.key
Secrets__SqlitePath=/shared/secrets/scadabridge-secrets.db

Option B — a production-only config layer that is not the committed dev base (e.g. an untracked appsettings.Production.json deployed alongside the binaries, or an orchestrator-injected config mount):

// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/scadabridge-master.key"
    },
    "SqlitePath": "/shared/secrets/scadabridge-secrets.db"
  }
}

Either way, the file/path referenced must exist and be identical on every central node before that node boots — the expander runs unconditionally and will throw (SecretNotFoundException / migration failure) if the store or key is missing.

Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well over network filesystems (SMB/NFS locking is unreliable, and even on a clustered block volume only one writer should be active at a time). The interim posture above is acceptable because:

Reads dominate (every boot + cache-refresh cycle on both nodes).
Writes are rare, human-initiated, and effectively single-writer in practice (an operator runs the CLI/UI against one node at a time).

It is not a substitute for real replication, and it is not safe if both nodes attempt concurrent writes. Do not build automation that writes secrets from both central nodes simultaneously.

Data Protection is independent — do not touch it here

ScadaBridge's cookie/session and hub-token protection already has its own clustered-key story: AddDataProtection().PersistKeysToDbContext<ScadaBridgeDbContext>() (see docs/components/Security.md), which shares the Data Protection key ring across both central nodes via the existing MS SQL ConfigurationDb. That mechanism is unrelated to ZB.MOM.WW.Secrets' envelope encryption (KEK + SQLite store) and must not be reconfigured as part of secrets-adoption work — doing so risks invalidating active sessions/cookies for an unrelated reason.

The G-7 hand-off

The posture above is an interim, ops-only workaround. The long-term shape, tracked as G-7 in scadaproj/components/secrets/GAPS.md, is one of:

A ConfigDb-backed ISecretStore — mirroring the pattern ScadaBridge already uses for the Data Protection key ring (PersistKeysToDbContext), giving both central nodes a single MS SQL-backed source of truth for secret rows instead of a shared SQLite file; or
The ZB.MOM.WW.Secrets.Akka replicator (LWW + anti-entropy resync + tombstones) referenced in the library's SPEC.

Both require new library code that does not exist yet. This runbook's posture is the bridge until one of those lands.

Dev/test/default posture (unchanged)

The committed default in appsettings.json is:

"Secrets": {
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true,
  "ResolveCacheTtl": "00:00:30"
}

SqlitePath is deliberately absent (changed with ZB.MOM.WW.Secrets 0.6.2, 2026-08-11). That version validates the setting at startup and rejects it unless it is both absolute and outside the application content root — the two rules are independent, and the path that destroyed the MxGateway key store on 2026-08-09 was absolute. The former committed value ("scadabridge-secrets.db") was relative and now fails the boot, so it was removed rather than replaced: an unset path takes the library's absolute per-user LocalApplicationData default, which satisfies both rules everywhere without a mount.

This is dev-safe: Source=Environment needs no filesystem key, and the per-user default is writable, so local dev and the WebApplicationFactory<Program> Host.Tests boot cleanly with no external mount.

Containerized deployments need care. The rule is lexical, so a path under the content root is rejected even when it is a bind mount that an image rebuild never touches. The docker rig hit exactly this: /app/data/scadabridge-secrets.db with content root /app. The fix was to mount each node's existing host directory a second time at /data and point Secrets__SqlitePath there — same underlying directory, so the store is found unmoved and nothing needs migrating. Local dev and docker-compose environments supply concrete secret values via the whole-key environment override (e.g. ScadaBridge__Database__ConfigurationDb), which bypasses ${secret:...} resolution entirely, so the expander is effectively a no-op there. The File-KEK + shared-volume posture in this runbook applies only to real clustered production deployments of the central pair — it must never be baked into the committed dev base, because the expander runs unconditionally at every Central boot and would break dev/CI if pointed at a nonexistent /shared mount.

9.2 KiB Raw Blame History