ScadaBridge/docs/operations/2026-07-16-secrets-clustered-master-key.md

# Secrets: Clustered Master-Key Posture (Central Pair)

> **Update 2026-08-07 (truth sweep):** two claims below are stale.
> (1) "there is no built-in cross-node replication today" — cross-node replication
> has since shipped: ScadaBridge adopted opt-in **SQL-Server hub replication** for
> the host secret store (commit `8e12f994`, "feat(secrets): opt-in SQL-Server hub
> replication for the host secret store"; `ZB.MOM.WW.Secrets` 0.2.x Replicator
> packages), covered by the shared library's clustered-secrets runbook
> (`scadaproj/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`).
> (2) the G-8 KEK-rotation runbook is no longer "not yet built" — it ships with the
> shared library at `scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`
> (lib 0.1.3, `Rewrap`/`rewrap-all`). The interim shared-volume posture below
> remains valid but is no longer the only option.

## Purpose

`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
pre-host expander that runs at every Central-role boot, before `StartupValidator`
(Host-003). It reads rows from an envelope-encrypted SQLite store
(`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
`Secrets:MasterKey:Source`.

ScadaBridge's central role is an Akka.NET-clustered **pair** (`central-a` /
`central-b`, see [failover-procedures.md](./failover-procedures.md)) — either node
can be active, and both boot independently. This runbook covers what a production
deployment of the central pair (and any site node that is ever given a `${secret:}`
token) needs so that secret resolution behaves identically no matter which node is
running. It does **not** change any code; it is an operations/deployment posture,
delivered out-of-band from the committed config.

## The two hard requirements

For the pre-host expander to resolve the same plaintext secret on every node:

1. **Identical KEK on every node.** All central nodes must unwrap the store with
   the exact same master key. A per-node KEK (e.g. two different DPAPI-protected
   keys, one per Windows box) would make each node decrypt the *other* node's
   ciphertext rows to garbage.
2. **Identical store rows on every node.** All central nodes must read the same
   SQLite database (same file, or a replicated/shared copy with the same rows) —
   not two independently-seeded stores that happen to use the same KEK.

`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
— there is no built-in cross-node replication today. Meeting both requirements in
production is a deployment concern, covered below.

## Recommended interim posture (G-5)

Until real replication exists (G-7, below), the recommended production posture is:

- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
  key file that is identical on every central node** — a base64-encoded 32-byte
  key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed to each
  node's filesystem/secret-mount by the deployment tooling, and **never committed**
  to the repo. Treat it with the same discipline as any other production secret
  (restrictive file ACLs, no logging, rotated via the KEK-rotation runbook — G-8,
  not yet built).
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
  that both central nodes mount, so every node's `SqliteSecretsStoreMigrator` opens
  and reads the same rows.

Writes to the store are rare and human-driven — an operator using the
`/admin/secrets` UI (G-6) or the `ZB.MOM.WW.Secrets` CLI on one node — while reads
happen on every boot and on the `ResolveCacheTtl` refresh cycle on both nodes. The
access pattern is read-mostly / effectively single-writer, which is what makes a
shared SQLite volume viable as an interim posture (see caveat below).

## How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time —
**never** by editing the committed `appsettings.json` or
`appsettings.Central.json`. Two acceptable delivery mechanisms:

**Option A — environment variable overrides** (Windows Service / NSSM env block,
container `env_file`, etc.), applied identically on `central-a` and `central-b`:

```
# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/scadabridge-master.key
Secrets__SqlitePath=/shared/secrets/scadabridge-secrets.db
```

**Option B — a production-only config layer** that is *not* the committed dev
base (e.g. an untracked `appsettings.Production.json` deployed alongside the
binaries, or an orchestrator-injected config mount):

```jsonc
// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/scadabridge-master.key"
    },
    "SqlitePath": "/shared/secrets/scadabridge-secrets.db"
  }
}
```

Either way, the file/path referenced must exist and be identical on every central
node **before** that node boots — the expander runs unconditionally and will throw
(`SecretNotFoundException` / migration failure) if the store or key is missing.

## Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
block volume only one writer should be active at a time). The interim posture
above is acceptable because:

- Reads dominate (every boot + cache-refresh cycle on both nodes).
- Writes are rare, human-initiated, and effectively single-writer in practice
  (an operator runs the CLI/UI against one node at a time).

It is **not** a substitute for real replication, and it is not safe if both nodes
attempt concurrent writes. Do not build automation that writes secrets from both
central nodes simultaneously.

## Data Protection is independent — do not touch it here

ScadaBridge's cookie/session and hub-token protection already has its own
clustered-key story: `AddDataProtection().PersistKeysToDbContext<ScadaBridgeDbContext>()`
(see `docs/components/Security.md`), which shares the Data Protection key ring
across both central nodes via the existing MS SQL `ConfigurationDb`. That mechanism
is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption (KEK + SQLite store) and
must **not** be reconfigured as part of secrets-adoption work — doing so risks
invalidating active sessions/cookies for an unrelated reason.

## The G-7 hand-off

The posture above is an interim, ops-only workaround. The long-term shape,
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is one of:

- A ConfigDb-backed `ISecretStore` — mirroring the pattern ScadaBridge already
  uses for the Data Protection key ring (`PersistKeysToDbContext`), giving both
  central nodes a single MS SQL-backed source of truth for secret rows instead of
  a shared SQLite file; or
- The `ZB.MOM.WW.Secrets.Akka` replicator (LWW + anti-entropy resync +
  tombstones) referenced in the library's SPEC.

Both require new library code that does not exist yet. This runbook's posture is
the bridge until one of those lands.

## Dev/test/default posture (unchanged)

The committed default in `appsettings.json` is:

```json
"Secrets": {
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true,
  "ResolveCacheTtl": "00:00:30"
}
```

**`SqlitePath` is deliberately absent** (changed with `ZB.MOM.WW.Secrets` 0.6.2, 2026-08-11).
That version validates the setting at startup and rejects it unless it is both **absolute** and
**outside the application content root** — the two rules are independent, and the path that
destroyed the MxGateway key store on 2026-08-09 was absolute. The former committed value
(`"scadabridge-secrets.db"`) was relative and now fails the boot, so it was removed rather than
replaced: an unset path takes the library's absolute per-user `LocalApplicationData` default,
which satisfies both rules everywhere without a mount.

This is dev-safe: `Source=Environment` needs no filesystem key, and the per-user default is
writable, so local dev and the `WebApplicationFactory<Program>` `Host.Tests` boot cleanly with no
external mount.

**Containerized deployments need care.** The rule is lexical, so a path under the content root is
rejected even when it is a *bind mount* that an image rebuild never touches. The docker rig hit
exactly this: `/app/data/scadabridge-secrets.db` with content root `/app`. The fix was to mount
each node's existing host directory a **second** time at `/data` and point `Secrets__SqlitePath`
there — same underlying directory, so the store is found unmoved and nothing needs migrating.
Local dev and docker-compose environments supply concrete secret values via the
**whole-key environment override** (e.g. `ScadaBridge__Database__ConfigurationDb`),
which bypasses `${secret:...}` resolution entirely, so the expander is effectively
a no-op there. The File-KEK + shared-volume posture in this runbook applies only
to real clustered production deployments of the central pair — it must never be
baked into the committed dev base, because the expander runs unconditionally at
every Central boot and would break dev/CI if pointed at a nonexistent `/shared`
mount.