lmxopcua/docs/operations/2026-07-16-secrets-clustered-master-key.md

# Secrets: Clustered Master-Key Posture (All Roles)

## Purpose

`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node
boot, before the host is built. It reads rows from an envelope-encrypted SQLite
store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also
registered unconditionally, independent of node role.

OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })`
in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`):
`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a
fused admin+driver node, or a driver-only node). This runbook covers what a
production deployment needs so that secret resolution behaves identically on
**every node regardless of role** — not just admin nodes. That matters here more
than it might elsewhere: the pre-host expander and the runtime resolver both run
unconditionally on driver-only nodes too, and driver-only nodes are the ones that
will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at
runtime, not just at boot. It does **not** change any code; it is an
operations/deployment posture, delivered out-of-band from the committed config.

## The two hard requirements

For the pre-host expander (and the runtime resolver) to resolve the same
plaintext secret on every node, no matter its role:

1. **Identical KEK on every node.** All nodes — admin, driver, and any fused
   combination — must unwrap the store with the exact same master key. A
   per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
   decrypt every *other* node's ciphertext rows to garbage.
2. **The same store rows on every node of a pair.** Nodes must read the same rows —
   not independently-seeded stores that merely share a KEK. Either mount one SQLite file
   (see "Alternative: shared volume" below) or, preferred, turn on Akka secret replication
   so each pair converges its own rows.

The base `ZB.MOM.WW.Secrets` store is a plain local SQLite `ISecretStore` whose default
replicator is `NoOpSecretReplicator`: **with `Secrets:Replication:Enabled=false` (the
default) nothing replicates and each node's store is private.** That is what this page
used to describe as the only option. It is no longer: pair replication exists
(`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`, lib pin `0.6.2` in `Directory.Packages.props`;
gated by `SecretsRegistration`, see below).

## Recommended posture: pair-local Akka replication

Opt in with `Secrets:Replication:Enabled=true` on **every node** (default `false`, so absent
config means a plain local SQLite store). `SecretsRegistration.AddOtOpcUaSecrets`
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`) then registers
`AddZbSecretsAkkaReplication` **instead of** `AddZbSecrets` (the two are alternatives, never
additive) and a `SecretReplicationStarter` hosted service resolves the store at boot so a
node that never reads a secret still joins anti-entropy. Keys: `Secrets:Replication:Enabled`,
`:AnnounceInterval` (default `00:00:30`; each node publishes a name/revision/timestamp/tombstone
manifest per interval, no ciphertext), `:ActorName` (`zb-secret-replication`).

- **Scope is PAIR-LOCAL.** Since mesh Phase 6 each application cluster is its own 2-node Akka
  mesh and replication rides that mesh's DistributedPubSub, so a secret converges **within its
  pair and nowhere else**. There is no fleet-wide sync (site nodes have no SQL hub). Seed each
  pair independently: central via `/admin/secrets`, driver-only site pairs (no AdminUI) via the
  secrets CLI against one node's store, or a ciphertext-only bundle import. See
  `docs/Redundancy.md` §"Secrets replication is pair-local".
- **Both nodes of a pair must share the same KEK** — rows replicate verbatim (ciphertext, wrapped
  DEK and `kek_id` untouched), never re-encrypted.
- **Deletes replicate** as soft tombstones (last-writer-wins on `updated_utc`/`revision`).
- **Re-wraps do not replicate** (a re-wrap preserves `revision`/`updated_utc`, so it is invisible to
  last-writer-wins). KEK rotation therefore runs against each node's store — see the next section.
- The store should sit on a **persistent per-node path** (`Secrets:SqlitePath`, e.g. the node's
  `/app/data` volume in containers); the relative default lands inside the image layer and is lost on
  recreate.
- Evidence: live-gated PASS 4/4 on the three-mesh docker-dev rig (convergence <= 7 s, no cross-pair
  leakage in any direction, delete < 1 s, clean restart without resurrection) —
  `docs/plans/2026-08-07-secrets-pair-local-live-gate.md`. Replication stays default-off in the product.

Whichever store layout is used, the KEK is delivered per node as before:
**`Secrets:MasterKey:Source = File`** with `FilePath` pointing at a read-only base64 32-byte key
(`openssl rand -base64 32`), distributed identically to every node of the pair by deployment tooling and
**never committed**. Writes are rare and human-driven (`/admin/secrets` on admin nodes, or the
CLI); reads happen at every node's boot and each `ResolveCacheTtl` refresh.

## Rotating the KEK

Rotation re-wraps each row's per-secret DEK under the new KEK (`wrapped_dek`, `wrap_nonce`,
`wrap_tag`, `kek_id` change; the sealed body, `revision` and `updated_utc` do not). The
authoritative operator procedure and troubleshooting table live with the library:
`scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`, driven by the `secret rewrap-all`
CLI verb (in that repo; not packed). The OtOpcUa-specific points:

1. **Back up every store first** (copy each node's SQLite file) — the backup is the complete undo.
2. **Stage the new KEK on every node of the pair** (same file path / env var as `Secrets:MasterKey`
   resolves) but do not restart yet. Never pass key values on a command line; use
   `--old-key-env VAR` / `--old-key-file PATH` and, if the new key is not the node's configured
   provider, `--new-key-file`.
3. **Run `secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY` once per store, i.e. on EACH node.**
   Because re-wraps do not replicate, rotating one node leaves its partner's rows on the old
   `kek_id`. With replication on, do not leave the pair half-rotated: a node whose rows carry a KEK
   it cannot unwrap fails closed (`Row wrapped by unknown KEK`) on resolve.
4. **Quiesce first**: stop (or drain) the node while its store is rewrapped. Rotating a pair
   node-by-node while the other node keeps serving works for resolves, but do not author secrets
   anywhere in the pair mid-rotation — a fresh write is wrapped under whichever KEK the writing node
   currently uses and replicates verbatim to a partner that may not yet hold that KEK.
5. **Cut over**: confirm `Secrets:MasterKey` resolves the new key on both nodes, start them, and
   confirm a `${secret:}` value resolves (the pre-host expander fails the boot loudly if not).
6. **Destroy the old key** only after both stores resolve cleanly. Re-running `rewrap-all` is
   idempotent (`alreadyCurrent`), so an interrupted pass is resumed by running it again.
7. Rollback: restore the step-1 backups on both nodes and revert the KEK source.

Pair rotation across **separate pairs** (central, site-a, site-b) is independent per pair; there is no
fleet-wide order.

## Alternative: shared SQLite volume (no replication)

The earlier interim posture — every node of a pair mounting one `Secrets:SqlitePath` on a shared
or replicated volume — still works for a deployment that keeps `Secrets:Replication:Enabled=false`:
reads dominate and writes are effectively single-writer. See the caveat below; it is not safe with
concurrent writers, and it is impractical for site pairs whose nodes share no volume.

## How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time —
**never** by editing the committed `appsettings.json` or the role-overlay files
(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`).
Those committed overlays are also consumed by local dev and the
`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path
into them would break every dev/test/CI boot. Two acceptable delivery
mechanisms instead:

**Option A — environment variable overrides** (Windows Service / NSSM env block,
container `env_file`, etc.), applied identically on every node regardless of role:

```
# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db
```

**Option B — a production-only config layer** that is *not* the committed dev
base (e.g. an untracked `appsettings.Production.json` deployed alongside the
binaries, or an orchestrator-injected config mount):

```jsonc
// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/otopcua-master.key"
    },
    "SqlitePath": "/shared/secrets/otopcua-secrets.db"
  }
}
```

Either way, the file/path referenced must exist and be identical on every node
**before** that node boots — the pre-host expander runs unconditionally on every
role and will throw (`SecretNotFoundException` / migration failure) if the store
or key is missing.

## Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
block volume only one writer should be active at a time). The shared-volume alternative
above is acceptable because:

- Reads dominate (every node's boot + cache-refresh cycle, across every role).
- Writes are rare, human-initiated, and effectively single-writer in practice
  (an operator runs the CLI/UI against one admin node at a time).

It is **not** a substitute for real replication (Akka pair replication, above), and it is not safe if multiple
nodes attempt concurrent writes. Do not build automation that writes secrets
from more than one node simultaneously.

## Data Protection is independent — do not touch it here

OtOpcUa's cookie/session protection already has its own clustered-key story:
`AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>()`
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`),
which shares the Data Protection key ring across every node via the existing
ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption
(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
work — doing so risks invalidating active sessions for an unrelated reason.

## The G-7 hand-off (status)

G-7 (`scadaproj/components/secrets/GAPS.md`) asked for real cross-node replication. It was
built in two flavours in the library (`ZB.MOM.WW.Secrets.Replicator.SqlServer` and
`.AkkaDotNet`, plus a gRPC replicator package). **OtOpcUa production uses the Akka pair-local
option** (the decision recorded in scadaproj#3), wired by `SecretsRegistration` as described above;
the ConfigDb-backed store this page used to anticipate is not what shipped here. Driver-only nodes
have no ConfigDb since mesh Phase 4, which is a further reason a ConfigDb secret store would not
reach them.

## Dev/test/default posture (unchanged)

The committed default in `appsettings.json` is:

```json
"Secrets": {
  "SqlitePath": "otopcua-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true
}
```

This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
path is relative to the working directory, so local dev, the role-overlay
appsettings (`appsettings.admin.json`, `appsettings.driver.json`,
`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration
tests all boot cleanly with no external mount. The File-KEK + shared-volume
posture in this runbook applies only to real clustered production deployments —
it must never be baked into the committed dev/role-overlay base, because the
expander runs unconditionally at every node boot (any role) and would break
dev/CI if pointed at a nonexistent `/shared` mount.