lmxopcua/docs/operations/2026-07-16-secrets-clustered-master-key.md

Secrets: Clustered Master-Key Posture (All Roles)

Purpose

ZB.MOM.WW.Secrets resolves ${secret:...} tokens in appsettings.*.json via a pre-host expander (SecretReferenceExpander.ExpandConfigurationAsync, wired in src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs) that runs at every OtOpcUa node boot, before the host is built. It reads rows from an envelope-encrypted SQLite store (Secrets:SqlitePath) unwrapped with a key-encryption key (KEK) sourced per Secrets:MasterKey:Source. The runtime ISecretResolver (AddZbSecrets) is also registered unconditionally, independent of node role.

OtOpcUa is Akka.NET-clustered (builder.Services.AddAkka("otopcua", (ab, sp) => { ... }) in Program.cs), with roles parsed by RoleParser (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs): admin, driver, dev. A node can carry any combination of these roles (e.g. a fused admin+driver node, or a driver-only node). This runbook covers what a production deployment needs so that secret resolution behaves identically on every node regardless of role — not just admin nodes. That matters here more than it might elsewhere: the pre-host expander and the runtime resolver both run unconditionally on driver-only nodes too, and driver-only nodes are the ones that will resolve Layer-B DriverConfig secret references (coming in Slice 2) at runtime, not just at boot. It does not change any code; it is an operations/deployment posture, delivered out-of-band from the committed config.

The two hard requirements

For the pre-host expander (and the runtime resolver) to resolve the same plaintext secret on every node, no matter its role:

1. Identical KEK on every node. All nodes — admin, driver, and any fused combination — must unwrap the store with the exact same master key. A per-node KEK (e.g. a per-box DPAPI-protected key) would make each node decrypt every other node's ciphertext rows to garbage.
2. The same store rows on every node of a pair. Nodes must read the same rows — not independently-seeded stores that merely share a KEK. Either mount one SQLite file (see "Alternative: shared volume" below) or, preferred, turn on Akka secret replication so each pair converges its own rows.

The base ZB.MOM.WW.Secrets store is a plain local SQLite ISecretStore whose default replicator is NoOpSecretReplicator: with Secrets:Replication:Enabled=false (the default) nothing replicates and each node's store is private. That is what this page used to describe as the only option. It is no longer: pair replication exists (ZB.MOM.WW.Secrets.Replicator.AkkaDotNet, lib pin 0.6.2 in Directory.Packages.props; gated by SecretsRegistration, see below).

Recommended posture: pair-local Akka replication

Opt in with Secrets:Replication:Enabled=true on every node (default false, so absent config means a plain local SQLite store). SecretsRegistration.AddOtOpcUaSecrets (src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs) then registers AddZbSecretsAkkaReplication instead of AddZbSecrets (the two are alternatives, never additive) and a SecretReplicationStarter hosted service resolves the store at boot so a node that never reads a secret still joins anti-entropy. Keys: Secrets:Replication:Enabled, :AnnounceInterval (default 00:00:30; each node publishes a name/revision/timestamp/tombstone manifest per interval, no ciphertext), :ActorName (zb-secret-replication).

• Scope is PAIR-LOCAL. Since mesh Phase 6 each application cluster is its own 2-node Akka mesh and replication rides that mesh's DistributedPubSub, so a secret converges within its pair and nowhere else. There is no fleet-wide sync (site nodes have no SQL hub). Seed each pair independently: central via /admin/secrets, driver-only site pairs (no AdminUI) via the secrets CLI against one node's store, or a ciphertext-only bundle import. See docs/Redundancy.md §"Secrets replication is pair-local".
• Both nodes of a pair must share the same KEK — rows replicate verbatim (ciphertext, wrapped DEK and kek_id untouched), never re-encrypted.
• Deletes replicate as soft tombstones (last-writer-wins on updated_utc/revision).
• Re-wraps do not replicate (a re-wrap preserves revision/updated_utc, so it is invisible to last-writer-wins). KEK rotation therefore runs against each node's store — see the next section.
• The store should sit on a persistent per-node path (Secrets:SqlitePath, e.g. the node's /app/data volume in containers); the relative default lands inside the image layer and is lost on recreate.
• Evidence: live-gated PASS 4/4 on the three-mesh docker-dev rig (convergence <= 7 s, no cross-pair leakage in any direction, delete < 1 s, clean restart without resurrection) — docs/plans/2026-08-07-secrets-pair-local-live-gate.md. Replication stays default-off in the product.

Whichever store layout is used, the KEK is delivered per node as before: Secrets:MasterKey:Source = File with FilePath pointing at a read-only base64 32-byte key (openssl rand -base64 32), distributed identically to every node of the pair by deployment tooling and never committed. Writes are rare and human-driven (/admin/secrets on admin nodes, or the CLI); reads happen at every node's boot and each ResolveCacheTtl refresh.

Rotating the KEK

Rotation re-wraps each row's per-secret DEK under the new KEK (wrapped_dek, wrap_nonce, wrap_tag, kek_id change; the sealed body, revision and updated_utc do not). The authoritative operator procedure and troubleshooting table live with the library: scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md, driven by the secret rewrap-all CLI verb (in that repo; not packed). The OtOpcUa-specific points:

1. Back up every store first (copy each node's SQLite file) — the backup is the complete undo.
2. Stage the new KEK on every node of the pair (same file path / env var as Secrets:MasterKey resolves) but do not restart yet. Never pass key values on a command line; use --old-key-env VAR / --old-key-file PATH and, if the new key is not the node's configured provider, --new-key-file.
3. Run secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY once per store, i.e. on EACH node. Because re-wraps do not replicate, rotating one node leaves its partner's rows on the old kek_id. With replication on, do not leave the pair half-rotated: a node whose rows carry a KEK it cannot unwrap fails closed (Row wrapped by unknown KEK) on resolve.
4. Quiesce first: stop (or drain) the node while its store is rewrapped. Rotating a pair node-by-node while the other node keeps serving works for resolves, but do not author secrets anywhere in the pair mid-rotation — a fresh write is wrapped under whichever KEK the writing node currently uses and replicates verbatim to a partner that may not yet hold that KEK.
5. Cut over: confirm Secrets:MasterKey resolves the new key on both nodes, start them, and confirm a ${secret:} value resolves (the pre-host expander fails the boot loudly if not).
6. Destroy the old key only after both stores resolve cleanly. Re-running rewrap-all is idempotent (alreadyCurrent), so an interrupted pass is resumed by running it again.
7. Rollback: restore the step-1 backups on both nodes and revert the KEK source.

Pair rotation across separate pairs (central, site-a, site-b) is independent per pair; there is no fleet-wide order.

Alternative: shared SQLite volume (no replication)

The earlier interim posture — every node of a pair mounting one Secrets:SqlitePath on a shared or replicated volume — still works for a deployment that keeps Secrets:Replication:Enabled=false: reads dominate and writes are effectively single-writer. See the caveat below; it is not safe with concurrent writers, and it is impractical for site pairs whose nodes share no volume.

How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time — never by editing the committed appsettings.json or the role-overlay files (appsettings.admin.json, appsettings.driver.json, appsettings.admin-driver.json). Those committed overlays are also consumed by local dev and the TwoNodeClusterHarness integration tests, so hardcoding a Source=File path into them would break every dev/test/CI boot. Two acceptable delivery mechanisms instead:

Option A — environment variable overrides (Windows Service / NSSM env block, container env_file, etc.), applied identically on every node regardless of role:

# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db

Option B — a production-only config layer that is not the committed dev base (e.g. an untracked appsettings.Production.json deployed alongside the binaries, or an orchestrator-injected config mount):

// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/otopcua-master.key"
    },
    "SqlitePath": "/shared/secrets/otopcua-secrets.db"
  }
}

Either way, the file/path referenced must exist and be identical on every node before that node boots — the pre-host expander runs unconditionally on every role and will throw (SecretNotFoundException / migration failure) if the store or key is missing.

Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well over network filesystems (SMB/NFS locking is unreliable, and even on a clustered block volume only one writer should be active at a time). The shared-volume alternative above is acceptable because:

• Reads dominate (every node's boot + cache-refresh cycle, across every role).
• Writes are rare, human-initiated, and effectively single-writer in practice (an operator runs the CLI/UI against one admin node at a time).

It is not a substitute for real replication (Akka pair replication, above), and it is not safe if multiple nodes attempt concurrent writes. Do not build automation that writes secrets from more than one node simultaneously.

Data Protection is independent — do not touch it here

OtOpcUa's cookie/session protection already has its own clustered-key story: AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>() (src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75), which shares the Data Protection key ring across every node via the existing ConfigDb. That mechanism is unrelated to ZB.MOM.WW.Secrets' envelope encryption (KEK + SQLite store) and must not be reconfigured as part of secrets-adoption work — doing so risks invalidating active sessions for an unrelated reason.

The G-7 hand-off (status)

G-7 (scadaproj/components/secrets/GAPS.md) asked for real cross-node replication. It was built in two flavours in the library (ZB.MOM.WW.Secrets.Replicator.SqlServer and .AkkaDotNet, plus a gRPC replicator package). OtOpcUa production uses the Akka pair-local option (the decision recorded in scadaproj#3), wired by SecretsRegistration as described above; the ConfigDb-backed store this page used to anticipate is not what shipped here. Driver-only nodes have no ConfigDb since mesh Phase 4, which is a further reason a ConfigDb secret store would not reach them.

Dev/test/default posture (unchanged)

The committed default in appsettings.json is:

"Secrets": {
  "SqlitePath": "otopcua-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true
}

This is dev-safe: Source=Environment needs no filesystem key, and the SQLite path is relative to the working directory, so local dev, the role-overlay appsettings (appsettings.admin.json, appsettings.driver.json, appsettings.admin-driver.json), and the TwoNodeClusterHarness integration tests all boot cleanly with no external mount. The File-KEK + shared-volume posture in this runbook applies only to real clustered production deployments — it must never be baked into the committed dev/role-overlay base, because the expander runs unconditionally at every node boot (any role) and would break dev/CI if pointed at a nonexistent /shared mount.