lmxopcua/docs/operations/2026-07-16-secrets-clustered-master-key.md

Secrets: Clustered Master-Key Posture (All Roles)

Purpose

ZB.MOM.WW.Secrets resolves ${secret:...} tokens in appsettings.*.json via a pre-host expander (SecretReferenceExpander.ExpandConfigurationAsync, wired in src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs) that runs at every OtOpcUa node boot, before the host is built. It reads rows from an envelope-encrypted SQLite store (Secrets:SqlitePath) unwrapped with a key-encryption key (KEK) sourced per Secrets:MasterKey:Source. The runtime ISecretResolver (AddZbSecrets) is also registered unconditionally, independent of node role.

OtOpcUa is Akka.NET-clustered (builder.Services.AddAkka("otopcua", (ab, sp) => { ... }) in Program.cs), with roles parsed by RoleParser (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs): admin, driver, dev. A node can carry any combination of these roles (e.g. a fused admin+driver node, or a driver-only node). This runbook covers what a production deployment needs so that secret resolution behaves identically on every node regardless of role — not just admin nodes. That matters here more than it might elsewhere: the pre-host expander and the runtime resolver both run unconditionally on driver-only nodes too, and driver-only nodes are the ones that will resolve Layer-B DriverConfig secret references (coming in Slice 2) at runtime, not just at boot. It does not change any code; it is an operations/deployment posture, delivered out-of-band from the committed config.

The two hard requirements

For the pre-host expander (and the runtime resolver) to resolve the same plaintext secret on every node, no matter its role:

1. Identical KEK on every node. All nodes — admin, driver, and any fused combination — must unwrap the store with the exact same master key. A per-node KEK (e.g. a per-box DPAPI-protected key) would make each node decrypt every other node's ciphertext rows to garbage.
2. Identical store rows on every node. All nodes must read the same SQLite database (same file, or a replicated/shared copy with the same rows) — not independently-seeded stores that happen to share a KEK.

ZB.MOM.WW.Secrets ships a SQLite-only ISecretStore with a NoOpSecretReplicator — there is no built-in cross-node replication today. Meeting both requirements in production is a deployment concern, covered below.

Recommended interim posture (G-5)

Until real replication exists (G-7, below), the recommended production posture is:

• Secrets:MasterKey:Source = File, with FilePath pointing at a read-only key file that is identical on every node of every role — a base64-encoded 32-byte key, generated out-of-band (e.g. openssl rand -base64 32), distributed to each node's filesystem/secret-mount by the deployment tooling, and never committed to the repo. Treat it with the same discipline as any other production secret (restrictive file ACLs, no logging, rotated via a future KEK-rotation runbook — not yet built).
• Secrets:SqlitePath pointing at a single shared or replicated volume that every node mounts (admin, driver, and dev/fused alike), so every node's migrator opens and reads the same rows at boot.

Writes to the store are rare and human-driven — an operator using the /admin/secrets UI (admin nodes only) or the ZB.MOM.WW.Secrets CLI — while reads happen on every node's boot and on the ResolveCacheTtl refresh cycle, regardless of role. The access pattern is read-mostly / effectively single-writer, which is what makes a shared SQLite volume viable as an interim posture (see caveat below).

How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time — never by editing the committed appsettings.json or the role-overlay files (appsettings.admin.json, appsettings.driver.json, appsettings.admin-driver.json). Those committed overlays are also consumed by local dev and the TwoNodeClusterHarness integration tests, so hardcoding a Source=File path into them would break every dev/test/CI boot. Two acceptable delivery mechanisms instead:

Option A — environment variable overrides (Windows Service / NSSM env block, container env_file, etc.), applied identically on every node regardless of role:

# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db

Option B — a production-only config layer that is not the committed dev base (e.g. an untracked appsettings.Production.json deployed alongside the binaries, or an orchestrator-injected config mount):

// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/otopcua-master.key"
    },
    "SqlitePath": "/shared/secrets/otopcua-secrets.db"
  }
}

Either way, the file/path referenced must exist and be identical on every node before that node boots — the pre-host expander runs unconditionally on every role and will throw (SecretNotFoundException / migration failure) if the store or key is missing.

Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well over network filesystems (SMB/NFS locking is unreliable, and even on a clustered block volume only one writer should be active at a time). The interim posture above is acceptable because:

• Reads dominate (every node's boot + cache-refresh cycle, across every role).
• Writes are rare, human-initiated, and effectively single-writer in practice (an operator runs the CLI/UI against one admin node at a time).

It is not a substitute for real replication, and it is not safe if multiple nodes attempt concurrent writes. Do not build automation that writes secrets from more than one node simultaneously.

Data Protection is independent — do not touch it here

OtOpcUa's cookie/session protection already has its own clustered-key story: AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>() (src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75), which shares the Data Protection key ring across every node via the existing ConfigDb. That mechanism is unrelated to ZB.MOM.WW.Secrets' envelope encryption (KEK + SQLite store) and must not be reconfigured as part of secrets-adoption work — doing so risks invalidating active sessions for an unrelated reason.

It is, however, the model for where the next iteration of secret storage should go — see the G-7 hand-off below.

The G-7 hand-off

The posture above is an interim, ops-only workaround. The long-term shape, tracked as G-7 in scadaproj/components/secrets/GAPS.md, is a ConfigDb-backed ISecretStore that mirrors the pattern OtOpcUa already uses for the Data Protection key ring:

services.AddDataProtection()
    .PersistKeysToDbContext<OtOpcUaConfigDbContext>()
    .SetApplicationName("OtOpcUa");

(src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75). OtOpcUaConfigDbContext already gives every node — admin, driver, and dev/fused alike — a single MS SQL-backed source of truth for the Data Protection key ring; the secret store is the natural next tenant of that same shared database instead of a shared SQLite file. Building this requires new ZB.MOM.WW.Secrets library code (a ConfigDb-backed ISecretStore implementation) that does not exist yet, overlaps the G-7 tracking item, and is explicitly deferred there — it is not built as part of this cut. This runbook's shared-SQLite-volume posture is the bridge until G-7 lands.

Dev/test/default posture (unchanged)

The committed default in appsettings.json is:

"Secrets": {
  "SqlitePath": "otopcua-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true
}

This is dev-safe: Source=Environment needs no filesystem key, and the SQLite path is relative to the working directory, so local dev, the role-overlay appsettings (appsettings.admin.json, appsettings.driver.json, appsettings.admin-driver.json), and the TwoNodeClusterHarness integration tests all boot cleanly with no external mount. The File-KEK + shared-volume posture in this runbook applies only to real clustered production deployments — it must never be baked into the committed dev/role-overlay base, because the expander runs unconditionally at every node boot (any role) and would break dev/CI if pointed at a nonexistent /shared mount.