# Secrets: Clustered Master-Key Posture (All Roles)

## Purpose

`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node
boot, before the host is built. It reads rows from an envelope-encrypted SQLite
store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also
registered unconditionally, independent of node role.

OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })`
in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`):
`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a
fused admin+driver node, or a driver-only node). This runbook covers what a
production deployment needs so that secret resolution behaves identically on
**every node regardless of role** — not just admin nodes. That matters here more
than it might elsewhere: the pre-host expander and the runtime resolver both run
unconditionally on driver-only nodes too, and driver-only nodes are the ones that
will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at
runtime, not just at boot. It does **not** change any code; it is an
operations/deployment posture, delivered out-of-band from the committed config.

## The two hard requirements

For the pre-host expander (and the runtime resolver) to resolve the same
plaintext secret on every node, no matter its role:

1. **Identical KEK on every node.** All nodes — admin, driver, and any fused
   combination — must unwrap the store with the exact same master key. A
   per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
   decrypt every *other* node's ciphertext rows to garbage.
2. **Identical store rows on every node.** All nodes must read the same SQLite
   database (same file, or a replicated/shared copy with the same rows) — not
   independently-seeded stores that happen to share a KEK.

`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
— there is no built-in cross-node replication today. Meeting both requirements in
production is a deployment concern, covered below.

## Recommended interim posture (G-5)

Until real replication exists (G-7, below), the recommended production posture is:

- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
  key file that is identical on every node of every role** — a base64-encoded
  32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
  to each node's filesystem/secret-mount by the deployment tooling, and **never
  committed** to the repo. Treat it with the same discipline as any other
  production secret (restrictive file ACLs, no logging, rotated via a future
  KEK-rotation runbook — not yet built).
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
  that every node mounts (admin, driver, and dev/fused alike), so every node's
  migrator opens and reads the same rows at boot.

Writes to the store are rare and human-driven — an operator using the
`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
regardless of role. The access pattern is read-mostly / effectively
single-writer, which is what makes a shared SQLite volume viable as an interim
posture (see caveat below).

## How it's delivered (do NOT commit these values)

The File-KEK + shared-store posture is supplied per-node at deployment time —
**never** by editing the committed `appsettings.json` or the role-overlay files
(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`).
Those committed overlays are also consumed by local dev and the
`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path
into them would break every dev/test/CI boot. Two acceptable delivery
mechanisms instead:

**Option A — environment variable overrides** (Windows Service / NSSM env block,
container `env_file`, etc.), applied identically on every node regardless of role:

```
# production deployment — do not commit to the dev appsettings
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db
```

**Option B — a production-only config layer** that is *not* the committed dev
base (e.g. an untracked `appsettings.Production.json` deployed alongside the
binaries, or an orchestrator-injected config mount):

```jsonc
// production deployment — do not commit to the dev appsettings
{
  "Secrets": {
    "MasterKey": {
      "Source": "File",
      "FilePath": "/run/secrets/otopcua-master.key"
    },
    "SqlitePath": "/shared/secrets/otopcua-secrets.db"
  }
}
```

Either way, the file/path referenced must exist and be identical on every node
**before** that node boots — the pre-host expander runs unconditionally on every
role and will throw (`SecretNotFoundException` / migration failure) if the store
or key is missing.

## Caveat: SQLite over a shared volume is not real replication

SQLite's file-locking model does not tolerate concurrent multi-writer access well
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
block volume only one writer should be active at a time). The interim posture
above is acceptable because:

- Reads dominate (every node's boot + cache-refresh cycle, across every role).
- Writes are rare, human-initiated, and effectively single-writer in practice
  (an operator runs the CLI/UI against one admin node at a time).

It is **not** a substitute for real replication, and it is not safe if multiple
nodes attempt concurrent writes. Do not build automation that writes secrets
from more than one node simultaneously.

## Data Protection is independent — do not touch it here

OtOpcUa's cookie/session protection already has its own clustered-key story:
`AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>()`
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`),
which shares the Data Protection key ring across every node via the existing
ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption
(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
work — doing so risks invalidating active sessions for an unrelated reason.

It is, however, the model for where the *next* iteration of secret storage
should go — see the G-7 hand-off below.

## The G-7 hand-off

The posture above is an interim, ops-only workaround. The long-term shape,
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
the Data Protection key ring:

```csharp
services.AddDataProtection()
    .PersistKeysToDbContext<OtOpcUaConfigDbContext>()
    .SetApplicationName("OtOpcUa");
```

(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
alike — a single MS SQL-backed source of truth for the Data Protection key ring;
the secret store is the natural next tenant of that same shared database instead
of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
built as part of this cut. This runbook's shared-SQLite-volume posture is the
bridge until G-7 lands.

## Dev/test/default posture (unchanged)

The committed default in `appsettings.json` is:

```json
"Secrets": {
  "SqlitePath": "otopcua-secrets.db",
  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
  "RunMigrationsOnStartup": true
}
```

This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
path is relative to the working directory, so local dev, the role-overlay
appsettings (`appsettings.admin.json`, `appsettings.driver.json`,
`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration
tests all boot cleanly with no external mount. The File-KEK + shared-volume
posture in this runbook applies only to real clustered production deployments —
it must never be baked into the committed dev/role-overlay base, because the
expander runs unconditionally at every node boot (any role) and would break
dev/CI if pointed at a nonexistent `/shared` mount.