docs: rewrite IncrementalSync + AddressSpace for v3, fix driver-specs/driver-stability tier claims, update Secrets runbook for pair replication + KEK rotation (#623)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
@@ -31,35 +31,90 @@ plaintext secret on every node, no matter its role:
|
||||
combination — must unwrap the store with the exact same master key. A
|
||||
per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
|
||||
decrypt every *other* node's ciphertext rows to garbage.
|
||||
2. **Identical store rows on every node.** All nodes must read the same SQLite
|
||||
database (same file, or a replicated/shared copy with the same rows) — not
|
||||
independently-seeded stores that happen to share a KEK.
|
||||
2. **The same store rows on every node of a pair.** Nodes must read the same rows —
|
||||
not independently-seeded stores that merely share a KEK. Either mount one SQLite file
|
||||
(see "Alternative: shared volume" below) or, preferred, turn on Akka secret replication
|
||||
so each pair converges its own rows.
|
||||
|
||||
`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
|
||||
— there is no built-in cross-node replication today. Meeting both requirements in
|
||||
production is a deployment concern, covered below.
|
||||
The base `ZB.MOM.WW.Secrets` store is a plain local SQLite `ISecretStore` whose default
|
||||
replicator is `NoOpSecretReplicator`: **with `Secrets:Replication:Enabled=false` (the
|
||||
default) nothing replicates and each node's store is private.** That is what this page
|
||||
used to describe as the only option. It is no longer: pair replication exists
|
||||
(`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`, lib pin `0.6.2` in `Directory.Packages.props`;
|
||||
gated by `SecretsRegistration`, see below).
|
||||
|
||||
## Recommended interim posture (G-5)
|
||||
## Recommended posture: pair-local Akka replication
|
||||
|
||||
Until real replication exists (G-7, below), the recommended production posture is:
|
||||
Opt in with `Secrets:Replication:Enabled=true` on **every node** (default `false`, so absent
|
||||
config means a plain local SQLite store). `SecretsRegistration.AddOtOpcUaSecrets`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`) then registers
|
||||
`AddZbSecretsAkkaReplication` **instead of** `AddZbSecrets` (the two are alternatives, never
|
||||
additive) and a `SecretReplicationStarter` hosted service resolves the store at boot so a
|
||||
node that never reads a secret still joins anti-entropy. Keys: `Secrets:Replication:Enabled`,
|
||||
`:AnnounceInterval` (default `00:00:30`; each node publishes a name/revision/timestamp/tombstone
|
||||
manifest per interval, no ciphertext), `:ActorName` (`zb-secret-replication`).
|
||||
|
||||
- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
|
||||
key file that is identical on every node of every role** — a base64-encoded
|
||||
32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
|
||||
to each node's filesystem/secret-mount by the deployment tooling, and **never
|
||||
committed** to the repo. Treat it with the same discipline as any other
|
||||
production secret (restrictive file ACLs, no logging, rotated via a future
|
||||
KEK-rotation runbook — not yet built).
|
||||
- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
|
||||
that every node mounts (admin, driver, and dev/fused alike), so every node's
|
||||
migrator opens and reads the same rows at boot.
|
||||
- **Scope is PAIR-LOCAL.** Since mesh Phase 6 each application cluster is its own 2-node Akka
|
||||
mesh and replication rides that mesh's DistributedPubSub, so a secret converges **within its
|
||||
pair and nowhere else**. There is no fleet-wide sync (site nodes have no SQL hub). Seed each
|
||||
pair independently: central via `/admin/secrets`, driver-only site pairs (no AdminUI) via the
|
||||
secrets CLI against one node's store, or a ciphertext-only bundle import. See
|
||||
`docs/Redundancy.md` §"Secrets replication is pair-local".
|
||||
- **Both nodes of a pair must share the same KEK** — rows replicate verbatim (ciphertext, wrapped
|
||||
DEK and `kek_id` untouched), never re-encrypted.
|
||||
- **Deletes replicate** as soft tombstones (last-writer-wins on `updated_utc`/`revision`).
|
||||
- **Re-wraps do not replicate** (a re-wrap preserves `revision`/`updated_utc`, so it is invisible to
|
||||
last-writer-wins). KEK rotation therefore runs against each node's store — see the next section.
|
||||
- The store should sit on a **persistent per-node path** (`Secrets:SqlitePath`, e.g. the node's
|
||||
`/app/data` volume in containers); the relative default lands inside the image layer and is lost on
|
||||
recreate.
|
||||
- Evidence: live-gated PASS 4/4 on the three-mesh docker-dev rig (convergence <= 7 s, no cross-pair
|
||||
leakage in any direction, delete < 1 s, clean restart without resurrection) —
|
||||
`docs/plans/2026-08-07-secrets-pair-local-live-gate.md`. Replication stays default-off in the product.
|
||||
|
||||
Writes to the store are rare and human-driven — an operator using the
|
||||
`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
|
||||
reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
|
||||
regardless of role. The access pattern is read-mostly / effectively
|
||||
single-writer, which is what makes a shared SQLite volume viable as an interim
|
||||
posture (see caveat below).
|
||||
Whichever store layout is used, the KEK is delivered per node as before:
|
||||
**`Secrets:MasterKey:Source = File`** with `FilePath` pointing at a read-only base64 32-byte key
|
||||
(`openssl rand -base64 32`), distributed identically to every node of the pair by deployment tooling and
|
||||
**never committed**. Writes are rare and human-driven (`/admin/secrets` on admin nodes, or the
|
||||
CLI); reads happen at every node's boot and each `ResolveCacheTtl` refresh.
|
||||
|
||||
## Rotating the KEK
|
||||
|
||||
Rotation re-wraps each row's per-secret DEK under the new KEK (`wrapped_dek`, `wrap_nonce`,
|
||||
`wrap_tag`, `kek_id` change; the sealed body, `revision` and `updated_utc` do not). The
|
||||
authoritative operator procedure and troubleshooting table live with the library:
|
||||
`scadaproj/ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`, driven by the `secret rewrap-all`
|
||||
CLI verb (in that repo; not packed). The OtOpcUa-specific points:
|
||||
|
||||
1. **Back up every store first** (copy each node's SQLite file) — the backup is the complete undo.
|
||||
2. **Stage the new KEK on every node of the pair** (same file path / env var as `Secrets:MasterKey`
|
||||
resolves) but do not restart yet. Never pass key values on a command line; use
|
||||
`--old-key-env VAR` / `--old-key-file PATH` and, if the new key is not the node's configured
|
||||
provider, `--new-key-file`.
|
||||
3. **Run `secret rewrap-all --old-key-env ZB_SECRETS_OLD_KEY` once per store, i.e. on EACH node.**
|
||||
Because re-wraps do not replicate, rotating one node leaves its partner's rows on the old
|
||||
`kek_id`. With replication on, do not leave the pair half-rotated: a node whose rows carry a KEK
|
||||
it cannot unwrap fails closed (`Row wrapped by unknown KEK`) on resolve.
|
||||
4. **Quiesce first**: stop (or drain) the node while its store is rewrapped. Rotating a pair
|
||||
node-by-node while the other node keeps serving works for resolves, but do not author secrets
|
||||
anywhere in the pair mid-rotation — a fresh write is wrapped under whichever KEK the writing node
|
||||
currently uses and replicates verbatim to a partner that may not yet hold that KEK.
|
||||
5. **Cut over**: confirm `Secrets:MasterKey` resolves the new key on both nodes, start them, and
|
||||
confirm a `${secret:}` value resolves (the pre-host expander fails the boot loudly if not).
|
||||
6. **Destroy the old key** only after both stores resolve cleanly. Re-running `rewrap-all` is
|
||||
idempotent (`alreadyCurrent`), so an interrupted pass is resumed by running it again.
|
||||
7. Rollback: restore the step-1 backups on both nodes and revert the KEK source.
|
||||
|
||||
Pair rotation across **separate pairs** (central, site-a, site-b) is independent per pair; there is no
|
||||
fleet-wide order.
|
||||
|
||||
## Alternative: shared SQLite volume (no replication)
|
||||
|
||||
The earlier interim posture — every node of a pair mounting one `Secrets:SqlitePath` on a shared
|
||||
or replicated volume — still works for a deployment that keeps `Secrets:Replication:Enabled=false`:
|
||||
reads dominate and writes are effectively single-writer. See the caveat below; it is not safe with
|
||||
concurrent writers, and it is not available to DB-less driver-only site nodes that have no shared
|
||||
volume with their partner.
|
||||
|
||||
## How it's delivered (do NOT commit these values)
|
||||
|
||||
@@ -107,14 +162,14 @@ or key is missing.
|
||||
|
||||
SQLite's file-locking model does not tolerate concurrent multi-writer access well
|
||||
over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
|
||||
block volume only one writer should be active at a time). The interim posture
|
||||
block volume only one writer should be active at a time). The shared-volume alternative
|
||||
above is acceptable because:
|
||||
|
||||
- Reads dominate (every node's boot + cache-refresh cycle, across every role).
|
||||
- Writes are rare, human-initiated, and effectively single-writer in practice
|
||||
(an operator runs the CLI/UI against one admin node at a time).
|
||||
|
||||
It is **not** a substitute for real replication, and it is not safe if multiple
|
||||
It is **not** a substitute for real replication (Akka pair replication, above), and it is not safe if multiple
|
||||
nodes attempt concurrent writes. Do not build automation that writes secrets
|
||||
from more than one node simultaneously.
|
||||
|
||||
@@ -128,31 +183,15 @@ ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryptio
|
||||
(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
|
||||
work — doing so risks invalidating active sessions for an unrelated reason.
|
||||
|
||||
It is, however, the model for where the *next* iteration of secret storage
|
||||
should go — see the G-7 hand-off below.
|
||||
## The G-7 hand-off (status)
|
||||
|
||||
## The G-7 hand-off
|
||||
|
||||
The posture above is an interim, ops-only workaround. The long-term shape,
|
||||
tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
|
||||
ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
|
||||
the Data Protection key ring:
|
||||
|
||||
```csharp
|
||||
services.AddDataProtection()
|
||||
.PersistKeysToDbContext<OtOpcUaConfigDbContext>()
|
||||
.SetApplicationName("OtOpcUa");
|
||||
```
|
||||
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
|
||||
`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
|
||||
alike — a single MS SQL-backed source of truth for the Data Protection key ring;
|
||||
the secret store is the natural next tenant of that same shared database instead
|
||||
of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
|
||||
code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
|
||||
overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
|
||||
built as part of this cut. This runbook's shared-SQLite-volume posture is the
|
||||
bridge until G-7 lands.
|
||||
G-7 (`scadaproj/components/secrets/GAPS.md`) asked for real cross-node replication. It was
|
||||
built in two flavours in the library (`ZB.MOM.WW.Secrets.Replicator.SqlServer` and
|
||||
`.AkkaDotNet`, plus a gRPC replicator package). **OtOpcUa production uses the Akka pair-local
|
||||
option** (the decision recorded in scadaproj#3), wired by `SecretsRegistration` as described above;
|
||||
the ConfigDb-backed store this page used to anticipate is not what shipped here. Driver-only nodes
|
||||
have no ConfigDb since mesh Phase 4, which is a further reason a ConfigDb secret store would not
|
||||
reach them.
|
||||
|
||||
## Dev/test/default posture (unchanged)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user