scadaproj/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md

Operating clustered secrets

How to run ZB.MOM.WW.Secrets across more than one node. Covers the shared SQL-Server store, the SQL-Server hub, and Akka peer-to-peer replication.

Companion: kek-rotation.md.

---

0. The one rule that breaks everything if you get it wrong

Every node must resolve the same master KEK.

Rows are ciphertext everywhere — in the database, in a backup, on the wire. Only the KEK decrypts them, and it never leaves the node. So if node B has a different master key than node A:

• replication still "works" — rows arrive and are stored;
• every resolve on B of a secret written on A fails closed with a kek_id mismatch.

The symptom looks like data corruption and is not. Before enabling any topology below, confirm the same key material is present on every node:

# On each node — compare the digests, never print the key itself.
sha256sum /run/secrets/zb-kek
# or, for the environment source:
printf '%s' "$ZB_SECRETS_MASTER_KEY" | sha256sum

---

1. Shared SQL-Server store

One database, one copy of each row. Prefer this unless a node must keep resolving secrets while partitioned from the database.

Provision

The schema is created automatically by the first node that starts (idempotent, and safe if several start at once). You only need a database and a login:

CREATE DATABASE ZbSecrets;
GO
USE ZbSecrets;
CREATE LOGIN zbsecrets WITH PASSWORD = '<strong password>';
CREATE USER  zbsecrets FOR LOGIN zbsecrets;
GO
-- The app creates and owns the zbsecrets schema; it needs permission to do so.
GRANT CREATE SCHEMA, CREATE TABLE TO zbsecrets;
ALTER ROLE db_owner ADD MEMBER zbsecrets;   -- or grant on the schema once it exists
GO

To pre-create the schema instead (so the app can run with narrower rights), run the DDL from SqlServerSecretsSchema.CreateSchemaDdl("zbsecrets") and then grant only SELECT, INSERT, UPDATE on [zbsecrets].[secret] and [zbsecrets].[schema_version].

DELETE is deliberately not required. Deletes are tombstones — an UPDATE — so a compromised app credential cannot destroy secret history.

Configure

"Secrets": {
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "SqlServer": {
    "ConnectionString": "Server=sql01;Database=ZbSecrets;User Id=zbsecrets;Password=${secret:sql/zbsecrets-password};TrustServerCertificate=False",
    "SchemaName": "zbsecrets",
    "CommandTimeout": "00:00:30"
  }
}

The connection string is itself a credential — deliver it by ${secret:} reference, an orchestrator secret, or an environment variable. Never commit it.

Verify

secret set smoke/test hello --description "delete me"   # on node A
secret get smoke/test                                    # on node B — expect: hello
secret rm  smoke/test

Rotation

Run rewrap-all once, against the shared store, with resolve traffic quiesced.

---

2. SQL-Server hub replication

Each node keeps its local SQLite store and syncs against a shared hub.

Same provisioning as topology 1. Configure with AddZbSecretsSqlServerReplication and:

"Secrets": {
  "SqlitePath": "/var/lib/zb/secrets.db",
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "SqlServer": {
    "ConnectionString": "...",
    "SyncInterval": "00:00:30",
    "SyncOnStartup": true
  }
}

SyncInterval is the convergence bound, not the propagation latency — a live write reaches the hub in milliseconds; the interval governs how fast a dropped write or a rejoining node is repaired.

Rotation

Run rewrap-all against the hub and against every node's local SQLite store. A re-wrap does not bump the revision (by design — otherwise every node's independent re-wrap would churn replication), so it does not propagate.

Verifying convergence

Secret hub sync converged: pulled N row(s) from the hub, pushed M row(s) to it.

Logged at Information only when something actually moved. A healthy steady-state cluster logs nothing. Continuous non-zero pushes on every tick means two nodes are fighting over the same secret — look for a script writing the same name on a loop.

---

3. Akka cluster peer-to-peer

No shared database. Requires the application to register its ActorSystem in DI.

"Secrets": {
  "SqlitePath": "/var/lib/zb/secrets.db",
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "Replication": {
    "AnnounceInterval": "00:00:30",
    "ActorName": "zb-secret-replication"
  }
}

Merge the serializer config when creating the actor system:

Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
ActorSystem.Create("my-system", config);

Optional but recommended: without it the protocol still round-trips under Akka's default JSON serializer, but the serializer carrying secret ciphertext becomes an inherited default rather than a deliberate choice.

Securing the transport

Rows on the wire are ciphertext and the KEK never travels, so a passive eavesdropper learns nothing but secret names, revisions, and timestamps. That is still metadata worth protecting, and an unauthenticated cluster port is a much larger problem than secret replication — configure Akka.Remote TLS and cluster authentication as you would for any other cluster traffic.

Rotation

Run rewrap-all on each node's local store. Re-wraps do not replicate (see above).

Announce traffic

One manifest per node per interval: names, revisions, timestamps, tombstone flags — no ciphertext. Cost grows as O(nodes × secrets), so a large cluster with a large secret set should lengthen AnnounceInterval rather than shorten it.

---

Choosing between them

	Shared store	SQL hub	Akka peer-to-peer
Resolves while cut off from the DB	✗ (cache TTL only)	✓	✓ (no DB at all)
Distributed failure modes	none	LWW, sweep lag	LWW, sweep lag
Rotation runs	once	hub + each node	each node
Needs a shared database	✓	✓	✗
Needs an Akka cluster	✗	✗	✓

Start at the left and move right only when a stated requirement forces it.

---

Production enablement (decided 2026-08-07)

Which topology each app runs in production is decided — tracked as scadaproj#3, design at docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md (scadaproj):

• OtOpcUa: Akka peer-to-peer, pair-local. Each application cluster is its own 2-node Akka mesh (mesh Phase 6), so replication converges within a pair and nowhere else — seed each pair independently (central via /admin/secrets; site pairs via the secret CLI or ciphertext-only bundle import). This topology is live-gated as such — PASS 4/4 on the three-mesh rig (OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md, lmxopcua master @ 1bd79c8e): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and no cross-pair propagation proven with seeded isolation controls on the other pairs rather than by mere absence. It supersedes the 2026-07-18 six-node gate as evidence, which predates the mesh Phase 6 partition.
• ScadaBridge: central gRPC hub (§2's SQL-hub mode is NOT used). Site servers never talk to the DB directly — they talk to central for everything — so the SqlServer hub's site-side connection string violates the topology rule. The ZB.MOM.WW.Secrets.Replicator.Grpc package (central-hosted secrets_hub.v1, pull-only: updates originate at central only, by design — exactly two RPCs, GetManifest/GetSecrets, with no write RPC at the contract level) replaces it. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 — see ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md (ScadaBridge main @ f6c3f7c5): convergence in 5 s, byte-identical and decrypt-verified on both site-a nodes; a cold boot with central down serves last-known-good and resumes; tombstones propagate ≤9 s with no resurrection; auth negatives deny uniformly with zero credential material in the logs. The gRPC mode is cached on the site clusters exactly like §2's hub mode — each site node keeps a full local store, serves last-known-good secrets while central is unreachable, and converges on the first successful sweep. The code path is merged and gated; production enablement is still a separate operator action — Secrets:Replication:Enabled remains default-off, and Secrets:Replication:Mode defaults to SqlServer (legacy-equivalent), so Grpc must be selected deliberately.
  • The central pair runs on ONE shared SQL-Server store (§1's AddZbSecretsSqlServerStore; scadaproj#4, CLOSED 2026-08-07, Secrets family 0.5.x, ScadaBridge main @ e697477c). Central in Grpc mode requires Secrets:SqlServer:ConnectionString — the boot fails closed naming the key if it is blank, and refuses a ${secret:} value (the store's own connection string can never be a secret reference — bootstrap circularity; deliver it as a literal or environment value, never committed). Both hub instances serve the one store, so their manifests are identical by construction — the #4 divergence (an authenticated hub answering an empty manifest while its partner held every secret) is structurally unreproducible. Sites are unchanged (local SQLite, pull-only sweep). Live gate PASS 5/5: ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md.
  • Site-side hub endpoint failover: Secrets:GrpcHub:FallbackEndpoints (family 0.5.0). List the second central node's hub endpoint there. The failover reader tries endpoints in rotation on transport-level gRPC errors, with sticky preference for the last endpoint that answered. Same-store-only rule: every listed endpoint must serve the SAME store — for ScadaBridge that is the central pair over the one shared SQL store. Never list endpoints backed by independent stores: failing over to an emptier hub is a silent convergence stop, the exact defect #4 recorded.
  • Minimum version for central pairs: 0.5.1. 0.5.0's boot-time SqlServerSecretsStoreMigrator lost a concurrent first-boot race on a virgin database (error 2714 crashed one node when both centrals provisioned the empty DB simultaneously — the normal first deploy of a pair). 0.5.1 retries the race (1205/2714/1913/2627) and was re-drilled live: both centrals booting in the same second against a virgin DB, clean.
• KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override, never committed. The committed dev KEK is dev-only. Verify per pair with the §0 digest compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
• Rotation fan-out: re-wraps do not replicate, so secret rewrap-all runs against each independent store. On OtOpcUa that is both nodes of every pair. On ScadaBridge it is ONE run against central's shared SQL store — the CLI auto-targets it by reading central's own appsettings (Secrets:SqlServer:ConnectionString), and one run covers both central nodes because there is only one store — plus both nodes of every site pair. (The old "both central nodes" caveat is gone with the shared store; scadaproj#4.) Note that the pull-only channel does not re-push re-wrapped rows either: a site node's own store must be re-wrapped directly. The KEK doctor is the recovery path for a missed node.

---

Troubleshooting

SecretDecryptionException / kek_id mismatch after enabling replication. The KEKs differ between nodes. Compare digests as in §0. Do not "fix" it by re-writing the secret on the failing node — that hides the mismatch until the next secret.

A secret written on node A never appears on node B. Check, in order: (1) are both nodes actually in the cluster / pointed at the same hub database and schema; (2) has one AnnounceInterval / SyncInterval elapsed; (3) is there a warning-level log about a failed publish or a failed sweep. A dropped publish is expected and self-repairing — a sweep that fails every interval is not.

A deleted secret came back. Deletes are tombstones and propagate as newer rows, so this should not happen through replication. It does happen if a node's local store was restored from a backup taken before the delete: the restore reintroduces the row with an old timestamp, loses to LWW, and the tombstone re-wins. If instead the row is live again, someone re-created it — check the audit log for the write.

The migration refuses to run. SecretStoreMigrationException naming a version newer than supported means another node is running a newer build against the same store. Upgrade this node; do not downgrade the store.