# Operating clustered secrets

How to run `ZB.MOM.WW.Secrets` across more than one node. Covers the shared SQL-Server store, the
SQL-Server hub, and Akka peer-to-peer replication.

Companion: [`kek-rotation.md`](kek-rotation.md).

---

## 0. The one rule that breaks everything if you get it wrong

**Every node must resolve the same master KEK.**

Rows are ciphertext everywhere — in the database, in a backup, on the wire. Only the KEK decrypts
them, and it never leaves the node. So if node B has a different master key than node A:

- replication still "works" — rows arrive and are stored;
- every resolve on B of a secret written on A **fails closed** with a `kek_id` mismatch.

The symptom looks like data corruption and is not. Before enabling any topology below, confirm the
same key material is present on every node:

```bash
# On each node — compare the digests, never print the key itself.
sha256sum /run/secrets/zb-kek
# or, for the environment source:
printf '%s' "$ZB_SECRETS_MASTER_KEY" | sha256sum
```

---

## 1. Shared SQL-Server store

One database, one copy of each row. Prefer this unless a node must keep resolving secrets while
partitioned from the database.

### Provision

The schema is created automatically by the first node that starts (idempotent, and safe if several
start at once). You only need a database and a login:

```sql
CREATE DATABASE ZbSecrets;
GO
USE ZbSecrets;
CREATE LOGIN zbsecrets WITH PASSWORD = '<strong password>';
CREATE USER  zbsecrets FOR LOGIN zbsecrets;
GO
-- The app creates and owns the zbsecrets schema; it needs permission to do so.
GRANT CREATE SCHEMA, CREATE TABLE TO zbsecrets;
ALTER ROLE db_owner ADD MEMBER zbsecrets;   -- or grant on the schema once it exists
GO
```

To pre-create the schema instead (so the app can run with narrower rights), run the DDL from
`SqlServerSecretsSchema.CreateSchemaDdl("zbsecrets")` and then grant only
`SELECT, INSERT, UPDATE` on `[zbsecrets].[secret]` and `[zbsecrets].[schema_version]`.

> `DELETE` is deliberately **not** required. Deletes are tombstones — an `UPDATE` — so a compromised
> app credential cannot destroy secret history.

### Configure

```jsonc
"Secrets": {
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "SqlServer": {
    "ConnectionString": "Server=sql01;Database=ZbSecrets;User Id=zbsecrets;Password=${secret:sql/zbsecrets-password};TrustServerCertificate=False",
    "SchemaName": "zbsecrets",
    "CommandTimeout": "00:00:30"
  }
}
```

The connection string is itself a credential — deliver it by `${secret:}` reference, an orchestrator
secret, or an environment variable. Never commit it.

### Verify

```bash
secret set smoke/test hello --description "delete me"   # on node A
secret get smoke/test                                    # on node B — expect: hello
secret rm  smoke/test
```

### Rotation

Run `rewrap-all` **once**, against the shared store, with resolve traffic quiesced.

---

## 2. SQL-Server hub replication

Each node keeps its local SQLite store and syncs against a shared hub.

Same provisioning as topology 1. Configure with `AddZbSecretsSqlServerReplication` and:

```jsonc
"Secrets": {
  "SqlitePath": "/var/lib/zb/secrets.db",
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "SqlServer": {
    "ConnectionString": "...",
    "SyncInterval": "00:00:30",
    "SyncOnStartup": true
  }
}
```

`SyncInterval` is the convergence bound, not the propagation latency — a live write reaches the hub
in milliseconds; the interval governs how fast a *dropped* write or a rejoining node is repaired.

### Rotation

Run `rewrap-all` against the hub **and** against every node's local SQLite store. A re-wrap does not
bump the revision (by design — otherwise every node's independent re-wrap would churn replication),
so it does **not** propagate.

### Verifying convergence

```
Secret hub sync converged: pulled N row(s) from the hub, pushed M row(s) to it.
```

Logged at Information only when something actually moved. A healthy steady-state cluster logs
nothing. Continuous non-zero pushes on every tick means two nodes are fighting over the same secret
— look for a script writing the same name on a loop.

---

## 3. Akka cluster peer-to-peer

No shared database. Requires the application to register its `ActorSystem` in DI.

```jsonc
"Secrets": {
  "SqlitePath": "/var/lib/zb/secrets.db",
  "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
  "Replication": {
    "AnnounceInterval": "00:00:30",
    "ActorName": "zb-secret-replication"
  }
}
```

Merge the serializer config when creating the actor system:

```csharp
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
ActorSystem.Create("my-system", config);
```

Optional but recommended: without it the protocol still round-trips under Akka's default JSON
serializer, but the serializer carrying secret ciphertext becomes an inherited default rather than
a deliberate choice.

### Securing the transport

Rows on the wire are ciphertext and the KEK never travels, so a passive eavesdropper learns
nothing but secret *names*, revisions, and timestamps. That is still metadata worth protecting, and
an unauthenticated cluster port is a much larger problem than secret replication — configure
Akka.Remote TLS and cluster authentication as you would for any other cluster traffic.

### Rotation

Run `rewrap-all` on **each node's** local store. Re-wraps do not replicate (see above).

### Announce traffic

One manifest per node per interval: names, revisions, timestamps, tombstone flags — no ciphertext.
Cost grows as O(nodes × secrets), so a large cluster with a large secret set should lengthen
`AnnounceInterval` rather than shorten it.

---

## Choosing between them

| | Shared store | SQL hub | Akka peer-to-peer |
|---|---|---|---|
| Resolves while cut off from the DB | ✗ (cache TTL only) | ✓ | ✓ (no DB at all) |
| Distributed failure modes | none | LWW, sweep lag | LWW, sweep lag |
| Rotation runs | once | hub + each node | each node |
| Needs a shared database | ✓ | ✓ | ✗ |
| Needs an Akka cluster | ✗ | ✗ | ✓ |

Start at the left and move right only when a stated requirement forces it.

---

## Production enablement (decided 2026-08-07)

Which topology each app runs in production is decided — tracked as
[scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design at
`docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md` (scadaproj):

- **OtOpcUa: Akka peer-to-peer, pair-local.** Each application cluster is its own 2-node Akka
  mesh (mesh Phase 6), so replication converges within a pair and nowhere else — seed each pair
  independently (central via `/admin/secrets`; site pairs via the `secret` CLI or ciphertext-only
  bundle import). **This topology is live-gated as such** — PASS 4/4 on the three-mesh rig
  (`OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`, `lmxopcua master @ 1bd79c8e`):
  pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair
  propagation proven with seeded isolation controls on the other pairs** rather than by mere
  absence. It supersedes the 2026-07-18 six-node gate as evidence, which predates the mesh
  Phase 6 partition.
- **ScadaBridge: central gRPC hub (§2's SQL-hub mode is NOT used).** Site servers never talk to
  the DB directly — they talk to central for everything — so the SqlServer hub's site-side
  connection string violates the topology rule. The `ZB.MOM.WW.Secrets.Replicator.Grpc` package
  (central-hosted `secrets_hub.v1`, **pull-only: updates originate at central only, by design** —
  exactly two RPCs, `GetManifest`/`GetSecrets`, with no write RPC at the contract level) replaces
  it. **It is BUILT, SHIPPED (`0.4.1`) and LIVE-GATED PASS 4/4** — see
  `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md` (`ScadaBridge main @ f6c3f7c5`):
  convergence in 5 s, byte-identical and decrypt-verified on both site-a nodes; a cold boot with
  central down serves last-known-good and resumes; tombstones propagate ≤9 s with no resurrection;
  auth negatives deny uniformly with zero credential material in the logs. The gRPC mode is
  **cached on the site clusters** exactly like §2's hub mode — each site node keeps a full local
  store, serves last-known-good secrets while central is unreachable, and converges on the first
  successful sweep. **The code path is merged and gated; production enablement is still a separate
  operator action** — `Secrets:Replication:Enabled` remains default-off, and `Secrets:Replication:Mode`
  defaults to `SqlServer` (legacy-equivalent), so `Grpc` must be selected deliberately.
  - **The central pair runs on ONE shared SQL-Server store** (§1's `AddZbSecretsSqlServerStore`;
    [scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4), CLOSED 2026-08-07,
    Secrets family `0.5.x`, `ScadaBridge main @ e697477c`). Central in Grpc mode **requires**
    `Secrets:SqlServer:ConnectionString` — the boot fails closed naming the key if it is blank,
    and refuses a `${secret:}` value (the store's own connection string can never be a secret
    reference — bootstrap circularity; deliver it as a literal or environment value, never
    committed). Both hub instances serve the one store, so their manifests are identical **by
    construction** — the #4 divergence (an authenticated hub answering an empty manifest while
    its partner held every secret) is structurally unreproducible. Sites are unchanged (local
    SQLite, pull-only sweep). Live gate PASS 5/5:
    `ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`.
  - **Site-side hub endpoint failover: `Secrets:GrpcHub:FallbackEndpoints`** (family `0.5.0`).
    List the **second central node's** hub endpoint there. The failover reader tries endpoints
    in rotation on transport-level gRPC errors, with sticky preference for the last endpoint
    that answered. **Same-store-only rule:** every listed endpoint must serve the SAME store —
    for ScadaBridge that is the central pair over the one shared SQL store. Never list endpoints
    backed by independent stores: failing over to an emptier hub is a silent convergence stop,
    the exact defect #4 recorded.
  - **Minimum version for central pairs: `0.5.1`.** `0.5.0`'s boot-time
    `SqlServerSecretsStoreMigrator` lost a concurrent first-boot race on a virgin database
    (error 2714 crashed one node when both centrals provisioned the empty DB simultaneously —
    the normal first deploy of a pair). `0.5.1` retries the race (1205/2714/1913/2627) and was
    re-drilled live: both centrals booting in the same second against a virgin DB, clean.
- **KEK: one fleet-wide production KEK, delivered per node via appsettings/environment override,
  never committed.** The committed dev KEK is dev-only. Verify per pair with the §0 digest
  compare. A single fleet KEK is what makes bundle-based seeding between pairs possible.
- **Rotation fan-out:** re-wraps do not replicate, so `secret rewrap-all` runs against each
  independent store. On **OtOpcUa** that is both nodes of every pair. On **ScadaBridge** it is
  **ONE run against central's shared SQL store** — the CLI auto-targets it by reading central's
  own appsettings (`Secrets:SqlServer:ConnectionString`), and one run covers both central nodes
  because there is only one store — **plus both nodes of every site pair**. (The old "both
  central nodes" caveat is gone with the shared store; scadaproj#4.) Note that the pull-only
  channel does not re-push re-wrapped rows either: a site node's own store must be re-wrapped
  directly. The KEK doctor is the recovery path for a missed node.

---

## Troubleshooting

**`SecretDecryptionException` / `kek_id` mismatch after enabling replication.**
The KEKs differ between nodes. Compare digests as in §0. Do not "fix" it by re-writing the secret on
the failing node — that hides the mismatch until the next secret.

**A secret written on node A never appears on node B.**
Check, in order: (1) are both nodes actually in the cluster / pointed at the same hub database and
schema; (2) has one `AnnounceInterval` / `SyncInterval` elapsed; (3) is there a warning-level log
about a failed publish or a failed sweep. A dropped publish is expected and self-repairing — a
sweep that fails every interval is not.

**A deleted secret came back.**
Deletes are tombstones and propagate as newer rows, so this should not happen through replication.
It does happen if a node's local store was restored from a backup taken before the delete: the
restore reintroduces the row with an old timestamp, loses to LWW, and the tombstone re-wins. If
instead the row is *live* again, someone re-created it — check the audit log for the write.

**The migration refuses to run.** `SecretStoreMigrationException` naming a version newer than
supported means another node is running a newer build against the same store. Upgrade this node;
do not downgrade the store.