feat(secrets): cluster replication via SQL Server and Akka.NET (G-7, 0.2.0)
Secrets were per-node SQLite, so a secret written on one node was invisible to the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka replicator" fork to build only the former; both are built here so the choice is a deployment decision (availability vs partition tolerance) rather than a library limitation. Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over distributed pub/sub). Core gains ISecretsStoreMigrator, one shared SecretLastWriterWins predicate so no two stores can disagree on a tie, the transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was inert and local writes would not have propagated at all. Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural divergence between the stores fails) and a 9-test in-process 2-node Akka cluster over real remoting. A post-build review caught six defects, all fixed and now covered: both replication modes could not resolve from the container (no test had built one), an unbounded fetch that broke past SQL Server's 2100-parameter cap, a poison row that aborted the rest of its batch forever, Enum.Parse on peer input that could restart the actor in a loop, null crypto blobs crossing the trust boundary, and a silently dropped pull-read failure. Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -11,6 +11,8 @@ plaintext back on demand — from application code, from configuration, or from
|
||||
| `ZB.MOM.WW.Secrets.Abstractions` | Contracts only (`ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, `ISecretCacheInvalidator`, `ISecretActorAccessor`), value types, exceptions. Dependency-light. |
|
||||
| `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
|
||||
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
|
||||
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | Cluster-wide secrets via SQL Server — as one shared store, or as a hub each node syncs against. |
|
||||
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | Peer-to-peer replication over an Akka.NET cluster — partition-tolerant, no shared database. |
|
||||
|
||||
A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).
|
||||
|
||||
@@ -78,8 +80,73 @@ SQL-Server store; once per node for per-node SQLite on a shared KEK). See the op
|
||||
|
||||
## Clustered deployments
|
||||
|
||||
Secrets storage is local SQLite by default. The schema already carries the
|
||||
`revision` / `updated_utc` / tombstone columns and an `ISecretReplicator` seam for
|
||||
cluster replication, but the Akka replicator (`ZB.MOM.WW.Secrets.Akka`) is a deferred
|
||||
follow-on. **When replicating across a node pair, every node must resolve the _same_ master
|
||||
KEK** (e.g. the same mounted key file) — a row wrapped by an unknown KEK fails closed.
|
||||
Storage is local SQLite by default, which is correct for a single-process app and wrong the
|
||||
moment a secret written on one node has to be readable on another. Three topologies are
|
||||
supported; pick the least complicated one that meets the requirement.
|
||||
|
||||
> **Every topology requires the same master KEK on every node.** The database and the wire carry
|
||||
> ciphertext only, so a node with a different key cannot decrypt what another wrote — it fails
|
||||
> closed on a `kek_id` mismatch, which looks like corruption but is a deployment mistake. Mount the
|
||||
> same key file, or supply the same key environment variable, everywhere.
|
||||
|
||||
### 1. Shared SQL-Server store — *the default choice*
|
||||
|
||||
Every node points at one database. There is exactly one copy of each row, so there is no
|
||||
replication, no reconciliation, no split-brain, and no clock skew to reason about.
|
||||
|
||||
```csharp
|
||||
builder.Services.AddZbSecretsSqlServerStore(builder.Configuration);
|
||||
```
|
||||
|
||||
```jsonc
|
||||
"Secrets": {
|
||||
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
||||
"SqlServer": {
|
||||
"ConnectionString": "Server=sql01;Database=ZbSecrets;...",
|
||||
"SchemaName": "zbsecrets"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Trade-off:** availability is coupled to that database. A node cut off from it falls back only to
|
||||
the resolver's short in-memory cache. Rotation (`rewrap-all`) runs **once**, against the one store.
|
||||
|
||||
### 2. SQL-Server hub replication
|
||||
|
||||
Each node keeps its local store and stays converged with a shared hub: writes publish immediately,
|
||||
and a background sweep reconciles **both directions** on an interval (default 30s).
|
||||
|
||||
```csharp
|
||||
builder.Services.AddZbSecretsSqlServerReplication(builder.Configuration);
|
||||
```
|
||||
|
||||
Choose this over (1) when a node must keep **resolving** secrets while cut off from the hub — reads
|
||||
never leave the node, so a hub outage degrades propagation rather than availability.
|
||||
|
||||
### 3. Akka cluster peer-to-peer
|
||||
|
||||
No shared database at all. Nodes broadcast writes over distributed pub/sub and repair divergence
|
||||
with a periodic manifest exchange.
|
||||
|
||||
```csharp
|
||||
builder.Services.AddZbSecretsAkkaReplication(builder.Configuration);
|
||||
// Recommended — pins the serializer that carries ciphertext:
|
||||
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
|
||||
```
|
||||
|
||||
Requires the application to register its `ActorSystem` in DI. The local store is SQLite. This is
|
||||
the right answer for air-gapped or intermittently-connected sites; it is the wrong answer if every
|
||||
node can always reach a shared database, because you pay real distributed-systems behaviour for it.
|
||||
|
||||
### What replication costs you (2 and 3)
|
||||
|
||||
- **Eventual consistency.** Rows converge within the sweep interval, not instantly.
|
||||
- **Last-writer-wins.** Two nodes writing the same secret concurrently: the later `updated_utc`
|
||||
wins and the other write is discarded. There is no merge and no conflict report.
|
||||
- **Publishing is best-effort.** A failed broadcast is logged, not thrown — the local write is
|
||||
already durable and anti-entropy repairs it. Writes never fail because a peer was unreachable.
|
||||
- **KEK rotation does not replicate.** A re-wrap deliberately leaves the revision untouched so it
|
||||
stays invisible to last-writer-wins. Run `rewrap-all` against **each** independent store.
|
||||
|
||||
See [`docs/operations/clustered-secrets.md`](docs/operations/clustered-secrets.md) for operator
|
||||
setup, and [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md) for rotation.
|
||||
|
||||
Reference in New Issue
Block a user