diff --git a/CLAUDE.md b/CLAUDE.md index 2862380..29efeed 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -148,7 +148,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | -| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication DESIGNED + PLANNED** — fork resolved to **build the shared SQL-Server `ISecretStore`** (Akka replicator deferred phase-2), executable plan `docs/plans/2026-07-17-secrets-g7-*`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. G-7 has no `ISecretStore` SQL implementation yet | Shared `ZB.MOM.WW.Secrets` lib (3 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | +| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 164 pass / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` is packed + vulnerability-scanned clean but NOT yet published to the feed**, and no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | | LocalDb (embedded cache + 2-node sync) | Built (3 pkgs `0.1.0`, **published to the Gitea feed** 2026-07-18, restore-verified from a scratch consumer; no app adoption yet) | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a diff --git a/README.md b/README.md index 036d2b5..c42740e 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ it produces. | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | **Built shared library** — readiness / liveness / active-node probes | | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | **Built shared library** — OTel metrics / traces + Serilog | | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | **Built shared library** — options validation + startup preflight | -| [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | **Built shared library** — AES-256-GCM secret store + `${secret:}` resolution + CLI | +| [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | **Built shared library** — AES-256-GCM secret store + `${secret:}` resolution + CLI + SQL-Server / Akka cluster replication | | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | **Built shared library** — Galaxy object-hierarchy SQL browse + gRPC service | | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | **Built shared library** — embedded SQLite cache + 2-node gRPC replication | @@ -70,7 +70,7 @@ Status verified 2026-07-18 against the feed + consumer branches (see [Roadmap](# | Config + validation | `0.1.0` | all four | [`components/configuration/`](components/configuration/) | | Observability (metrics / traces / logs) | `0.1.0` | all four | [`components/observability/`](components/observability/) | | Health (readiness / liveness / active-node) | `0.1.0` | all four | [`components/health/`](components/health/) | -| Secrets (encrypted store + `${secret:}`) | `0.1.0`–`0.1.3` | all four @ `0.1.2` (`0.1.3` published, not yet consumed) | [`components/secrets/`](components/secrets/) | +| Secrets (encrypted store + `${secret:}`) | `0.1.0`–`0.1.3` (`0.2.0` packed, unpublished) | all four @ `0.1.2` (`0.1.3` published, not yet consumed) | [`components/secrets/`](components/secrets/) | | Galaxy Repository (SQL browse + gRPC) | `0.1.0`, `0.2.0` | HistorianGateway + mxaccessgw | _(design in `docs/plans/`)_ | | LocalDb (embedded cache + 2-node sync) | `0.1.0` | none yet | _(design in `docs/plans/`)_ | @@ -172,13 +172,18 @@ ZB_LDAP_IT=1 dotnet test # requires a reachable GLAuth (e.g. a sister repo's i pending follow-on — also stale). - ✅ Secrets — `ZB.MOM.WW.Secrets` (+ `.Abstractions`, `.Ui`) published through **`0.1.3`** (KEK rotation, 2026-07-18); all four apps still pinned at `0.1.2`. +- ✅ Secrets G-7 clustered replication — **built 2026-07-18 at `0.2.0`**, both fork options: + `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared store *and* local-plus-hub) and + `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (peer-to-peer, partition-tolerant). Verified against a + real SQL Server 2022 and an in-process 2-node Akka cluster. Packed, **not yet published**. - ✅ LocalDb — `ZB.MOM.WW.LocalDb` (+ `.Contracts`, `.Replication`) built at `0.1.0` and **published to the feed 2026-07-18** (restore-verified from a scratch consumer). **Open** -- ⬜ Execute the G-7 clustered-replication plan (shared SQL-Server `ISecretStore`) — [`components/secrets/GAPS.md`](components/secrets/GAPS.md). -- ⬜ Bump the four Secrets consumers `0.1.2` → `0.1.3` (KEK rotation + the same security fix). +- ⬜ Publish Secrets `0.2.0` (5 packages) to the feed, then adopt a clustered topology in + ScadaBridge / OtOpcUa — config + a shared KEK, no code change. [`components/secrets/GAPS.md`](components/secrets/GAPS.md). +- ⬜ Bump the four Secrets consumers `0.1.2` → `0.1.3`/`0.2.0` (KEK rotation + the same security fix). - ⬜ Clear the pre-existing `AngleSharp` NU1902 build break in ScadaBridge `CentralUI.Tests` and HistorianGateway `Tests` — both fail full-solution builds under `TreatWarningsAsErrors`. - ⬜ Adopt `ZB.MOM.WW.LocalDb` in an app — no consumer references it yet. diff --git a/ZB.MOM.WW.Secrets/Directory.Build.props b/ZB.MOM.WW.Secrets/Directory.Build.props index 0b2b340..c3118d2 100644 --- a/ZB.MOM.WW.Secrets/Directory.Build.props +++ b/ZB.MOM.WW.Secrets/Directory.Build.props @@ -5,7 +5,7 @@ enable enable latest - 0.1.3 + 0.2.0 true README.md diff --git a/ZB.MOM.WW.Secrets/Directory.Packages.props b/ZB.MOM.WW.Secrets/Directory.Packages.props index a08993c..1b9476f 100644 --- a/ZB.MOM.WW.Secrets/Directory.Packages.props +++ b/ZB.MOM.WW.Secrets/Directory.Packages.props @@ -16,8 +16,25 @@ which carries advisory GHSA-2m69-gcr7-jv3q (NU1903). Pin the patched 2.1.12 native lib. --> + + + + + + + + + + - + + @@ -28,6 +45,8 @@ + + diff --git a/ZB.MOM.WW.Secrets/README.md b/ZB.MOM.WW.Secrets/README.md index e3f6e8c..1e74986 100644 --- a/ZB.MOM.WW.Secrets/README.md +++ b/ZB.MOM.WW.Secrets/README.md @@ -11,6 +11,8 @@ plaintext back on demand — from application code, from configuration, or from | `ZB.MOM.WW.Secrets.Abstractions` | Contracts only (`ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, `ISecretCacheInvalidator`, `ISecretActorAccessor`), value types, exceptions. Dependency-light. | | `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. | | `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. | +| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | Cluster-wide secrets via SQL Server — as one shared store, or as a hub each node syncs against. | +| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | Peer-to-peer replication over an Akka.NET cluster — partition-tolerant, no shared database. | A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed). @@ -78,8 +80,73 @@ SQL-Server store; once per node for per-node SQLite on a shared KEK). See the op ## Clustered deployments -Secrets storage is local SQLite by default. The schema already carries the -`revision` / `updated_utc` / tombstone columns and an `ISecretReplicator` seam for -cluster replication, but the Akka replicator (`ZB.MOM.WW.Secrets.Akka`) is a deferred -follow-on. **When replicating across a node pair, every node must resolve the _same_ master -KEK** (e.g. the same mounted key file) — a row wrapped by an unknown KEK fails closed. +Storage is local SQLite by default, which is correct for a single-process app and wrong the +moment a secret written on one node has to be readable on another. Three topologies are +supported; pick the least complicated one that meets the requirement. + +> **Every topology requires the same master KEK on every node.** The database and the wire carry +> ciphertext only, so a node with a different key cannot decrypt what another wrote — it fails +> closed on a `kek_id` mismatch, which looks like corruption but is a deployment mistake. Mount the +> same key file, or supply the same key environment variable, everywhere. + +### 1. Shared SQL-Server store — *the default choice* + +Every node points at one database. There is exactly one copy of each row, so there is no +replication, no reconciliation, no split-brain, and no clock skew to reason about. + +```csharp +builder.Services.AddZbSecretsSqlServerStore(builder.Configuration); +``` + +```jsonc +"Secrets": { + "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" }, + "SqlServer": { + "ConnectionString": "Server=sql01;Database=ZbSecrets;...", + "SchemaName": "zbsecrets" + } +} +``` + +**Trade-off:** availability is coupled to that database. A node cut off from it falls back only to +the resolver's short in-memory cache. Rotation (`rewrap-all`) runs **once**, against the one store. + +### 2. SQL-Server hub replication + +Each node keeps its local store and stays converged with a shared hub: writes publish immediately, +and a background sweep reconciles **both directions** on an interval (default 30s). + +```csharp +builder.Services.AddZbSecretsSqlServerReplication(builder.Configuration); +``` + +Choose this over (1) when a node must keep **resolving** secrets while cut off from the hub — reads +never leave the node, so a hub outage degrades propagation rather than availability. + +### 3. Akka cluster peer-to-peer + +No shared database at all. Nodes broadcast writes over distributed pub/sub and repair divergence +with a periodic manifest exchange. + +```csharp +builder.Services.AddZbSecretsAkkaReplication(builder.Configuration); +// Recommended — pins the serializer that carries ciphertext: +Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig); +``` + +Requires the application to register its `ActorSystem` in DI. The local store is SQLite. This is +the right answer for air-gapped or intermittently-connected sites; it is the wrong answer if every +node can always reach a shared database, because you pay real distributed-systems behaviour for it. + +### What replication costs you (2 and 3) + +- **Eventual consistency.** Rows converge within the sweep interval, not instantly. +- **Last-writer-wins.** Two nodes writing the same secret concurrently: the later `updated_utc` + wins and the other write is discarded. There is no merge and no conflict report. +- **Publishing is best-effort.** A failed broadcast is logged, not thrown — the local write is + already durable and anti-entropy repairs it. Writes never fail because a peer was unreachable. +- **KEK rotation does not replicate.** A re-wrap deliberately leaves the revision untouched so it + stays invisible to last-writer-wins. Run `rewrap-all` against **each** independent store. + +See [`docs/operations/clustered-secrets.md`](docs/operations/clustered-secrets.md) for operator +setup, and [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md) for rotation. diff --git a/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx b/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx index f74e9c1..5f821d9 100644 --- a/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx +++ b/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.slnx @@ -2,11 +2,15 @@ + + + + diff --git a/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md b/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md new file mode 100644 index 0000000..88d4e42 --- /dev/null +++ b/ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md @@ -0,0 +1,210 @@ +# Operating clustered secrets + +How to run `ZB.MOM.WW.Secrets` across more than one node. Covers the shared SQL-Server store, the +SQL-Server hub, and Akka peer-to-peer replication. + +Companion: [`kek-rotation.md`](kek-rotation.md). + +--- + +## 0. The one rule that breaks everything if you get it wrong + +**Every node must resolve the same master KEK.** + +Rows are ciphertext everywhere — in the database, in a backup, on the wire. Only the KEK decrypts +them, and it never leaves the node. So if node B has a different master key than node A: + +- replication still "works" — rows arrive and are stored; +- every resolve on B of a secret written on A **fails closed** with a `kek_id` mismatch. + +The symptom looks like data corruption and is not. Before enabling any topology below, confirm the +same key material is present on every node: + +```bash +# On each node — compare the digests, never print the key itself. +sha256sum /run/secrets/zb-kek +# or, for the environment source: +printf '%s' "$ZB_SECRETS_MASTER_KEY" | sha256sum +``` + +--- + +## 1. Shared SQL-Server store + +One database, one copy of each row. Prefer this unless a node must keep resolving secrets while +partitioned from the database. + +### Provision + +The schema is created automatically by the first node that starts (idempotent, and safe if several +start at once). You only need a database and a login: + +```sql +CREATE DATABASE ZbSecrets; +GO +USE ZbSecrets; +CREATE LOGIN zbsecrets WITH PASSWORD = ''; +CREATE USER zbsecrets FOR LOGIN zbsecrets; +GO +-- The app creates and owns the zbsecrets schema; it needs permission to do so. +GRANT CREATE SCHEMA, CREATE TABLE TO zbsecrets; +ALTER ROLE db_owner ADD MEMBER zbsecrets; -- or grant on the schema once it exists +GO +``` + +To pre-create the schema instead (so the app can run with narrower rights), run the DDL from +`SqlServerSecretsSchema.CreateSchemaDdl("zbsecrets")` and then grant only +`SELECT, INSERT, UPDATE` on `[zbsecrets].[secret]` and `[zbsecrets].[schema_version]`. + +> `DELETE` is deliberately **not** required. Deletes are tombstones — an `UPDATE` — so a compromised +> app credential cannot destroy secret history. + +### Configure + +```jsonc +"Secrets": { + "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" }, + "SqlServer": { + "ConnectionString": "Server=sql01;Database=ZbSecrets;User Id=zbsecrets;Password=${secret:sql/zbsecrets-password};TrustServerCertificate=False", + "SchemaName": "zbsecrets", + "CommandTimeout": "00:00:30" + } +} +``` + +The connection string is itself a credential — deliver it by `${secret:}` reference, an orchestrator +secret, or an environment variable. Never commit it. + +### Verify + +```bash +secret set smoke/test hello --description "delete me" # on node A +secret get smoke/test # on node B — expect: hello +secret rm smoke/test +``` + +### Rotation + +Run `rewrap-all` **once**, against the shared store, with resolve traffic quiesced. + +--- + +## 2. SQL-Server hub replication + +Each node keeps its local SQLite store and syncs against a shared hub. + +Same provisioning as topology 1. Configure with `AddZbSecretsSqlServerReplication` and: + +```jsonc +"Secrets": { + "SqlitePath": "/var/lib/zb/secrets.db", + "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" }, + "SqlServer": { + "ConnectionString": "...", + "SyncInterval": "00:00:30", + "SyncOnStartup": true + } +} +``` + +`SyncInterval` is the convergence bound, not the propagation latency — a live write reaches the hub +in milliseconds; the interval governs how fast a *dropped* write or a rejoining node is repaired. + +### Rotation + +Run `rewrap-all` against the hub **and** against every node's local SQLite store. A re-wrap does not +bump the revision (by design — otherwise every node's independent re-wrap would churn replication), +so it does **not** propagate. + +### Verifying convergence + +``` +Secret hub sync converged: pulled N row(s) from the hub, pushed M row(s) to it. +``` + +Logged at Information only when something actually moved. A healthy steady-state cluster logs +nothing. Continuous non-zero pushes on every tick means two nodes are fighting over the same secret +— look for a script writing the same name on a loop. + +--- + +## 3. Akka cluster peer-to-peer + +No shared database. Requires the application to register its `ActorSystem` in DI. + +```jsonc +"Secrets": { + "SqlitePath": "/var/lib/zb/secrets.db", + "MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" }, + "Replication": { + "AnnounceInterval": "00:00:30", + "ActorName": "zb-secret-replication" + } +} +``` + +Merge the serializer config when creating the actor system: + +```csharp +Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig); +ActorSystem.Create("my-system", config); +``` + +Optional but recommended: without it the protocol still round-trips under Akka's default JSON +serializer, but the serializer carrying secret ciphertext becomes an inherited default rather than +a deliberate choice. + +### Securing the transport + +Rows on the wire are ciphertext and the KEK never travels, so a passive eavesdropper learns +nothing but secret *names*, revisions, and timestamps. That is still metadata worth protecting, and +an unauthenticated cluster port is a much larger problem than secret replication — configure +Akka.Remote TLS and cluster authentication as you would for any other cluster traffic. + +### Rotation + +Run `rewrap-all` on **each node's** local store. Re-wraps do not replicate (see above). + +### Announce traffic + +One manifest per node per interval: names, revisions, timestamps, tombstone flags — no ciphertext. +Cost grows as O(nodes × secrets), so a large cluster with a large secret set should lengthen +`AnnounceInterval` rather than shorten it. + +--- + +## Choosing between them + +| | Shared store | SQL hub | Akka peer-to-peer | +|---|---|---|---| +| Resolves while cut off from the DB | ✗ (cache TTL only) | ✓ | ✓ (no DB at all) | +| Distributed failure modes | none | LWW, sweep lag | LWW, sweep lag | +| Rotation runs | once | hub + each node | each node | +| Needs a shared database | ✓ | ✓ | ✗ | +| Needs an Akka cluster | ✗ | ✗ | ✓ | + +Start at the left and move right only when a stated requirement forces it. + +--- + +## Troubleshooting + +**`SecretDecryptionException` / `kek_id` mismatch after enabling replication.** +The KEKs differ between nodes. Compare digests as in §0. Do not "fix" it by re-writing the secret on +the failing node — that hides the mismatch until the next secret. + +**A secret written on node A never appears on node B.** +Check, in order: (1) are both nodes actually in the cluster / pointed at the same hub database and +schema; (2) has one `AnnounceInterval` / `SyncInterval` elapsed; (3) is there a warning-level log +about a failed publish or a failed sweep. A dropped publish is expected and self-repairing — a +sweep that fails every interval is not. + +**A deleted secret came back.** +Deletes are tombstones and propagate as newer rows, so this should not happen through replication. +It does happen if a node's local store was restored from a backup taken before the delete: the +restore reintroduces the row with an old timestamp, loses to LWW, and the tombstone re-wins. If +instead the row is *live* again, someone re-created it — check the audit log for the write. + +**The migration refuses to run.** `SecretStoreMigrationException` naming a version newer than +supported means another node is running a newer build against the same store. Upgrade this node; +do not downgrade the store. diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretReplicator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretReplicator.cs index a11dce8..190e705 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretReplicator.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretReplicator.cs @@ -2,7 +2,8 @@ namespace ZB.MOM.WW.Secrets.Abstractions; /// /// Publishes an encrypted row to cluster peers. Core registers a no-op implementation; -/// ZB.MOM.WW.Secrets.Akka (deferred) provides the real cluster implementation. +/// ZB.MOM.WW.Secrets.Replication.Akka (peer-to-peer over the cluster) and +/// ZB.MOM.WW.Secrets.Replication.SqlServer (via a shared hub database) provide the real ones. /// public interface ISecretReplicator { diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretsStoreMigrator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretsStoreMigrator.cs new file mode 100644 index 0000000..1040d36 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/ISecretsStoreMigrator.cs @@ -0,0 +1,22 @@ +namespace ZB.MOM.WW.Secrets.Abstractions; + +/// +/// Provisions the schema backing an . Implementations are idempotent — +/// running a migration repeatedly is safe — and refuse to run against a store whose on-disk schema +/// version is newer than the build supports, rather than silently downgrading it. +/// +/// +/// This seam exists so the startup migration hosted service and the CLI stay store-agnostic: the +/// SQLite store, the shared SQL-Server store, and any future provider are all provisioned through +/// the same call, and swapping the store never touches the migration wiring. +/// +public interface ISecretsStoreMigrator +{ + /// Creates or updates the secret-store schema to the version this build supports. + /// A token to cancel the operation. + /// A task that completes when the schema is at the supported version. + /// + /// The store's schema version is newer than this build supports. + /// + Task MigrateAsync(CancellationToken cancellationToken); +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/SecretLastWriterWins.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/SecretLastWriterWins.cs new file mode 100644 index 0000000..3381329 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Abstractions/SecretLastWriterWins.cs @@ -0,0 +1,58 @@ +namespace ZB.MOM.WW.Secrets.Abstractions; + +/// +/// The single definition of the last-writer-wins (LWW) ordering used to reconcile a secret row +/// against a peer's copy: newer wins, and +/// breaks a timestamp tie. +/// +/// +/// +/// Every store's and every replication transport +/// reconciler routes its "is the incoming row newer?" decision through here. Keeping one definition +/// is a correctness requirement, not a tidiness one: a cluster can mix stores (a node backed by the +/// shared SQL-Server store alongside nodes on local SQLite), and two implementations that disagree +/// about a tie would converge to different rows on different nodes — a silent split-brain +/// that no single-node test can catch. +/// +/// +/// The comparison is deliberately strict: a row that merely ties the local one is not newer +/// and is ignored. That makes replication idempotent — re-delivering a row a node already has is a +/// no-op rather than a write — so at-least-once transports (pub/sub redelivery, an anti-entropy +/// sweep racing a live broadcast) cost nothing and cannot flap a row back and forth. +/// +/// +public static class SecretLastWriterWins +{ + /// + /// Returns if a row at (, + /// ) should overwrite a local row at + /// (, ). + /// + /// The incoming row's last-updated timestamp. + /// The incoming row's revision. + /// The local row's last-updated timestamp. + /// The local row's revision. + /// if the incoming row is strictly newer. + public static bool IsNewer( + DateTimeOffset incomingUpdatedUtc, + long incomingRevision, + DateTimeOffset localUpdatedUtc, + long localRevision) => + incomingUpdatedUtc > localUpdatedUtc || + (incomingUpdatedUtc == localUpdatedUtc && incomingRevision > localRevision); + + /// + /// Returns if should overwrite the local row + /// described by . + /// + /// The manifest entry received from a peer. + /// The corresponding local manifest entry. + /// if the incoming entry is strictly newer. + public static bool IsNewer(SecretManifestEntry incoming, SecretManifestEntry local) + { + ArgumentNullException.ThrowIfNull(incoming); + ArgumentNullException.ThrowIfNull(local); + + return IsNewer(incoming.UpdatedUtc, incoming.Revision, local.UpdatedUtc, local.Revision); + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli/Program.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli/Program.cs index e0ab45e..5423f81 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli/Program.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Cli/Program.cs @@ -1,12 +1,10 @@ using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.Secrets.Abstractions; using ZB.MOM.WW.Secrets.Cli; using ZB.MOM.WW.Secrets.DependencyInjection; using ZB.MOM.WW.Secrets.MasterKey; -using ZB.MOM.WW.Secrets.Sqlite; // Headless `secret` CLI: set / get / list / rm / rotate over the ZB.MOM.WW secrets store. // The heavy lifting lives in SecretCommands; Program only builds the host, runs the schema @@ -20,8 +18,9 @@ builder.Services.AddZbSecrets(builder.Configuration, "Secrets"); using IHost host = builder.Build(); -// Ensure the schema exists before any store operation. -await host.Services.GetRequiredService() +// Ensure the schema exists before any store operation. Resolved through the ISecretsStoreMigrator +// seam so the CLI provisions whichever store the app configured (local SQLite or shared SQL Server). +await host.Services.GetRequiredService() .MigrateAsync(CancellationToken.None); var commands = new SecretCommands( diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretReplicator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretReplicator.cs new file mode 100644 index 0000000..9f8d4f6 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretReplicator.cs @@ -0,0 +1,33 @@ +using Akka.Actor; +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet; + +/// +/// Publishes local secret writes to cluster peers by handing them to this node's +/// . +/// +/// +/// The hand-off is a Tell, so completes as soon as the row is in +/// the actor's mailbox rather than when peers have applied it. That is the correct contract here: +/// the caller (ReplicatingSecretStore) has already durably written the row locally and treats +/// publishing as best-effort, and blocking a write until every peer acknowledged would make each +/// node's availability depend on all the others. Delivery is guaranteed by anti-entropy, not by +/// this call. +/// +/// This node's replication actor. +public sealed class AkkaSecretReplicator(IActorRef replicationActor) : ISecretReplicator +{ + private readonly IActorRef _replicationActor = + replicationActor ?? throw new ArgumentNullException(nameof(replicationActor)); + + /// + public Task PublishAsync(StoredSecret row, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(row); + + _replicationActor.Tell(new SecretReplicationActor.PublishLocalWrite(row)); + + return Task.CompletedTask; + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretsReplication.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretsReplication.cs new file mode 100644 index 0000000..d3e508f --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretsReplication.cs @@ -0,0 +1,41 @@ +using Akka.Configuration; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet; + +/// +/// Akka configuration this package contributes. +/// +public static class AkkaSecretsReplication +{ + /// + /// HOCON binding the secret-replication wire protocol to + /// . Merge it into the application's Akka + /// configuration when creating the ActorSystem: + /// + /// Config config = AkkaSecretsReplication.SerializationConfig + /// .WithFallback(myAppConfig); + /// ActorSystem.Create("my-system", config); + /// + /// + /// + /// Optional — the protocol DTOs round-trip under Akka's default JSON serializer without it — but + /// recommended, because it makes the serializer carrying secret ciphertext an explicit choice + /// rather than an inherited default, and pins the manifest strings that a future protocol + /// version would bump. + /// + public static Config SerializationConfig { get; } = ConfigurationFactory.ParseString($$""" + akka.actor { + serializers { + zb-secrets-replication = "{{typeof(SecretReplicationSerializer).AssemblyQualifiedName}}" + } + serialization-bindings { + "ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol.SecretRowsMessage, ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" = zb-secrets-replication + "ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol.SecretManifestAnnounce, ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" = zb-secrets-replication + "ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol.SecretPullRequest, ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" = zb-secrets-replication + } + serialization-identifiers { + "{{typeof(SecretReplicationSerializer).AssemblyQualifiedName}}" = {{SecretReplicationSerializer.SerializerId}} + } + } + """); +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretsReplicationOptions.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretsReplicationOptions.cs new file mode 100644 index 0000000..ca354ff --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/AkkaSecretsReplicationOptions.cs @@ -0,0 +1,40 @@ +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet; + +/// +/// Configuration for peer-to-peer secret replication over the Akka cluster. +/// +public sealed class AkkaSecretsReplicationOptions +{ + /// + /// How often this node broadcasts its manifest for anti-entropy. Defaults to 30 seconds. + /// + /// + /// This is the convergence bound, not the propagation latency: a live write reaches peers in + /// milliseconds, and this interval only governs how quickly a dropped write or a + /// rejoining node is repaired. The traffic is one manifest per node per interval — names, + /// revisions, and timestamps, no ciphertext — so shortening it is cheap for a small cluster and + /// grows as O(nodes × secrets). + /// + public TimeSpan AnnounceInterval { get; set; } = TimeSpan.FromSeconds(30); + + /// Name of the replication actor within the actor system. Defaults to zb-secret-replication. + public string ActorName { get; set; } = "zb-secret-replication"; + + /// + /// Throws if the options are unusable. + /// + /// The announce interval is not positive, or the actor name is blank. + public void Validate() + { + if (AnnounceInterval <= TimeSpan.Zero) + { + throw new InvalidOperationException( + $"Secret replication AnnounceInterval must be positive (was {AnnounceInterval})."); + } + + if (string.IsNullOrWhiteSpace(ActorName)) + { + throw new InvalidOperationException("Secret replication ActorName must not be blank."); + } + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/DependencyInjection/AkkaSecretsServiceCollectionExtensions.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/DependencyInjection/AkkaSecretsServiceCollectionExtensions.cs new file mode 100644 index 0000000..97e7383 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/DependencyInjection/AkkaSecretsServiceCollectionExtensions.cs @@ -0,0 +1,124 @@ +using Akka.Actor; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.DependencyInjection; +using ZB.MOM.WW.Secrets.Sqlite; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection; + +/// +/// Dependency-injection entry point for peer-to-peer secret replication over an Akka.NET cluster. +/// +public static class AkkaSecretsServiceCollectionExtensions +{ + /// + /// Registers secret replication over the cluster: each node keeps its own local store, local + /// writes broadcast to peers, and a periodic manifest exchange repairs any divergence in both + /// directions. + /// + /// + /// + /// Choose this transport when a node must keep resolving secrets while partitioned from + /// the rest of the cluster — reads never leave the node, so an isolated site keeps running on its + /// last-known-good state and re-converges when the partition heals. If every node can always + /// reach a shared database, the SQL-Server package's shared-store mode is simpler and has no + /// distributed failure modes at all; prefer it unless partition tolerance is a stated requirement. + /// + /// + /// Requirements. The application must (1) register its in DI + /// before the first secret resolve, and (2) give every node the same KEK — rows carry + /// ciphertext only, so a node with a different master key fails closed on a kek_id + /// mismatch. Merging into the actor + /// system's configuration is recommended. + /// + /// + /// KEK rotation does not replicate. A re-wrap deliberately leaves the revision untouched + /// so it stays invisible to last-writer-wins; run rewrap-all against every node's local + /// store. + /// + /// + /// The service collection to add to. + /// The application configuration. + /// Configuration section holding the core secrets options. + /// Configuration section holding the replication options. + /// The same instance, for chaining. + public static IServiceCollection AddZbSecretsAkkaReplication( + this IServiceCollection services, + IConfiguration config, + string secretsSectionPath = "Secrets", + string replicationSectionPath = "Secrets:Replication") + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(config); + ArgumentException.ThrowIfNullOrWhiteSpace(secretsSectionPath); + ArgumentException.ThrowIfNullOrWhiteSpace(replicationSectionPath); + + IConfigurationSection section = config.GetSection(replicationSectionPath); + services.Configure(section); + + // Validate eagerly so a bad interval fails at startup, not on the first announce tick. + var eager = new AkkaSecretsReplicationOptions(); + section.Bind(eager); + eager.Validate(); + + // The local store + migrator come from AddZbSecrets (SQLite by default). + services.AddZbSecrets(config, secretsSectionPath); + + // The actor is created lazily on first use rather than at registration: the ActorSystem is + // typically registered by the application AFTER this call, and a cluster node may also not be + // ready to join at the moment the container is built. + services.TryAddSingleton(sp => new SecretReplicationActorProvider( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetService(), + sp.GetRequiredService>().Value)); + + services.TryAddSingleton(sp => + new AkkaSecretReplicator(sp.GetRequiredService().ActorRef)); + + // Decorate the local store so every write publishes. Resolved from the concrete SQLite store, + // not ISecretStore — asking the container for ISecretStore here would resolve this very + // decorator and recurse forever. + services.AddSingleton(sp => new ReplicatingSecretStore( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService>())); + + return services; + } +} + +/// +/// Creates this node's on first access and hands out its ref. +/// +/// +/// The indirection exists because the actor must be created lazily — the application usually +/// registers its after AddZbSecretsAkkaReplication, so eagerly +/// resolving one at registration time would fail. Creation is guarded by +/// with so +/// concurrent first writes cannot spawn two actors on the same node. +/// +/// The application's actor system. +/// The node's local secret store — the undecorated one. +/// Resolver-cache seam. +/// Replication options. +public sealed class SecretReplicationActorProvider( + ActorSystem system, + ISecretStore store, + ISecretCacheInvalidator? cacheInvalidator, + AkkaSecretsReplicationOptions options) +{ + private readonly Lazy _actorRef = new( + () => system.ActorOf( + SecretReplicationActor.Props(store, cacheInvalidator, options.AnnounceInterval), + options.ActorName), + LazyThreadSafetyMode.ExecutionAndPublication); + + /// This node's replication actor, created on first access. + public IActorRef ActorRef => _actorRef.Value; +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/Protocol/SecretReplicationMessages.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/Protocol/SecretReplicationMessages.cs new file mode 100644 index 0000000..4f2a68e --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/Protocol/SecretReplicationMessages.cs @@ -0,0 +1,215 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol; + +/// +/// The wire representation of one encrypted secret row. +/// +/// +/// +/// This is a deliberate hand-written mirror of rather than the type +/// itself, and the duplication is the point. It makes the trust boundary structural: the wire +/// contract is a flat set of primitives that a reader can check against "ciphertext only, never the +/// KEK" in one glance, and a field added to in future cannot start +/// crossing the network merely by existing — someone has to add it here on purpose. +/// +/// +/// All fields are primitives (, arrays, , +/// ) so the DTO round-trips through any serializer, including Akka's default JSON +/// one if an application never registers . Timestamps are +/// ISO-8601 "O" strings for the same reason the stores use them: last-writer-wins compares +/// them for equality, so the representation has to be exact rather than merely close. +/// +/// +public sealed record SecretRowDto +{ + /// The normalized secret name. + public required string Name { get; init; } + + /// Optional human-readable description. + public string? Description { get; init; } + + /// The enum name. + public required string ContentType { get; init; } + + /// AES-256-GCM ciphertext of the secret plaintext. + public required byte[] Ciphertext { get; init; } + + /// Nonce sealing . + public required byte[] Nonce { get; init; } + + /// Authentication tag for . + public required byte[] Tag { get; init; } + + /// The per-secret data key, wrapped by the KEK. The KEK itself never travels. + public required byte[] WrappedDek { get; init; } + + /// Nonce used to wrap . + public required byte[] WrapNonce { get; init; } + + /// Authentication tag for . + public required byte[] WrapTag { get; init; } + + /// Identifier of the KEK that wrapped the DEK — an identifier, not key material. + public required string KekId { get; init; } + + /// Monotonic revision. + public required long Revision { get; init; } + + /// Whether the row is a tombstone. + public bool IsDeleted { get; init; } + + /// ISO-8601 deletion timestamp, or . + public string? DeletedUtc { get; init; } + + /// ISO-8601 creation timestamp. + public required string CreatedUtc { get; init; } + + /// ISO-8601 last-update timestamp — half of the last-writer-wins ordering key. + public required string UpdatedUtc { get; init; } + + /// Principal that created the row, if known. + public string? CreatedBy { get; init; } + + /// Principal that last updated the row, if known. + public string? UpdatedBy { get; init; } + + /// Projects a stored row onto the wire contract. + /// The encrypted row. + /// The wire representation. + public static SecretRowDto FromStoredSecret(StoredSecret row) + { + ArgumentNullException.ThrowIfNull(row); + + return new SecretRowDto + { + Name = row.Name.Value, + Description = row.Description, + ContentType = row.ContentType.ToString(), + Ciphertext = row.Ciphertext, + Nonce = row.Nonce, + Tag = row.Tag, + WrappedDek = row.WrappedDek, + WrapNonce = row.WrapNonce, + WrapTag = row.WrapTag, + KekId = row.KekId, + Revision = row.Revision, + IsDeleted = row.IsDeleted, + DeletedUtc = row.DeletedUtc?.ToString("O"), + CreatedUtc = row.CreatedUtc.ToString("O"), + UpdatedUtc = row.UpdatedUtc.ToString("O"), + CreatedBy = row.CreatedBy, + UpdatedBy = row.UpdatedBy, + }; + } + + /// Rebuilds the stored row from the wire contract. + /// The encrypted row. + /// The name or content type is not valid. + public StoredSecret ToStoredSecret() => new() + { + // SecretName re-validates on construction, so a malformed or path-traversing name from a + // peer is rejected here rather than reaching the store. + Name = new SecretName(Name), + Description = Description, + ContentType = ParseContentType(ContentType), + // `required` on a byte[] means "present in the payload", NOT "non-null" — a peer sending an + // explicit JSON null satisfies it and yields a null array, which would then blow up deep in + // the store's parameter binding instead of being rejected at the boundary. + Ciphertext = RequireBlob(Ciphertext, nameof(Ciphertext)), + Nonce = RequireBlob(Nonce, nameof(Nonce)), + Tag = RequireBlob(Tag, nameof(Tag)), + WrappedDek = RequireBlob(WrappedDek, nameof(WrappedDek)), + WrapNonce = RequireBlob(WrapNonce, nameof(WrapNonce)), + WrapTag = RequireBlob(WrapTag, nameof(WrapTag)), + KekId = KekId ?? throw new ArgumentException("Replicated secret row has a null KekId."), + Revision = Revision, + IsDeleted = IsDeleted, + DeletedUtc = DeletedUtc is null ? null : ParseUtc(DeletedUtc), + CreatedUtc = ParseUtc(CreatedUtc), + UpdatedUtc = ParseUtc(UpdatedUtc), + CreatedBy = CreatedBy, + UpdatedBy = UpdatedBy, + }; + + private static DateTimeOffset ParseUtc(string value) => DateTimeOffset.Parse( + value, + System.Globalization.CultureInfo.InvariantCulture, + System.Globalization.DateTimeStyles.RoundtripKind); + + private static byte[] RequireBlob(byte[]? value, string field) => + value ?? throw new ArgumentException($"Replicated secret row has a null {field}.", field); + + // Enum.Parse is wrong for untrusted input on two counts: a numeric string that overflows the + // underlying type throws OverflowException (which the receiving actor's ArgumentException filter + // would not catch, restarting it in a redelivery loop), and any in-range number is accepted even + // if it names no member — persisting an undefined enum value that later readers assume is valid. + private static SecretContentType ParseContentType(string value) => + Enum.TryParse(value, out SecretContentType parsed) && Enum.IsDefined(parsed) + ? parsed + : throw new ArgumentException( + $"Replicated secret row has an unrecognized content type '{value}'.", nameof(value)); +} + +/// One manifest entry on the wire: enough to decide who is newer, with no ciphertext. +public sealed record SecretManifestEntryDto +{ + /// The normalized secret name. + public required string Name { get; init; } + + /// Monotonic revision. + public required long Revision { get; init; } + + /// ISO-8601 last-update timestamp. + public required string UpdatedUtc { get; init; } + + /// Whether the row is a tombstone. + public bool IsDeleted { get; init; } + + /// Projects a manifest entry onto the wire contract. + /// The manifest entry. + /// The wire representation. + public static SecretManifestEntryDto FromEntry(SecretManifestEntry entry) + { + ArgumentNullException.ThrowIfNull(entry); + + return new SecretManifestEntryDto + { + Name = entry.Name.Value, + Revision = entry.Revision, + UpdatedUtc = entry.UpdatedUtc.ToString("O"), + IsDeleted = entry.IsDeleted, + }; + } + + /// Rebuilds the manifest entry from the wire contract. + /// The manifest entry. + public SecretManifestEntry ToEntry() => new() + { + Name = new SecretName(Name), + Revision = Revision, + UpdatedUtc = DateTimeOffset.Parse( + UpdatedUtc, + System.Globalization.CultureInfo.InvariantCulture, + System.Globalization.DateTimeStyles.RoundtripKind), + IsDeleted = IsDeleted, + }; +} + +/// +/// A node broadcasting its full inventory so peers can work out, in one round trip, both what they +/// need from it and what it needs from them. +/// +/// The sender's manifest. +public sealed record SecretManifestAnnounce(IReadOnlyList Entries); + +/// A peer asking for the encrypted rows behind names it found it was missing or behind on. +/// The requested secret names. +public sealed record SecretPullRequest(IReadOnlyList Names); + +/// +/// Encrypted rows delivered to a peer — the reply to a , the +/// unsolicited push half of an anti-entropy exchange, and the live broadcast of a fresh local write. +/// +/// The encrypted rows. +public sealed record SecretRowsMessage(IReadOnlyList Rows); diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs new file mode 100644 index 0000000..8e571cc --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationActor.cs @@ -0,0 +1,303 @@ +using Akka.Actor; +using Akka.Cluster.Tools.PublishSubscribe; +using Akka.Event; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet; + +/// +/// One per node. Owns this node's participation in secret replication: broadcasts local writes, +/// periodically announces its manifest, and services peers' pulls. +/// +/// +/// +/// The protocol is deliberately small — three messages, no leader, no singleton: +/// +/// +/// +/// Live push. A local write publishes a to the topic. Peers +/// apply it last-writer-wins. This is the fast path — propagation in milliseconds — and it is +/// allowed to fail. +/// +/// +/// Anti-entropy. Every node periodically publishes a . +/// A peer receiving it computes both directions in one pass: it pushes back rows it holds +/// newer, and asks () for rows it is missing or behind on. This is +/// the correctness path — it repairs anything the fast path dropped, in either direction. +/// +/// +/// Pull service. A is answered with the requested rows. +/// +/// +/// +/// There is no coordinator because there does not need to be one: every exchange is idempotent and +/// commutative under last-writer-wins, so nodes converge regardless of message order, duplication, +/// or which subset of peers is currently reachable. A node that was partitioned keeps serving its +/// local store the whole time and re-converges on the first announce after the partition heals — +/// which is the entire reason to choose this transport over the shared SQL-Server store. +/// +/// +public sealed class SecretReplicationActor : ReceiveActor, IWithTimers +{ + /// The distributed pub/sub topic secret replication traffic rides on. + public const string Topic = "zb-mom-ww-secrets"; + + private readonly ISecretStore _store; + private readonly SecretReplicationReconciler _reconciler; + private readonly TimeSpan _announceInterval; + private readonly IActorRef _mediator; + private readonly ILoggingAdapter _log = Context.GetLogger(); + + /// + public ITimerScheduler Timers { get; set; } = null!; + + /// Creates the replication actor. + /// The node's local secret store. + /// Resolver-cache seam, evicted for every applied row. + /// How often this node announces its manifest. + public SecretReplicationActor( + ISecretStore store, + ISecretCacheInvalidator? cacheInvalidator, + TimeSpan announceInterval) + { + _store = store ?? throw new ArgumentNullException(nameof(store)); + _reconciler = new SecretReplicationReconciler( + _store, + cacheInvalidator, + onRowFailed: (name, ex) => _log.Warning( + ex, "Discarding replicated secret row {0} that could not be applied locally.", name.Value)); + _announceInterval = announceInterval; + _mediator = DistributedPubSub.Get(Context.System).Mediator; + + _mediator.Tell(new Subscribe(Topic, Self)); + + Receive(HandlePublishLocalWrite); + Receive(HandleRows); + Receive(HandleManifestAnnounce); + Receive(HandlePullRequest); + Receive(_ => HandleAnnounceTick()); + Receive(_ => _log.Debug("Subscribed to secret replication topic {0}.", Topic)); + Receive(f => + _log.Warning(f.Cause, "Secret anti-entropy exchange with a peer failed; retrying next announce.")); + Receive(_ => { /* successful exchange, piped back to self — nothing to do. */ }); + } + + /// Props for the replication actor. + /// The node's local secret store. + /// Resolver-cache seam. + /// How often this node announces its manifest. + /// Props creating the actor. + public static Props Props( + ISecretStore store, ISecretCacheInvalidator? cacheInvalidator, TimeSpan announceInterval) => + Akka.Actor.Props.Create(() => + new SecretReplicationActor(store, cacheInvalidator, announceInterval)); + + /// + protected override void PreStart() => + Timers.StartPeriodicTimer(AnnounceTick.Instance, AnnounceTick.Instance, _announceInterval, _announceInterval); + + // A local write, handed over by AkkaSecretReplicator. Published from inside the actor so the + // message carries Self as sender — that is what lets a peer reply directly to this node instead + // of broadcasting its answer to everyone. + private void HandlePublishLocalWrite(PublishLocalWrite message) + { + _mediator.Tell(new Publish( + Topic, + new SecretRowsMessage([SecretRowDto.FromStoredSecret(message.Row)]))); + } + + private void HandleRows(SecretRowsMessage message) + { + if (IsFromSelf()) + { + return; + } + + IReadOnlyList rows = Materialize(message.Rows); + + if (rows.Count == 0) + { + return; + } + + // PipeTo nothing: applying is fire-and-forget from the mailbox's point of view. Failures are + // logged, and anti-entropy re-delivers anything that did not land. + _reconciler.ApplyAsync(rows, CancellationToken.None) + .ContinueWith( + task => _log.Warning( + task.Exception, + "Failed to apply {0} replicated secret row(s) from a peer; anti-entropy will retry.", + rows.Count), + CancellationToken.None, + TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + } + + private void HandleManifestAnnounce(SecretManifestAnnounce message) + { + if (IsFromSelf()) + { + return; + } + + IActorRef peer = Sender; + IReadOnlyList remote = [.. message.Entries.Select(e => e.ToEntry())]; + + ReconcileWithPeerAsync(peer, remote).PipeTo(Self, failure: ex => new ExchangeFailed(ex)); + } + + private void HandlePullRequest(SecretPullRequest message) + { + if (IsFromSelf()) + { + return; + } + + IActorRef peer = Sender; + // Captured before the continuation: Self is context-dependent and there is no guarantee the + // actor context is still current when the continuation runs (Akka analyzer AK1005). + IActorRef self = Self; + IReadOnlyList names = ParseNames(message.Names); + + _reconciler.ReadLocalAsync(names, CancellationToken.None) + .ContinueWith( + task => + { + if (task.IsFaulted) + { + // Logged rather than swallowed: the peer is waiting for a reply it will never + // get, and without this the only symptom is a secret that quietly never + // converges. The peer re-asks on its next announce. + _log.Warning( + task.Exception, + "Failed to read {0} local secret row(s) for a peer's pull request.", + names.Count); + return; + } + + if (task.IsCompletedSuccessfully && task.Result.Count > 0) + { + peer.Tell( + new SecretRowsMessage([.. task.Result.Select(SecretRowDto.FromStoredSecret)]), + self); + } + }, + CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + } + + private void HandleAnnounceTick() + { + _store.GetManifestAsync(CancellationToken.None) + .ContinueWith( + task => + { + if (task.IsCompletedSuccessfully) + { + _mediator.Tell(new Publish( + Topic, + new SecretManifestAnnounce( + [.. task.Result.Select(SecretManifestEntryDto.FromEntry)]))); + } + }, + CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously, + TaskScheduler.Default); + } + + // Both halves of an anti-entropy exchange, computed from one manifest: push what the peer is + // behind on, ask for what we are behind on. + private async Task ReconcileWithPeerAsync( + IActorRef peer, IReadOnlyList remote) + { + IReadOnlyList localManifest = + await _store.GetManifestAsync(CancellationToken.None).ConfigureAwait(false); + + IReadOnlyList push = + SecretReplicationReconciler.ComputePushSet(localManifest, remote); + + if (push.Count > 0) + { + IReadOnlyList rows = + await _reconciler.ReadLocalAsync(push, CancellationToken.None).ConfigureAwait(false); + + if (rows.Count > 0) + { + peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), Self); + } + } + + IReadOnlyList pull = + SecretReplicationReconciler.ComputePullSet(localManifest, remote); + + if (pull.Count > 0) + { + peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), Self); + } + + return ExchangeDone.Instance; + } + + // DistributedPubSub delivers a node's own publications back to it. Dropping them here is what + // stops a broadcast from being re-applied locally and, worse, echoed onward. + private bool IsFromSelf() => Sender.Equals(Self); + + // A peer's rows are untrusted input: a malformed name or content type must drop that row, not + // fault the actor and stall replication for every other secret. + private IReadOnlyList Materialize(IReadOnlyList dtos) + { + var rows = new List(dtos.Count); + + foreach (SecretRowDto dto in dtos) + { + try + { + rows.Add(dto.ToStoredSecret()); + } + catch (Exception ex) when (ex is ArgumentException or FormatException) + { + _log.Warning(ex, "Discarding a malformed replicated secret row from a peer."); + } + } + + return rows; + } + + private IReadOnlyList ParseNames(IReadOnlyList names) + { + var parsed = new List(names.Count); + + foreach (string name in names) + { + try + { + parsed.Add(new SecretName(name)); + } + catch (ArgumentException ex) + { + _log.Warning(ex, "Discarding a malformed secret name in a peer's pull request."); + } + } + + return parsed; + } + + /// Instruction from to broadcast a local write. + /// The encrypted row to broadcast. + public sealed record PublishLocalWrite(StoredSecret Row); + + private sealed record AnnounceTick + { + public static readonly AnnounceTick Instance = new(); + } + + private sealed record ExchangeFailed(Exception Cause); + + private sealed record ExchangeDone + { + public static readonly ExchangeDone Instance = new(); + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationSerializer.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationSerializer.cs new file mode 100644 index 0000000..d8b3679 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/SecretReplicationSerializer.cs @@ -0,0 +1,84 @@ +using System.Runtime.Serialization; +using System.Text.Json; +using Akka.Actor; +using Akka.Serialization; +using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet; + +/// +/// Explicit System.Text.Json serializer for the secret-replication wire protocol. +/// +/// +/// +/// Registering this is optional but recommended. The protocol DTOs are all primitives, so +/// Akka's default JSON serializer round-trips them correctly if an application never wires this up — +/// but relying on the default means secret ciphertext is serialized by whatever the cluster's +/// fallback happens to be, which is not a decision worth leaving implicit for this payload. Binding +/// it explicitly also pins the manifest strings below as the versioning boundary: a future protocol +/// change adds a manifest, so a mixed-version cluster fails loudly on an unknown manifest instead of +/// silently mis-deserializing a secret. +/// +/// +/// Wire it up by merging into the +/// application's Akka configuration — see that member for the HOCON. +/// +/// +/// The actor system this serializer belongs to. +public sealed class SecretReplicationSerializer(ExtendedActorSystem system) : SerializerWithStringManifest(system) +{ + /// Stable serializer id. Must not collide with other serializers in the cluster. + public const int SerializerId = 7710; + + private const string RowsManifest = "zbs-rows-v1"; + private const string ManifestAnnounceManifest = "zbs-manifest-v1"; + private const string PullRequestManifest = "zbs-pull-v1"; + + private static readonly JsonSerializerOptions JsonOptions = new() + { + // Explicitly NOT indented and NOT camel-cased: the wire form should be compact and stable, + // not pretty. Property names are the C# names, pinned by the manifest version. + WriteIndented = false, + }; + + /// + public override int Identifier => SerializerId; + + /// + public override string Manifest(object o) => o switch + { + SecretRowsMessage => RowsManifest, + SecretManifestAnnounce => ManifestAnnounceManifest, + SecretPullRequest => PullRequestManifest, + _ => throw new ArgumentException( + $"{nameof(SecretReplicationSerializer)} cannot serialize {o?.GetType().FullName ?? "null"}.", + nameof(o)), + }; + + /// + public override byte[] ToBinary(object obj) => obj switch + { + SecretRowsMessage rows => JsonSerializer.SerializeToUtf8Bytes(rows, JsonOptions), + SecretManifestAnnounce announce => JsonSerializer.SerializeToUtf8Bytes(announce, JsonOptions), + SecretPullRequest pull => JsonSerializer.SerializeToUtf8Bytes(pull, JsonOptions), + _ => throw new ArgumentException( + $"{nameof(SecretReplicationSerializer)} cannot serialize {obj?.GetType().FullName ?? "null"}.", + nameof(obj)), + }; + + /// + public override object FromBinary(byte[] bytes, string manifest) => manifest switch + { + RowsManifest => Deserialize(bytes, manifest), + ManifestAnnounceManifest => Deserialize(bytes, manifest), + PullRequestManifest => Deserialize(bytes, manifest), + _ => throw new SerializationException( + $"Unknown secret-replication manifest '{manifest}'. This usually means a peer is running a " + + "newer protocol version than this node."), + }; + + private static T Deserialize(byte[] bytes, string manifest) => + JsonSerializer.Deserialize(bytes, JsonOptions) + ?? throw new SerializationException( + $"Secret-replication payload with manifest '{manifest}' deserialized to null."); +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.csproj b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.csproj new file mode 100644 index 0000000..92b9f1e --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.csproj @@ -0,0 +1,41 @@ + + + + net10.0 + enable + enable + true + true + + + + true + ZB.MOM.WW.Secrets.Replicator.AkkaDotNet + ZB.MOM.WW + Peer-to-peer secret replication over an Akka.NET cluster for the ZB.MOM.WW SCADA family — partition-tolerant, ciphertext-only. + https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets + https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/DependencyInjection/SqlServerSecretsServiceCollectionExtensions.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/DependencyInjection/SqlServerSecretsServiceCollectionExtensions.cs new file mode 100644 index 0000000..030a00f --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/DependencyInjection/SqlServerSecretsServiceCollectionExtensions.cs @@ -0,0 +1,141 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.DependencyInjection; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection; + +/// +/// Dependency-injection entry points for backing a cluster's secrets with SQL Server, in either of +/// the two supported topologies. +/// +/// +/// +/// Both require the same KEK on every node. The database stores ciphertext only, so a node +/// whose master key differs cannot decrypt rows another node wrote — it fails closed on resolve with +/// a kek_id mismatch, which reads like data corruption but is a deployment error. Mount the +/// same key file or supply the same key environment variable everywhere. +/// +/// +public static class SqlServerSecretsServiceCollectionExtensions +{ + /// + /// Shared-store mode. Points every node's at one shared + /// SQL-Server database. There is exactly one copy of each row, so there is nothing to replicate + /// and nothing to reconcile — no split-brain, no anti-entropy lag, no clock skew. + /// + /// + /// Prefer this unless a node must keep resolving secrets while partitioned from the + /// shared database. The trade is availability: a node cut off from the database falls back only + /// to the resolver's short in-memory cache. If that is unacceptable, use + /// (local store + hub) or the Akka package + /// (peer-to-peer, no shared database at all). + /// + /// The service collection to add to. + /// The application configuration. + /// Configuration section holding the core secrets options. + /// Configuration section holding the SQL-Server options. + /// The same instance, for chaining. + public static IServiceCollection AddZbSecretsSqlServerStore( + this IServiceCollection services, + IConfiguration config, + string secretsSectionPath = "Secrets", + string sqlServerSectionPath = "Secrets:SqlServer") + { + RegisterSqlServerCore(services, config, sqlServerSectionPath); + + // Registered BEFORE AddZbSecrets so its TryAdd calls find these already present and leave + // them alone — that is the whole mechanism by which the SQL-Server store displaces SQLite. + services.TryAddSingleton(sp => sp.GetRequiredService()); + services.TryAddSingleton(sp => + sp.GetRequiredService()); + + services.AddZbSecrets(config, secretsSectionPath); + + return services; + } + + /// + /// Hub-replication mode. Each node keeps its own local store (SQLite by default) and + /// stays converged with a shared SQL-Server hub: local writes publish to the hub immediately, and + /// a background sweep reconciles both directions on an interval. + /// + /// + /// Choose this over when a node must keep resolving + /// secrets while cut off from the hub — reads are served from local state, so a hub outage + /// degrades propagation rather than availability. The cost is real distributed-systems + /// behaviour: rows converge eventually (bounded by the sweep interval), conflicting concurrent + /// writes to the same secret resolve last-writer-wins, and KEK rotation must be run against the + /// hub and each node's local store, because a re-wrap deliberately does not replicate. + /// + /// The service collection to add to. + /// The application configuration. + /// Configuration section holding the core secrets options. + /// Configuration section holding the SQL-Server options. + /// The same instance, for chaining. + public static IServiceCollection AddZbSecretsSqlServerReplication( + this IServiceCollection services, + IConfiguration config, + string secretsSectionPath = "Secrets", + string sqlServerSectionPath = "Secrets:SqlServer") + { + RegisterSqlServerCore(services, config, sqlServerSectionPath); + + services.TryAddSingleton(sp => sp.GetRequiredService()); + + services.TryAddSingleton(sp => + new SqlServerSecretReplicator(sp.GetRequiredService())); + + // The local store + its migrator come from AddZbSecrets (SQLite by default). + services.AddZbSecrets(config, secretsSectionPath); + + // Decorate the local store so every write publishes. Resolved from the concrete SQLite store + // rather than ISecretStore, because ISecretStore is what we are replacing — asking the + // container for it here would resolve this very decorator and recurse forever. + services.AddSingleton(sp => new ReplicatingSecretStore( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService>())); + + // The hub's schema must exist too — the node's own migrator only provisions the local store. + services.AddHostedService(); + services.AddHostedService(); + + return services; + } + + // Registers the pieces both modes need: validated options, connection factory, hub store, and + // hub migrator. + private static void RegisterSqlServerCore( + IServiceCollection services, + IConfiguration config, + string sqlServerSectionPath) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(config); + ArgumentException.ThrowIfNullOrWhiteSpace(sqlServerSectionPath); + + IConfigurationSection section = config.GetSection(sqlServerSectionPath); + services.Configure(section); + + // Bind and validate NOW as well as through the options pipeline: a missing connection string + // or a bad schema name should fail at startup with a clear message, not on the first resolve. + var eager = new SqlServerSecretsOptions(); + section.Bind(eager); + eager.Validate(); + + services.TryAddSingleton(sp => + new SecretsSqlServerConnectionFactory( + sp.GetRequiredService>().Value)); + + services.TryAddSingleton(sp => + new SqlServerSecretStore(sp.GetRequiredService())); + + services.TryAddSingleton(sp => + new SqlServerSecretsStoreMigrator(sp.GetRequiredService())); + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ISecretReplicationHub.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ISecretReplicationHub.cs new file mode 100644 index 0000000..b9bc7bd --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ISecretReplicationHub.cs @@ -0,0 +1,34 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// The subset of store operations hub replication needs from the shared database: compare +/// inventories, fetch rows in bulk, and accept a row verbatim. +/// +/// +/// Narrower than on purpose. The hub is never resolved from, never +/// written through (which would stamp the hub's own revision +/// and bounce the row back), and never rewrapped by a syncing node — so exposing those operations +/// would only create ways to use it wrongly. It also makes the sync sweep testable against an +/// in-process fake instead of requiring a live SQL Server for every convergence case. +/// +public interface ISecretReplicationHub +{ + /// Returns the hub's manifest — names, revisions, timestamps, tombstone flags. + /// A token to cancel the operation. + /// The hub's manifest entries. + Task> GetManifestAsync(CancellationToken ct); + + /// Fetches several encrypted rows by name; names absent from the hub are omitted. + /// The names to fetch. + /// A token to cancel the operation. + /// The matching encrypted rows. + Task> GetManyAsync(IReadOnlyList names, CancellationToken ct); + + /// Applies a row to the hub verbatim, last-writer-wins. + /// The encrypted row. + /// A token to cancel the operation. + /// A task that completes when the row has been applied or ignored. + Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct); +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SecretsSqlServerConnectionFactory.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SecretsSqlServerConnectionFactory.cs new file mode 100644 index 0000000..d0195b9 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SecretsSqlServerConnectionFactory.cs @@ -0,0 +1,57 @@ +using Microsoft.Data.SqlClient; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Opens connections to the shared SQL-Server secret database and carries the settings every +/// command needs (schema name, command timeout). +/// +/// The validated SQL-Server store options. +public sealed class SecretsSqlServerConnectionFactory(SqlServerSecretsOptions options) +{ + private readonly SqlServerSecretsOptions _options = + options ?? throw new ArgumentNullException(nameof(options)); + + /// The bracket-quoted, allow-list-validated schema prefix for object names. + public string SchemaName => _options.SchemaName; + + /// Command timeout, in whole seconds, for every command issued against this store. + public int CommandTimeoutSeconds => Math.Max(1, (int)_options.CommandTimeout.TotalSeconds); + + /// Opens a connection to the shared secret database. + /// A token to cancel the connect. + /// An opened connection. The caller owns disposal. + public async Task OpenConnectionAsync(CancellationToken cancellationToken) + { + var connection = new SqlConnection(_options.ConnectionString); + try + { + await connection.OpenAsync(cancellationToken).ConfigureAwait(false); + return connection; + } + catch + { + await connection.DisposeAsync().ConfigureAwait(false); + throw; + } + } + + /// + /// Creates a command on with the configured timeout applied. + /// + /// The open connection. + /// The command text. + /// The ambient transaction, if any. + /// The configured command. The caller owns disposal. + public SqlCommand CreateCommand( + SqlConnection connection, string commandText, SqlTransaction? transaction = null) + { + ArgumentNullException.ThrowIfNull(connection); + + SqlCommand command = connection.CreateCommand(); + command.CommandText = commandText; + command.CommandTimeout = CommandTimeoutSeconds; + command.Transaction = transaction; + return command; + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerHubMigrationHostedService.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerHubMigrationHostedService.cs new file mode 100644 index 0000000..755f2b7 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerHubMigrationHostedService.cs @@ -0,0 +1,33 @@ +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Options; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Provisions the shared hub schema at startup in hub-replication mode. +/// +/// +/// Separate from the core migration hosted service on purpose: in hub mode there are two +/// stores to provision — the node's local one (handled by the core service through the +/// ISecretsStoreMigrator seam) and the shared hub, which no node "owns". Every node runs this +/// idempotently at startup, so the hub exists as soon as any node has started, with no separate +/// provisioning step for an operator to forget. +/// +/// The hub migrator. +/// Core secrets options, for the shared startup-migration switch. +internal sealed class SqlServerHubMigrationHostedService( + SqlServerSecretsStoreMigrator migrator, + IOptions options) : IHostedService +{ + /// + public async Task StartAsync(CancellationToken cancellationToken) + { + if (options.Value.RunMigrationsOnStartup) + { + await migrator.MigrateAsync(cancellationToken).ConfigureAwait(false); + } + } + + /// + public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask; +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretReplicator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretReplicator.cs new file mode 100644 index 0000000..44b796a --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretReplicator.cs @@ -0,0 +1,26 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Publishes local writes to the shared SQL-Server hub, so a secret written on one node is +/// immediately visible to every other node that syncs against the same hub. +/// +/// +/// The row is written through , not +/// — the hub must record the row verbatim, keeping +/// the originating node's revision and updated_utc. An upsert would stamp the hub's own +/// revision and timestamp, and the row would then look newer than the writer's own copy, so it +/// would bounce straight back on the next sync. +/// +/// The shared hub store. +public sealed class SqlServerSecretReplicator(ISecretReplicationHub hub) : ISecretReplicator +{ + /// + public Task PublishAsync(StoredSecret row, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(row); + + return hub.ApplyReplicatedAsync(row, ct); + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretStore.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretStore.cs new file mode 100644 index 0000000..ed4f869 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretStore.cs @@ -0,0 +1,398 @@ +using System.Data; +using System.Globalization; +using Microsoft.Data.SqlClient; +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// SQL-Server-backed . Holds the same envelope-encrypted +/// rows as the SQLite store, in a database every node of a cluster can +/// reach — so one write is visible everywhere with no replication at all. +/// +/// +/// +/// The database holds ciphertext only. The KEK stays per-node and out of the database, so a +/// DBA, a backup tape, or a replica cannot decrypt anything. The corollary is a hard operational +/// constraint: every node must resolve the same KEK, because a row whose kek_id does +/// not match the local provider fails closed on resolve. +/// +/// +/// Semantics are deliberately identical to SqliteSecretStore — revision bump on upsert, +/// created stamps preserved on overwrite, verbatim last-writer-wins apply, compare-and-swap rewrap. +/// A cluster can run both stores at once (nodes on local SQLite replicating against a shared hub), +/// so a divergence between the two would be a silent convergence bug. +/// +/// +/// Factory for connections to the shared secret database. +public sealed class SqlServerSecretStore(SecretsSqlServerConnectionFactory connectionFactory) + : ISecretStore, ISecretReplicationHub +{ + // All columns of the secret table, in schema order, for full-row reads. + private const string AllColumns = + "name, description, content_type, ciphertext, nonce, tag, wrapped_dek, wrap_nonce, wrap_tag, " + + "kek_id, revision, is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by"; + + // The safe metadata projection — deliberately excludes every ciphertext / crypto BLOB column. + private const string MetadataColumns = + "name, description, content_type, kek_id, revision, is_deleted, created_utc, updated_utc, created_by, updated_by"; + + private string Secret => $"[{connectionFactory.SchemaName}].[{SqlServerSecretsSchema.SecretTable}]"; + + /// + public async Task GetAsync(SecretName name, CancellationToken ct) + { + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + await using SqlCommand command = connectionFactory.CreateCommand( + connection, $"SELECT {AllColumns} FROM {Secret} WHERE name = @name;"); + command.Parameters.AddWithValue("@name", name.Value); + + await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false); + + return await reader.ReadAsync(ct).ConfigureAwait(false) + ? ReadStoredSecret(reader) + : null; + } + + /// + public async Task UpsertAsync(StoredSecret row, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(row); + + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + string now = DateTimeOffset.UtcNow.ToString("O"); + + // MERGE establishes revision 0 / created==updated==now on insert; on match it overwrites the + // crypto material in place, bumps the revision, refreshes updated_utc/updated_by, and clears + // any tombstone — but deliberately preserves created_utc / created_by (the SQLite store's + // ON CONFLICT semantics, expressed in T-SQL). + // + // HOLDLOCK is not optional: without it MERGE takes only an update lock on the key range and + // two concurrent nodes upserting the same new name can both miss the match and both INSERT, + // producing a primary-key violation instead of one insert + one update. + await using SqlCommand command = connectionFactory.CreateCommand(connection, $""" + MERGE {Secret} WITH (HOLDLOCK) AS target + USING (SELECT @name AS name) AS source + ON target.name = source.name + WHEN MATCHED THEN UPDATE SET + description = @description, + content_type = @content_type, + ciphertext = @ciphertext, + nonce = @nonce, + tag = @tag, + wrapped_dek = @wrapped_dek, + wrap_nonce = @wrap_nonce, + wrap_tag = @wrap_tag, + kek_id = @kek_id, + revision = target.revision + 1, + updated_utc = @now, + updated_by = @updated_by, + is_deleted = 0, + deleted_utc = NULL + WHEN NOT MATCHED THEN INSERT ( + name, description, content_type, ciphertext, nonce, tag, + wrapped_dek, wrap_nonce, wrap_tag, kek_id, revision, + is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by) + VALUES ( + @name, @description, @content_type, @ciphertext, @nonce, @tag, + @wrapped_dek, @wrap_nonce, @wrap_tag, @kek_id, 0, + 0, NULL, @now, @now, @created_by, @updated_by); + """); + BindCryptoColumns(command, row); + command.Parameters.AddWithValue("@now", now); + command.Parameters.AddWithValue("@created_by", (object?)row.CreatedBy ?? DBNull.Value); + command.Parameters.AddWithValue("@updated_by", (object?)row.UpdatedBy ?? DBNull.Value); + + await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + + /// + public async Task DeleteAsync(SecretName name, string? actor, CancellationToken ct) + { + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + string now = DateTimeOffset.UtcNow.ToString("O"); + + await using SqlCommand command = connectionFactory.CreateCommand(connection, $""" + UPDATE {Secret} SET + is_deleted = 1, + deleted_utc = @now, + revision = revision + 1, + updated_utc = @now, + updated_by = @actor + WHERE name = @name AND is_deleted = 0; + """); + command.Parameters.AddWithValue("@now", now); + command.Parameters.AddWithValue("@actor", (object?)actor ?? DBNull.Value); + command.Parameters.AddWithValue("@name", name.Value); + + int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + return rowsAffected > 0; + } + + /// + public async Task> ListAsync(bool includeDeleted, CancellationToken ct) + { + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + await using SqlCommand command = connectionFactory.CreateCommand( + connection, + $"SELECT {MetadataColumns} FROM {Secret} WHERE (@include_deleted = 1 OR is_deleted = 0) ORDER BY name;"); + command.Parameters.AddWithValue("@include_deleted", includeDeleted ? 1 : 0); + + var results = new List(); + await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await reader.ReadAsync(ct).ConfigureAwait(false)) + { + results.Add(new SecretMetadata + { + Name = new SecretName(reader.GetString(0)), + Description = reader.IsDBNull(1) ? null : reader.GetString(1), + ContentType = Enum.Parse(reader.GetString(2)), + KekId = reader.GetString(3), + Revision = reader.GetInt64(4), + IsDeleted = reader.GetBoolean(5), + CreatedUtc = ParseUtc(reader.GetString(6)), + UpdatedUtc = ParseUtc(reader.GetString(7)), + CreatedBy = reader.IsDBNull(8) ? null : reader.GetString(8), + UpdatedBy = reader.IsDBNull(9) ? null : reader.GetString(9), + }); + } + + return results; + } + + /// + public async Task> GetManifestAsync(CancellationToken ct) + { + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + await using SqlCommand command = connectionFactory.CreateCommand( + connection, $"SELECT name, revision, updated_utc, is_deleted FROM {Secret} ORDER BY name;"); + + var results = new List(); + await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await reader.ReadAsync(ct).ConfigureAwait(false)) + { + results.Add(new SecretManifestEntry + { + Name = new SecretName(reader.GetString(0)), + Revision = reader.GetInt64(1), + UpdatedUtc = ParseUtc(reader.GetString(2)), + IsDeleted = reader.GetBoolean(3), + }); + } + + return results; + } + + /// + public async Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(row); + + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + await using SqlTransaction transaction = (SqlTransaction) + await connection.BeginTransactionAsync(IsolationLevel.Serializable, ct).ConfigureAwait(false); + + // Read the local (updated_utc, revision) so we can apply last-writer-wins. UPDLOCK/HOLDLOCK + // keeps another node from slipping a newer row in between this read and the write below — + // Serializable alone would not lock a key range that currently has no row. + await using (SqlCommand read = connectionFactory.CreateCommand( + connection, + $"SELECT updated_utc, revision FROM {Secret} WITH (UPDLOCK, HOLDLOCK) WHERE name = @name;", + transaction)) + { + read.Parameters.AddWithValue("@name", row.Name.Value); + + await using SqlDataReader reader = await read.ExecuteReaderAsync(ct).ConfigureAwait(false); + if (await reader.ReadAsync(ct).ConfigureAwait(false)) + { + DateTimeOffset localUpdated = ParseUtc(reader.GetString(0)); + long localRevision = reader.GetInt64(1); + + // Routed through the shared predicate so every store agrees on ties. + if (!SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, localUpdated, localRevision)) + { + await reader.CloseAsync().ConfigureAwait(false); + await transaction.CommitAsync(ct).ConfigureAwait(false); + return; + } + } + } + + // Apply the incoming row VERBATIM — its own revision, timestamps, tombstone flag, and crypto + // material — with no revision bump (this is the replication path, not a local write). + await using (SqlCommand upsert = connectionFactory.CreateCommand(connection, $""" + MERGE {Secret} WITH (HOLDLOCK) AS target + USING (SELECT @name AS name) AS source + ON target.name = source.name + WHEN MATCHED THEN UPDATE SET + description = @description, + content_type = @content_type, + ciphertext = @ciphertext, + nonce = @nonce, + tag = @tag, + wrapped_dek = @wrapped_dek, + wrap_nonce = @wrap_nonce, + wrap_tag = @wrap_tag, + kek_id = @kek_id, + revision = @revision, + is_deleted = @is_deleted, + deleted_utc = @deleted_utc, + created_utc = @created_utc, + updated_utc = @updated_utc, + created_by = @created_by, + updated_by = @updated_by + WHEN NOT MATCHED THEN INSERT ( + name, description, content_type, ciphertext, nonce, tag, + wrapped_dek, wrap_nonce, wrap_tag, kek_id, revision, + is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by) + VALUES ( + @name, @description, @content_type, @ciphertext, @nonce, @tag, + @wrapped_dek, @wrap_nonce, @wrap_tag, @kek_id, @revision, + @is_deleted, @deleted_utc, @created_utc, @updated_utc, @created_by, @updated_by); + """, transaction)) + { + BindCryptoColumns(upsert, row); + upsert.Parameters.AddWithValue("@revision", row.Revision); + upsert.Parameters.AddWithValue("@is_deleted", row.IsDeleted ? 1 : 0); + upsert.Parameters.AddWithValue("@deleted_utc", (object?)row.DeletedUtc?.ToString("O") ?? DBNull.Value); + upsert.Parameters.AddWithValue("@created_utc", row.CreatedUtc.ToString("O")); + upsert.Parameters.AddWithValue("@updated_utc", row.UpdatedUtc.ToString("O")); + upsert.Parameters.AddWithValue("@created_by", (object?)row.CreatedBy ?? DBNull.Value); + upsert.Parameters.AddWithValue("@updated_by", (object?)row.UpdatedBy ?? DBNull.Value); + + await upsert.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + + await transaction.CommitAsync(ct).ConfigureAwait(false); + } + + /// + public async Task ApplyRewrapAsync( + StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(rewrappedRow); + ArgumentNullException.ThrowIfNull(expectedCurrentWrappedDek); + + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + // Overwrite ONLY the KEK-wrap envelope + kek_id. Revision, updated_utc/by, the sealed body, + // and the tombstone state are deliberately untouched — a re-wrap is not a logical change, so + // it must not bump the revision or updated timestamp (see ISecretStore.ApplyRewrapAsync). + // + // Compare-and-swap on wrapped_dek: a concurrent set/rotate generates a fresh (random) DEK + // wrap, so this matches 0 rows instead of pairing this stale re-wrap with a newer body — + // which would leave the row permanently undecryptable. The caller re-processes a 0-row miss. + await using SqlCommand command = connectionFactory.CreateCommand(connection, $""" + UPDATE {Secret} SET + wrapped_dek = @wrapped_dek, + wrap_nonce = @wrap_nonce, + wrap_tag = @wrap_tag, + kek_id = @kek_id + WHERE name = @name AND wrapped_dek = @expected_wrapped_dek; + """); + command.Parameters.AddWithValue("@wrapped_dek", rewrappedRow.WrappedDek); + command.Parameters.AddWithValue("@wrap_nonce", rewrappedRow.WrapNonce); + command.Parameters.AddWithValue("@wrap_tag", rewrappedRow.WrapTag); + command.Parameters.AddWithValue("@kek_id", rewrappedRow.KekId); + command.Parameters.AddWithValue("@name", rewrappedRow.Name.Value); + command.Parameters.AddWithValue("@expected_wrapped_dek", expectedCurrentWrappedDek); + + int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + return rowsAffected > 0; + } + + /// + /// Fetches several rows by name in one round trip — the replication fetch path. Names not + /// present are simply absent from the result. + /// + /// The names to fetch. + /// A token to cancel the operation. + /// The matching encrypted rows. + public async Task> GetManyAsync( + IReadOnlyList names, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(names); + + if (names.Count == 0) + { + return []; + } + + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false); + + // One parameter per name — still fully parameterized, no name text is interpolated. + string[] placeholders = [.. names.Select((_, i) => $"@n{i.ToString(CultureInfo.InvariantCulture)}")]; + + await using SqlCommand command = connectionFactory.CreateCommand( + connection, + $"SELECT {AllColumns} FROM {Secret} WHERE name IN ({string.Join(", ", placeholders)});"); + + for (int i = 0; i < names.Count; i++) + { + command.Parameters.AddWithValue(placeholders[i], names[i].Value); + } + + var results = new List(names.Count); + await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false); + while (await reader.ReadAsync(ct).ConfigureAwait(false)) + { + results.Add(ReadStoredSecret(reader)); + } + + return results; + } + + // Binds the identity, description, content-type, KEK id, and all six crypto BLOB columns + // shared by every insert path. + private static void BindCryptoColumns(SqlCommand command, StoredSecret row) + { + command.Parameters.AddWithValue("@name", row.Name.Value); + command.Parameters.AddWithValue("@description", (object?)row.Description ?? DBNull.Value); + command.Parameters.AddWithValue("@content_type", row.ContentType.ToString()); + command.Parameters.AddWithValue("@ciphertext", row.Ciphertext); + command.Parameters.AddWithValue("@nonce", row.Nonce); + command.Parameters.AddWithValue("@tag", row.Tag); + command.Parameters.AddWithValue("@wrapped_dek", row.WrappedDek); + command.Parameters.AddWithValue("@wrap_nonce", row.WrapNonce); + command.Parameters.AddWithValue("@wrap_tag", row.WrapTag); + command.Parameters.AddWithValue("@kek_id", row.KekId); + } + + private static StoredSecret ReadStoredSecret(SqlDataReader reader) => new() + { + Name = new SecretName(reader.GetString(0)), + Description = reader.IsDBNull(1) ? null : reader.GetString(1), + ContentType = Enum.Parse(reader.GetString(2)), + Ciphertext = reader.GetFieldValue(3), + Nonce = reader.GetFieldValue(4), + Tag = reader.GetFieldValue(5), + WrappedDek = reader.GetFieldValue(6), + WrapNonce = reader.GetFieldValue(7), + WrapTag = reader.GetFieldValue(8), + KekId = reader.GetString(9), + Revision = reader.GetInt64(10), + IsDeleted = reader.GetBoolean(11), + DeletedUtc = reader.IsDBNull(12) ? null : ParseUtc(reader.GetString(12)), + CreatedUtc = ParseUtc(reader.GetString(13)), + UpdatedUtc = ParseUtc(reader.GetString(14)), + CreatedBy = reader.IsDBNull(15) ? null : reader.GetString(15), + UpdatedBy = reader.IsDBNull(16) ? null : reader.GetString(16), + }; + + private static DateTimeOffset ParseUtc(string value) => + DateTimeOffset.Parse(value, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind); +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretSyncService.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretSyncService.cs new file mode 100644 index 0000000..2ce60dc --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretSyncService.cs @@ -0,0 +1,158 @@ +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Keeps a node's local secret store converged with the shared SQL-Server hub, in hub-replication +/// mode. Runs a bidirectional anti-entropy sweep on an interval: pulls rows the hub holds newer, +/// pushes rows the node holds newer. +/// +/// +/// +/// This is the durability guarantee behind best-effort publishing. Live publishes make a write +/// visible in milliseconds; this sweep is what makes it certain. A node that was offline +/// while a peer wrote catches up on its next sweep, and a write made while the hub was unreachable +/// is pushed on the next sweep instead of being lost. +/// +/// +/// The sweep never throws out of : a hub outage must degrade the node to +/// "serving its last-known-good local secrets" rather than crash the host. Failures are logged and +/// retried on the next tick. +/// +/// +public sealed partial class SqlServerSecretSyncService : BackgroundService +{ + private readonly SecretReplicationReconciler _reconciler; + private readonly ISecretStore _local; + private readonly ISecretReplicationHub _hub; + private readonly SqlServerSecretsOptions _options; + private readonly ILogger _logger; + private readonly TimeProvider _timeProvider; + + /// Creates the sync service. + /// The node's local store. + /// The shared hub store. + /// Resolver-cache seam, evicted for every applied row. + /// SQL-Server replication options. + /// Logger for sweep outcomes and failures. + /// Clock, injectable so tests need not wait a real interval. + public SqlServerSecretSyncService( + ISecretStore local, + ISecretReplicationHub hub, + ISecretCacheInvalidator? cacheInvalidator, + IOptions options, + ILogger logger, + TimeProvider? timeProvider = null) + { + ArgumentNullException.ThrowIfNull(options); + + _local = local ?? throw new ArgumentNullException(nameof(local)); + _hub = hub ?? throw new ArgumentNullException(nameof(hub)); + _options = options.Value; + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _timeProvider = timeProvider ?? TimeProvider.System; + _reconciler = new SecretReplicationReconciler( + _local, + cacheInvalidator, + onRowFailed: (name, ex) => LogRowDiscarded(ex, name.Value)); + } + + /// + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + if (_options.SyncOnStartup) + { + await SweepSafelyAsync(stoppingToken).ConfigureAwait(false); + } + + using PeriodicTimer timer = new(_options.SyncInterval, _timeProvider); + + while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false)) + { + await SweepSafelyAsync(stoppingToken).ConfigureAwait(false); + } + } + + /// + /// Runs one bidirectional reconcile against the hub. + /// + /// A token to cancel the sweep. + /// How many rows were pulled from and pushed to the hub. + internal async Task<(int Pulled, int Pushed)> SweepAsync(CancellationToken ct) + { + IReadOnlyList hubManifest = + await _hub.GetManifestAsync(ct).ConfigureAwait(false); + + // Pull first: adopting the hub's newer rows before computing the push set means a row the hub + // already had newer is not then pushed straight back at it. + int pulled = await _reconciler + .ReconcileAsync(hubManifest, (names, token) => _hub.GetManyAsync(names, token), ct) + .ConfigureAwait(false); + + IReadOnlyList localManifest = + await _local.GetManifestAsync(ct).ConfigureAwait(false); + + IReadOnlyList push = + SecretReplicationReconciler.ComputePushSet(localManifest, hubManifest); + + int pushed = 0; + + if (push.Count > 0) + { + IReadOnlyList rows = + await _reconciler.ReadLocalAsync(push, ct).ConfigureAwait(false); + + foreach (StoredSecret row in rows) + { + // Verbatim, like the replicator — the hub records the originating node's revision. + await _hub.ApplyReplicatedAsync(row, ct).ConfigureAwait(false); + pushed++; + } + } + + return (pulled, pushed); + } + + private async Task SweepSafelyAsync(CancellationToken ct) + { + try + { + (int pulled, int pushed) = await SweepAsync(ct).ConfigureAwait(false); + + if (pulled > 0 || pushed > 0) + { + LogSweepConverged(pulled, pushed); + } + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + // Host shutdown — not a failure. + } + catch (Exception ex) + { + LogSweepFailed(ex); + } + } + + [LoggerMessage( + EventId = 1, + Level = LogLevel.Information, + Message = "Secret hub sync converged: pulled {Pulled} row(s) from the hub, pushed {Pushed} row(s) to it.")] + private partial void LogSweepConverged(int pulled, int pushed); + + [LoggerMessage( + EventId = 2, + Level = LogLevel.Warning, + Message = "Secret hub sync failed; the node continues serving its local store and will retry on the next interval.")] + private partial void LogSweepFailed(Exception exception); + + [LoggerMessage( + EventId = 3, + Level = LogLevel.Warning, + Message = "Discarding replicated secret row {SecretName} that could not be applied locally; the rest of the batch continues.")] + private partial void LogRowDiscarded(Exception exception, string secretName); +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsOptions.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsOptions.cs new file mode 100644 index 0000000..a8c4b87 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsOptions.cs @@ -0,0 +1,121 @@ +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Configuration for the shared SQL-Server secret store and its hub replication. Bound from a +/// configuration section by the AddZbSecretsSqlServer* extensions. +/// +public sealed class SqlServerSecretsOptions +{ + private string _schemaName = SqlServerSecretsSchema.DefaultSchemaName; + + /// + /// Connection string for the shared secret database. Required. Supply it through a + /// ${secret:} / secret: reference like every other connection string in the family + /// — the shared store holds ciphertext only, but its credentials are still credentials. + /// + public string ConnectionString { get; set; } = string.Empty; + + /// + /// SQL schema the secret tables live in. Defaults to zbsecrets so the secret tables never + /// collide with an application's own dbo objects when they share a database. + /// + /// + /// The value is not a plain identifier (letters, digits, and underscore, not starting with a + /// digit, at most 128 characters). + /// + /// + /// Validated on assignment rather than at use: the schema name is interpolated into DDL and + /// query text (T-SQL cannot parameterize an object name), so the allow-list is the injection + /// boundary and it must be impossible to get an unvalidated value into the property at all. + /// + public string SchemaName + { + get => _schemaName; + set => _schemaName = ValidateIdentifier(value); + } + + /// + /// How often the sync service reconciles the local store against the hub, in hub-replication + /// mode. Defaults to 30 seconds. Ignored in shared-store mode (there is nothing to reconcile). + /// + public TimeSpan SyncInterval { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// When (the default), hub-replication mode runs a reconcile at startup + /// before the first interval elapses, so a node that was down while a peer wrote does not serve + /// stale secrets for a whole interval after it comes back. + /// + public bool SyncOnStartup { get; set; } = true; + + /// + /// Command timeout applied to every store operation. Defaults to 30 seconds — long enough for a + /// contended shared database, short enough that a hung hub surfaces as an error rather than + /// stalling a resolve indefinitely. + /// + public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// Throws if the options are unusable. Called by the DI extensions at registration time so a + /// misconfigured node fails at startup rather than on the first secret resolve. + /// + /// The connection string is missing, or an interval is not positive. + public void Validate() + { + if (string.IsNullOrWhiteSpace(ConnectionString)) + { + throw new InvalidOperationException( + "SQL-Server secret store requires a ConnectionString (Secrets:SqlServer:ConnectionString)."); + } + + if (SyncInterval <= TimeSpan.Zero) + { + throw new InvalidOperationException( + $"SQL-Server secret replication SyncInterval must be positive (was {SyncInterval})."); + } + + if (CommandTimeout <= TimeSpan.Zero) + { + throw new InvalidOperationException( + $"SQL-Server secret store CommandTimeout must be positive (was {CommandTimeout})."); + } + } + + /// + /// Strict allow-list for a SQL identifier that will be interpolated into DDL or query text. + /// Rejects everything that is not [A-Za-z_][A-Za-z0-9_]{0,127} — no brackets, quotes, + /// dots, or spaces — so no bracket-escape (]]) trick can break out of the quoting applied + /// at the use site. Uses the ASCII-only character tests deliberately, so homoglyph and + /// full-width look-alikes are rejected rather than accepted as "letters". + /// + /// The candidate identifier. + /// The same value, once validated. + /// The value is not a plain identifier. + internal static string ValidateIdentifier(string value) + { + ArgumentException.ThrowIfNullOrWhiteSpace(value); + + if (value.Length > 128) + { + throw new ArgumentException( + $"SQL schema name must be at most 128 characters (was {value.Length}).", nameof(value)); + } + + if (!char.IsAsciiLetter(value[0]) && value[0] != '_') + { + throw new ArgumentException( + $"SQL schema name must start with a letter or underscore (was '{value}').", nameof(value)); + } + + foreach (char c in value) + { + if (!char.IsAsciiLetterOrDigit(c) && c != '_') + { + throw new ArgumentException( + $"SQL schema name may contain only letters, digits, and underscore (was '{value}').", + nameof(value)); + } + } + + return value; + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsSchema.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsSchema.cs new file mode 100644 index 0000000..8a62da6 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsSchema.cs @@ -0,0 +1,100 @@ +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Schema constants and DDL for the shared SQL-Server secret store. Mirrors +/// ZB.MOM.WW.Secrets.Sqlite.SqliteSecretsSchema column-for-column so a row means exactly the +/// same thing in either store. +/// +/// +/// +/// Timestamps are ISO-8601 nvarchar, not datetimeoffset. This looks wrong for a +/// SQL-Server schema and is deliberate: the same +/// rows replicate between this store and SQLite (which stores "O"-format text), and +/// last-writer-wins compares those timestamps for equality. datetimeoffset rounds to 100ns +/// while .NET DateTimeOffset ticks are also 100ns but the round-trip through the driver is +/// not guaranteed bit-identical across providers — a sub-tick difference would turn a tie into a +/// spurious "newer", and two nodes would then overwrite each other forever. Storing the exact +/// round-trippable text both providers already agree on removes the failure mode entirely. +/// +/// +/// The cost is that ORDER BY updated_utc is a lexicographic sort. That is still chronological +/// for "O"-format UTC strings (fixed-width, zero-padded, always +00:00), which is what +/// the store writes. +/// +/// +public static class SqlServerSecretsSchema +{ + /// + /// The schema version this build creates and supports. The migrator refuses a database whose + /// recorded version is newer than this. + /// + public const int CurrentVersion = 1; + + /// Default schema the secret tables live in. + public const string DefaultSchemaName = "zbsecrets"; + + /// Name of the single-row table tracking the applied schema version. + public const string SchemaVersionTable = "schema_version"; + + /// Name of the table storing envelope-encrypted secret records. + public const string SecretTable = "secret"; + + /// + /// Returns the idempotent DDL that provisions the schema, both tables, and the supporting index + /// in . + /// + /// The already-validated SQL schema name to create the objects in. + /// The DDL batch. + /// + /// is interpolated, not parameterized — T-SQL has no parameter form + /// for an object name. It is re-validated here against the same strict identifier allow-list + /// enforces, and bracket-quoted below. The + /// duplicate check is deliberate: this method is public, so it must defend itself rather + /// than trust that every caller routed through the options type. + /// + /// is not a plain identifier. + public static string CreateSchemaDdl(string schemaName) + { + SqlServerSecretsOptions.ValidateIdentifier(schemaName); + + return $""" + IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}') + EXEC(N'CREATE SCHEMA [{schemaName}]'); + + IF OBJECT_ID(N'[{schemaName}].[{SchemaVersionTable}]', N'U') IS NULL + CREATE TABLE [{schemaName}].[{SchemaVersionTable}] ( + id INT NOT NULL PRIMARY KEY CHECK (id = 1), + version INT NOT NULL, + applied_utc NVARCHAR(33) NOT NULL + ); + + IF OBJECT_ID(N'[{schemaName}].[{SecretTable}]', N'U') IS NULL + CREATE TABLE [{schemaName}].[{SecretTable}] ( + name NVARCHAR(450) NOT NULL PRIMARY KEY, + description NVARCHAR(MAX) NULL, + content_type NVARCHAR(64) NOT NULL, + ciphertext VARBINARY(MAX) NOT NULL, + nonce VARBINARY(MAX) NOT NULL, + tag VARBINARY(MAX) NOT NULL, + wrapped_dek VARBINARY(MAX) NOT NULL, + wrap_nonce VARBINARY(MAX) NOT NULL, + wrap_tag VARBINARY(MAX) NOT NULL, + kek_id NVARCHAR(256) NOT NULL, + revision BIGINT NOT NULL CONSTRAINT DF_secret_revision DEFAULT 0, + is_deleted BIT NOT NULL CONSTRAINT DF_secret_is_deleted DEFAULT 0, + deleted_utc NVARCHAR(33) NULL, + created_utc NVARCHAR(33) NOT NULL, + updated_utc NVARCHAR(33) NOT NULL, + created_by NVARCHAR(256) NULL, + updated_by NVARCHAR(256) NULL + ); + + IF NOT EXISTS ( + SELECT 1 FROM sys.indexes + WHERE name = N'ix_secret_updated_utc' + AND object_id = OBJECT_ID(N'[{schemaName}].[{SecretTable}]')) + CREATE INDEX ix_secret_updated_utc + ON [{schemaName}].[{SecretTable}] (updated_utc); + """; + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsStoreMigrator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsStoreMigrator.cs new file mode 100644 index 0000000..653f0f5 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/SqlServerSecretsStoreMigrator.cs @@ -0,0 +1,125 @@ +using System.Data; +using System.Globalization; +using Microsoft.Data.SqlClient; +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer; + +/// +/// Creates the shared SQL-Server secret schema and records the applied version. Idempotent: safe to +/// run on every node at every startup, including concurrently. Refuses to run against a database +/// whose recorded version is newer than this build supports. +/// +/// Factory for connections to the shared secret database. +public sealed class SqlServerSecretsStoreMigrator(SecretsSqlServerConnectionFactory connectionFactory) + : ISecretsStoreMigrator +{ + // Concurrent Serializable DDL against the system catalogs can pick a deadlock victim (error + // 1205) even though the IF NOT EXISTS guards make the logic itself safe. Every node runs this at + // every startup, so a virgin database being provisioned by N nodes at once is the NORMAL case, + // not a rare race — and an unretried victim aborts that node's whole host at startup. Retrying + // is correct precisely because the migration is idempotent. + private const int MaxDeadlockRetries = 5; + private const int SqlDeadlockVictimErrorNumber = 1205; + + /// + /// + /// The recorded schema version is newer than . + /// + public async Task MigrateAsync(CancellationToken cancellationToken) + { + for (int attempt = 1; ; attempt++) + { + try + { + await MigrateOnceAsync(cancellationToken).ConfigureAwait(false); + return; + } + catch (SqlException ex) + when (ex.Number == SqlDeadlockVictimErrorNumber && attempt < MaxDeadlockRetries) + { + // Staggered back-off so two victims do not retry in lockstep and deadlock again. + await Task.Delay(TimeSpan.FromMilliseconds(100 * attempt), cancellationToken) + .ConfigureAwait(false); + } + } + } + + private async Task MigrateOnceAsync(CancellationToken cancellationToken) + { + string schema = connectionFactory.SchemaName; + + await using SqlConnection connection = + await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false); + + // Serializable: unlike the single-writer SQLite store, EVERY node in the cluster runs this + // migration at startup, so concurrent first-boots are the normal case rather than a rare + // race. The isolation level plus the IF NOT EXISTS guards make a simultaneous provision by + // two nodes converge on one schema instead of one of them failing with "object exists". + await using SqlTransaction transaction = (SqlTransaction) + await connection.BeginTransactionAsync(IsolationLevel.Serializable, cancellationToken) + .ConfigureAwait(false); + + int existingVersion = + await ReadExistingSchemaVersionAsync(connection, transaction, schema, cancellationToken) + .ConfigureAwait(false); + + if (existingVersion > SqlServerSecretsSchema.CurrentVersion) + { + throw new SecretStoreMigrationException( + $"Shared secret database schema version {existingVersion} is newer than supported version " + + $"{SqlServerSecretsSchema.CurrentVersion}. Upgrade this node before it writes to the shared store."); + } + + await using (SqlCommand ddl = connectionFactory.CreateCommand( + connection, SqlServerSecretsSchema.CreateSchemaDdl(schema), transaction)) + { + await ddl.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false); + } + + await using (SqlCommand version = connectionFactory.CreateCommand( + connection, + $""" + MERGE [{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}] AS target + USING (SELECT 1 AS id, @version AS version, @applied_utc AS applied_utc) AS source + ON target.id = source.id + WHEN MATCHED THEN + UPDATE SET version = source.version, applied_utc = source.applied_utc + WHEN NOT MATCHED THEN + INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc); + """, + transaction)) + { + version.Parameters.AddWithValue("@version", SqlServerSecretsSchema.CurrentVersion); + version.Parameters.AddWithValue("@applied_utc", DateTimeOffset.UtcNow.ToString("O")); + await version.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false); + } + + await transaction.CommitAsync(cancellationToken).ConfigureAwait(false); + } + + private static async Task ReadExistingSchemaVersionAsync( + SqlConnection connection, + SqlTransaction transaction, + string schema, + CancellationToken cancellationToken) + { + await using SqlCommand command = connection.CreateCommand(); + command.Transaction = transaction; + + // OBJECT_ID returns NULL for an absent table, so this reads 0 on a virgin database without + // needing the table to exist first. + command.CommandText = $""" + IF OBJECT_ID(N'[{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}]', N'U') IS NULL + SELECT 0; + ELSE + SELECT ISNULL((SELECT version FROM [{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}] WHERE id = 1), 0); + """; + + object? version = await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false); + + return version is null || version == DBNull.Value + ? 0 + : Convert.ToInt32(version, CultureInfo.InvariantCulture); + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ZB.MOM.WW.Secrets.Replicator.SqlServer.csproj b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ZB.MOM.WW.Secrets.Replicator.SqlServer.csproj new file mode 100644 index 0000000..d20597b --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ZB.MOM.WW.Secrets.Replicator.SqlServer.csproj @@ -0,0 +1,39 @@ + + + + net10.0 + enable + enable + true + true + + + + true + ZB.MOM.WW.Secrets.Replicator.SqlServer + ZB.MOM.WW + Shared SQL-Server secret store and hub replication for the ZB.MOM.WW SCADA family — cluster-wide, ciphertext-only secrets. + https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets + https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets + + + + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/NoOpSecretReplicator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/NoOpSecretReplicator.cs index 04cfe87..7c5f637 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/NoOpSecretReplicator.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/NoOpSecretReplicator.cs @@ -4,8 +4,9 @@ namespace ZB.MOM.WW.Secrets.DependencyInjection; /// /// The single-node default : publishing a row is a no-op because -/// there are no peers to broadcast to. The deferred ZB.MOM.WW.Secrets.Akka package replaces -/// this registration with a real cluster replicator for clustered deployments. +/// there are no peers to broadcast to. The ZB.MOM.WW.Secrets.Replication.Akka and +/// ZB.MOM.WW.Secrets.Replication.SqlServer packages replace this registration with a real +/// replicator for clustered deployments. /// public sealed class NoOpSecretReplicator : ISecretReplicator { diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsMigrationHostedService.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsMigrationHostedService.cs index bf58beb..11a8d50 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsMigrationHostedService.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsMigrationHostedService.cs @@ -1,16 +1,21 @@ using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Options; -using ZB.MOM.WW.Secrets.Sqlite; +using ZB.MOM.WW.Secrets.Abstractions; namespace ZB.MOM.WW.Secrets.DependencyInjection; /// -/// Runs the secrets SQLite schema migration at application startup when +/// Runs the secret-store schema migration at application startup when /// is . The migration is /// idempotent, so repeated restarts are safe. /// +/// +/// Depends on the seam rather than a concrete migrator, so the +/// same hosted service provisions the local SQLite store or the shared SQL-Server store depending +/// only on which migrator the application registered. +/// internal sealed class SecretsMigrationHostedService( - SqliteSecretsStoreMigrator migrator, + ISecretsStoreMigrator migrator, IOptions options) : IHostedService { /// diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsServiceCollectionExtensions.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsServiceCollectionExtensions.cs index 20c7187..3609977 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsServiceCollectionExtensions.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/DependencyInjection/SecretsServiceCollectionExtensions.cs @@ -51,9 +51,15 @@ public static class SecretsServiceCollectionExtensions services.TryAddSingleton(sp => new AesGcmEnvelopeCipher(sp.GetRequiredService())); - services.TryAddSingleton(sp => + // Registered BOTH concretely and behind the interface, forwarding to the one singleton. + // The concrete registration is what lets a replication package resolve the *undecorated* + // local store when it replaces ISecretStore with a decorator — asking the container for + // ISecretStore there would resolve the decorator itself and recurse forever. + services.TryAddSingleton(sp => new SqliteSecretStore(sp.GetRequiredService())); + services.TryAddSingleton(sp => sp.GetRequiredService()); + services.TryAddSingleton(); // ONE shared resolver instance backs both the read seam (ISecretResolver) and the @@ -81,10 +87,16 @@ public static class SecretsServiceCollectionExtensions services.TryAddSingleton(sp => sp.GetRequiredService()); // Migrator: singleton, constructed from the already-registered connection factory. Needed - // before any store operations so the schema exists. + // before any store operations so the schema exists. Registered BOTH concretely (for callers + // that want the SQLite migrator specifically) and behind the ISecretsStoreMigrator seam that + // the hosted service and CLI consume — a replication package that swaps the store registers + // its own ISecretsStoreMigrator BEFORE calling AddZbSecrets, and TryAdd leaves it in place. services.TryAddSingleton(sp => new SqliteSecretsStoreMigrator(sp.GetRequiredService())); + services.TryAddSingleton(sp => + sp.GetRequiredService()); + // Hosted service that runs migrations on startup when SecretsOptions.RunMigrationsOnStartup. services.AddHostedService(); diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/ReplicatingSecretStore.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/ReplicatingSecretStore.cs new file mode 100644 index 0000000..ca6fa2a --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/ReplicatingSecretStore.cs @@ -0,0 +1,115 @@ +using Microsoft.Extensions.Logging; +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replication; + +/// +/// Decorates a local so every successful local write is published to +/// peers through . +/// +/// +/// +/// This is what turns the seam from a declaration into behaviour. +/// Decorating the store — rather than calling the replicator from each write site — means the CLI, +/// the Blazor UI, and any future caller all replicate automatically, and no new write path can +/// forget to. +/// +/// +/// Publishing is best-effort. A transport failure is logged and swallowed: the local write is +/// already durable, and refusing it because a peer was unreachable would make every node a single +/// point of failure for every other. Propagation is then guaranteed by the transport's periodic +/// anti-entropy sweep instead, which is why every transport in this library reconciles +/// bidirectionally — a dropped publish is repaired by the next sweep rather than lost. +/// +/// +/// deliberately does not publish (that would echo a peer's row +/// straight back and, across three or more nodes, never stop). Neither does +/// : a re-wrap does not bump the revision, so it is invisible to +/// last-writer-wins by design and a peer would discard it — KEK rotation is run once per independent +/// store instead. +/// +/// +/// The local store doing the actual persistence. +/// The transport that publishes rows to peers. +/// Logger for publish failures. +public sealed class ReplicatingSecretStore( + ISecretStore inner, + ISecretReplicator replicator, + ILogger logger) : ISecretStore +{ + /// + public Task GetAsync(SecretName name, CancellationToken ct) => + inner.GetAsync(name, ct); + + /// + public Task> ListAsync(bool includeDeleted, CancellationToken ct) => + inner.ListAsync(includeDeleted, ct); + + /// + public Task> GetManifestAsync(CancellationToken ct) => + inner.GetManifestAsync(ct); + + /// + public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) => + inner.ApplyReplicatedAsync(row, ct); + + /// + public Task ApplyRewrapAsync( + StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) => + inner.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek, ct); + + /// + public async Task UpsertAsync(StoredSecret row, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(row); + + await inner.UpsertAsync(row, ct).ConfigureAwait(false); + await PublishCurrentAsync(row.Name, ct).ConfigureAwait(false); + } + + /// + public async Task DeleteAsync(SecretName name, string? actor, CancellationToken ct) + { + bool deleted = await inner.DeleteAsync(name, actor, ct).ConfigureAwait(false); + + if (deleted) + { + // The tombstone is the thing being replicated — a delete propagates as a newer row, not + // as an absence, so peers converge on "deleted" instead of resurrecting it. + await PublishCurrentAsync(name, ct).ConfigureAwait(false); + } + + return deleted; + } + + // Re-reads the row the store actually persisted and publishes THAT, never the caller's input: + // the store assigns the revision and updated_utc, and those two fields are exactly what peers + // order by. Publishing the caller's copy would broadcast a row whose ordering keys are wrong. + private async Task PublishCurrentAsync(SecretName name, CancellationToken ct) + { + StoredSecret? persisted = await inner.GetAsync(name, ct).ConfigureAwait(false); + + if (persisted is null) + { + return; + } + + try + { + await replicator.PublishAsync(persisted, ct).ConfigureAwait(false); + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + throw; + } + catch (Exception ex) + { + // Deliberately swallowed — see the class remarks. Anti-entropy repairs it. + logger.LogWarning( + ex, + "Failed to publish secret {SecretName} to peers; the local write succeeded and " + + "anti-entropy will propagate it on the next sweep.", + name.Value); + } + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/SecretReplicationReconciler.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/SecretReplicationReconciler.cs new file mode 100644 index 0000000..6d44f01 --- /dev/null +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Replication/SecretReplicationReconciler.cs @@ -0,0 +1,228 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replication; + +/// +/// Transport-agnostic anti-entropy: compares a peer's manifest against the local store, pulls only +/// the rows the peer holds a strictly newer copy of, and applies them last-writer-wins. +/// +/// +/// +/// This is the half of replication that is pure logic, deliberately separated from how the +/// peer is reached. The SQL-Server package points it at a shared hub database; the Akka package +/// points it at a cluster peer over pub/sub. Both get identical convergence behaviour, and the +/// interesting edge cases (ties, tombstones, a peer that is behind) are testable offline without a +/// database or a cluster. +/// +/// +/// Reconciliation is a pull, never a push: it only ever writes rows the peer says are newer, so +/// running it against a peer that is behind is a no-op rather than a regression. It is also safe to +/// run concurrently with live replication — re-checks +/// the same predicate under its own transaction, so a row that a broadcast delivered first is simply +/// ignored the second time. +/// +/// +/// The local store to bring up to date. +/// +/// Optional resolver-cache seam. When supplied, every applied name is evicted so a node picks up a +/// peer's write within the reconcile interval instead of serving a stale plaintext for the rest of +/// the cache TTL. +/// +/// +/// Optional callback invoked when a single replicated row could not be applied. The row is skipped +/// and the rest of the batch proceeds; without a callback the failure is silent, so transports +/// should supply one that logs. +/// +public sealed class SecretReplicationReconciler( + ISecretStore local, + ISecretCacheInvalidator? cacheInvalidator = null, + Action? onRowFailed = null) +{ + /// + /// Returns the names for which holds a strictly newer row than + /// — including names absent locally, and including tombstones + /// (a delete propagates as a newer row, not as an absence). + /// + /// The local store's manifest. + /// The peer's manifest. + /// The names to pull from the peer, in manifest order. + public static IReadOnlyList ComputePullSet( + IReadOnlyList localManifest, + IReadOnlyList remote) + { + ArgumentNullException.ThrowIfNull(localManifest); + ArgumentNullException.ThrowIfNull(remote); + + Dictionary localByName = + localManifest.ToDictionary(entry => entry.Name.Value, StringComparer.Ordinal); + + var pull = new List(); + + foreach (SecretManifestEntry remoteEntry in remote) + { + // Absent locally, or the peer's copy wins LWW. A name we hold and the peer does not is + // deliberately NOT handled here: this side pulls, the peer's own reconcile pushes. + if (!localByName.TryGetValue(remoteEntry.Name.Value, out SecretManifestEntry? localEntry) || + SecretLastWriterWins.IsNewer(remoteEntry, localEntry)) + { + pull.Add(remoteEntry.Name); + } + } + + return pull; + } + + /// + /// Returns the names the local store holds a strictly newer row for than + /// — the mirror of , used to push local writes a peer is missing. + /// + /// + /// Every transport here reconciles in both directions, and this is why: publishing a write + /// is best-effort (see ReplicatingSecretStore), so without a push sweep a write made while + /// a peer was unreachable would sit on one node forever — the peer never learns the name exists, + /// so its own pull can never ask for it. + /// + /// The local store's manifest. + /// The peer's manifest. + /// The names to push to the peer. + public static IReadOnlyList ComputePushSet( + IReadOnlyList localManifest, + IReadOnlyList remote) => + // Exactly the pull computation with the two sides swapped: "what does the peer need from me" + // is "what would I need, if I were the peer". + ComputePullSet(remote, localManifest); + + /// + /// Reconciles the local store against , fetching and applying + /// every row the peer holds a newer copy of. + /// + /// The peer's manifest. + /// + /// Fetches the encrypted rows for the requested names from the peer. May return fewer rows than + /// requested (a row deleted between manifest and fetch); extra rows are applied normally. + /// + /// A token to cancel the operation. + /// The number of rows applied locally. + public async Task ReconcileAsync( + IReadOnlyList remoteManifest, + Func, CancellationToken, Task>> fetchAsync, + CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(remoteManifest); + ArgumentNullException.ThrowIfNull(fetchAsync); + + IReadOnlyList localManifest = + await local.GetManifestAsync(ct).ConfigureAwait(false); + + IReadOnlyList pull = ComputePullSet(localManifest, remoteManifest); + + if (pull.Count == 0) + { + return 0; + } + + int applied = 0; + + // Chunked, not fetched in one shot. A cold-starting node's pull set is the peer's ENTIRE + // inventory, and an unbounded fetch breaks on real backends — SQL Server caps a command at + // 2100 parameters, so a hub with more than that many secrets would fail the same way every + // interval and the node would never receive a single row. Chunking also bounds the size of + // one replication message on the wire. + foreach (IReadOnlyList batch in Chunk(pull, FetchBatchSize)) + { + IReadOnlyList rows = await fetchAsync(batch, ct).ConfigureAwait(false); + applied += await ApplyAsync(rows, ct).ConfigureAwait(false); + } + + return applied; + } + + /// + /// Names fetched per round trip. Well under SQL Server's 2100-parameter command limit, and small + /// enough that one replication message stays a sane size. + /// + internal const int FetchBatchSize = 500; + + private static IEnumerable> Chunk( + IReadOnlyList names, int size) + { + for (int offset = 0; offset < names.Count; offset += size) + { + yield return [.. names.Skip(offset).Take(size)]; + } + } + + /// + /// Reads the local rows for , skipping any that vanished between the + /// manifest and the read. Used by transports to assemble a push batch. + /// + /// The names to read. + /// A token to cancel the operation. + /// The encrypted rows that still exist locally. + public async Task> ReadLocalAsync( + IReadOnlyList names, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(names); + + var rows = new List(names.Count); + + foreach (SecretName name in names) + { + StoredSecret? row = await local.GetAsync(name, ct).ConfigureAwait(false); + + if (row is not null) + { + rows.Add(row); + } + } + + return rows; + } + + /// + /// Applies encrypted rows received from a peer, last-writer-wins, evicting the resolver cache for + /// each. Used both by and by transports that receive live broadcasts. + /// + /// The encrypted rows to apply. + /// A token to cancel the operation. + /// The number of rows passed to the store. + public async Task ApplyAsync(IReadOnlyList rows, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(rows); + + int applied = 0; + + foreach (StoredSecret row in rows) + { + try + { + await local.ApplyReplicatedAsync(row, ct).ConfigureAwait(false); + applied++; + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + throw; + } + catch (Exception ex) + { + // Isolated per row, because a batch is not a transaction and must not be all-or- + // nothing. One row a peer cannot supply cleanly (a hostile or buggy peer, a + // constraint violation) would otherwise abandon every row after it — and since the + // pull set is recomputed identically each sweep, in the same manifest order, the + // SAME row would poison the SAME batch forever and every name behind it would never + // converge. Drop the row, keep the batch. + onRowFailed?.Invoke(row.Name, ex); + continue; + } + finally + { + // Evicted unconditionally: the store may have ignored the row as stale, but an + // eviction only costs the next resolve a re-read, whereas skipping one when the row + // DID land serves a stale plaintext until the TTL expires. + cacheInvalidator?.Invalidate(row.Name); + } + } + + return applied; + } +} diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretStore.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretStore.cs index 64a2f44..e8a8745 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretStore.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretStore.cs @@ -200,12 +200,9 @@ public sealed class SqliteSecretStore(SecretsSqliteConnectionFactory connectionF DateTimeOffset localUpdated = ParseUtc(reader.GetString(0)); long localRevision = reader.GetInt64(1); - // Incoming wins only if strictly newer by (updated_utc, then revision). - bool incomingIsNewer = - row.UpdatedUtc > localUpdated || - (row.UpdatedUtc == localUpdated && row.Revision > localRevision); - - if (!incomingIsNewer) + // Incoming wins only if strictly newer by (updated_utc, then revision). Routed through + // the shared predicate so every store agrees on ties — see SecretLastWriterWins. + if (!SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, localUpdated, localRevision)) { await transaction.CommitAsync(ct).ConfigureAwait(false); return; diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretsStoreMigrator.cs b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretsStoreMigrator.cs index 1198f3e..5d03594 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretsStoreMigrator.cs +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/Sqlite/SqliteSecretsStoreMigrator.cs @@ -10,6 +10,7 @@ namespace ZB.MOM.WW.Secrets.Sqlite; /// newer than this build supports. /// public sealed class SqliteSecretsStoreMigrator(SecretsSqliteConnectionFactory connectionFactory) + : ISecretsStoreMigrator { /// Applies the schema migration to the secret store. /// Cancellation token. diff --git a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.csproj b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.csproj index 759ce67..64c4d5a 100644 --- a/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.csproj +++ b/ZB.MOM.WW.Secrets/src/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.csproj @@ -13,6 +13,7 @@ + diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/HostileWireInputTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/HostileWireInputTests.cs new file mode 100644 index 0000000..f487244 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/HostileWireInputTests.cs @@ -0,0 +1,81 @@ +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests; + +/// +/// The DTO is the trust boundary for anything a peer sends. Every rejection here must surface as an +/// , because that is the exception type the receiving actor filters +/// on — anything else escapes its handler, restarts the actor, and (since the row is redelivered) +/// loops. +/// +public sealed class HostileWireInputTests +{ + private static SecretRowDto Valid() => new() + { + Name = "app/ok", + ContentType = "Text", + Ciphertext = [1], + Nonce = [2], + Tag = [3], + WrappedDek = [4], + WrapNonce = [5], + WrapTag = [6], + KekId = "sha256:x", + Revision = 0, + CreatedUtc = "2026-01-01T00:00:00.0000000+00:00", + UpdatedUtc = "2026-01-01T00:00:00.0000000+00:00", + }; + + [Fact] + public void A_valid_row_materializes() + { + StoredSecret row = Valid().ToStoredSecret(); + + Assert.Equal("app/ok", row.Name.Value); + Assert.Equal(SecretContentType.Text, row.ContentType); + } + + [Theory] + [InlineData("../../etc/passwd")] + [InlineData("/rooted")] + [InlineData("has space")] + [InlineData("")] + public void A_malformed_name_is_rejected(string name) + { + Assert.Throws(() => (Valid() with { Name = name }).ToStoredSecret()); + } + + [Theory] + // An in-range number names no member — Enum.Parse would happily produce (SecretContentType)4096 + // and the store would persist it for later readers that assume a defined value. + [InlineData("4096")] + // Overflows the underlying type: Enum.Parse throws OverflowException, which is NOT an + // ArgumentException, so it would escape the actor's filter and restart it. + [InlineData("99999999999999999999")] + [InlineData("NotAContentType")] + [InlineData("")] + public void An_unrecognized_content_type_is_rejected_as_an_ArgumentException(string contentType) + { + Assert.Throws( + () => (Valid() with { ContentType = contentType }).ToStoredSecret()); + } + + [Fact] + public void A_null_crypto_blob_is_rejected_at_the_boundary() + { + // `required byte[]` means "present in the payload", not "non-null" — an explicit JSON null + // satisfies it. Caught here, or it blows up deep in the store's parameter binding instead. + Assert.Throws( + () => (Valid() with { Ciphertext = null! }).ToStoredSecret()); + Assert.Throws( + () => (Valid() with { WrappedDek = null! }).ToStoredSecret()); + } + + [Fact] + public void A_malformed_timestamp_is_rejected() + { + Assert.ThrowsAny( + () => (Valid() with { UpdatedUtc = "not-a-date" }).ToStoredSecret()); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/SecretReplicationSerializerTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/SecretReplicationSerializerTests.cs new file mode 100644 index 0000000..d4738d9 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/SecretReplicationSerializerTests.cs @@ -0,0 +1,160 @@ +using Akka.Actor; +using Akka.Serialization; +using Akka.TestKit.Xunit2; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests; + +/// +/// Wire-contract tests. Secret ciphertext crosses the network as these bytes, so the round-trip is +/// asserted field-by-field rather than by equality alone — a silently dropped crypto field would +/// produce a row that stores fine and fails to decrypt much later, on another node. +/// +public sealed class SecretReplicationSerializerTests : TestKit +{ + public SecretReplicationSerializerTests() + : base(AkkaSecretsReplication.SerializationConfig) + { + } + + private static StoredSecret Row(string name) => new() + { + Name = new SecretName(name), + Description = "a description", + ContentType = SecretContentType.ConnectionString, + Ciphertext = [1, 2, 3, 250], + Nonce = [4, 5, 6], + Tag = [7, 8, 9], + WrappedDek = [10, 11, 12], + WrapNonce = [13, 14, 15], + WrapTag = [16, 17, 18], + KekId = "sha256:abc", + Revision = 42, + IsDeleted = true, + DeletedUtc = new DateTimeOffset(2026, 3, 2, 1, 0, 0, TimeSpan.Zero), + CreatedUtc = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), + UpdatedUtc = new DateTimeOffset(2026, 2, 1, 0, 0, 0, TimeSpan.Zero), + CreatedBy = "alice", + UpdatedBy = "bob", + }; + + private object RoundTrip(object message) + { + Serializer serializer = Sys.Serialization.FindSerializerFor(message); + Assert.IsType(serializer); + + byte[] bytes = serializer.ToBinary(message); + string manifest = ((SerializerWithStringManifest)serializer).Manifest(message); + + return ((SerializerWithStringManifest)serializer).FromBinary(bytes, manifest); + } + + [Fact] + public void SecretRowsMessage_round_trips_every_field() + { + StoredSecret original = Row("app/db-conn"); + var message = new SecretRowsMessage([SecretRowDto.FromStoredSecret(original)]); + + var restored = (SecretRowsMessage)RoundTrip(message); + StoredSecret result = restored.Rows.Single().ToStoredSecret(); + + Assert.Equal(original.Name.Value, result.Name.Value); + Assert.Equal(original.Description, result.Description); + Assert.Equal(original.ContentType, result.ContentType); + Assert.Equal(original.Ciphertext, result.Ciphertext); + Assert.Equal(original.Nonce, result.Nonce); + Assert.Equal(original.Tag, result.Tag); + Assert.Equal(original.WrappedDek, result.WrappedDek); + Assert.Equal(original.WrapNonce, result.WrapNonce); + Assert.Equal(original.WrapTag, result.WrapTag); + Assert.Equal(original.KekId, result.KekId); + Assert.Equal(original.Revision, result.Revision); + Assert.Equal(original.IsDeleted, result.IsDeleted); + Assert.Equal(original.DeletedUtc, result.DeletedUtc); + Assert.Equal(original.CreatedUtc, result.CreatedUtc); + Assert.Equal(original.UpdatedUtc, result.UpdatedUtc); + Assert.Equal(original.CreatedBy, result.CreatedBy); + Assert.Equal(original.UpdatedBy, result.UpdatedBy); + } + + [Fact] + public void Timestamps_survive_the_wire_exactly() + { + // Last-writer-wins compares UpdatedUtc for EQUALITY to break revision ties. A serializer that + // rounded to the millisecond would turn a tie into a spurious "newer" and two nodes would + // then overwrite each other forever, so tick-exactness is a correctness requirement. + StoredSecret original = Row("app/precise") with + { + UpdatedUtc = new DateTimeOffset(2026, 2, 1, 12, 34, 56, TimeSpan.Zero).AddTicks(1234567), + }; + + var restored = (SecretRowsMessage)RoundTrip( + new SecretRowsMessage([SecretRowDto.FromStoredSecret(original)])); + + Assert.Equal(original.UpdatedUtc, restored.Rows.Single().ToStoredSecret().UpdatedUtc); + Assert.Equal(original.UpdatedUtc.Ticks, restored.Rows.Single().ToStoredSecret().UpdatedUtc.Ticks); + } + + [Fact] + public void ManifestAnnounce_round_trips() + { + var message = new SecretManifestAnnounce( + [ + SecretManifestEntryDto.FromEntry(new SecretManifestEntry + { + Name = new SecretName("a/one"), + Revision = 3, + UpdatedUtc = new DateTimeOffset(2026, 5, 5, 5, 5, 5, TimeSpan.Zero), + IsDeleted = true, + }), + ]); + + var restored = (SecretManifestAnnounce)RoundTrip(message); + SecretManifestEntry entry = restored.Entries.Single().ToEntry(); + + Assert.Equal("a/one", entry.Name.Value); + Assert.Equal(3, entry.Revision); + Assert.True(entry.IsDeleted); + Assert.Equal(new DateTimeOffset(2026, 5, 5, 5, 5, 5, TimeSpan.Zero), entry.UpdatedUtc); + } + + [Fact] + public void PullRequest_round_trips() + { + var restored = (SecretPullRequest)RoundTrip(new SecretPullRequest(["a/one", "b/two"])); + + Assert.Equal(["a/one", "b/two"], restored.Names); + } + + [Fact] + public void An_unknown_manifest_fails_loudly_rather_than_guessing() + { + // A mixed-version cluster must fail on an unrecognized payload, not mis-deserialize a secret. + var serializer = (SerializerWithStringManifest)Sys.Serialization + .FindSerializerFor(new SecretPullRequest([])); + + Assert.Throws( + () => serializer.FromBinary([1, 2, 3], "zbs-from-the-future-v9")); + } + + [Fact] + public void The_wire_contract_carries_no_field_that_could_hold_key_material() + { + // Structural guard on the trust boundary: the DTO's surface is fixed and reviewed. If someone + // adds a property to SecretRowDto, this test fails and forces a deliberate look at whether + // the new field is safe to put on the wire. + string[] properties = [.. typeof(SecretRowDto) + .GetProperties() + .Select(p => p.Name) + .Order(StringComparer.Ordinal)]; + + Assert.Equal( + [ + "Ciphertext", "ContentType", "CreatedBy", "CreatedUtc", "DeletedUtc", "Description", + "IsDeleted", "KekId", "Name", "Nonce", "Revision", "Tag", "UpdatedBy", "UpdatedUtc", + "WrapNonce", "WrapTag", "WrappedDek", + ], + properties); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs new file mode 100644 index 0000000..30e7da7 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/TwoNodeClusterReplicationTests.cs @@ -0,0 +1,354 @@ +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.Sqlite; + +namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests; + +/// +/// Two real cluster nodes, two real SQLite stores, real remoting over loopback — the gate the G-7 +/// design called for, run in-process so it belongs to the ordinary offline suite instead of needing +/// a deployed rig. +/// +/// +/// Nothing here is faked: each node runs its own with Akka.Remote on its +/// own port, they form a cluster, and rows travel through the actual serializer and the actual +/// distributed pub/sub mediator. A mock-based test of this behaviour would prove almost nothing — +/// the failure modes worth catching (self-echo, sender loss across the mediator, tombstones that do +/// not propagate, a partition that never re-converges) only appear when messages really cross nodes. +/// +public sealed class TwoNodeClusterReplicationTests : IAsyncLifetime +{ + private readonly List _dbPaths = []; + private ActorSystem _systemA = null!; + private ActorSystem _systemB = null!; + private SqliteSecretStore _storeA = null!; + private SqliteSecretStore _storeB = null!; + private ISecretStore _writableA = null!; + private IActorRef _replicatorB = null!; + + // Short so anti-entropy assertions do not dominate the suite's runtime. + private static readonly TimeSpan AnnounceInterval = TimeSpan.FromMilliseconds(300); + private static readonly TimeSpan Patience = TimeSpan.FromSeconds(20); + + public async Task InitializeAsync() + { + _storeA = CreateStore(); + _storeB = CreateStore(); + + // Port 0 lets the OS assign free ports, so parallel test runs cannot collide. + _systemA = ActorSystem.Create("zb-secrets-cluster", ClusterConfig(port: 0)); + int portA = ClusterPort(_systemA); + + // Node A seeds itself; node B joins it. + _systemA.Dispose(); + _systemA = ActorSystem.Create("zb-secrets-cluster", ClusterConfig(portA, seedPort: portA)); + _systemB = ActorSystem.Create("zb-secrets-cluster", ClusterConfig(port: 0, seedPort: portA)); + + await AwaitClusterUpAsync(_systemA, expectedMembers: 2); + await AwaitClusterUpAsync(_systemB, expectedMembers: 2); + + IActorRef replicatorA = _systemA.ActorOf( + SecretReplicationActor.Props(_storeA, null, AnnounceInterval), "zb-secret-replication"); + _replicatorB = _systemB.ActorOf( + SecretReplicationActor.Props(_storeB, null, AnnounceInterval), "zb-secret-replication"); + + _writableA = new ReplicatingSecretStore( + _storeA, new AkkaSecretReplicator(replicatorA), NullLogger.Instance); + + // The mediator gossips subscriptions between nodes; publishing before both sides are + // registered would silently drop the message. + await AwaitBothSubscribedAsync(); + } + + private SqliteSecretStore CreateStore() + { + string path = Path.Combine(Path.GetTempPath(), $"zb-akka-{Guid.NewGuid():N}.db"); + _dbPaths.Add(path); + var factory = new SecretsSqliteConnectionFactory(path); + new SqliteSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None).GetAwaiter().GetResult(); + return new SqliteSecretStore(factory); + } + + private static Config ClusterConfig(int port, int? seedPort = null) + { + string seeds = seedPort is null + ? "[]" + : $"[\"akka.tcp://zb-secrets-cluster@127.0.0.1:{seedPort}\"]"; + + return ConfigurationFactory.ParseString($$""" + akka { + loglevel = WARNING + actor.provider = cluster + remote.dot-netty.tcp { + hostname = "127.0.0.1" + public-hostname = "127.0.0.1" + port = {{port}} + } + cluster { + seed-nodes = {{seeds}} + downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider" + } + } + """).WithFallback(AkkaSecretsReplication.SerializationConfig); + } + + private static int ClusterPort(ActorSystem system) => + Cluster.Get(system).SelfAddress.Port + ?? throw new InvalidOperationException("Cluster address has no port."); + + private static async Task AwaitClusterUpAsync(ActorSystem system, int expectedMembers) + { + Cluster cluster = Cluster.Get(system); + DateTime deadline = DateTime.UtcNow + Patience; + + while (DateTime.UtcNow < deadline) + { + if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers) + { + return; + } + + await Task.Delay(100); + } + + throw new TimeoutException( + $"Cluster did not reach {expectedMembers} Up members within {Patience}."); + } + + // Writes a throwaway secret on A and waits for it to land on B. That round trip is the only + // reliable signal that pub/sub subscriptions have gossiped across both nodes. + private async Task AwaitBothSubscribedAsync() + { + var probe = new SecretName("warmup/probe"); + DateTime deadline = DateTime.UtcNow + Patience; + + while (DateTime.UtcNow < deadline) + { + await _writableA.UpsertAsync(Row("warmup/probe", 0x01), CancellationToken.None); + + if (await WaitForAsync(_storeB, probe, _ => true, TimeSpan.FromMilliseconds(500))) + { + await _storeA.DeleteAsync(probe, "warmup", CancellationToken.None); + await _storeB.DeleteAsync(probe, "warmup", CancellationToken.None); + return; + } + } + + throw new TimeoutException("Nodes did not establish pub/sub subscriptions in time."); + } + + private static StoredSecret Row(string name, byte marker) => new() + { + Name = new SecretName(name), + ContentType = SecretContentType.Text, + Ciphertext = [marker], + Nonce = [1], + Tag = [2], + WrappedDek = [3], + WrapNonce = [4], + WrapTag = [5], + KekId = "sha256:shared-kek", + Revision = 0, + CreatedUtc = DateTimeOffset.UtcNow, + UpdatedUtc = DateTimeOffset.UtcNow, + }; + + private static async Task WaitForAsync( + ISecretStore store, SecretName name, Func predicate, TimeSpan? timeout = null) + { + DateTime deadline = DateTime.UtcNow + (timeout ?? Patience); + + while (DateTime.UtcNow < deadline) + { + StoredSecret? row = await store.GetAsync(name, CancellationToken.None); + + if (row is not null && predicate(row)) + { + return true; + } + + await Task.Delay(50); + } + + return false; + } + + [Fact] + public async Task A_secret_written_on_node_A_becomes_resolvable_on_node_B() + { + var name = new SecretName("db/password"); + await _writableA.UpsertAsync(Row("db/password", 0xAA), CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0xAA), + "Node B never received the secret written on node A."); + } + + [Fact] + public async Task The_ciphertext_arrives_byte_identical() + { + var name = new SecretName("app/exact"); + await _writableA.UpsertAsync(Row("app/exact", 0x7F), CancellationToken.None); + + Assert.True(await WaitForAsync(_storeB, name, _ => true)); + + StoredSecret onA = (await _storeA.GetAsync(name, CancellationToken.None))!; + StoredSecret onB = (await _storeB.GetAsync(name, CancellationToken.None))!; + + Assert.Equal(onA.Ciphertext, onB.Ciphertext); + Assert.Equal(onA.WrappedDek, onB.WrappedDek); + Assert.Equal(onA.KekId, onB.KekId); + // Verbatim: the peer keeps the originating node's revision and timestamp, or the row would + // look newer on B than on A and bounce back. + Assert.Equal(onA.Revision, onB.Revision); + Assert.Equal(onA.UpdatedUtc, onB.UpdatedUtc); + } + + [Fact] + public async Task A_delete_on_node_A_tombstones_the_row_on_node_B() + { + var name = new SecretName("app/doomed"); + await _writableA.UpsertAsync(Row("app/doomed", 0xBB), CancellationToken.None); + Assert.True(await WaitForAsync(_storeB, name, _ => true)); + + await _writableA.DeleteAsync(name, "operator", CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeB, name, r => r.IsDeleted), + "The delete never propagated to node B."); + } + + [Fact] + public async Task An_update_overwrites_the_earlier_value_on_the_peer() + { + var name = new SecretName("app/rotating"); + await _writableA.UpsertAsync(Row("app/rotating", 0x01), CancellationToken.None); + Assert.True(await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0x01)); + + await _writableA.UpsertAsync(Row("app/rotating", 0x02), CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0x02 && r.Revision == 1), + "Node B did not converge on the updated value."); + } + + [Fact] + public async Task Anti_entropy_repairs_a_write_that_never_got_broadcast() + { + // Simulates a dropped publish — the write goes straight to node A's store, bypassing the + // replicator entirely, so the ONLY route to node B is the periodic manifest exchange. This is + // the case that makes best-effort publishing safe. + var name = new SecretName("app/missed-broadcast"); + await _storeA.UpsertAsync(Row("app/missed-broadcast", 0xCC), CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0xCC), + "Anti-entropy failed to repair a write that was never broadcast."); + } + + [Fact] + public async Task Anti_entropy_pulls_in_the_other_direction_too() + { + // The mirror case: node B holds a row node A has never heard of. A cannot request a name it + // does not know exists, so this only converges if the exchange pushes as well as pulls. + var name = new SecretName("app/born-on-b"); + await _storeB.UpsertAsync(Row("app/born-on-b", 0xDD), CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeA, name, r => r.Ciphertext[0] == 0xDD), + "Node A never learned about a secret that originated on node B."); + } + + [Fact] + public async Task A_node_that_joins_late_catches_up_on_everything() + { + // Partition-heal / cold-start: the whole reason to choose this transport over a shared store. + await _writableA.UpsertAsync(Row("bulk/one", 0x11), CancellationToken.None); + await _writableA.UpsertAsync(Row("bulk/two", 0x22), CancellationToken.None); + await _writableA.UpsertAsync(Row("bulk/three", 0x33), CancellationToken.None); + + SqliteSecretStore lateStore = CreateStore(); + _systemB.ActorOf( + SecretReplicationActor.Props(lateStore, null, AnnounceInterval), "late-joiner"); + + Assert.True(await WaitForAsync(lateStore, new SecretName("bulk/one"), r => r.Ciphertext[0] == 0x11)); + Assert.True(await WaitForAsync(lateStore, new SecretName("bulk/two"), r => r.Ciphertext[0] == 0x22)); + Assert.True(await WaitForAsync(lateStore, new SecretName("bulk/three"), r => r.Ciphertext[0] == 0x33)); + } + + [Fact] + public async Task A_broadcast_round_trip_does_not_drift_the_originating_nodes_revision() + { + // Note on what this does and does NOT prove: the revision would also stay put if the self-echo + // filter were removed, because re-applying an identical row ties on (updated_utc, revision) + // and last-writer-wins rejects a tie. So this covers the drift symptom via defence in depth, + // not the filter itself. The filter is what stops the echo being forwarded onward, which + // needs three nodes to observe — worth adding if a third node ever joins this rig. + var name = new SecretName("app/no-echo"); + await _writableA.UpsertAsync(Row("app/no-echo", 0xEE), CancellationToken.None); + Assert.True(await WaitForAsync(_storeB, name, _ => true)); + + await Task.Delay(AnnounceInterval * 4); + + StoredSecret onA = (await _storeA.GetAsync(name, CancellationToken.None))!; + Assert.Equal(0, onA.Revision); + } + + [Fact] + public async Task A_malformed_row_from_a_peer_is_discarded_without_stalling_replication() + { + // Peer input is untrusted: a path-traversing name must be dropped, and — critically — the + // actor must keep serving everything else afterwards. + _replicatorB.Tell(new Protocol.SecretRowsMessage( + [ + new Protocol.SecretRowDto + { + Name = "../../etc/passwd", + ContentType = "Text", + Ciphertext = [1], Nonce = [1], Tag = [1], + WrappedDek = [1], WrapNonce = [1], WrapTag = [1], + KekId = "sha256:x", + Revision = 0, + CreatedUtc = DateTimeOffset.UtcNow.ToString("O"), + UpdatedUtc = DateTimeOffset.UtcNow.ToString("O"), + }, + ])); + + var name = new SecretName("app/after-malformed"); + await _writableA.UpsertAsync(Row("app/after-malformed", 0x5A), CancellationToken.None); + + Assert.True( + await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0x5A), + "Replication stalled after a malformed row was received."); + } + + public async Task DisposeAsync() + { + await CoordinatedShutdown.Get(_systemA).Run(CoordinatedShutdown.ClrExitReason.Instance); + await CoordinatedShutdown.Get(_systemB).Run(CoordinatedShutdown.ClrExitReason.Instance); + + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + + foreach (string db in _dbPaths) + { + foreach (string path in new[] { db, db + "-wal", db + "-shm" }) + { + try + { + if (File.Exists(path)) + { + File.Delete(path); + } + } + catch (IOException) + { + // Best-effort temp cleanup. + } + } + } + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.csproj b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.csproj new file mode 100644 index 0000000..b51027a --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.csproj @@ -0,0 +1,25 @@ + + + + false + + + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/DependencyInjection/AddZbSecretsSqlServerTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/DependencyInjection/AddZbSecretsSqlServerTests.cs new file mode 100644 index 0000000..2904da0 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/DependencyInjection/AddZbSecretsSqlServerTests.cs @@ -0,0 +1,111 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection; +using ZB.MOM.WW.Secrets.Sqlite; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.DependencyInjection; + +/// +/// Container-resolution tests for both SQL-Server topologies. +/// +/// +/// These exist because a code review caught that neither mode could resolve at all: the decorators +/// ask for the concrete , which the core package only registered +/// behind . Every unit test passed and both modes were dead on arrival, +/// because nothing built a provider. Resolving the graph is the assertion that matters. +/// +public sealed class AddZbSecretsSqlServerTests +{ + private static IConfiguration Config(string sqlitePath) => + new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["Secrets:SqlitePath"] = sqlitePath, + ["Secrets:RunMigrationsOnStartup"] = "false", + ["Secrets:MasterKey:Source"] = "Environment", + ["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_TEST_KEY", + ["Secrets:SqlServer:ConnectionString"] = "Server=unused;Database=x;", + }).Build(); + + private static string TempDb() => Path.Combine(Path.GetTempPath(), $"zb-di-{Guid.NewGuid():N}.db"); + + [Fact] + public void SharedStore_mode_resolves_the_SqlServer_store_as_ISecretStore() + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddZbSecretsSqlServerStore(Config(TempDb())); + + using ServiceProvider provider = services.BuildServiceProvider(); + + // The SQL-Server store displaces SQLite — no decorator, because there is nothing to replicate. + Assert.IsType(provider.GetRequiredService()); + Assert.IsType(provider.GetRequiredService()); + } + + [Fact] + public void HubReplication_mode_resolves_a_decorated_local_store() + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddZbSecretsSqlServerReplication(Config(TempDb())); + + using ServiceProvider provider = services.BuildServiceProvider(); + + Assert.IsType(provider.GetRequiredService()); + Assert.IsType(provider.GetRequiredService()); + // The local store is still provisioned by the core SQLite migrator, not the hub's. + Assert.IsType(provider.GetRequiredService()); + } + + [Fact] + public void HubReplication_mode_registers_both_the_hub_migration_and_the_sync_service() + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddZbSecretsSqlServerReplication(Config(TempDb())); + + using ServiceProvider provider = services.BuildServiceProvider(); + IHostedService[] hosted = [.. provider.GetServices()]; + + // The hub schema is nobody's "own" store, so every node must provision it at startup too. + Assert.Contains(hosted, h => h is SqlServerSecretSyncService); + Assert.Single(hosted, h => h.GetType().Name == "SqlServerHubMigrationHostedService"); + } + + [Fact] + public void The_decorator_wraps_the_undecorated_local_store_rather_than_itself() + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddZbSecretsSqlServerReplication(Config(TempDb())); + + using ServiceProvider provider = services.BuildServiceProvider(); + + // Guards against the self-referential registration that would stack-overflow on first use. + var decorated = (ReplicatingSecretStore)provider.GetRequiredService(); + Assert.NotNull(decorated); + Assert.NotSame(decorated, provider.GetRequiredService()); + } + + [Fact] + public void A_missing_connection_string_fails_at_registration_not_at_first_resolve() + { + IConfiguration config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["Secrets:SqlitePath"] = TempDb(), + }) + .Build(); + + var services = new ServiceCollection(); + services.AddLogging(); + + InvalidOperationException ex = Assert.Throws( + () => services.AddZbSecretsSqlServerStore(config)); + + Assert.Contains("ConnectionString", ex.Message, StringComparison.Ordinal); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Fakes/SqliteBackedHub.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Fakes/SqliteBackedHub.cs new file mode 100644 index 0000000..2f85574 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Fakes/SqliteBackedHub.cs @@ -0,0 +1,51 @@ +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replicator.SqlServer; +using ZB.MOM.WW.Secrets.Sqlite; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Fakes; + +/// +/// An backed by a real SQLite store, standing in for the shared +/// SQL-Server hub in offline convergence tests. +/// +/// +/// Deliberately backed by a real store rather than a dictionary: the behaviour under test is +/// last-writer-wins convergence, which lives in ApplyReplicatedAsync. A hand-rolled fake would +/// be re-implementing the very logic the tests exist to check, and would happily agree with a broken +/// reconciler. The live SQL-Server tests then confirm the T-SQL store behaves the same way. +/// +/// The SQLite store standing in for the hub database. +public sealed class SqliteBackedHub(SqliteSecretStore store) : ISecretReplicationHub +{ + /// Number of times the sweep asked this hub for rows — asserts the fetch was skipped. + public int GetManyCallCount { get; private set; } + + /// + public Task> GetManifestAsync(CancellationToken ct) => + store.GetManifestAsync(ct); + + /// + public async Task> GetManyAsync( + IReadOnlyList names, CancellationToken ct) + { + GetManyCallCount++; + + var rows = new List(names.Count); + + foreach (SecretName name in names) + { + StoredSecret? row = await store.GetAsync(name, ct); + + if (row is not null) + { + rows.Add(row); + } + } + + return rows; + } + + /// + public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) => + store.ApplyReplicatedAsync(row, ct); +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/LiveSqlServerFixture.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/LiveSqlServerFixture.cs new file mode 100644 index 0000000..fb8e457 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/LiveSqlServerFixture.cs @@ -0,0 +1,68 @@ +using Microsoft.Data.SqlClient; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Live; + +/// +/// Shared connection details for the env-gated live SQL-Server suite. +/// +/// +/// Gated on SECRETS_SQLSERVER_CONNSTR and skips cleanly when it is unset, matching the family +/// live-test idiom — the offline suite must stay runnable on any machine. Set it to a database the +/// test is allowed to create and drop a schema in, for example: +/// Server=10.100.0.35,31433;Database=ZbSecretsTest;User Id=sa;Password=...;TrustServerCertificate=True. +/// +public static class LiveSqlServer +{ + /// Environment variable holding the live connection string. + public const string ConnectionStringVariable = "SECRETS_SQLSERVER_CONNSTR"; + + /// The configured connection string, or when the suite is not enabled. + public static string? ConnectionString => + Environment.GetEnvironmentVariable(ConnectionStringVariable); + + /// Whether the live suite should run. + public static bool IsEnabled => !string.IsNullOrWhiteSpace(ConnectionString); + + /// Skip reason shown when the suite is not enabled. + public const string SkipReason = + "Live SQL-Server suite disabled; set SECRETS_SQLSERVER_CONNSTR to enable."; + + /// + /// Builds options against a throwaway schema so parallel runs and repeat runs cannot collide, + /// and so a failed run leaves no trace in a shared database. + /// + /// The unique schema name for this test. + /// Options targeting that schema. + public static SqlServerSecretsOptions OptionsFor(string schemaName) => new() + { + ConnectionString = ConnectionString!, + SchemaName = schemaName, + }; + + /// Generates a unique, allow-list-legal schema name for one test class. + /// A fresh schema name. + public static string NewSchemaName() => $"zbtest_{Guid.NewGuid():N}"[..32]; + + /// Drops the throwaway schema and its tables. + /// The schema to drop. + /// A task that completes when the schema is gone. + public static async Task DropSchemaAsync(string schemaName) + { + if (!IsEnabled) + { + return; + } + + await using var connection = new SqlConnection(ConnectionString); + await connection.OpenAsync(); + + await using SqlCommand command = connection.CreateCommand(); + command.CommandText = $""" + IF OBJECT_ID(N'[{schemaName}].[secret]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[secret]; + IF OBJECT_ID(N'[{schemaName}].[schema_version]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[schema_version]; + IF EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}') EXEC(N'DROP SCHEMA [{schemaName}]'); + """; + + await command.ExecuteNonQueryAsync(); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/SqlServerSecretStoreLiveTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/SqlServerSecretStoreLiveTests.cs new file mode 100644 index 0000000..46a0a02 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/Live/SqlServerSecretStoreLiveTests.cs @@ -0,0 +1,365 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Live; + +/// +/// The SQLite store's test suite, ported case-for-case onto the SQL-Server store. +/// +/// +/// Ported rather than newly written on purpose. The two stores must be behaviourally identical — +/// a cluster can hold both at once — so the strongest available check is that the assertions +/// written against SQLite pass unchanged here. Anywhere the T-SQL diverges in observable behaviour, +/// one of these fails. +/// +[Collection("live-sqlserver")] +public sealed class SqlServerSecretStoreLiveTests : IAsyncLifetime +{ + private readonly string _schema = LiveSqlServer.NewSchemaName(); + private SqlServerSecretStore _store = null!; + + public async Task InitializeAsync() + { + if (!LiveSqlServer.IsEnabled) + { + return; + } + + var factory = new SecretsSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema)); + await new SqlServerSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None); + _store = new SqlServerSecretStore(factory); + } + + public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema); + + private static StoredSecret MakeSecret( + string name, + long revision = 0, + byte[]? ciphertext = null, + DateTimeOffset? updatedUtc = null, + DateTimeOffset? createdUtc = null, + bool isDeleted = false, + DateTimeOffset? deletedUtc = null, + string? createdBy = "alice", + string? updatedBy = "alice") => new() + { + Name = new SecretName(name), + Description = "desc", + ContentType = SecretContentType.ConnectionString, + Ciphertext = ciphertext ?? [1, 2, 3], + Nonce = [4, 5, 6], + Tag = [7, 8, 9], + WrappedDek = [10, 11, 12], + WrapNonce = [13, 14, 15], + WrapTag = [16, 17, 18], + KekId = "kek-1", + Revision = revision, + IsDeleted = isDeleted, + DeletedUtc = deletedUtc, + CreatedUtc = createdUtc ?? DateTimeOffset.UtcNow, + UpdatedUtc = updatedUtc ?? DateTimeOffset.UtcNow, + CreatedBy = createdBy, + UpdatedBy = updatedBy, + }; + + [SkippableFact] + public async Task Upsert_Then_Get_RoundTrips() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("app/db-conn"), CancellationToken.None); + + StoredSecret? got = await _store.GetAsync(new SecretName("app/db-conn"), CancellationToken.None); + + Assert.NotNull(got); + Assert.Equal("app/db-conn", got!.Name.Value); + Assert.Equal("desc", got.Description); + Assert.Equal(SecretContentType.ConnectionString, got.ContentType); + Assert.Equal(new byte[] { 1, 2, 3 }, got.Ciphertext); + Assert.Equal(new byte[] { 4, 5, 6 }, got.Nonce); + Assert.Equal(new byte[] { 7, 8, 9 }, got.Tag); + Assert.Equal(new byte[] { 10, 11, 12 }, got.WrappedDek); + Assert.Equal(new byte[] { 13, 14, 15 }, got.WrapNonce); + Assert.Equal(new byte[] { 16, 17, 18 }, got.WrapTag); + Assert.Equal("kek-1", got.KekId); + Assert.Equal(0, got.Revision); + Assert.False(got.IsDeleted); + Assert.Null(got.DeletedUtc); + Assert.Equal("alice", got.CreatedBy); + Assert.Equal("alice", got.UpdatedBy); + } + + [SkippableFact] + public async Task Get_ReturnsNull_WhenAbsent() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + Assert.Null(await _store.GetAsync(new SecretName("nope"), CancellationToken.None)); + } + + [SkippableFact] + public async Task Upsert_Existing_OverwritesInPlace_BumpsRevision() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync( + MakeSecret("app/rotating", ciphertext: [1, 1, 1], createdBy: "alice", updatedBy: "alice"), + CancellationToken.None); + + StoredSecret afterFirst = + (await _store.GetAsync(new SecretName("app/rotating"), CancellationToken.None))!; + Assert.Equal(0, afterFirst.Revision); + DateTimeOffset originalCreatedUtc = afterFirst.CreatedUtc; + + await _store.UpsertAsync( + MakeSecret("app/rotating", ciphertext: [9, 9, 9], createdBy: "bob", updatedBy: "bob"), + CancellationToken.None); + + StoredSecret afterSecond = + (await _store.GetAsync(new SecretName("app/rotating"), CancellationToken.None))!; + Assert.Equal(1, afterSecond.Revision); + Assert.Equal(new byte[] { 9, 9, 9 }, afterSecond.Ciphertext); + Assert.Equal(originalCreatedUtc, afterSecond.CreatedUtc); + Assert.Equal("alice", afterSecond.CreatedBy); + Assert.Equal("bob", afterSecond.UpdatedBy); + Assert.False(afterSecond.IsDeleted); + } + + [SkippableFact] + public async Task List_ExcludesTombstoned_ByDefault() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("keep"), CancellationToken.None); + await _store.UpsertAsync(MakeSecret("gone"), CancellationToken.None); + await _store.DeleteAsync(new SecretName("gone"), "carol", CancellationToken.None); + + IReadOnlyList visible = + await _store.ListAsync(includeDeleted: false, CancellationToken.None); + Assert.DoesNotContain(visible, m => m.Name.Value == "gone"); + Assert.Contains(visible, m => m.Name.Value == "keep"); + + IReadOnlyList all = + await _store.ListAsync(includeDeleted: true, CancellationToken.None); + Assert.Contains(all, m => m.Name.Value == "gone"); + Assert.Contains(all, m => m.Name.Value == "keep"); + } + + [SkippableFact] + public async Task Delete_SetsTombstone_BumpsRevision_ReturnsFalseWhenAbsent() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("app/secret"), CancellationToken.None); + + Assert.True(await _store.DeleteAsync(new SecretName("app/secret"), "carol", CancellationToken.None)); + + StoredSecret tombstoned = + (await _store.GetAsync(new SecretName("app/secret"), CancellationToken.None))!; + Assert.True(tombstoned.IsDeleted); + Assert.NotNull(tombstoned.DeletedUtc); + Assert.Equal(1, tombstoned.Revision); + Assert.Equal("carol", tombstoned.UpdatedBy); + + Assert.False(await _store.DeleteAsync(new SecretName("app/secret"), "carol", CancellationToken.None)); + Assert.False(await _store.DeleteAsync(new SecretName("never-existed"), "carol", CancellationToken.None)); + } + + [SkippableFact] + public async Task GetManifest_ReturnsAllRows() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("a"), CancellationToken.None); + await _store.UpsertAsync(MakeSecret("b"), CancellationToken.None); + await _store.DeleteAsync(new SecretName("b"), "carol", CancellationToken.None); + + IReadOnlyList manifest = await _store.GetManifestAsync(CancellationToken.None); + + Assert.Equal(2, manifest.Count); + SecretManifestEntry a = manifest.Single(e => e.Name.Value == "a"); + SecretManifestEntry b = manifest.Single(e => e.Name.Value == "b"); + Assert.False(a.IsDeleted); + Assert.Equal(0, a.Revision); + Assert.True(b.IsDeleted); + Assert.Equal(1, b.Revision); + } + + [SkippableFact] + public async Task ApplyReplicated_AppliesNewer_IgnoresStale() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + DateTimeOffset t1 = new(2026, 1, 1, 0, 0, 0, TimeSpan.Zero); + DateTimeOffset t2 = new(2026, 1, 2, 0, 0, 0, TimeSpan.Zero); + DateTimeOffset t3 = new(2026, 1, 3, 0, 0, 0, TimeSpan.Zero); + + await _store.ApplyReplicatedAsync( + MakeSecret("repl", revision: 5, ciphertext: [5, 5, 5], updatedUtc: t2, createdUtc: t1), + CancellationToken.None); + + Assert.Equal(5, (await _store.GetAsync(new SecretName("repl"), CancellationToken.None))!.Revision); + + await _store.ApplyReplicatedAsync( + MakeSecret("repl", revision: 6, ciphertext: [6, 6, 6], updatedUtc: t3, createdUtc: t1), + CancellationToken.None); + + StoredSecret afterNewer = (await _store.GetAsync(new SecretName("repl"), CancellationToken.None))!; + Assert.Equal(6, afterNewer.Revision); + Assert.Equal(new byte[] { 6, 6, 6 }, afterNewer.Ciphertext); + Assert.Equal(t3, afterNewer.UpdatedUtc); + + await _store.ApplyReplicatedAsync( + MakeSecret("repl", revision: 4, ciphertext: [4, 4, 4], updatedUtc: t1, createdUtc: t1), + CancellationToken.None); + + StoredSecret afterStale = (await _store.GetAsync(new SecretName("repl"), CancellationToken.None))!; + Assert.Equal(6, afterStale.Revision); + Assert.Equal(new byte[] { 6, 6, 6 }, afterStale.Ciphertext); + } + + [SkippableFact] + public async Task ApplyRewrap_ChangesOnlyWrapEnvelopeAndKekId_PreservingRevisionBodyAndTimestamps() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync( + MakeSecret("app/rewrap", ciphertext: [1, 2, 3], createdUtc: new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero)), + CancellationToken.None); + await _store.UpsertAsync(MakeSecret("app/rewrap", ciphertext: [1, 2, 3]), CancellationToken.None); + + StoredSecret before = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!; + Assert.Equal(1, before.Revision); + + bool applied = await _store.ApplyRewrapAsync( + before with + { + WrappedDek = [90, 91, 92], + WrapNonce = [93, 94, 95], + WrapTag = [96, 97, 98], + KekId = "kek-2", + }, + before.WrappedDek, + CancellationToken.None); + Assert.True(applied); + + StoredSecret after = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!; + Assert.Equal(new byte[] { 90, 91, 92 }, after.WrappedDek); + Assert.Equal(new byte[] { 93, 94, 95 }, after.WrapNonce); + Assert.Equal(new byte[] { 96, 97, 98 }, after.WrapTag); + Assert.Equal("kek-2", after.KekId); + Assert.Equal(before.Revision, after.Revision); + Assert.Equal(before.UpdatedUtc, after.UpdatedUtc); + Assert.Equal(before.CreatedUtc, after.CreatedUtc); + Assert.Equal(before.UpdatedBy, after.UpdatedBy); + Assert.Equal(new byte[] { 1, 2, 3 }, after.Ciphertext); + Assert.Equal(before.Nonce, after.Nonce); + Assert.Equal(before.Tag, after.Tag); + } + + [SkippableFact] + public async Task ApplyRewrap_RewrapsTombstonedRows() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("app/dead"), CancellationToken.None); + await _store.DeleteAsync(new SecretName("app/dead"), "carol", CancellationToken.None); + + StoredSecret tomb = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!; + Assert.True(tomb.IsDeleted); + + Assert.True(await _store.ApplyRewrapAsync( + tomb with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" }, + tomb.WrappedDek, + CancellationToken.None)); + + StoredSecret after = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!; + Assert.Equal("kek-2", after.KekId); + Assert.True(after.IsDeleted); + Assert.Equal(tomb.Revision, after.Revision); + } + + [SkippableFact] + public async Task ApplyRewrap_ReturnsFalse_WhenRowAbsent() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + StoredSecret ghost = MakeSecret("never-existed") with { KekId = "kek-2" }; + + Assert.False(await _store.ApplyRewrapAsync(ghost, ghost.WrappedDek, CancellationToken.None)); + } + + [SkippableFact] + public async Task ApplyRewrap_ReturnsFalse_WhenCurrentWrapChanged_UnderConcurrentWrite() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("app/cas", ciphertext: [1, 2, 3]), CancellationToken.None); + StoredSecret before = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!; + + await _store.UpsertAsync( + MakeSecret("app/cas", ciphertext: [9, 9, 9]) with { WrappedDek = [77, 78, 79] }, + CancellationToken.None); + + bool applied = await _store.ApplyRewrapAsync( + before with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" }, + before.WrappedDek, + CancellationToken.None); + + Assert.False(applied); + StoredSecret after = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!; + Assert.Equal(new byte[] { 77, 78, 79 }, after.WrappedDek); + Assert.Equal("kek-1", after.KekId); + } + + [SkippableFact] + public async Task GetMany_returns_only_the_requested_rows() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + await _store.UpsertAsync(MakeSecret("a/one"), CancellationToken.None); + await _store.UpsertAsync(MakeSecret("b/two"), CancellationToken.None); + await _store.UpsertAsync(MakeSecret("c/three"), CancellationToken.None); + + IReadOnlyList rows = await _store.GetManyAsync( + [new SecretName("a/one"), new SecretName("c/three"), new SecretName("absent/name")], + CancellationToken.None); + + Assert.Equal(2, rows.Count); + Assert.Contains(rows, r => r.Name.Value == "a/one"); + Assert.Contains(rows, r => r.Name.Value == "c/three"); + } + + [SkippableFact] + public async Task GetMany_short_circuits_on_an_empty_request() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + // Guards the IN () clause: an empty list would otherwise produce invalid T-SQL. + Assert.Empty(await _store.GetManyAsync([], CancellationToken.None)); + } + + [SkippableFact] + public async Task Migrator_is_idempotent() + { + Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason); + + var factory = new SecretsSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema)); + var migrator = new SqlServerSecretsStoreMigrator(factory); + + // Every node runs this at every startup — re-running must be a no-op, not an error. + await migrator.MigrateAsync(CancellationToken.None); + await migrator.MigrateAsync(CancellationToken.None); + + await _store.UpsertAsync(MakeSecret("survives/migration"), CancellationToken.None); + await migrator.MigrateAsync(CancellationToken.None); + + Assert.NotNull(await _store.GetAsync(new SecretName("survives/migration"), CancellationToken.None)); + } +} + +/// +/// Serializes the live suite: the tests share one database server and each provisions its own +/// schema, so running them in parallel would multiply connection load for no benefit. +/// +[CollectionDefinition("live-sqlserver", DisableParallelization = true)] +public sealed class LiveSqlServerCollection; diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/SqlServerSecretSyncServiceTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/SqlServerSecretSyncServiceTests.cs new file mode 100644 index 0000000..7e31d4d --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/SqlServerSecretSyncServiceTests.cs @@ -0,0 +1,205 @@ +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Fakes; +using ZB.MOM.WW.Secrets.Sqlite; + +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests; + +/// +/// Convergence behaviour of the bidirectional hub sweep, exercised against real stores on both +/// sides so the assertions are about actual persisted state rather than mock interactions. +/// +public sealed class SqlServerSecretSyncServiceTests : IDisposable +{ + private readonly List _dbPaths = []; + private readonly SqliteSecretStore _local; + private readonly SqliteSecretStore _hubStore; + private readonly SqliteBackedHub _hub; + + public SqlServerSecretSyncServiceTests() + { + _local = CreateStore(); + _hubStore = CreateStore(); + _hub = new SqliteBackedHub(_hubStore); + } + + private SqliteSecretStore CreateStore() + { + string path = Path.Combine(Path.GetTempPath(), $"zb-sync-{Guid.NewGuid():N}.db"); + _dbPaths.Add(path); + var factory = new SecretsSqliteConnectionFactory(path); + new SqliteSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None).GetAwaiter().GetResult(); + return new SqliteSecretStore(factory); + } + + private SqlServerSecretSyncService CreateService() => new( + _local, + _hub, + cacheInvalidator: null, + Options.Create(new SqlServerSecretsOptions { ConnectionString = "Server=unused;" }), + NullLogger.Instance); + + private static StoredSecret Row(string name, byte marker) => new() + { + Name = new SecretName(name), + ContentType = SecretContentType.Text, + Ciphertext = [marker], + Nonce = [1], + Tag = [2], + WrappedDek = [3], + WrapNonce = [4], + WrapTag = [5], + KekId = "sha256:test", + Revision = 0, + CreatedUtc = DateTimeOffset.UtcNow, + UpdatedUtc = DateTimeOffset.UtcNow, + }; + + [Fact] + public async Task Sweep_pulls_a_secret_the_node_has_never_seen() + { + await _hubStore.UpsertAsync(Row("db/password", 0xAA), CancellationToken.None); + + (int pulled, int pushed) = await CreateService().SweepAsync(CancellationToken.None); + + Assert.Equal(1, pulled); + Assert.Equal(0, pushed); + + StoredSecret? local = await _local.GetAsync(new SecretName("db/password"), CancellationToken.None); + Assert.NotNull(local); + Assert.Equal(0xAA, local!.Ciphertext[0]); + } + + [Fact] + public async Task Sweep_pushes_a_local_write_the_hub_never_received() + { + // The case that makes best-effort publishing safe: the publish failed (hub was down), so the + // hub has never heard of this name and can never ask for it. Only a push sweep repairs this. + await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None); + + (int pulled, int pushed) = await CreateService().SweepAsync(CancellationToken.None); + + Assert.Equal(0, pulled); + Assert.Equal(1, pushed); + + StoredSecret? hub = await _hubStore.GetAsync(new SecretName("api/key"), CancellationToken.None); + Assert.NotNull(hub); + Assert.Equal(0xBB, hub!.Ciphertext[0]); + } + + [Fact] + public async Task Sweep_preserves_the_originating_revision_when_pushing() + { + // Pushed verbatim, not upserted: if the hub stamped its own revision the row would look newer + // than the writer's copy and bounce straight back on the next sweep. + await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None); + await _local.UpsertAsync(Row("api/key", 0xCC), CancellationToken.None); + + StoredSecret localRow = (await _local.GetAsync(new SecretName("api/key"), CancellationToken.None))!; + Assert.Equal(1, localRow.Revision); + + await CreateService().SweepAsync(CancellationToken.None); + + StoredSecret hubRow = (await _hubStore.GetAsync(new SecretName("api/key"), CancellationToken.None))!; + Assert.Equal(localRow.Revision, hubRow.Revision); + Assert.Equal(localRow.UpdatedUtc, hubRow.UpdatedUtc); + } + + [Fact] + public async Task Sweep_converges_both_directions_in_one_pass() + { + await _local.UpsertAsync(Row("only/local", 0x11), CancellationToken.None); + await _hubStore.UpsertAsync(Row("only/hub", 0x22), CancellationToken.None); + + (int pulled, int pushed) = await CreateService().SweepAsync(CancellationToken.None); + + Assert.Equal(1, pulled); + Assert.Equal(1, pushed); + + Assert.NotNull(await _local.GetAsync(new SecretName("only/hub"), CancellationToken.None)); + Assert.NotNull(await _hubStore.GetAsync(new SecretName("only/local"), CancellationToken.None)); + } + + [Fact] + public async Task Sweep_is_idempotent_so_a_converged_pair_stays_quiet() + { + await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None); + + SqlServerSecretSyncService service = CreateService(); + await service.SweepAsync(CancellationToken.None); + + (int pulled, int pushed) = await service.SweepAsync(CancellationToken.None); + + // A steady-state cluster must not write on every tick — that would churn the hub forever. + Assert.Equal(0, pulled); + Assert.Equal(0, pushed); + } + + [Fact] + public async Task Sweep_propagates_a_delete_as_a_tombstone() + { + await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None); + await CreateService().SweepAsync(CancellationToken.None); + + await _local.DeleteAsync(new SecretName("api/key"), "tester", CancellationToken.None); + await CreateService().SweepAsync(CancellationToken.None); + + StoredSecret hubRow = (await _hubStore.GetAsync(new SecretName("api/key"), CancellationToken.None))!; + Assert.True(hubRow.IsDeleted); + } + + [Fact] + public async Task Sweep_does_not_resurrect_a_secret_deleted_on_the_hub() + { + await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None); + await CreateService().SweepAsync(CancellationToken.None); + + await _hubStore.DeleteAsync(new SecretName("api/key"), "admin", CancellationToken.None); + await CreateService().SweepAsync(CancellationToken.None); + + StoredSecret localRow = (await _local.GetAsync(new SecretName("api/key"), CancellationToken.None))!; + Assert.True(localRow.IsDeleted); + } + + [Fact] + public async Task Sweep_skips_the_hub_fetch_when_nothing_is_newer() + { + await CreateService().SweepAsync(CancellationToken.None); + + Assert.Equal(0, _hub.GetManyCallCount); + } + + [Fact] + public async Task Sweep_lets_the_newer_side_win_a_conflicting_write() + { + await _local.UpsertAsync(Row("api/key", 0x11), CancellationToken.None); + await CreateService().SweepAsync(CancellationToken.None); + + // Both sides then write independently; the hub's write is later. + await _local.UpsertAsync(Row("api/key", 0x22), CancellationToken.None); + await Task.Delay(10); + await _hubStore.UpsertAsync(Row("api/key", 0x33), CancellationToken.None); + + await CreateService().SweepAsync(CancellationToken.None); + + StoredSecret localRow = (await _local.GetAsync(new SecretName("api/key"), CancellationToken.None))!; + Assert.Equal(0x33, localRow.Ciphertext[0]); + } + + public void Dispose() + { + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + + foreach (string db in _dbPaths) + { + foreach (string path in new[] { db, db + "-wal", db + "-shm" }) + { + if (File.Exists(path)) + { + File.Delete(path); + } + } + } + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/SqlServerSecretsOptionsTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/SqlServerSecretsOptionsTests.cs new file mode 100644 index 0000000..32985d9 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/SqlServerSecretsOptionsTests.cs @@ -0,0 +1,96 @@ +namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests; + +/// +/// The schema-name allow-list is the injection boundary for the one value that cannot be +/// parameterized (a T-SQL object name), so it gets adversarial coverage rather than a happy path. +/// +public sealed class SqlServerSecretsOptionsTests +{ + [Theory] + [InlineData("zbsecrets")] + [InlineData("ZbSecrets")] + [InlineData("_private")] + [InlineData("schema_1")] + public void SchemaName_accepts_plain_identifiers(string value) + { + var options = new SqlServerSecretsOptions { SchemaName = value }; + + Assert.Equal(value, options.SchemaName); + } + + [Theory] + // The bracket-escape trick that would otherwise break out of the [quoting] at the use site. + [InlineData("evil]]; DROP TABLE secret; --")] + [InlineData("dbo.secret")] + [InlineData("with space")] + [InlineData("quote'name")] + [InlineData("semi;colon")] + [InlineData("1leading_digit")] + [InlineData("")] + [InlineData(" ")] + public void SchemaName_rejects_anything_that_is_not_a_plain_identifier(string value) + { + Assert.Throws(() => new SqlServerSecretsOptions { SchemaName = value }); + } + + [Fact] + public void SchemaName_rejects_an_over_long_identifier() + { + Assert.Throws(() => + new SqlServerSecretsOptions { SchemaName = new string('a', 129) }); + } + + [Fact] + public void Validate_rejects_a_missing_connection_string() + { + InvalidOperationException ex = + Assert.Throws(() => new SqlServerSecretsOptions().Validate()); + + Assert.Contains("ConnectionString", ex.Message, StringComparison.Ordinal); + } + + [Fact] + public void Validate_rejects_a_non_positive_sync_interval() + { + var options = new SqlServerSecretsOptions + { + ConnectionString = "Server=x;", + SyncInterval = TimeSpan.Zero, + }; + + Assert.Throws(options.Validate); + } + + [Fact] + public void Validate_rejects_a_non_positive_command_timeout() + { + var options = new SqlServerSecretsOptions + { + ConnectionString = "Server=x;", + CommandTimeout = TimeSpan.FromSeconds(-1), + }; + + Assert.Throws(options.Validate); + } + + [Fact] + public void Validate_accepts_a_fully_configured_instance() + { + var options = new SqlServerSecretsOptions { ConnectionString = "Server=x;Database=y;" }; + + options.Validate(); + } + + [Fact] + public void CreateSchemaDdl_quotes_the_schema_name_everywhere_it_appears() + { + string ddl = SqlServerSecretsSchema.CreateSchemaDdl("zbsecrets"); + + Assert.Contains("[zbsecrets].[secret]", ddl, StringComparison.Ordinal); + Assert.Contains("[zbsecrets].[schema_version]", ddl, StringComparison.Ordinal); + // Every DDL statement is guarded, so re-running the migration on a provisioned database is + // a no-op rather than an error. + Assert.Contains("IF NOT EXISTS", ddl, StringComparison.Ordinal); + Assert.Contains("IS NULL", ddl, StringComparison.Ordinal); + } +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.csproj b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.csproj new file mode 100644 index 0000000..7b7e234 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.csproj @@ -0,0 +1,27 @@ + + + + false + + + + + + + + + + + + + + + + + + + + + + + diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Tests/Fakes/RecordingCacheInvalidator.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Tests/Fakes/RecordingCacheInvalidator.cs new file mode 100644 index 0000000..8ed7622 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Tests/Fakes/RecordingCacheInvalidator.cs @@ -0,0 +1,15 @@ +using ZB.MOM.WW.Secrets.Abstractions; + +namespace ZB.MOM.WW.Secrets.Tests.Fakes; + +/// Records every eviction so tests can assert the resolver cache was cleared. +public sealed class RecordingCacheInvalidator : ISecretCacheInvalidator +{ + private readonly List _invalidated = []; + + /// The names evicted, in call order. + public IReadOnlyList Invalidated => _invalidated; + + /// + public void Invalidate(SecretName name) => _invalidated.Add(name); +} diff --git a/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Tests/Replication/SecretReplicationReconcilerTests.cs b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Tests/Replication/SecretReplicationReconcilerTests.cs new file mode 100644 index 0000000..23da668 --- /dev/null +++ b/ZB.MOM.WW.Secrets/tests/ZB.MOM.WW.Secrets.Tests/Replication/SecretReplicationReconcilerTests.cs @@ -0,0 +1,265 @@ +using ZB.MOM.WW.Secrets.Abstractions; +using ZB.MOM.WW.Secrets.Replication; +using ZB.MOM.WW.Secrets.Sqlite; +using ZB.MOM.WW.Secrets.Tests.Fakes; + +namespace ZB.MOM.WW.Secrets.Tests.Replication; + +/// +/// Convergence rules for the transport-agnostic reconciler. These are the cases both replication +/// packages inherit, so they are pinned here once against a real SQLite store rather than a fake — +/// a reconciler that agrees with a mock but not with the store would be worthless. +/// +public sealed class SecretReplicationReconcilerTests : IDisposable +{ + private readonly string _dbPath = Path.Combine(Path.GetTempPath(), $"zb-recon-{Guid.NewGuid():N}.db"); + private readonly SecretsSqliteConnectionFactory _factory; + private readonly SqliteSecretStore _store; + + public SecretReplicationReconcilerTests() + { + _factory = new SecretsSqliteConnectionFactory(_dbPath); + new SqliteSecretsStoreMigrator(_factory).MigrateAsync(CancellationToken.None).GetAwaiter().GetResult(); + _store = new SqliteSecretStore(_factory); + } + + private static readonly DateTimeOffset T0 = new(2026, 7, 18, 12, 0, 0, TimeSpan.Zero); + + private static SecretManifestEntry Entry(string name, long revision, DateTimeOffset updated, bool deleted = false) => + new() { Name = new SecretName(name), Revision = revision, UpdatedUtc = updated, IsDeleted = deleted }; + + private static StoredSecret Row(string name, long revision, DateTimeOffset updated, bool deleted = false) => new() + { + Name = new SecretName(name), + ContentType = SecretContentType.Text, + Ciphertext = [1, 2, 3], + Nonce = [4], + Tag = [5], + WrappedDek = [6], + WrapNonce = [7], + WrapTag = [8], + KekId = "sha256:test", + Revision = revision, + IsDeleted = deleted, + DeletedUtc = deleted ? updated : null, + CreatedUtc = T0, + UpdatedUtc = updated, + }; + + [Fact] + public void ComputePullSet_pulls_names_absent_locally() + { + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [], + remote: [Entry("db/password", 0, T0)]); + + Assert.Equal(["db/password"], pull.Select(n => n.Value)); + } + + [Fact] + public void ComputePullSet_pulls_a_newer_remote_row() + { + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [Entry("api/key", 1, T0)], + remote: [Entry("api/key", 2, T0.AddSeconds(1))]); + + Assert.Equal(["api/key"], pull.Select(n => n.Value)); + } + + [Fact] + public void ComputePullSet_breaks_a_timestamp_tie_on_revision() + { + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [Entry("api/key", 1, T0)], + remote: [Entry("api/key", 2, T0)]); + + Assert.Equal(["api/key"], pull.Select(n => n.Value)); + } + + [Fact] + public void ComputePullSet_ignores_an_identical_row_so_redelivery_is_free() + { + // At-least-once transports redeliver. A tie must not be "newer", or two nodes would + // ping-pong the same row forever. + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [Entry("api/key", 2, T0)], + remote: [Entry("api/key", 2, T0)]); + + Assert.Empty(pull); + } + + [Fact] + public void ComputePullSet_ignores_a_peer_that_is_behind() + { + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [Entry("api/key", 5, T0.AddMinutes(1))], + remote: [Entry("api/key", 2, T0)]); + + Assert.Empty(pull); + } + + [Fact] + public void ComputePullSet_pulls_a_tombstone_so_deletes_propagate() + { + // A delete is a newer row, not an absence — otherwise a deleted secret would be resurrected + // by the next anti-entropy sweep from a node that still has it. + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [Entry("api/key", 1, T0)], + remote: [Entry("api/key", 2, T0.AddSeconds(1), deleted: true)]); + + Assert.Equal(["api/key"], pull.Select(n => n.Value)); + } + + [Fact] + public void ComputePullSet_does_not_pull_a_name_only_the_local_side_has() + { + // The local side only ever pulls. Pushing local-only names is the peer's own reconcile. + IReadOnlyList pull = SecretReplicationReconciler.ComputePullSet( + localManifest: [Entry("local/only", 0, T0)], + remote: []); + + Assert.Empty(pull); + } + + [Fact] + public async Task ReconcileAsync_applies_only_the_rows_the_peer_holds_newer() + { + await _store.UpsertAsync(Row("shared/a", 0, T0), CancellationToken.None); + + StoredSecret localA = (await _store.GetAsync(new SecretName("shared/a"), CancellationToken.None))!; + + var reconciler = new SecretReplicationReconciler(_store); + List? requested = null; + + int applied = await reconciler.ReconcileAsync( + remoteManifest: + [ + // Behind the local copy — must not be fetched. + Entry("shared/a", localA.Revision, localA.UpdatedUtc.AddMinutes(-1)), + // Absent locally — must be fetched. + Entry("shared/b", 3, T0.AddHours(1)), + ], + fetchAsync: (names, _) => + { + requested = [.. names]; + return Task.FromResult>([Row("shared/b", 3, T0.AddHours(1))]); + }, + CancellationToken.None); + + Assert.Equal(1, applied); + Assert.Equal(["shared/b"], requested!.Select(n => n.Value)); + + StoredSecret? b = await _store.GetAsync(new SecretName("shared/b"), CancellationToken.None); + Assert.NotNull(b); + // Applied VERBATIM — the peer's revision, not a locally bumped one. + Assert.Equal(3, b!.Revision); + } + + [Fact] + public async Task ReconcileAsync_skips_the_fetch_entirely_when_nothing_is_newer() + { + var reconciler = new SecretReplicationReconciler(_store); + bool fetched = false; + + int applied = await reconciler.ReconcileAsync( + remoteManifest: [], + fetchAsync: (_, _) => + { + fetched = true; + return Task.FromResult>([]); + }, + CancellationToken.None); + + Assert.Equal(0, applied); + Assert.False(fetched); + } + + [Fact] + public async Task ApplyAsync_evicts_the_resolver_cache_for_every_row() + { + var invalidator = new RecordingCacheInvalidator(); + var reconciler = new SecretReplicationReconciler(_store, invalidator); + + await reconciler.ApplyAsync([Row("a/one", 1, T0), Row("b/two", 1, T0)], CancellationToken.None); + + Assert.Equal(["a/one", "b/two"], invalidator.Invalidated.Select(n => n.Value)); + } + + [Fact] + public async Task ApplyAsync_evicts_even_when_the_store_rejected_the_row_as_stale() + { + // Serving a stale plaintext for the rest of the TTL is worse than a redundant re-read. + await _store.UpsertAsync(Row("api/key", 0, T0), CancellationToken.None); + + var invalidator = new RecordingCacheInvalidator(); + var reconciler = new SecretReplicationReconciler(_store, invalidator); + + await reconciler.ApplyAsync([Row("api/key", 0, T0.AddYears(-1))], CancellationToken.None); + + Assert.Equal(["api/key"], invalidator.Invalidated.Select(n => n.Value)); + } + + [Fact] + public async Task ApplyAsync_skips_a_poison_row_and_keeps_applying_the_rest() + { + // A batch is not a transaction. Without per-row isolation, one row a peer cannot supply + // cleanly abandons every row after it — and since the pull set is recomputed identically + // each sweep, in the same manifest order, the same row poisons the same batch forever. + var failures = new List(); + var reconciler = new SecretReplicationReconciler( + _store, cacheInvalidator: null, onRowFailed: (name, _) => failures.Add(name)); + + // A null ciphertext cannot be persisted (NOT NULL column) — stands in for any bad row. + StoredSecret poison = Row("bad/row", 1, T0) with { Ciphertext = null! }; + + int applied = await reconciler.ApplyAsync( + [Row("good/first", 1, T0), poison, Row("good/second", 1, T0)], + CancellationToken.None); + + Assert.Equal(2, applied); + Assert.Equal(["bad/row"], failures.Select(n => n.Value)); + Assert.NotNull(await _store.GetAsync(new SecretName("good/first"), CancellationToken.None)); + Assert.NotNull(await _store.GetAsync(new SecretName("good/second"), CancellationToken.None)); + } + + [Fact] + public async Task ReconcileAsync_fetches_in_bounded_batches() + { + // An unbounded fetch breaks on real backends: SQL Server caps a command at 2100 parameters, + // so a cold-starting node pulling a large hub's entire inventory would fail identically + // every interval and never receive a single row. + const int total = SecretReplicationReconciler.FetchBatchSize + 20; + + List remote = [.. Enumerable.Range(0, total) + .Select(i => Entry($"bulk/{i}", 1, T0))]; + + var batchSizes = new List(); + var reconciler = new SecretReplicationReconciler(_store); + + int applied = await reconciler.ReconcileAsync( + remote, + fetchAsync: (names, _) => + { + batchSizes.Add(names.Count); + return Task.FromResult>( + [.. names.Select(n => Row(n.Value, 1, T0))]); + }, + CancellationToken.None); + + Assert.Equal(total, applied); + Assert.Equal(2, batchSizes.Count); + Assert.All(batchSizes, size => Assert.True(size <= SecretReplicationReconciler.FetchBatchSize)); + } + + public void Dispose() + { + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + foreach (string path in new[] { _dbPath, _dbPath + "-wal", _dbPath + "-shm" }) + { + if (File.Exists(path)) + { + File.Delete(path); + } + } + } +} diff --git a/components/secrets/GAPS.md b/components/secrets/GAPS.md index bb58f67..69e5ae9 100644 --- a/components/secrets/GAPS.md +++ b/components/secrets/GAPS.md @@ -23,16 +23,63 @@ construction). fix until bumped. - **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take `Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`. -- **G-7 clustered replication is designed + planned, not built** — plan at - `docs/plans/2026-07-17-secrets-g7-*` (resolved to build the shared SQL-Server `ISecretStore`; - Akka replicator deferred to phase 2). No `ISecretStore` SQL-Server implementation exists yet. -- Library suite: **97 passing, 1 skipped**, green 2026-07-18. +- **G-7 clustered replication is BUILT (2026-07-18, lib `0.2.0`) — both options, not just one.** + The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build + covers **both**, as two new packages: + - **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** — `SqlServerSecretStore : ISecretStore` (all 7 + members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and *two* + topologies: `AddZbSecretsSqlServerStore` (one shared store, Option A) and + `AddZbSecretsSqlServerReplication` (local store + hub, with a bidirectional sweep). + - **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** — peer-to-peer over distributed pub/sub + (Option B): live broadcast + periodic manifest anti-entropy, an explicit + `SerializerWithStringManifest`, ciphertext-only wire DTOs. + - **Core gained the pieces both need:** `ISecretsStoreMigrator` (so the hosted service + CLI are + store-agnostic), `SecretLastWriterWins` (one definition of the LWW tie-break, shared by every + store and reconciler), `SecretReplicationReconciler`, and `ReplicatingSecretStore`. + - **The `ISecretReplicator` seam had no caller** — nothing invoked `PublishAsync`, so local writes + would never have propagated. `ReplicatingSecretStore` decorates the store, which fixes it for + the UI, the CLI, and any future write path at once. + - Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`); + `AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace. +- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests + against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural + divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster** + (real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions, + late-joiner catch-up, and malformed-peer-input handling. ### Open -- ⬜ Bump the four apps `0.1.2` → `0.1.3` to pick up KEK rotation + the SQLitePCLRaw security fix. -- ⬜ Execute the G-7 plan (shared SQL-Server `ISecretStore`). +- ⬜ Bump the four apps `0.1.2` → `0.1.3`/`0.2.0` to pick up KEK rotation + the SQLitePCLRaw security fix. +- ⬜ Publish `0.2.0` (5 packages) to the Gitea feed — packed and scanned clean, **not yet pushed**. +- ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed). - ⬜ Per-app live wonder gates for the G-8 rotation path. +- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the + drift symptom, not the forwarding filter itself (noted in the test). + +### Post-build code review (2026-07-18) — 6 defects found and fixed + +An independent review caught real bugs that a green suite had not: + +- **CRITICAL — both replication modes were dead on arrival.** The decorators resolve the *undecorated* + local store by concrete type, but core registered `SqliteSecretStore` only behind `ISecretStore`. + Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with + `No service for type 'SqliteSecretStore'`), fixed, and covered by new DI-resolution tests. +- **Unbounded fetch.** A cold-starting node pulls the peer's entire inventory in one call; past + ~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the + node never converges. Now chunked at 500. +- **Poison row aborted the batch.** One unapplyable row abandoned every row after it, and since the + pull set recomputes in the same order each sweep, it would poison the same batch forever. Now + isolated per row and logged. +- **`Enum.Parse` on peer input.** An overflowing numeric string throws `OverflowException`, which the + actor's `ArgumentException` filter did not catch → actor restart loop on redelivery; and any + in-range number was accepted as an undefined enum value. Now `TryParse` + `IsDefined`. +- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*). +- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic. + +One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's +`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and +three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as +written. ## Execution status (2026-07-16) @@ -146,15 +193,26 @@ mounted key)** for clustered pairs (hard requirement — same KEK on every node) Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI) and map each app's admin role onto `secrets:manage` / `secrets:reveal`. -### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17) -The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A — -a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared -Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure -modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred -phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the -`ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable -plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md) -+ [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.** +### G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib `0.2.0`) +The SPEC's "shared SQL-Server store **vs** Akka replicator" fork was designed as *either/or* with +Option A recommended and Option B deferred. **Both were built** — the fork is now a deployment +choice rather than a library one, and the two packages are independent (an app takes whichever +matches its availability requirement, or neither). + +| Package | Topologies | When | +|---|---|---| +| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | shared store (Option A) · local + hub | every node can reach a shared DB | +| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | peer-to-peer over the cluster (Option B) | a node must resolve while **partitioned** | + +Three constraints hold across all of them, and each is documented at the API and in the runbook: +**same KEK on every node** (a mismatch fails closed and looks like corruption); **ciphertext only +crosses trust boundaries**; **re-wraps do not replicate**, so `rewrap-all` runs once per independent +store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no +merge — stated plainly in the README rather than left for an operator to discover. + +Design + plan: [`…-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md) ++ [`…-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). +Operator runbook: [`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md). ### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`) The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in diff --git a/docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md b/docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md index 42a31ba..3d95566 100644 --- a/docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md +++ b/docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md @@ -1,5 +1,12 @@ # G-7 — Clustered secret replication: design & fork resolution +> **✅ SUPERSEDED BY THE BUILD, 2026-07-18.** This document resolved the fork to Option A and +> deferred Option B. On the user's instruction **both were built**, as +> `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared-store *and* hub modes) and +> `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`. The fork analysis below is still the right way to +> *choose* between them — it is now a deployment decision, not a build one. The "Decision" section's +> YAGNI reasoning for deferring Option B no longer applies. + > **Status:** design-only (G-7 in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md)). > Companion executable plan: [`2026-07-17-secrets-g7-sqlserver-store.md`](2026-07-17-secrets-g7-sqlserver-store.md). > Prerequisite G-2…G-6 have landed for both clustered apps (ScadaBridge, OtOpcUa); G-8 (KEK diff --git a/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md b/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md index 7c8cebc..a964062 100644 --- a/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md +++ b/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md @@ -1,5 +1,28 @@ # G-7 (Option A) — Shared SQL-Server `ISecretStore` Implementation Plan +> **✅ EXECUTED 2026-07-18 — and the scope was widened on the user's instruction.** This plan covered +> Option A only, with Option B (Akka) deferred. The user directed that **both** be built, packaged as +> `ZB.MOM.WW.Secrets.Replicator.SqlServer` and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` — so the +> "Deferred phase-2" section below is **also built**, and the naming here (`SqlServer/` inside the +> core package; `ZB.MOM.WW.Secrets.Akka`) is superseded by two standalone packages. +> +> Other deltas from what was planned, all deliberate: +> - **Two SQL-Server topologies, not one.** Shared-store (as planned) *plus* a local-store-with-hub +> mode, so the package's `Replicator` name is honest and partition tolerance is available without +> Akka. +> - **`SecretLastWriterWins` extracted to Abstractions.** The plan flagged Task 2 high-risk because +> the LWW semantics "MUST match SQLite exactly"; a shared predicate makes that structural instead +> of a review obligation. +> - **`ReplicatingSecretStore` added.** Discovered during the build that nothing ever called +> `ISecretReplicator.PublishAsync` — the seam was inert. A store decorator fixes every write path. +> - **`ISecretReplicationHub` added**, so the hub sweep is testable offline rather than requiring a +> live SQL Server for every convergence case. +> - **Anti-entropy is bidirectional.** Pull-only would strand a write made while a peer was down: +> the peer never learns the name exists, so it can never ask for it. +> +> Verified: 164 pass / 1 skip / 0 warnings, including 15 live tests against a real SQL Server 2022 +> and a 9-test in-process 2-node Akka cluster. See [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md) §G-7. + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or > subagent-driven-development) to implement this plan task-by-task. diff --git a/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md.tasks.json b/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md.tasks.json index e77a427..86be216 100644 --- a/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md.tasks.json +++ b/docs/plans/2026-07-17-secrets-g7-sqlserver-store.md.tasks.json @@ -1,11 +1,39 @@ { "planPath": "docs/plans/2026-07-17-secrets-g7-sqlserver-store.md", + "status": "executed 2026-07-18 with widened scope (both options built as .Replicator.* packages)", "tasks": [ - {"id": 0, "subject": "Task 1: SQL-Server schema + migrator", "status": "pending"}, - {"id": 1, "subject": "Task 2: SqlServerSecretStore : ISecretStore", "status": "pending", "blockedBy": [0]}, - {"id": 2, "subject": "Task 3: DI store-selection wiring", "status": "pending", "blockedBy": [1]}, - {"id": 3, "subject": "Task 4: Docs + adoption notes", "status": "pending", "blockedBy": [1]} + { + "id": 0, + "subject": "Task 1: SQL-Server schema + migrator", + "status": "completed" + }, + { + "id": 1, + "subject": "Task 2: SqlServerSecretStore : ISecretStore", + "status": "completed", + "blockedBy": [ + 0 + ] + }, + { + "id": 2, + "subject": "Task 3: DI store-selection wiring", + "status": "completed", + "blockedBy": [ + 1 + ], + "note": "Delivered as AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication in a standalone package rather than a SecretsOptions.Store enum in core." + }, + { + "id": 3, + "subject": "Task 4: Docs + adoption notes", + "status": "completed", + "blockedBy": [ + 1 + ], + "note": "docs/operations/clustered-secrets.md covers all three topologies." + } ], - "deferredPhase2": "Option B — ZB.MOM.WW.Secrets.Akka replicator (build only on a stated partition-tolerance requirement); see plan doc.", - "lastUpdated": "2026-07-17" + "deferredPhase2": "BUILT 2026-07-18 as ZB.MOM.WW.Secrets.Replicator.AkkaDotNet (no longer deferred).", + "lastUpdated": "2026-07-18" }