feat(secrets): cluster replication via SQL Server and Akka.NET (G-7, 0.2.0)
Secrets were per-node SQLite, so a secret written on one node was invisible to the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka replicator" fork to build only the former; both are built here so the choice is a deployment decision (availability vs partition tolerance) rather than a library limitation. Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over distributed pub/sub). Core gains ISecretsStoreMigrator, one shared SecretLastWriterWins predicate so no two stores can disagree on a tie, the transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was inert and local writes would not have propagated at all. Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural divergence between the stores fails) and a 9-test in-process 2-node Akka cluster over real remoting. A post-build review caught six defects, all fixed and now covered: both replication modes could not resolve from the container (no test had built one), an unbounded fetch that broke past SQL Server's 2100-parameter cap, a poison row that aborted the rest of its batch forever, Enum.Parse on peer input that could restart the actor in a loop, null crypto blobs crossing the trust boundary, and a silently dropped pull-read failure. Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -148,7 +148,7 @@ each project's **code-verified current state**, and the **gaps** between. See
|
||||
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
|
||||
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
|
||||
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
|
||||
| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication DESIGNED + PLANNED** — fork resolved to **build the shared SQL-Server `ISecretStore`** (Akka replicator deferred phase-2), executable plan `docs/plans/2026-07-17-secrets-g7-*`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. G-7 has no `ISecretStore` SQL implementation yet | Shared `ZB.MOM.WW.Secrets` lib (3 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
|
||||
| Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 164 pass / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. All 4 apps still pinned at `0.1.2` → bump to consume KEK rotation + the security fix. **`0.2.0` is packed + vulnerability-scanned clean but NOT yet published to the feed**, and no app has adopted a clustered topology yet | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
|
||||
| LocalDb (embedded cache + 2-node sync) | Built (3 pkgs `0.1.0`, **published to the Gitea feed** 2026-07-18, restore-verified from a scratch consumer; no app adoption yet) | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |
|
||||
|
||||
The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
|
||||
|
||||
@@ -24,7 +24,7 @@ it produces.
|
||||
| [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | **Built shared library** — readiness / liveness / active-node probes |
|
||||
| [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | **Built shared library** — OTel metrics / traces + Serilog |
|
||||
| [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | **Built shared library** — options validation + startup preflight |
|
||||
| [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | **Built shared library** — AES-256-GCM secret store + `${secret:}` resolution + CLI |
|
||||
| [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | **Built shared library** — AES-256-GCM secret store + `${secret:}` resolution + CLI + SQL-Server / Akka cluster replication |
|
||||
| [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | **Built shared library** — Galaxy object-hierarchy SQL browse + gRPC service |
|
||||
| [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | **Built shared library** — embedded SQLite cache + 2-node gRPC replication |
|
||||
|
||||
@@ -70,7 +70,7 @@ Status verified 2026-07-18 against the feed + consumer branches (see [Roadmap](#
|
||||
| Config + validation | `0.1.0` | all four | [`components/configuration/`](components/configuration/) |
|
||||
| Observability (metrics / traces / logs) | `0.1.0` | all four | [`components/observability/`](components/observability/) |
|
||||
| Health (readiness / liveness / active-node) | `0.1.0` | all four | [`components/health/`](components/health/) |
|
||||
| Secrets (encrypted store + `${secret:}`) | `0.1.0`–`0.1.3` | all four @ `0.1.2` (`0.1.3` published, not yet consumed) | [`components/secrets/`](components/secrets/) |
|
||||
| Secrets (encrypted store + `${secret:}`) | `0.1.0`–`0.1.3` (`0.2.0` packed, unpublished) | all four @ `0.1.2` (`0.1.3` published, not yet consumed) | [`components/secrets/`](components/secrets/) |
|
||||
| Galaxy Repository (SQL browse + gRPC) | `0.1.0`, `0.2.0` | HistorianGateway + mxaccessgw | _(design in `docs/plans/`)_ |
|
||||
| LocalDb (embedded cache + 2-node sync) | `0.1.0` | none yet | _(design in `docs/plans/`)_ |
|
||||
|
||||
@@ -172,13 +172,18 @@ ZB_LDAP_IT=1 dotnet test # requires a reachable GLAuth (e.g. a sister repo's i
|
||||
pending follow-on — also stale).
|
||||
- ✅ Secrets — `ZB.MOM.WW.Secrets` (+ `.Abstractions`, `.Ui`) published through **`0.1.3`** (KEK rotation,
|
||||
2026-07-18); all four apps still pinned at `0.1.2`.
|
||||
- ✅ Secrets G-7 clustered replication — **built 2026-07-18 at `0.2.0`**, both fork options:
|
||||
`ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared store *and* local-plus-hub) and
|
||||
`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (peer-to-peer, partition-tolerant). Verified against a
|
||||
real SQL Server 2022 and an in-process 2-node Akka cluster. Packed, **not yet published**.
|
||||
- ✅ LocalDb — `ZB.MOM.WW.LocalDb` (+ `.Contracts`, `.Replication`) built at `0.1.0` and **published to the
|
||||
feed 2026-07-18** (restore-verified from a scratch consumer).
|
||||
|
||||
**Open**
|
||||
|
||||
- ⬜ Execute the G-7 clustered-replication plan (shared SQL-Server `ISecretStore`) — [`components/secrets/GAPS.md`](components/secrets/GAPS.md).
|
||||
- ⬜ Bump the four Secrets consumers `0.1.2` → `0.1.3` (KEK rotation + the same security fix).
|
||||
- ⬜ Publish Secrets `0.2.0` (5 packages) to the feed, then adopt a clustered topology in
|
||||
ScadaBridge / OtOpcUa — config + a shared KEK, no code change. [`components/secrets/GAPS.md`](components/secrets/GAPS.md).
|
||||
- ⬜ Bump the four Secrets consumers `0.1.2` → `0.1.3`/`0.2.0` (KEK rotation + the same security fix).
|
||||
- ⬜ Clear the pre-existing `AngleSharp` NU1902 build break in ScadaBridge `CentralUI.Tests`
|
||||
and HistorianGateway `Tests` — both fail full-solution builds under `TreatWarningsAsErrors`.
|
||||
- ⬜ Adopt `ZB.MOM.WW.LocalDb` in an app — no consumer references it yet.
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
<Nullable>enable</Nullable>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<LangVersion>latest</LangVersion>
|
||||
<Version>0.1.3</Version>
|
||||
<Version>0.2.0</Version>
|
||||
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
|
||||
<PackageReadmeFile>README.md</PackageReadmeFile>
|
||||
</PropertyGroup>
|
||||
|
||||
@@ -16,8 +16,25 @@
|
||||
which carries advisory GHSA-2m69-gcr7-jv3q (NU1903). Pin the patched 2.1.12 native lib. -->
|
||||
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
|
||||
|
||||
<!-- Shared SQL-Server secret store (ZB.MOM.WW.Secrets.Replication.SqlServer).
|
||||
6.0.2 matches the version ScadaBridge and ZB.MOM.WW.GalaxyRepository already resolve, so a
|
||||
consumer taking both gets one SqlClient rather than a downgrade warning. -->
|
||||
<PackageVersion Include="Microsoft.Data.SqlClient" Version="6.0.2" />
|
||||
|
||||
<!-- Akka cluster replication (ZB.MOM.WW.Secrets.Replication.Akka).
|
||||
1.5.62 matches ScadaBridge's pinned Akka.NET line — the clustered consumer must not be
|
||||
forced into an Akka version bump just to adopt secret replication. -->
|
||||
<PackageVersion Include="Akka" Version="1.5.62" />
|
||||
<PackageVersion Include="Akka.Cluster" Version="1.5.62" />
|
||||
<PackageVersion Include="Akka.Cluster.Tools" Version="1.5.62" />
|
||||
<PackageVersion Include="Akka.Remote" Version="1.5.62" />
|
||||
<PackageVersion Include="Akka.TestKit.Xunit2" Version="1.5.62" />
|
||||
|
||||
<!-- Cryptography -->
|
||||
<PackageVersion Include="System.Security.Cryptography.ProtectedData" Version="9.0.0" />
|
||||
<!-- 9.0.4, not 9.0.0: Microsoft.Data.SqlClient 6.0.2 pulls System.Configuration.ConfigurationManager
|
||||
9.0.4, which requires ProtectedData >= 9.0.4. With transitive pinning on, the lower central
|
||||
version is a hard NU1109 downgrade error rather than a silent resolve. -->
|
||||
<PackageVersion Include="System.Security.Cryptography.ProtectedData" Version="9.0.4" />
|
||||
|
||||
<!-- Extensions -->
|
||||
<PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.7" />
|
||||
@@ -28,6 +45,8 @@
|
||||
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.Extensions.Hosting" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.Extensions.Logging" Version="10.0.7" />
|
||||
|
||||
<!-- ASP.NET Core Authentication / Authorization -->
|
||||
<PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.7" />
|
||||
|
||||
@@ -11,6 +11,8 @@ plaintext back on demand — from application code, from configuration, or from
|
||||
| `ZB.MOM.WW.Secrets.Abstractions` | Contracts only (`ISecretStore`, `ISecretResolver`, `IMasterKeyProvider`, `ISecretCipher`, `ISecretReplicator`, `ISecretCacheInvalidator`, `ISecretActorAccessor`), value types, exceptions. Dependency-light. |
|
||||
| `ZB.MOM.WW.Secrets` | Implementation: AES-256-GCM envelope cipher, env/file/DPAPI master-key providers, SQLite store, TTL resolver (audited), `${secret:}` config expander, `AddZbSecrets`. |
|
||||
| `ZB.MOM.WW.Secrets.Ui` | Blazor RCL on `ZB.MOM.WW.Theme`: list / add / rotate / delete + policy-gated, audited reveal. |
|
||||
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | Cluster-wide secrets via SQL Server — as one shared store, or as a hub each node syncs against. |
|
||||
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | Peer-to-peer replication over an Akka.NET cluster — partition-tolerant, no shared database. |
|
||||
|
||||
A `secret` CLI (`set` / `get` / `list` / `rm` / `rotate` / `rewrap-all`) ships in the repo (not packed).
|
||||
|
||||
@@ -78,8 +80,73 @@ SQL-Server store; once per node for per-node SQLite on a shared KEK). See the op
|
||||
|
||||
## Clustered deployments
|
||||
|
||||
Secrets storage is local SQLite by default. The schema already carries the
|
||||
`revision` / `updated_utc` / tombstone columns and an `ISecretReplicator` seam for
|
||||
cluster replication, but the Akka replicator (`ZB.MOM.WW.Secrets.Akka`) is a deferred
|
||||
follow-on. **When replicating across a node pair, every node must resolve the _same_ master
|
||||
KEK** (e.g. the same mounted key file) — a row wrapped by an unknown KEK fails closed.
|
||||
Storage is local SQLite by default, which is correct for a single-process app and wrong the
|
||||
moment a secret written on one node has to be readable on another. Three topologies are
|
||||
supported; pick the least complicated one that meets the requirement.
|
||||
|
||||
> **Every topology requires the same master KEK on every node.** The database and the wire carry
|
||||
> ciphertext only, so a node with a different key cannot decrypt what another wrote — it fails
|
||||
> closed on a `kek_id` mismatch, which looks like corruption but is a deployment mistake. Mount the
|
||||
> same key file, or supply the same key environment variable, everywhere.
|
||||
|
||||
### 1. Shared SQL-Server store — *the default choice*
|
||||
|
||||
Every node points at one database. There is exactly one copy of each row, so there is no
|
||||
replication, no reconciliation, no split-brain, and no clock skew to reason about.
|
||||
|
||||
```csharp
|
||||
builder.Services.AddZbSecretsSqlServerStore(builder.Configuration);
|
||||
```
|
||||
|
||||
```jsonc
|
||||
"Secrets": {
|
||||
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
||||
"SqlServer": {
|
||||
"ConnectionString": "Server=sql01;Database=ZbSecrets;...",
|
||||
"SchemaName": "zbsecrets"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Trade-off:** availability is coupled to that database. A node cut off from it falls back only to
|
||||
the resolver's short in-memory cache. Rotation (`rewrap-all`) runs **once**, against the one store.
|
||||
|
||||
### 2. SQL-Server hub replication
|
||||
|
||||
Each node keeps its local store and stays converged with a shared hub: writes publish immediately,
|
||||
and a background sweep reconciles **both directions** on an interval (default 30s).
|
||||
|
||||
```csharp
|
||||
builder.Services.AddZbSecretsSqlServerReplication(builder.Configuration);
|
||||
```
|
||||
|
||||
Choose this over (1) when a node must keep **resolving** secrets while cut off from the hub — reads
|
||||
never leave the node, so a hub outage degrades propagation rather than availability.
|
||||
|
||||
### 3. Akka cluster peer-to-peer
|
||||
|
||||
No shared database at all. Nodes broadcast writes over distributed pub/sub and repair divergence
|
||||
with a periodic manifest exchange.
|
||||
|
||||
```csharp
|
||||
builder.Services.AddZbSecretsAkkaReplication(builder.Configuration);
|
||||
// Recommended — pins the serializer that carries ciphertext:
|
||||
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
|
||||
```
|
||||
|
||||
Requires the application to register its `ActorSystem` in DI. The local store is SQLite. This is
|
||||
the right answer for air-gapped or intermittently-connected sites; it is the wrong answer if every
|
||||
node can always reach a shared database, because you pay real distributed-systems behaviour for it.
|
||||
|
||||
### What replication costs you (2 and 3)
|
||||
|
||||
- **Eventual consistency.** Rows converge within the sweep interval, not instantly.
|
||||
- **Last-writer-wins.** Two nodes writing the same secret concurrently: the later `updated_utc`
|
||||
wins and the other write is discarded. There is no merge and no conflict report.
|
||||
- **Publishing is best-effort.** A failed broadcast is logged, not thrown — the local write is
|
||||
already durable and anti-entropy repairs it. Writes never fail because a peer was unreachable.
|
||||
- **KEK rotation does not replicate.** A re-wrap deliberately leaves the revision untouched so it
|
||||
stays invisible to last-writer-wins. Run `rewrap-all` against **each** independent store.
|
||||
|
||||
See [`docs/operations/clustered-secrets.md`](docs/operations/clustered-secrets.md) for operator
|
||||
setup, and [`docs/operations/kek-rotation.md`](docs/operations/kek-rotation.md) for rotation.
|
||||
|
||||
@@ -2,11 +2,15 @@
|
||||
<Folder Name="/src/">
|
||||
<Project Path="src/ZB.MOM.WW.Secrets.Abstractions/ZB.MOM.WW.Secrets.Abstractions.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Secrets/ZB.MOM.WW.Secrets.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Secrets.Replicator.SqlServer/ZB.MOM.WW.Secrets.Replicator.SqlServer.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Secrets.Ui/ZB.MOM.WW.Secrets.Ui.csproj" />
|
||||
<Project Path="src/ZB.MOM.WW.Secrets.Cli/ZB.MOM.WW.Secrets.Cli.csproj" />
|
||||
</Folder>
|
||||
<Folder Name="/tests/">
|
||||
<Project Path="tests/ZB.MOM.WW.Secrets.Tests/ZB.MOM.WW.Secrets.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests/ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests/ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.Secrets.Ui.Tests/ZB.MOM.WW.Secrets.Ui.Tests.csproj" />
|
||||
</Folder>
|
||||
</Solution>
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
# Operating clustered secrets
|
||||
|
||||
How to run `ZB.MOM.WW.Secrets` across more than one node. Covers the shared SQL-Server store, the
|
||||
SQL-Server hub, and Akka peer-to-peer replication.
|
||||
|
||||
Companion: [`kek-rotation.md`](kek-rotation.md).
|
||||
|
||||
---
|
||||
|
||||
## 0. The one rule that breaks everything if you get it wrong
|
||||
|
||||
**Every node must resolve the same master KEK.**
|
||||
|
||||
Rows are ciphertext everywhere — in the database, in a backup, on the wire. Only the KEK decrypts
|
||||
them, and it never leaves the node. So if node B has a different master key than node A:
|
||||
|
||||
- replication still "works" — rows arrive and are stored;
|
||||
- every resolve on B of a secret written on A **fails closed** with a `kek_id` mismatch.
|
||||
|
||||
The symptom looks like data corruption and is not. Before enabling any topology below, confirm the
|
||||
same key material is present on every node:
|
||||
|
||||
```bash
|
||||
# On each node — compare the digests, never print the key itself.
|
||||
sha256sum /run/secrets/zb-kek
|
||||
# or, for the environment source:
|
||||
printf '%s' "$ZB_SECRETS_MASTER_KEY" | sha256sum
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Shared SQL-Server store
|
||||
|
||||
One database, one copy of each row. Prefer this unless a node must keep resolving secrets while
|
||||
partitioned from the database.
|
||||
|
||||
### Provision
|
||||
|
||||
The schema is created automatically by the first node that starts (idempotent, and safe if several
|
||||
start at once). You only need a database and a login:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE ZbSecrets;
|
||||
GO
|
||||
USE ZbSecrets;
|
||||
CREATE LOGIN zbsecrets WITH PASSWORD = '<strong password>';
|
||||
CREATE USER zbsecrets FOR LOGIN zbsecrets;
|
||||
GO
|
||||
-- The app creates and owns the zbsecrets schema; it needs permission to do so.
|
||||
GRANT CREATE SCHEMA, CREATE TABLE TO zbsecrets;
|
||||
ALTER ROLE db_owner ADD MEMBER zbsecrets; -- or grant on the schema once it exists
|
||||
GO
|
||||
```
|
||||
|
||||
To pre-create the schema instead (so the app can run with narrower rights), run the DDL from
|
||||
`SqlServerSecretsSchema.CreateSchemaDdl("zbsecrets")` and then grant only
|
||||
`SELECT, INSERT, UPDATE` on `[zbsecrets].[secret]` and `[zbsecrets].[schema_version]`.
|
||||
|
||||
> `DELETE` is deliberately **not** required. Deletes are tombstones — an `UPDATE` — so a compromised
|
||||
> app credential cannot destroy secret history.
|
||||
|
||||
### Configure
|
||||
|
||||
```jsonc
|
||||
"Secrets": {
|
||||
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
||||
"SqlServer": {
|
||||
"ConnectionString": "Server=sql01;Database=ZbSecrets;User Id=zbsecrets;Password=${secret:sql/zbsecrets-password};TrustServerCertificate=False",
|
||||
"SchemaName": "zbsecrets",
|
||||
"CommandTimeout": "00:00:30"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The connection string is itself a credential — deliver it by `${secret:}` reference, an orchestrator
|
||||
secret, or an environment variable. Never commit it.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
secret set smoke/test hello --description "delete me" # on node A
|
||||
secret get smoke/test # on node B — expect: hello
|
||||
secret rm smoke/test
|
||||
```
|
||||
|
||||
### Rotation
|
||||
|
||||
Run `rewrap-all` **once**, against the shared store, with resolve traffic quiesced.
|
||||
|
||||
---
|
||||
|
||||
## 2. SQL-Server hub replication
|
||||
|
||||
Each node keeps its local SQLite store and syncs against a shared hub.
|
||||
|
||||
Same provisioning as topology 1. Configure with `AddZbSecretsSqlServerReplication` and:
|
||||
|
||||
```jsonc
|
||||
"Secrets": {
|
||||
"SqlitePath": "/var/lib/zb/secrets.db",
|
||||
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
||||
"SqlServer": {
|
||||
"ConnectionString": "...",
|
||||
"SyncInterval": "00:00:30",
|
||||
"SyncOnStartup": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`SyncInterval` is the convergence bound, not the propagation latency — a live write reaches the hub
|
||||
in milliseconds; the interval governs how fast a *dropped* write or a rejoining node is repaired.
|
||||
|
||||
### Rotation
|
||||
|
||||
Run `rewrap-all` against the hub **and** against every node's local SQLite store. A re-wrap does not
|
||||
bump the revision (by design — otherwise every node's independent re-wrap would churn replication),
|
||||
so it does **not** propagate.
|
||||
|
||||
### Verifying convergence
|
||||
|
||||
```
|
||||
Secret hub sync converged: pulled N row(s) from the hub, pushed M row(s) to it.
|
||||
```
|
||||
|
||||
Logged at Information only when something actually moved. A healthy steady-state cluster logs
|
||||
nothing. Continuous non-zero pushes on every tick means two nodes are fighting over the same secret
|
||||
— look for a script writing the same name on a loop.
|
||||
|
||||
---
|
||||
|
||||
## 3. Akka cluster peer-to-peer
|
||||
|
||||
No shared database. Requires the application to register its `ActorSystem` in DI.
|
||||
|
||||
```jsonc
|
||||
"Secrets": {
|
||||
"SqlitePath": "/var/lib/zb/secrets.db",
|
||||
"MasterKey": { "Source": "File", "FilePath": "/run/secrets/zb-kek" },
|
||||
"Replication": {
|
||||
"AnnounceInterval": "00:00:30",
|
||||
"ActorName": "zb-secret-replication"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Merge the serializer config when creating the actor system:
|
||||
|
||||
```csharp
|
||||
Config config = AkkaSecretsReplication.SerializationConfig.WithFallback(myAppConfig);
|
||||
ActorSystem.Create("my-system", config);
|
||||
```
|
||||
|
||||
Optional but recommended: without it the protocol still round-trips under Akka's default JSON
|
||||
serializer, but the serializer carrying secret ciphertext becomes an inherited default rather than
|
||||
a deliberate choice.
|
||||
|
||||
### Securing the transport
|
||||
|
||||
Rows on the wire are ciphertext and the KEK never travels, so a passive eavesdropper learns
|
||||
nothing but secret *names*, revisions, and timestamps. That is still metadata worth protecting, and
|
||||
an unauthenticated cluster port is a much larger problem than secret replication — configure
|
||||
Akka.Remote TLS and cluster authentication as you would for any other cluster traffic.
|
||||
|
||||
### Rotation
|
||||
|
||||
Run `rewrap-all` on **each node's** local store. Re-wraps do not replicate (see above).
|
||||
|
||||
### Announce traffic
|
||||
|
||||
One manifest per node per interval: names, revisions, timestamps, tombstone flags — no ciphertext.
|
||||
Cost grows as O(nodes × secrets), so a large cluster with a large secret set should lengthen
|
||||
`AnnounceInterval` rather than shorten it.
|
||||
|
||||
---
|
||||
|
||||
## Choosing between them
|
||||
|
||||
| | Shared store | SQL hub | Akka peer-to-peer |
|
||||
|---|---|---|---|
|
||||
| Resolves while cut off from the DB | ✗ (cache TTL only) | ✓ | ✓ (no DB at all) |
|
||||
| Distributed failure modes | none | LWW, sweep lag | LWW, sweep lag |
|
||||
| Rotation runs | once | hub + each node | each node |
|
||||
| Needs a shared database | ✓ | ✓ | ✗ |
|
||||
| Needs an Akka cluster | ✗ | ✗ | ✓ |
|
||||
|
||||
Start at the left and move right only when a stated requirement forces it.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`SecretDecryptionException` / `kek_id` mismatch after enabling replication.**
|
||||
The KEKs differ between nodes. Compare digests as in §0. Do not "fix" it by re-writing the secret on
|
||||
the failing node — that hides the mismatch until the next secret.
|
||||
|
||||
**A secret written on node A never appears on node B.**
|
||||
Check, in order: (1) are both nodes actually in the cluster / pointed at the same hub database and
|
||||
schema; (2) has one `AnnounceInterval` / `SyncInterval` elapsed; (3) is there a warning-level log
|
||||
about a failed publish or a failed sweep. A dropped publish is expected and self-repairing — a
|
||||
sweep that fails every interval is not.
|
||||
|
||||
**A deleted secret came back.**
|
||||
Deletes are tombstones and propagate as newer rows, so this should not happen through replication.
|
||||
It does happen if a node's local store was restored from a backup taken before the delete: the
|
||||
restore reintroduces the row with an old timestamp, loses to LWW, and the tombstone re-wins. If
|
||||
instead the row is *live* again, someone re-created it — check the audit log for the write.
|
||||
|
||||
**The migration refuses to run.** `SecretStoreMigrationException` naming a version newer than
|
||||
supported means another node is running a newer build against the same store. Upgrade this node;
|
||||
do not downgrade the store.
|
||||
@@ -2,7 +2,8 @@ namespace ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Publishes an encrypted row to cluster peers. Core registers a no-op implementation;
|
||||
/// <c>ZB.MOM.WW.Secrets.Akka</c> (deferred) provides the real cluster implementation.
|
||||
/// <c>ZB.MOM.WW.Secrets.Replication.Akka</c> (peer-to-peer over the cluster) and
|
||||
/// <c>ZB.MOM.WW.Secrets.Replication.SqlServer</c> (via a shared hub database) provide the real ones.
|
||||
/// </summary>
|
||||
public interface ISecretReplicator
|
||||
{
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
namespace ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// Provisions the schema backing an <see cref="ISecretStore"/>. Implementations are idempotent —
|
||||
/// running a migration repeatedly is safe — and refuse to run against a store whose on-disk schema
|
||||
/// version is <em>newer</em> than the build supports, rather than silently downgrading it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This seam exists so the startup migration hosted service and the CLI stay store-agnostic: the
|
||||
/// SQLite store, the shared SQL-Server store, and any future provider are all provisioned through
|
||||
/// the same call, and swapping the store never touches the migration wiring.
|
||||
/// </remarks>
|
||||
public interface ISecretsStoreMigrator
|
||||
{
|
||||
/// <summary>Creates or updates the secret-store schema to the version this build supports.</summary>
|
||||
/// <param name="cancellationToken">A token to cancel the operation.</param>
|
||||
/// <returns>A task that completes when the schema is at the supported version.</returns>
|
||||
/// <exception cref="SecretStoreMigrationException">
|
||||
/// The store's schema version is newer than this build supports.
|
||||
/// </exception>
|
||||
Task MigrateAsync(CancellationToken cancellationToken);
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
namespace ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
/// <summary>
|
||||
/// The single definition of the last-writer-wins (LWW) ordering used to reconcile a secret row
|
||||
/// against a peer's copy: newer <see cref="StoredSecret.UpdatedUtc"/> wins, and
|
||||
/// <see cref="StoredSecret.Revision"/> breaks a timestamp tie.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Every store's <see cref="ISecretStore.ApplyReplicatedAsync"/> and every replication transport
|
||||
/// reconciler routes its "is the incoming row newer?" decision through here. Keeping one definition
|
||||
/// is a correctness requirement, not a tidiness one: a cluster can mix stores (a node backed by the
|
||||
/// shared SQL-Server store alongside nodes on local SQLite), and two implementations that disagree
|
||||
/// about a tie would converge to <em>different</em> rows on different nodes — a silent split-brain
|
||||
/// that no single-node test can catch.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The comparison is deliberately <b>strict</b>: a row that merely ties the local one is not newer
|
||||
/// and is ignored. That makes replication idempotent — re-delivering a row a node already has is a
|
||||
/// no-op rather than a write — so at-least-once transports (pub/sub redelivery, an anti-entropy
|
||||
/// sweep racing a live broadcast) cost nothing and cannot flap a row back and forth.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SecretLastWriterWins
|
||||
{
|
||||
/// <summary>
|
||||
/// Returns <see langword="true"/> if a row at (<paramref name="incomingUpdatedUtc"/>,
|
||||
/// <paramref name="incomingRevision"/>) should overwrite a local row at
|
||||
/// (<paramref name="localUpdatedUtc"/>, <paramref name="localRevision"/>).
|
||||
/// </summary>
|
||||
/// <param name="incomingUpdatedUtc">The incoming row's last-updated timestamp.</param>
|
||||
/// <param name="incomingRevision">The incoming row's revision.</param>
|
||||
/// <param name="localUpdatedUtc">The local row's last-updated timestamp.</param>
|
||||
/// <param name="localRevision">The local row's revision.</param>
|
||||
/// <returns><see langword="true"/> if the incoming row is strictly newer.</returns>
|
||||
public static bool IsNewer(
|
||||
DateTimeOffset incomingUpdatedUtc,
|
||||
long incomingRevision,
|
||||
DateTimeOffset localUpdatedUtc,
|
||||
long localRevision) =>
|
||||
incomingUpdatedUtc > localUpdatedUtc ||
|
||||
(incomingUpdatedUtc == localUpdatedUtc && incomingRevision > localRevision);
|
||||
|
||||
/// <summary>
|
||||
/// Returns <see langword="true"/> if <paramref name="incoming"/> should overwrite the local row
|
||||
/// described by <paramref name="local"/>.
|
||||
/// </summary>
|
||||
/// <param name="incoming">The manifest entry received from a peer.</param>
|
||||
/// <param name="local">The corresponding local manifest entry.</param>
|
||||
/// <returns><see langword="true"/> if the incoming entry is strictly newer.</returns>
|
||||
public static bool IsNewer(SecretManifestEntry incoming, SecretManifestEntry local)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(incoming);
|
||||
ArgumentNullException.ThrowIfNull(local);
|
||||
|
||||
return IsNewer(incoming.UpdatedUtc, incoming.Revision, local.UpdatedUtc, local.Revision);
|
||||
}
|
||||
}
|
||||
@@ -1,12 +1,10 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Cli;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.MasterKey;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
// Headless `secret` CLI: set / get / list / rm / rotate over the ZB.MOM.WW secrets store.
|
||||
// The heavy lifting lives in SecretCommands; Program only builds the host, runs the schema
|
||||
@@ -20,8 +18,9 @@ builder.Services.AddZbSecrets(builder.Configuration, "Secrets");
|
||||
|
||||
using IHost host = builder.Build();
|
||||
|
||||
// Ensure the schema exists before any store operation.
|
||||
await host.Services.GetRequiredService<SqliteSecretsStoreMigrator>()
|
||||
// Ensure the schema exists before any store operation. Resolved through the ISecretsStoreMigrator
|
||||
// seam so the CLI provisions whichever store the app configured (local SQLite or shared SQL Server).
|
||||
await host.Services.GetRequiredService<ISecretsStoreMigrator>()
|
||||
.MigrateAsync(CancellationToken.None);
|
||||
|
||||
var commands = new SecretCommands(
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
using Akka.Actor;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
|
||||
|
||||
/// <summary>
|
||||
/// Publishes local secret writes to cluster peers by handing them to this node's
|
||||
/// <see cref="SecretReplicationActor"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The hand-off is a <c>Tell</c>, so <see cref="PublishAsync"/> completes as soon as the row is in
|
||||
/// the actor's mailbox rather than when peers have applied it. That is the correct contract here:
|
||||
/// the caller (<c>ReplicatingSecretStore</c>) has already durably written the row locally and treats
|
||||
/// publishing as best-effort, and blocking a write until every peer acknowledged would make each
|
||||
/// node's availability depend on all the others. Delivery is guaranteed by anti-entropy, not by
|
||||
/// this call.
|
||||
/// </remarks>
|
||||
/// <param name="replicationActor">This node's replication actor.</param>
|
||||
public sealed class AkkaSecretReplicator(IActorRef replicationActor) : ISecretReplicator
|
||||
{
|
||||
private readonly IActorRef _replicationActor =
|
||||
replicationActor ?? throw new ArgumentNullException(nameof(replicationActor));
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task PublishAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
_replicationActor.Tell(new SecretReplicationActor.PublishLocalWrite(row));
|
||||
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
using Akka.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
|
||||
|
||||
/// <summary>
|
||||
/// Akka configuration this package contributes.
|
||||
/// </summary>
|
||||
public static class AkkaSecretsReplication
|
||||
{
|
||||
/// <summary>
|
||||
/// HOCON binding the secret-replication wire protocol to
|
||||
/// <see cref="SecretReplicationSerializer"/>. Merge it into the application's Akka
|
||||
/// configuration when creating the <c>ActorSystem</c>:
|
||||
/// <code>
|
||||
/// Config config = AkkaSecretsReplication.SerializationConfig
|
||||
/// .WithFallback(myAppConfig);
|
||||
/// ActorSystem.Create("my-system", config);
|
||||
/// </code>
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Optional — the protocol DTOs round-trip under Akka's default JSON serializer without it — but
|
||||
/// recommended, because it makes the serializer carrying secret ciphertext an explicit choice
|
||||
/// rather than an inherited default, and pins the manifest strings that a future protocol
|
||||
/// version would bump.
|
||||
/// </remarks>
|
||||
public static Config SerializationConfig { get; } = ConfigurationFactory.ParseString($$"""
|
||||
akka.actor {
|
||||
serializers {
|
||||
zb-secrets-replication = "{{typeof(SecretReplicationSerializer).AssemblyQualifiedName}}"
|
||||
}
|
||||
serialization-bindings {
|
||||
"ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol.SecretRowsMessage, ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" = zb-secrets-replication
|
||||
"ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol.SecretManifestAnnounce, ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" = zb-secrets-replication
|
||||
"ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol.SecretPullRequest, ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" = zb-secrets-replication
|
||||
}
|
||||
serialization-identifiers {
|
||||
"{{typeof(SecretReplicationSerializer).AssemblyQualifiedName}}" = {{SecretReplicationSerializer.SerializerId}}
|
||||
}
|
||||
}
|
||||
""");
|
||||
}
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
|
||||
|
||||
/// <summary>
|
||||
/// Configuration for peer-to-peer secret replication over the Akka cluster.
|
||||
/// </summary>
|
||||
public sealed class AkkaSecretsReplicationOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// How often this node broadcasts its manifest for anti-entropy. Defaults to 30 seconds.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is the convergence bound, not the propagation latency: a live write reaches peers in
|
||||
/// milliseconds, and this interval only governs how quickly a <em>dropped</em> write or a
|
||||
/// rejoining node is repaired. The traffic is one manifest per node per interval — names,
|
||||
/// revisions, and timestamps, no ciphertext — so shortening it is cheap for a small cluster and
|
||||
/// grows as O(nodes × secrets).
|
||||
/// </remarks>
|
||||
public TimeSpan AnnounceInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>Name of the replication actor within the actor system. Defaults to <c>zb-secret-replication</c>.</summary>
|
||||
public string ActorName { get; set; } = "zb-secret-replication";
|
||||
|
||||
/// <summary>
|
||||
/// Throws if the options are unusable.
|
||||
/// </summary>
|
||||
/// <exception cref="InvalidOperationException">The announce interval is not positive, or the actor name is blank.</exception>
|
||||
public void Validate()
|
||||
{
|
||||
if (AnnounceInterval <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"Secret replication AnnounceInterval must be positive (was {AnnounceInterval}).");
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(ActorName))
|
||||
{
|
||||
throw new InvalidOperationException("Secret replication ActorName must not be blank.");
|
||||
}
|
||||
}
|
||||
}
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Dependency-injection entry point for peer-to-peer secret replication over an Akka.NET cluster.
|
||||
/// </summary>
|
||||
public static class AkkaSecretsServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// Registers secret replication over the cluster: each node keeps its own local store, local
|
||||
/// writes broadcast to peers, and a periodic manifest exchange repairs any divergence in both
|
||||
/// directions.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Choose this transport when a node must keep resolving secrets while <b>partitioned</b> from
|
||||
/// the rest of the cluster — reads never leave the node, so an isolated site keeps running on its
|
||||
/// last-known-good state and re-converges when the partition heals. If every node can always
|
||||
/// reach a shared database, the SQL-Server package's shared-store mode is simpler and has no
|
||||
/// distributed failure modes at all; prefer it unless partition tolerance is a stated requirement.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Requirements.</b> The application must (1) register its <see cref="ActorSystem"/> in DI
|
||||
/// before the first secret resolve, and (2) give every node the <b>same KEK</b> — rows carry
|
||||
/// ciphertext only, so a node with a different master key fails closed on a <c>kek_id</c>
|
||||
/// mismatch. Merging <see cref="AkkaSecretsReplication.SerializationConfig"/> into the actor
|
||||
/// system's configuration is recommended.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>KEK rotation does not replicate.</b> A re-wrap deliberately leaves the revision untouched
|
||||
/// so it stays invisible to last-writer-wins; run <c>rewrap-all</c> against every node's local
|
||||
/// store.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to add to.</param>
|
||||
/// <param name="config">The application configuration.</param>
|
||||
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
|
||||
/// <param name="replicationSectionPath">Configuration section holding the replication options.</param>
|
||||
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
|
||||
public static IServiceCollection AddZbSecretsAkkaReplication(
|
||||
this IServiceCollection services,
|
||||
IConfiguration config,
|
||||
string secretsSectionPath = "Secrets",
|
||||
string replicationSectionPath = "Secrets:Replication")
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(secretsSectionPath);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(replicationSectionPath);
|
||||
|
||||
IConfigurationSection section = config.GetSection(replicationSectionPath);
|
||||
services.Configure<AkkaSecretsReplicationOptions>(section);
|
||||
|
||||
// Validate eagerly so a bad interval fails at startup, not on the first announce tick.
|
||||
var eager = new AkkaSecretsReplicationOptions();
|
||||
section.Bind(eager);
|
||||
eager.Validate();
|
||||
|
||||
// The local store + migrator come from AddZbSecrets (SQLite by default).
|
||||
services.AddZbSecrets(config, secretsSectionPath);
|
||||
|
||||
// The actor is created lazily on first use rather than at registration: the ActorSystem is
|
||||
// typically registered by the application AFTER this call, and a cluster node may also not be
|
||||
// ready to join at the moment the container is built.
|
||||
services.TryAddSingleton<SecretReplicationActorProvider>(sp => new SecretReplicationActorProvider(
|
||||
sp.GetRequiredService<ActorSystem>(),
|
||||
sp.GetRequiredService<SqliteSecretStore>(),
|
||||
sp.GetService<ISecretCacheInvalidator>(),
|
||||
sp.GetRequiredService<IOptions<AkkaSecretsReplicationOptions>>().Value));
|
||||
|
||||
services.TryAddSingleton<ISecretReplicator>(sp =>
|
||||
new AkkaSecretReplicator(sp.GetRequiredService<SecretReplicationActorProvider>().ActorRef));
|
||||
|
||||
// Decorate the local store so every write publishes. Resolved from the concrete SQLite store,
|
||||
// not ISecretStore — asking the container for ISecretStore here would resolve this very
|
||||
// decorator and recurse forever.
|
||||
services.AddSingleton<ISecretStore>(sp => new ReplicatingSecretStore(
|
||||
sp.GetRequiredService<SqliteSecretStore>(),
|
||||
sp.GetRequiredService<ISecretReplicator>(),
|
||||
sp.GetRequiredService<ILogger<ReplicatingSecretStore>>()));
|
||||
|
||||
return services;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates this node's <see cref="SecretReplicationActor"/> on first access and hands out its ref.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The indirection exists because the actor must be created lazily — the application usually
|
||||
/// registers its <see cref="ActorSystem"/> after <c>AddZbSecretsAkkaReplication</c>, so eagerly
|
||||
/// resolving one at registration time would fail. Creation is guarded by
|
||||
/// <see cref="Lazy{T}"/> with <see cref="LazyThreadSafetyMode.ExecutionAndPublication"/> so
|
||||
/// concurrent first writes cannot spawn two actors on the same node.
|
||||
/// </remarks>
|
||||
/// <param name="system">The application's actor system.</param>
|
||||
/// <param name="store">The node's local secret store — the undecorated one.</param>
|
||||
/// <param name="cacheInvalidator">Resolver-cache seam.</param>
|
||||
/// <param name="options">Replication options.</param>
|
||||
public sealed class SecretReplicationActorProvider(
|
||||
ActorSystem system,
|
||||
ISecretStore store,
|
||||
ISecretCacheInvalidator? cacheInvalidator,
|
||||
AkkaSecretsReplicationOptions options)
|
||||
{
|
||||
private readonly Lazy<IActorRef> _actorRef = new(
|
||||
() => system.ActorOf(
|
||||
SecretReplicationActor.Props(store, cacheInvalidator, options.AnnounceInterval),
|
||||
options.ActorName),
|
||||
LazyThreadSafetyMode.ExecutionAndPublication);
|
||||
|
||||
/// <summary>This node's replication actor, created on first access.</summary>
|
||||
public IActorRef ActorRef => _actorRef.Value;
|
||||
}
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol;
|
||||
|
||||
/// <summary>
|
||||
/// The wire representation of one encrypted secret row.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is a deliberate hand-written mirror of <see cref="StoredSecret"/> rather than the type
|
||||
/// itself, and the duplication is the point. It makes the trust boundary <b>structural</b>: the wire
|
||||
/// contract is a flat set of primitives that a reader can check against "ciphertext only, never the
|
||||
/// KEK" in one glance, and a field added to <see cref="StoredSecret"/> in future cannot start
|
||||
/// crossing the network merely by existing — someone has to add it here on purpose.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// All fields are primitives (<see cref="string"/>, <see cref="byte"/> arrays, <see cref="long"/>,
|
||||
/// <see cref="bool"/>) so the DTO round-trips through any serializer, including Akka's default JSON
|
||||
/// one if an application never registers <see cref="SecretReplicationSerializer"/>. Timestamps are
|
||||
/// ISO-8601 <c>"O"</c> strings for the same reason the stores use them: last-writer-wins compares
|
||||
/// them for equality, so the representation has to be exact rather than merely close.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed record SecretRowDto
|
||||
{
|
||||
/// <summary>The normalized secret name.</summary>
|
||||
public required string Name { get; init; }
|
||||
|
||||
/// <summary>Optional human-readable description.</summary>
|
||||
public string? Description { get; init; }
|
||||
|
||||
/// <summary>The <see cref="SecretContentType"/> enum name.</summary>
|
||||
public required string ContentType { get; init; }
|
||||
|
||||
/// <summary>AES-256-GCM ciphertext of the secret plaintext.</summary>
|
||||
public required byte[] Ciphertext { get; init; }
|
||||
|
||||
/// <summary>Nonce sealing <see cref="Ciphertext"/>.</summary>
|
||||
public required byte[] Nonce { get; init; }
|
||||
|
||||
/// <summary>Authentication tag for <see cref="Ciphertext"/>.</summary>
|
||||
public required byte[] Tag { get; init; }
|
||||
|
||||
/// <summary>The per-secret data key, wrapped by the KEK. The KEK itself never travels.</summary>
|
||||
public required byte[] WrappedDek { get; init; }
|
||||
|
||||
/// <summary>Nonce used to wrap <see cref="WrappedDek"/>.</summary>
|
||||
public required byte[] WrapNonce { get; init; }
|
||||
|
||||
/// <summary>Authentication tag for <see cref="WrappedDek"/>.</summary>
|
||||
public required byte[] WrapTag { get; init; }
|
||||
|
||||
/// <summary>Identifier of the KEK that wrapped the DEK — an identifier, not key material.</summary>
|
||||
public required string KekId { get; init; }
|
||||
|
||||
/// <summary>Monotonic revision.</summary>
|
||||
public required long Revision { get; init; }
|
||||
|
||||
/// <summary>Whether the row is a tombstone.</summary>
|
||||
public bool IsDeleted { get; init; }
|
||||
|
||||
/// <summary>ISO-8601 deletion timestamp, or <see langword="null"/>.</summary>
|
||||
public string? DeletedUtc { get; init; }
|
||||
|
||||
/// <summary>ISO-8601 creation timestamp.</summary>
|
||||
public required string CreatedUtc { get; init; }
|
||||
|
||||
/// <summary>ISO-8601 last-update timestamp — half of the last-writer-wins ordering key.</summary>
|
||||
public required string UpdatedUtc { get; init; }
|
||||
|
||||
/// <summary>Principal that created the row, if known.</summary>
|
||||
public string? CreatedBy { get; init; }
|
||||
|
||||
/// <summary>Principal that last updated the row, if known.</summary>
|
||||
public string? UpdatedBy { get; init; }
|
||||
|
||||
/// <summary>Projects a stored row onto the wire contract.</summary>
|
||||
/// <param name="row">The encrypted row.</param>
|
||||
/// <returns>The wire representation.</returns>
|
||||
public static SecretRowDto FromStoredSecret(StoredSecret row)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
return new SecretRowDto
|
||||
{
|
||||
Name = row.Name.Value,
|
||||
Description = row.Description,
|
||||
ContentType = row.ContentType.ToString(),
|
||||
Ciphertext = row.Ciphertext,
|
||||
Nonce = row.Nonce,
|
||||
Tag = row.Tag,
|
||||
WrappedDek = row.WrappedDek,
|
||||
WrapNonce = row.WrapNonce,
|
||||
WrapTag = row.WrapTag,
|
||||
KekId = row.KekId,
|
||||
Revision = row.Revision,
|
||||
IsDeleted = row.IsDeleted,
|
||||
DeletedUtc = row.DeletedUtc?.ToString("O"),
|
||||
CreatedUtc = row.CreatedUtc.ToString("O"),
|
||||
UpdatedUtc = row.UpdatedUtc.ToString("O"),
|
||||
CreatedBy = row.CreatedBy,
|
||||
UpdatedBy = row.UpdatedBy,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Rebuilds the stored row from the wire contract.</summary>
|
||||
/// <returns>The encrypted row.</returns>
|
||||
/// <exception cref="ArgumentException">The name or content type is not valid.</exception>
|
||||
public StoredSecret ToStoredSecret() => new()
|
||||
{
|
||||
// SecretName re-validates on construction, so a malformed or path-traversing name from a
|
||||
// peer is rejected here rather than reaching the store.
|
||||
Name = new SecretName(Name),
|
||||
Description = Description,
|
||||
ContentType = ParseContentType(ContentType),
|
||||
// `required` on a byte[] means "present in the payload", NOT "non-null" — a peer sending an
|
||||
// explicit JSON null satisfies it and yields a null array, which would then blow up deep in
|
||||
// the store's parameter binding instead of being rejected at the boundary.
|
||||
Ciphertext = RequireBlob(Ciphertext, nameof(Ciphertext)),
|
||||
Nonce = RequireBlob(Nonce, nameof(Nonce)),
|
||||
Tag = RequireBlob(Tag, nameof(Tag)),
|
||||
WrappedDek = RequireBlob(WrappedDek, nameof(WrappedDek)),
|
||||
WrapNonce = RequireBlob(WrapNonce, nameof(WrapNonce)),
|
||||
WrapTag = RequireBlob(WrapTag, nameof(WrapTag)),
|
||||
KekId = KekId ?? throw new ArgumentException("Replicated secret row has a null KekId."),
|
||||
Revision = Revision,
|
||||
IsDeleted = IsDeleted,
|
||||
DeletedUtc = DeletedUtc is null ? null : ParseUtc(DeletedUtc),
|
||||
CreatedUtc = ParseUtc(CreatedUtc),
|
||||
UpdatedUtc = ParseUtc(UpdatedUtc),
|
||||
CreatedBy = CreatedBy,
|
||||
UpdatedBy = UpdatedBy,
|
||||
};
|
||||
|
||||
private static DateTimeOffset ParseUtc(string value) => DateTimeOffset.Parse(
|
||||
value,
|
||||
System.Globalization.CultureInfo.InvariantCulture,
|
||||
System.Globalization.DateTimeStyles.RoundtripKind);
|
||||
|
||||
private static byte[] RequireBlob(byte[]? value, string field) =>
|
||||
value ?? throw new ArgumentException($"Replicated secret row has a null {field}.", field);
|
||||
|
||||
// Enum.Parse is wrong for untrusted input on two counts: a numeric string that overflows the
|
||||
// underlying type throws OverflowException (which the receiving actor's ArgumentException filter
|
||||
// would not catch, restarting it in a redelivery loop), and any in-range number is accepted even
|
||||
// if it names no member — persisting an undefined enum value that later readers assume is valid.
|
||||
private static SecretContentType ParseContentType(string value) =>
|
||||
Enum.TryParse(value, out SecretContentType parsed) && Enum.IsDefined(parsed)
|
||||
? parsed
|
||||
: throw new ArgumentException(
|
||||
$"Replicated secret row has an unrecognized content type '{value}'.", nameof(value));
|
||||
}
|
||||
|
||||
/// <summary>One manifest entry on the wire: enough to decide who is newer, with no ciphertext.</summary>
|
||||
public sealed record SecretManifestEntryDto
|
||||
{
|
||||
/// <summary>The normalized secret name.</summary>
|
||||
public required string Name { get; init; }
|
||||
|
||||
/// <summary>Monotonic revision.</summary>
|
||||
public required long Revision { get; init; }
|
||||
|
||||
/// <summary>ISO-8601 last-update timestamp.</summary>
|
||||
public required string UpdatedUtc { get; init; }
|
||||
|
||||
/// <summary>Whether the row is a tombstone.</summary>
|
||||
public bool IsDeleted { get; init; }
|
||||
|
||||
/// <summary>Projects a manifest entry onto the wire contract.</summary>
|
||||
/// <param name="entry">The manifest entry.</param>
|
||||
/// <returns>The wire representation.</returns>
|
||||
public static SecretManifestEntryDto FromEntry(SecretManifestEntry entry)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(entry);
|
||||
|
||||
return new SecretManifestEntryDto
|
||||
{
|
||||
Name = entry.Name.Value,
|
||||
Revision = entry.Revision,
|
||||
UpdatedUtc = entry.UpdatedUtc.ToString("O"),
|
||||
IsDeleted = entry.IsDeleted,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>Rebuilds the manifest entry from the wire contract.</summary>
|
||||
/// <returns>The manifest entry.</returns>
|
||||
public SecretManifestEntry ToEntry() => new()
|
||||
{
|
||||
Name = new SecretName(Name),
|
||||
Revision = Revision,
|
||||
UpdatedUtc = DateTimeOffset.Parse(
|
||||
UpdatedUtc,
|
||||
System.Globalization.CultureInfo.InvariantCulture,
|
||||
System.Globalization.DateTimeStyles.RoundtripKind),
|
||||
IsDeleted = IsDeleted,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A node broadcasting its full inventory so peers can work out, in one round trip, both what they
|
||||
/// need from it and what it needs from them.
|
||||
/// </summary>
|
||||
/// <param name="Entries">The sender's manifest.</param>
|
||||
public sealed record SecretManifestAnnounce(IReadOnlyList<SecretManifestEntryDto> Entries);
|
||||
|
||||
/// <summary>A peer asking for the encrypted rows behind names it found it was missing or behind on.</summary>
|
||||
/// <param name="Names">The requested secret names.</param>
|
||||
public sealed record SecretPullRequest(IReadOnlyList<string> Names);
|
||||
|
||||
/// <summary>
|
||||
/// Encrypted rows delivered to a peer — the reply to a <see cref="SecretPullRequest"/>, the
|
||||
/// unsolicited push half of an anti-entropy exchange, and the live broadcast of a fresh local write.
|
||||
/// </summary>
|
||||
/// <param name="Rows">The encrypted rows.</param>
|
||||
public sealed record SecretRowsMessage(IReadOnlyList<SecretRowDto> Rows);
|
||||
+303
@@ -0,0 +1,303 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
|
||||
|
||||
/// <summary>
|
||||
/// One per node. Owns this node's participation in secret replication: broadcasts local writes,
|
||||
/// periodically announces its manifest, and services peers' pulls.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The protocol is deliberately small — three messages, no leader, no singleton:
|
||||
/// </para>
|
||||
/// <list type="number">
|
||||
/// <item><description>
|
||||
/// <b>Live push.</b> A local write publishes a <see cref="SecretRowsMessage"/> to the topic. Peers
|
||||
/// apply it last-writer-wins. This is the fast path — propagation in milliseconds — and it is
|
||||
/// allowed to fail.
|
||||
/// </description></item>
|
||||
/// <item><description>
|
||||
/// <b>Anti-entropy.</b> Every node periodically publishes a <see cref="SecretManifestAnnounce"/>.
|
||||
/// A peer receiving it computes <em>both</em> directions in one pass: it pushes back rows it holds
|
||||
/// newer, and asks (<see cref="SecretPullRequest"/>) for rows it is missing or behind on. This is
|
||||
/// the correctness path — it repairs anything the fast path dropped, in either direction.
|
||||
/// </description></item>
|
||||
/// <item><description>
|
||||
/// <b>Pull service.</b> A <see cref="SecretPullRequest"/> is answered with the requested rows.
|
||||
/// </description></item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// There is no coordinator because there does not need to be one: every exchange is idempotent and
|
||||
/// commutative under last-writer-wins, so nodes converge regardless of message order, duplication,
|
||||
/// or which subset of peers is currently reachable. A node that was partitioned keeps serving its
|
||||
/// local store the whole time and re-converges on the first announce after the partition heals —
|
||||
/// which is the entire reason to choose this transport over the shared SQL-Server store.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SecretReplicationActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
/// <summary>The distributed pub/sub topic secret replication traffic rides on.</summary>
|
||||
public const string Topic = "zb-mom-ww-secrets";
|
||||
|
||||
private readonly ISecretStore _store;
|
||||
private readonly SecretReplicationReconciler _reconciler;
|
||||
private readonly TimeSpan _announceInterval;
|
||||
private readonly IActorRef _mediator;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
|
||||
/// <inheritdoc />
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
/// <summary>Creates the replication actor.</summary>
|
||||
/// <param name="store">The node's local secret store.</param>
|
||||
/// <param name="cacheInvalidator">Resolver-cache seam, evicted for every applied row.</param>
|
||||
/// <param name="announceInterval">How often this node announces its manifest.</param>
|
||||
public SecretReplicationActor(
|
||||
ISecretStore store,
|
||||
ISecretCacheInvalidator? cacheInvalidator,
|
||||
TimeSpan announceInterval)
|
||||
{
|
||||
_store = store ?? throw new ArgumentNullException(nameof(store));
|
||||
_reconciler = new SecretReplicationReconciler(
|
||||
_store,
|
||||
cacheInvalidator,
|
||||
onRowFailed: (name, ex) => _log.Warning(
|
||||
ex, "Discarding replicated secret row {0} that could not be applied locally.", name.Value));
|
||||
_announceInterval = announceInterval;
|
||||
_mediator = DistributedPubSub.Get(Context.System).Mediator;
|
||||
|
||||
_mediator.Tell(new Subscribe(Topic, Self));
|
||||
|
||||
Receive<PublishLocalWrite>(HandlePublishLocalWrite);
|
||||
Receive<SecretRowsMessage>(HandleRows);
|
||||
Receive<SecretManifestAnnounce>(HandleManifestAnnounce);
|
||||
Receive<SecretPullRequest>(HandlePullRequest);
|
||||
Receive<AnnounceTick>(_ => HandleAnnounceTick());
|
||||
Receive<SubscribeAck>(_ => _log.Debug("Subscribed to secret replication topic {0}.", Topic));
|
||||
Receive<ExchangeFailed>(f =>
|
||||
_log.Warning(f.Cause, "Secret anti-entropy exchange with a peer failed; retrying next announce."));
|
||||
Receive<ExchangeDone>(_ => { /* successful exchange, piped back to self — nothing to do. */ });
|
||||
}
|
||||
|
||||
/// <summary>Props for the replication actor.</summary>
|
||||
/// <param name="store">The node's local secret store.</param>
|
||||
/// <param name="cacheInvalidator">Resolver-cache seam.</param>
|
||||
/// <param name="announceInterval">How often this node announces its manifest.</param>
|
||||
/// <returns>Props creating the actor.</returns>
|
||||
public static Props Props(
|
||||
ISecretStore store, ISecretCacheInvalidator? cacheInvalidator, TimeSpan announceInterval) =>
|
||||
Akka.Actor.Props.Create(() =>
|
||||
new SecretReplicationActor(store, cacheInvalidator, announceInterval));
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart() =>
|
||||
Timers.StartPeriodicTimer(AnnounceTick.Instance, AnnounceTick.Instance, _announceInterval, _announceInterval);
|
||||
|
||||
// A local write, handed over by AkkaSecretReplicator. Published from inside the actor so the
|
||||
// message carries Self as sender — that is what lets a peer reply directly to this node instead
|
||||
// of broadcasting its answer to everyone.
|
||||
private void HandlePublishLocalWrite(PublishLocalWrite message)
|
||||
{
|
||||
_mediator.Tell(new Publish(
|
||||
Topic,
|
||||
new SecretRowsMessage([SecretRowDto.FromStoredSecret(message.Row)])));
|
||||
}
|
||||
|
||||
private void HandleRows(SecretRowsMessage message)
|
||||
{
|
||||
if (IsFromSelf())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
IReadOnlyList<StoredSecret> rows = Materialize(message.Rows);
|
||||
|
||||
if (rows.Count == 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// PipeTo nothing: applying is fire-and-forget from the mailbox's point of view. Failures are
|
||||
// logged, and anti-entropy re-delivers anything that did not land.
|
||||
_reconciler.ApplyAsync(rows, CancellationToken.None)
|
||||
.ContinueWith(
|
||||
task => _log.Warning(
|
||||
task.Exception,
|
||||
"Failed to apply {0} replicated secret row(s) from a peer; anti-entropy will retry.",
|
||||
rows.Count),
|
||||
CancellationToken.None,
|
||||
TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
|
||||
TaskScheduler.Default);
|
||||
}
|
||||
|
||||
private void HandleManifestAnnounce(SecretManifestAnnounce message)
|
||||
{
|
||||
if (IsFromSelf())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
IActorRef peer = Sender;
|
||||
IReadOnlyList<SecretManifestEntry> remote = [.. message.Entries.Select(e => e.ToEntry())];
|
||||
|
||||
ReconcileWithPeerAsync(peer, remote).PipeTo(Self, failure: ex => new ExchangeFailed(ex));
|
||||
}
|
||||
|
||||
private void HandlePullRequest(SecretPullRequest message)
|
||||
{
|
||||
if (IsFromSelf())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
IActorRef peer = Sender;
|
||||
// Captured before the continuation: Self is context-dependent and there is no guarantee the
|
||||
// actor context is still current when the continuation runs (Akka analyzer AK1005).
|
||||
IActorRef self = Self;
|
||||
IReadOnlyList<SecretName> names = ParseNames(message.Names);
|
||||
|
||||
_reconciler.ReadLocalAsync(names, CancellationToken.None)
|
||||
.ContinueWith(
|
||||
task =>
|
||||
{
|
||||
if (task.IsFaulted)
|
||||
{
|
||||
// Logged rather than swallowed: the peer is waiting for a reply it will never
|
||||
// get, and without this the only symptom is a secret that quietly never
|
||||
// converges. The peer re-asks on its next announce.
|
||||
_log.Warning(
|
||||
task.Exception,
|
||||
"Failed to read {0} local secret row(s) for a peer's pull request.",
|
||||
names.Count);
|
||||
return;
|
||||
}
|
||||
|
||||
if (task.IsCompletedSuccessfully && task.Result.Count > 0)
|
||||
{
|
||||
peer.Tell(
|
||||
new SecretRowsMessage([.. task.Result.Select(SecretRowDto.FromStoredSecret)]),
|
||||
self);
|
||||
}
|
||||
},
|
||||
CancellationToken.None,
|
||||
TaskContinuationOptions.ExecuteSynchronously,
|
||||
TaskScheduler.Default);
|
||||
}
|
||||
|
||||
private void HandleAnnounceTick()
|
||||
{
|
||||
_store.GetManifestAsync(CancellationToken.None)
|
||||
.ContinueWith(
|
||||
task =>
|
||||
{
|
||||
if (task.IsCompletedSuccessfully)
|
||||
{
|
||||
_mediator.Tell(new Publish(
|
||||
Topic,
|
||||
new SecretManifestAnnounce(
|
||||
[.. task.Result.Select(SecretManifestEntryDto.FromEntry)])));
|
||||
}
|
||||
},
|
||||
CancellationToken.None,
|
||||
TaskContinuationOptions.ExecuteSynchronously,
|
||||
TaskScheduler.Default);
|
||||
}
|
||||
|
||||
// Both halves of an anti-entropy exchange, computed from one manifest: push what the peer is
|
||||
// behind on, ask for what we are behind on.
|
||||
private async Task<ExchangeDone> ReconcileWithPeerAsync(
|
||||
IActorRef peer, IReadOnlyList<SecretManifestEntry> remote)
|
||||
{
|
||||
IReadOnlyList<SecretManifestEntry> localManifest =
|
||||
await _store.GetManifestAsync(CancellationToken.None).ConfigureAwait(false);
|
||||
|
||||
IReadOnlyList<SecretName> push =
|
||||
SecretReplicationReconciler.ComputePushSet(localManifest, remote);
|
||||
|
||||
if (push.Count > 0)
|
||||
{
|
||||
IReadOnlyList<StoredSecret> rows =
|
||||
await _reconciler.ReadLocalAsync(push, CancellationToken.None).ConfigureAwait(false);
|
||||
|
||||
if (rows.Count > 0)
|
||||
{
|
||||
peer.Tell(new SecretRowsMessage([.. rows.Select(SecretRowDto.FromStoredSecret)]), Self);
|
||||
}
|
||||
}
|
||||
|
||||
IReadOnlyList<SecretName> pull =
|
||||
SecretReplicationReconciler.ComputePullSet(localManifest, remote);
|
||||
|
||||
if (pull.Count > 0)
|
||||
{
|
||||
peer.Tell(new SecretPullRequest([.. pull.Select(n => n.Value)]), Self);
|
||||
}
|
||||
|
||||
return ExchangeDone.Instance;
|
||||
}
|
||||
|
||||
// DistributedPubSub delivers a node's own publications back to it. Dropping them here is what
|
||||
// stops a broadcast from being re-applied locally and, worse, echoed onward.
|
||||
private bool IsFromSelf() => Sender.Equals(Self);
|
||||
|
||||
// A peer's rows are untrusted input: a malformed name or content type must drop that row, not
|
||||
// fault the actor and stall replication for every other secret.
|
||||
private IReadOnlyList<StoredSecret> Materialize(IReadOnlyList<SecretRowDto> dtos)
|
||||
{
|
||||
var rows = new List<StoredSecret>(dtos.Count);
|
||||
|
||||
foreach (SecretRowDto dto in dtos)
|
||||
{
|
||||
try
|
||||
{
|
||||
rows.Add(dto.ToStoredSecret());
|
||||
}
|
||||
catch (Exception ex) when (ex is ArgumentException or FormatException)
|
||||
{
|
||||
_log.Warning(ex, "Discarding a malformed replicated secret row from a peer.");
|
||||
}
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
private IReadOnlyList<SecretName> ParseNames(IReadOnlyList<string> names)
|
||||
{
|
||||
var parsed = new List<SecretName>(names.Count);
|
||||
|
||||
foreach (string name in names)
|
||||
{
|
||||
try
|
||||
{
|
||||
parsed.Add(new SecretName(name));
|
||||
}
|
||||
catch (ArgumentException ex)
|
||||
{
|
||||
_log.Warning(ex, "Discarding a malformed secret name in a peer's pull request.");
|
||||
}
|
||||
}
|
||||
|
||||
return parsed;
|
||||
}
|
||||
|
||||
/// <summary>Instruction from <see cref="AkkaSecretReplicator"/> to broadcast a local write.</summary>
|
||||
/// <param name="Row">The encrypted row to broadcast.</param>
|
||||
public sealed record PublishLocalWrite(StoredSecret Row);
|
||||
|
||||
private sealed record AnnounceTick
|
||||
{
|
||||
public static readonly AnnounceTick Instance = new();
|
||||
}
|
||||
|
||||
private sealed record ExchangeFailed(Exception Cause);
|
||||
|
||||
private sealed record ExchangeDone
|
||||
{
|
||||
public static readonly ExchangeDone Instance = new();
|
||||
}
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
using System.Runtime.Serialization;
|
||||
using System.Text.Json;
|
||||
using Akka.Actor;
|
||||
using Akka.Serialization;
|
||||
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
|
||||
|
||||
/// <summary>
|
||||
/// Explicit System.Text.Json serializer for the secret-replication wire protocol.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Registering this is <b>optional but recommended</b>. The protocol DTOs are all primitives, so
|
||||
/// Akka's default JSON serializer round-trips them correctly if an application never wires this up —
|
||||
/// but relying on the default means secret ciphertext is serialized by whatever the cluster's
|
||||
/// fallback happens to be, which is not a decision worth leaving implicit for this payload. Binding
|
||||
/// it explicitly also pins the manifest strings below as the versioning boundary: a future protocol
|
||||
/// change adds a manifest, so a mixed-version cluster fails loudly on an unknown manifest instead of
|
||||
/// silently mis-deserializing a secret.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Wire it up by merging <see cref="AkkaSecretsReplication.SerializationConfig"/> into the
|
||||
/// application's Akka configuration — see that member for the HOCON.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="system">The actor system this serializer belongs to.</param>
|
||||
public sealed class SecretReplicationSerializer(ExtendedActorSystem system) : SerializerWithStringManifest(system)
|
||||
{
|
||||
/// <summary>Stable serializer id. Must not collide with other serializers in the cluster.</summary>
|
||||
public const int SerializerId = 7710;
|
||||
|
||||
private const string RowsManifest = "zbs-rows-v1";
|
||||
private const string ManifestAnnounceManifest = "zbs-manifest-v1";
|
||||
private const string PullRequestManifest = "zbs-pull-v1";
|
||||
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
{
|
||||
// Explicitly NOT indented and NOT camel-cased: the wire form should be compact and stable,
|
||||
// not pretty. Property names are the C# names, pinned by the manifest version.
|
||||
WriteIndented = false,
|
||||
};
|
||||
|
||||
/// <inheritdoc />
|
||||
public override int Identifier => SerializerId;
|
||||
|
||||
/// <inheritdoc />
|
||||
public override string Manifest(object o) => o switch
|
||||
{
|
||||
SecretRowsMessage => RowsManifest,
|
||||
SecretManifestAnnounce => ManifestAnnounceManifest,
|
||||
SecretPullRequest => PullRequestManifest,
|
||||
_ => throw new ArgumentException(
|
||||
$"{nameof(SecretReplicationSerializer)} cannot serialize {o?.GetType().FullName ?? "null"}.",
|
||||
nameof(o)),
|
||||
};
|
||||
|
||||
/// <inheritdoc />
|
||||
public override byte[] ToBinary(object obj) => obj switch
|
||||
{
|
||||
SecretRowsMessage rows => JsonSerializer.SerializeToUtf8Bytes(rows, JsonOptions),
|
||||
SecretManifestAnnounce announce => JsonSerializer.SerializeToUtf8Bytes(announce, JsonOptions),
|
||||
SecretPullRequest pull => JsonSerializer.SerializeToUtf8Bytes(pull, JsonOptions),
|
||||
_ => throw new ArgumentException(
|
||||
$"{nameof(SecretReplicationSerializer)} cannot serialize {obj?.GetType().FullName ?? "null"}.",
|
||||
nameof(obj)),
|
||||
};
|
||||
|
||||
/// <inheritdoc />
|
||||
public override object FromBinary(byte[] bytes, string manifest) => manifest switch
|
||||
{
|
||||
RowsManifest => Deserialize<SecretRowsMessage>(bytes, manifest),
|
||||
ManifestAnnounceManifest => Deserialize<SecretManifestAnnounce>(bytes, manifest),
|
||||
PullRequestManifest => Deserialize<SecretPullRequest>(bytes, manifest),
|
||||
_ => throw new SerializationException(
|
||||
$"Unknown secret-replication manifest '{manifest}'. This usually means a peer is running a " +
|
||||
"newer protocol version than this node."),
|
||||
};
|
||||
|
||||
private static T Deserialize<T>(byte[] bytes, string manifest) =>
|
||||
JsonSerializer.Deserialize<T>(bytes, JsonOptions)
|
||||
?? throw new SerializationException(
|
||||
$"Secret-replication payload with manifest '{manifest}' deserialized to null.");
|
||||
}
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<GenerateDocumentationFile>true</GenerateDocumentationFile>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>true</IsPackable>
|
||||
<PackageId>ZB.MOM.WW.Secrets.Replicator.AkkaDotNet</PackageId>
|
||||
<Authors>ZB.MOM.WW</Authors>
|
||||
<Description>Peer-to-peer secret replication over an Akka.NET cluster for the ZB.MOM.WW SCADA family — partition-tolerant, ciphertext-only.</Description>
|
||||
<PackageProjectUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets</PackageProjectUrl>
|
||||
<RepositoryUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets</RepositoryUrl>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.Secrets\ZB.MOM.WW.Secrets.csproj" />
|
||||
<ProjectReference Include="..\ZB.MOM.WW.Secrets.Abstractions\ZB.MOM.WW.Secrets.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Akka" />
|
||||
<PackageReference Include="Akka.Cluster" />
|
||||
<PackageReference Include="Akka.Cluster.Tools" />
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Dependency-injection entry points for backing a cluster's secrets with SQL Server, in either of
|
||||
/// the two supported topologies.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Both require the same KEK on every node.</b> The database stores ciphertext only, so a node
|
||||
/// whose master key differs cannot decrypt rows another node wrote — it fails closed on resolve with
|
||||
/// a <c>kek_id</c> mismatch, which reads like data corruption but is a deployment error. Mount the
|
||||
/// same key file or supply the same key environment variable everywhere.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SqlServerSecretsServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// <b>Shared-store mode.</b> Points every node's <see cref="ISecretStore"/> at one shared
|
||||
/// SQL-Server database. There is exactly one copy of each row, so there is nothing to replicate
|
||||
/// and nothing to reconcile — no split-brain, no anti-entropy lag, no clock skew.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Prefer this unless a node must keep resolving secrets while <em>partitioned</em> from the
|
||||
/// shared database. The trade is availability: a node cut off from the database falls back only
|
||||
/// to the resolver's short in-memory cache. If that is unacceptable, use
|
||||
/// <see cref="AddZbSecretsSqlServerReplication"/> (local store + hub) or the Akka package
|
||||
/// (peer-to-peer, no shared database at all).
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to add to.</param>
|
||||
/// <param name="config">The application configuration.</param>
|
||||
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
|
||||
/// <param name="sqlServerSectionPath">Configuration section holding the SQL-Server options.</param>
|
||||
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
|
||||
public static IServiceCollection AddZbSecretsSqlServerStore(
|
||||
this IServiceCollection services,
|
||||
IConfiguration config,
|
||||
string secretsSectionPath = "Secrets",
|
||||
string sqlServerSectionPath = "Secrets:SqlServer")
|
||||
{
|
||||
RegisterSqlServerCore(services, config, sqlServerSectionPath);
|
||||
|
||||
// Registered BEFORE AddZbSecrets so its TryAdd calls find these already present and leave
|
||||
// them alone — that is the whole mechanism by which the SQL-Server store displaces SQLite.
|
||||
services.TryAddSingleton<ISecretStore>(sp => sp.GetRequiredService<SqlServerSecretStore>());
|
||||
services.TryAddSingleton<ISecretsStoreMigrator>(sp =>
|
||||
sp.GetRequiredService<SqlServerSecretsStoreMigrator>());
|
||||
|
||||
services.AddZbSecrets(config, secretsSectionPath);
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <b>Hub-replication mode.</b> Each node keeps its own local store (SQLite by default) and
|
||||
/// stays converged with a shared SQL-Server hub: local writes publish to the hub immediately, and
|
||||
/// a background sweep reconciles both directions on an interval.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Choose this over <see cref="AddZbSecretsSqlServerStore"/> when a node must keep resolving
|
||||
/// secrets while cut off from the hub — reads are served from local state, so a hub outage
|
||||
/// degrades propagation rather than availability. The cost is real distributed-systems
|
||||
/// behaviour: rows converge eventually (bounded by the sweep interval), conflicting concurrent
|
||||
/// writes to the same secret resolve last-writer-wins, and KEK rotation must be run against the
|
||||
/// hub <em>and</em> each node's local store, because a re-wrap deliberately does not replicate.
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to add to.</param>
|
||||
/// <param name="config">The application configuration.</param>
|
||||
/// <param name="secretsSectionPath">Configuration section holding the core secrets options.</param>
|
||||
/// <param name="sqlServerSectionPath">Configuration section holding the SQL-Server options.</param>
|
||||
/// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
|
||||
public static IServiceCollection AddZbSecretsSqlServerReplication(
|
||||
this IServiceCollection services,
|
||||
IConfiguration config,
|
||||
string secretsSectionPath = "Secrets",
|
||||
string sqlServerSectionPath = "Secrets:SqlServer")
|
||||
{
|
||||
RegisterSqlServerCore(services, config, sqlServerSectionPath);
|
||||
|
||||
services.TryAddSingleton<ISecretReplicationHub>(sp => sp.GetRequiredService<SqlServerSecretStore>());
|
||||
|
||||
services.TryAddSingleton<ISecretReplicator>(sp =>
|
||||
new SqlServerSecretReplicator(sp.GetRequiredService<ISecretReplicationHub>()));
|
||||
|
||||
// The local store + its migrator come from AddZbSecrets (SQLite by default).
|
||||
services.AddZbSecrets(config, secretsSectionPath);
|
||||
|
||||
// Decorate the local store so every write publishes. Resolved from the concrete SQLite store
|
||||
// rather than ISecretStore, because ISecretStore is what we are replacing — asking the
|
||||
// container for it here would resolve this very decorator and recurse forever.
|
||||
services.AddSingleton<ISecretStore>(sp => new ReplicatingSecretStore(
|
||||
sp.GetRequiredService<Secrets.Sqlite.SqliteSecretStore>(),
|
||||
sp.GetRequiredService<ISecretReplicator>(),
|
||||
sp.GetRequiredService<ILogger<ReplicatingSecretStore>>()));
|
||||
|
||||
// The hub's schema must exist too — the node's own migrator only provisions the local store.
|
||||
services.AddHostedService<SqlServerHubMigrationHostedService>();
|
||||
services.AddHostedService<SqlServerSecretSyncService>();
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
// Registers the pieces both modes need: validated options, connection factory, hub store, and
|
||||
// hub migrator.
|
||||
private static void RegisterSqlServerCore(
|
||||
IServiceCollection services,
|
||||
IConfiguration config,
|
||||
string sqlServerSectionPath)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(sqlServerSectionPath);
|
||||
|
||||
IConfigurationSection section = config.GetSection(sqlServerSectionPath);
|
||||
services.Configure<SqlServerSecretsOptions>(section);
|
||||
|
||||
// Bind and validate NOW as well as through the options pipeline: a missing connection string
|
||||
// or a bad schema name should fail at startup with a clear message, not on the first resolve.
|
||||
var eager = new SqlServerSecretsOptions();
|
||||
section.Bind(eager);
|
||||
eager.Validate();
|
||||
|
||||
services.TryAddSingleton(sp =>
|
||||
new SecretsSqlServerConnectionFactory(
|
||||
sp.GetRequiredService<IOptions<SqlServerSecretsOptions>>().Value));
|
||||
|
||||
services.TryAddSingleton(sp =>
|
||||
new SqlServerSecretStore(sp.GetRequiredService<SecretsSqlServerConnectionFactory>()));
|
||||
|
||||
services.TryAddSingleton(sp =>
|
||||
new SqlServerSecretsStoreMigrator(sp.GetRequiredService<SecretsSqlServerConnectionFactory>()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// The subset of store operations hub replication needs from the shared database: compare
|
||||
/// inventories, fetch rows in bulk, and accept a row verbatim.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Narrower than <see cref="ISecretStore"/> on purpose. The hub is never resolved from, never
|
||||
/// written through <see cref="ISecretStore.UpsertAsync"/> (which would stamp the hub's own revision
|
||||
/// and bounce the row back), and never rewrapped by a syncing node — so exposing those operations
|
||||
/// would only create ways to use it wrongly. It also makes the sync sweep testable against an
|
||||
/// in-process fake instead of requiring a live SQL Server for every convergence case.
|
||||
/// </remarks>
|
||||
public interface ISecretReplicationHub
|
||||
{
|
||||
/// <summary>Returns the hub's manifest — names, revisions, timestamps, tombstone flags.</summary>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The hub's manifest entries.</returns>
|
||||
Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct);
|
||||
|
||||
/// <summary>Fetches several encrypted rows by name; names absent from the hub are omitted.</summary>
|
||||
/// <param name="names">The names to fetch.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The matching encrypted rows.</returns>
|
||||
Task<IReadOnlyList<StoredSecret>> GetManyAsync(IReadOnlyList<SecretName> names, CancellationToken ct);
|
||||
|
||||
/// <summary>Applies a row to the hub verbatim, last-writer-wins.</summary>
|
||||
/// <param name="row">The encrypted row.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>A task that completes when the row has been applied or ignored.</returns>
|
||||
Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct);
|
||||
}
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
using Microsoft.Data.SqlClient;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Opens connections to the shared SQL-Server secret database and carries the settings every
|
||||
/// command needs (schema name, command timeout).
|
||||
/// </summary>
|
||||
/// <param name="options">The validated SQL-Server store options.</param>
|
||||
public sealed class SecretsSqlServerConnectionFactory(SqlServerSecretsOptions options)
|
||||
{
|
||||
private readonly SqlServerSecretsOptions _options =
|
||||
options ?? throw new ArgumentNullException(nameof(options));
|
||||
|
||||
/// <summary>The bracket-quoted, allow-list-validated schema prefix for object names.</summary>
|
||||
public string SchemaName => _options.SchemaName;
|
||||
|
||||
/// <summary>Command timeout, in whole seconds, for every command issued against this store.</summary>
|
||||
public int CommandTimeoutSeconds => Math.Max(1, (int)_options.CommandTimeout.TotalSeconds);
|
||||
|
||||
/// <summary>Opens a connection to the shared secret database.</summary>
|
||||
/// <param name="cancellationToken">A token to cancel the connect.</param>
|
||||
/// <returns>An opened connection. The caller owns disposal.</returns>
|
||||
public async Task<SqlConnection> OpenConnectionAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
var connection = new SqlConnection(_options.ConnectionString);
|
||||
try
|
||||
{
|
||||
await connection.OpenAsync(cancellationToken).ConfigureAwait(false);
|
||||
return connection;
|
||||
}
|
||||
catch
|
||||
{
|
||||
await connection.DisposeAsync().ConfigureAwait(false);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Creates a command on <paramref name="connection"/> with the configured timeout applied.
|
||||
/// </summary>
|
||||
/// <param name="connection">The open connection.</param>
|
||||
/// <param name="commandText">The command text.</param>
|
||||
/// <param name="transaction">The ambient transaction, if any.</param>
|
||||
/// <returns>The configured command. The caller owns disposal.</returns>
|
||||
public SqlCommand CreateCommand(
|
||||
SqlConnection connection, string commandText, SqlTransaction? transaction = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
SqlCommand command = connection.CreateCommand();
|
||||
command.CommandText = commandText;
|
||||
command.CommandTimeout = CommandTimeoutSeconds;
|
||||
command.Transaction = transaction;
|
||||
return command;
|
||||
}
|
||||
}
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Provisions the shared hub schema at startup in hub-replication mode.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Separate from the core migration hosted service on purpose: in hub mode there are <em>two</em>
|
||||
/// stores to provision — the node's local one (handled by the core service through the
|
||||
/// <c>ISecretsStoreMigrator</c> seam) and the shared hub, which no node "owns". Every node runs this
|
||||
/// idempotently at startup, so the hub exists as soon as any node has started, with no separate
|
||||
/// provisioning step for an operator to forget.
|
||||
/// </remarks>
|
||||
/// <param name="migrator">The hub migrator.</param>
|
||||
/// <param name="options">Core secrets options, for the shared startup-migration switch.</param>
|
||||
internal sealed class SqlServerHubMigrationHostedService(
|
||||
SqlServerSecretsStoreMigrator migrator,
|
||||
IOptions<SecretsOptions> options) : IHostedService
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public async Task StartAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
if (options.Value.RunMigrationsOnStartup)
|
||||
{
|
||||
await migrator.MigrateAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Publishes local writes to the shared SQL-Server hub, so a secret written on one node is
|
||||
/// immediately visible to every other node that syncs against the same hub.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The row is written through <see cref="ISecretStore.ApplyReplicatedAsync"/>, not
|
||||
/// <see cref="ISecretStore.UpsertAsync"/> — the hub must record the row <em>verbatim</em>, keeping
|
||||
/// the originating node's revision and <c>updated_utc</c>. An upsert would stamp the hub's own
|
||||
/// revision and timestamp, and the row would then look newer than the writer's own copy, so it
|
||||
/// would bounce straight back on the next sync.
|
||||
/// </remarks>
|
||||
/// <param name="hub">The shared hub store.</param>
|
||||
public sealed class SqlServerSecretReplicator(ISecretReplicationHub hub) : ISecretReplicator
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task PublishAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
return hub.ApplyReplicatedAsync(row, ct);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,398 @@
|
||||
using System.Data;
|
||||
using System.Globalization;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// SQL-Server-backed <see cref="ISecretStore"/>. Holds the same envelope-encrypted
|
||||
/// <see cref="StoredSecret"/> rows as the SQLite store, in a database every node of a cluster can
|
||||
/// reach — so one write is visible everywhere with no replication at all.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The database holds <b>ciphertext only</b>. The KEK stays per-node and out of the database, so a
|
||||
/// DBA, a backup tape, or a replica cannot decrypt anything. The corollary is a hard operational
|
||||
/// constraint: every node must resolve the <em>same</em> KEK, because a row whose <c>kek_id</c> does
|
||||
/// not match the local provider fails closed on resolve.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Semantics are deliberately identical to <c>SqliteSecretStore</c> — revision bump on upsert,
|
||||
/// created stamps preserved on overwrite, verbatim last-writer-wins apply, compare-and-swap rewrap.
|
||||
/// A cluster can run both stores at once (nodes on local SQLite replicating against a shared hub),
|
||||
/// so a divergence between the two would be a silent convergence bug.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="connectionFactory">Factory for connections to the shared secret database.</param>
|
||||
public sealed class SqlServerSecretStore(SecretsSqlServerConnectionFactory connectionFactory)
|
||||
: ISecretStore, ISecretReplicationHub
|
||||
{
|
||||
// All columns of the secret table, in schema order, for full-row reads.
|
||||
private const string AllColumns =
|
||||
"name, description, content_type, ciphertext, nonce, tag, wrapped_dek, wrap_nonce, wrap_tag, " +
|
||||
"kek_id, revision, is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by";
|
||||
|
||||
// The safe metadata projection — deliberately excludes every ciphertext / crypto BLOB column.
|
||||
private const string MetadataColumns =
|
||||
"name, description, content_type, kek_id, revision, is_deleted, created_utc, updated_utc, created_by, updated_by";
|
||||
|
||||
private string Secret => $"[{connectionFactory.SchemaName}].[{SqlServerSecretsSchema.SecretTable}]";
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct)
|
||||
{
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(
|
||||
connection, $"SELECT {AllColumns} FROM {Secret} WHERE name = @name;");
|
||||
command.Parameters.AddWithValue("@name", name.Value);
|
||||
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
|
||||
return await reader.ReadAsync(ct).ConfigureAwait(false)
|
||||
? ReadStoredSecret(reader)
|
||||
: null;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task UpsertAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
string now = DateTimeOffset.UtcNow.ToString("O");
|
||||
|
||||
// MERGE establishes revision 0 / created==updated==now on insert; on match it overwrites the
|
||||
// crypto material in place, bumps the revision, refreshes updated_utc/updated_by, and clears
|
||||
// any tombstone — but deliberately preserves created_utc / created_by (the SQLite store's
|
||||
// ON CONFLICT semantics, expressed in T-SQL).
|
||||
//
|
||||
// HOLDLOCK is not optional: without it MERGE takes only an update lock on the key range and
|
||||
// two concurrent nodes upserting the same new name can both miss the match and both INSERT,
|
||||
// producing a primary-key violation instead of one insert + one update.
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
MERGE {Secret} WITH (HOLDLOCK) AS target
|
||||
USING (SELECT @name AS name) AS source
|
||||
ON target.name = source.name
|
||||
WHEN MATCHED THEN UPDATE SET
|
||||
description = @description,
|
||||
content_type = @content_type,
|
||||
ciphertext = @ciphertext,
|
||||
nonce = @nonce,
|
||||
tag = @tag,
|
||||
wrapped_dek = @wrapped_dek,
|
||||
wrap_nonce = @wrap_nonce,
|
||||
wrap_tag = @wrap_tag,
|
||||
kek_id = @kek_id,
|
||||
revision = target.revision + 1,
|
||||
updated_utc = @now,
|
||||
updated_by = @updated_by,
|
||||
is_deleted = 0,
|
||||
deleted_utc = NULL
|
||||
WHEN NOT MATCHED THEN INSERT (
|
||||
name, description, content_type, ciphertext, nonce, tag,
|
||||
wrapped_dek, wrap_nonce, wrap_tag, kek_id, revision,
|
||||
is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by)
|
||||
VALUES (
|
||||
@name, @description, @content_type, @ciphertext, @nonce, @tag,
|
||||
@wrapped_dek, @wrap_nonce, @wrap_tag, @kek_id, 0,
|
||||
0, NULL, @now, @now, @created_by, @updated_by);
|
||||
""");
|
||||
BindCryptoColumns(command, row);
|
||||
command.Parameters.AddWithValue("@now", now);
|
||||
command.Parameters.AddWithValue("@created_by", (object?)row.CreatedBy ?? DBNull.Value);
|
||||
command.Parameters.AddWithValue("@updated_by", (object?)row.UpdatedBy ?? DBNull.Value);
|
||||
|
||||
await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct)
|
||||
{
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
string now = DateTimeOffset.UtcNow.ToString("O");
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE {Secret} SET
|
||||
is_deleted = 1,
|
||||
deleted_utc = @now,
|
||||
revision = revision + 1,
|
||||
updated_utc = @now,
|
||||
updated_by = @actor
|
||||
WHERE name = @name AND is_deleted = 0;
|
||||
""");
|
||||
command.Parameters.AddWithValue("@now", now);
|
||||
command.Parameters.AddWithValue("@actor", (object?)actor ?? DBNull.Value);
|
||||
command.Parameters.AddWithValue("@name", name.Value);
|
||||
|
||||
int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
return rowsAffected > 0;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct)
|
||||
{
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
$"SELECT {MetadataColumns} FROM {Secret} WHERE (@include_deleted = 1 OR is_deleted = 0) ORDER BY name;");
|
||||
command.Parameters.AddWithValue("@include_deleted", includeDeleted ? 1 : 0);
|
||||
|
||||
var results = new List<SecretMetadata>();
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
results.Add(new SecretMetadata
|
||||
{
|
||||
Name = new SecretName(reader.GetString(0)),
|
||||
Description = reader.IsDBNull(1) ? null : reader.GetString(1),
|
||||
ContentType = Enum.Parse<SecretContentType>(reader.GetString(2)),
|
||||
KekId = reader.GetString(3),
|
||||
Revision = reader.GetInt64(4),
|
||||
IsDeleted = reader.GetBoolean(5),
|
||||
CreatedUtc = ParseUtc(reader.GetString(6)),
|
||||
UpdatedUtc = ParseUtc(reader.GetString(7)),
|
||||
CreatedBy = reader.IsDBNull(8) ? null : reader.GetString(8),
|
||||
UpdatedBy = reader.IsDBNull(9) ? null : reader.GetString(9),
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct)
|
||||
{
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(
|
||||
connection, $"SELECT name, revision, updated_utc, is_deleted FROM {Secret} ORDER BY name;");
|
||||
|
||||
var results = new List<SecretManifestEntry>();
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
results.Add(new SecretManifestEntry
|
||||
{
|
||||
Name = new SecretName(reader.GetString(0)),
|
||||
Revision = reader.GetInt64(1),
|
||||
UpdatedUtc = ParseUtc(reader.GetString(2)),
|
||||
IsDeleted = reader.GetBoolean(3),
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
await using SqlTransaction transaction = (SqlTransaction)
|
||||
await connection.BeginTransactionAsync(IsolationLevel.Serializable, ct).ConfigureAwait(false);
|
||||
|
||||
// Read the local (updated_utc, revision) so we can apply last-writer-wins. UPDLOCK/HOLDLOCK
|
||||
// keeps another node from slipping a newer row in between this read and the write below —
|
||||
// Serializable alone would not lock a key range that currently has no row.
|
||||
await using (SqlCommand read = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
$"SELECT updated_utc, revision FROM {Secret} WITH (UPDLOCK, HOLDLOCK) WHERE name = @name;",
|
||||
transaction))
|
||||
{
|
||||
read.Parameters.AddWithValue("@name", row.Name.Value);
|
||||
|
||||
await using SqlDataReader reader = await read.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
if (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
DateTimeOffset localUpdated = ParseUtc(reader.GetString(0));
|
||||
long localRevision = reader.GetInt64(1);
|
||||
|
||||
// Routed through the shared predicate so every store agrees on ties.
|
||||
if (!SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, localUpdated, localRevision))
|
||||
{
|
||||
await reader.CloseAsync().ConfigureAwait(false);
|
||||
await transaction.CommitAsync(ct).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Apply the incoming row VERBATIM — its own revision, timestamps, tombstone flag, and crypto
|
||||
// material — with no revision bump (this is the replication path, not a local write).
|
||||
await using (SqlCommand upsert = connectionFactory.CreateCommand(connection, $"""
|
||||
MERGE {Secret} WITH (HOLDLOCK) AS target
|
||||
USING (SELECT @name AS name) AS source
|
||||
ON target.name = source.name
|
||||
WHEN MATCHED THEN UPDATE SET
|
||||
description = @description,
|
||||
content_type = @content_type,
|
||||
ciphertext = @ciphertext,
|
||||
nonce = @nonce,
|
||||
tag = @tag,
|
||||
wrapped_dek = @wrapped_dek,
|
||||
wrap_nonce = @wrap_nonce,
|
||||
wrap_tag = @wrap_tag,
|
||||
kek_id = @kek_id,
|
||||
revision = @revision,
|
||||
is_deleted = @is_deleted,
|
||||
deleted_utc = @deleted_utc,
|
||||
created_utc = @created_utc,
|
||||
updated_utc = @updated_utc,
|
||||
created_by = @created_by,
|
||||
updated_by = @updated_by
|
||||
WHEN NOT MATCHED THEN INSERT (
|
||||
name, description, content_type, ciphertext, nonce, tag,
|
||||
wrapped_dek, wrap_nonce, wrap_tag, kek_id, revision,
|
||||
is_deleted, deleted_utc, created_utc, updated_utc, created_by, updated_by)
|
||||
VALUES (
|
||||
@name, @description, @content_type, @ciphertext, @nonce, @tag,
|
||||
@wrapped_dek, @wrap_nonce, @wrap_tag, @kek_id, @revision,
|
||||
@is_deleted, @deleted_utc, @created_utc, @updated_utc, @created_by, @updated_by);
|
||||
""", transaction))
|
||||
{
|
||||
BindCryptoColumns(upsert, row);
|
||||
upsert.Parameters.AddWithValue("@revision", row.Revision);
|
||||
upsert.Parameters.AddWithValue("@is_deleted", row.IsDeleted ? 1 : 0);
|
||||
upsert.Parameters.AddWithValue("@deleted_utc", (object?)row.DeletedUtc?.ToString("O") ?? DBNull.Value);
|
||||
upsert.Parameters.AddWithValue("@created_utc", row.CreatedUtc.ToString("O"));
|
||||
upsert.Parameters.AddWithValue("@updated_utc", row.UpdatedUtc.ToString("O"));
|
||||
upsert.Parameters.AddWithValue("@created_by", (object?)row.CreatedBy ?? DBNull.Value);
|
||||
upsert.Parameters.AddWithValue("@updated_by", (object?)row.UpdatedBy ?? DBNull.Value);
|
||||
|
||||
await upsert.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
await transaction.CommitAsync(ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(rewrappedRow);
|
||||
ArgumentNullException.ThrowIfNull(expectedCurrentWrappedDek);
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Overwrite ONLY the KEK-wrap envelope + kek_id. Revision, updated_utc/by, the sealed body,
|
||||
// and the tombstone state are deliberately untouched — a re-wrap is not a logical change, so
|
||||
// it must not bump the revision or updated timestamp (see ISecretStore.ApplyRewrapAsync).
|
||||
//
|
||||
// Compare-and-swap on wrapped_dek: a concurrent set/rotate generates a fresh (random) DEK
|
||||
// wrap, so this matches 0 rows instead of pairing this stale re-wrap with a newer body —
|
||||
// which would leave the row permanently undecryptable. The caller re-processes a 0-row miss.
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(connection, $"""
|
||||
UPDATE {Secret} SET
|
||||
wrapped_dek = @wrapped_dek,
|
||||
wrap_nonce = @wrap_nonce,
|
||||
wrap_tag = @wrap_tag,
|
||||
kek_id = @kek_id
|
||||
WHERE name = @name AND wrapped_dek = @expected_wrapped_dek;
|
||||
""");
|
||||
command.Parameters.AddWithValue("@wrapped_dek", rewrappedRow.WrappedDek);
|
||||
command.Parameters.AddWithValue("@wrap_nonce", rewrappedRow.WrapNonce);
|
||||
command.Parameters.AddWithValue("@wrap_tag", rewrappedRow.WrapTag);
|
||||
command.Parameters.AddWithValue("@kek_id", rewrappedRow.KekId);
|
||||
command.Parameters.AddWithValue("@name", rewrappedRow.Name.Value);
|
||||
command.Parameters.AddWithValue("@expected_wrapped_dek", expectedCurrentWrappedDek);
|
||||
|
||||
int rowsAffected = await command.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
|
||||
return rowsAffected > 0;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fetches several rows by name in one round trip — the replication fetch path. Names not
|
||||
/// present are simply absent from the result.
|
||||
/// </summary>
|
||||
/// <param name="names">The names to fetch.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The matching encrypted rows.</returns>
|
||||
public async Task<IReadOnlyList<StoredSecret>> GetManyAsync(
|
||||
IReadOnlyList<SecretName> names, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(names);
|
||||
|
||||
if (names.Count == 0)
|
||||
{
|
||||
return [];
|
||||
}
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// One parameter per name — still fully parameterized, no name text is interpolated.
|
||||
string[] placeholders = [.. names.Select((_, i) => $"@n{i.ToString(CultureInfo.InvariantCulture)}")];
|
||||
|
||||
await using SqlCommand command = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
$"SELECT {AllColumns} FROM {Secret} WHERE name IN ({string.Join(", ", placeholders)});");
|
||||
|
||||
for (int i = 0; i < names.Count; i++)
|
||||
{
|
||||
command.Parameters.AddWithValue(placeholders[i], names[i].Value);
|
||||
}
|
||||
|
||||
var results = new List<StoredSecret>(names.Count);
|
||||
await using SqlDataReader reader = await command.ExecuteReaderAsync(ct).ConfigureAwait(false);
|
||||
while (await reader.ReadAsync(ct).ConfigureAwait(false))
|
||||
{
|
||||
results.Add(ReadStoredSecret(reader));
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// Binds the identity, description, content-type, KEK id, and all six crypto BLOB columns
|
||||
// shared by every insert path.
|
||||
private static void BindCryptoColumns(SqlCommand command, StoredSecret row)
|
||||
{
|
||||
command.Parameters.AddWithValue("@name", row.Name.Value);
|
||||
command.Parameters.AddWithValue("@description", (object?)row.Description ?? DBNull.Value);
|
||||
command.Parameters.AddWithValue("@content_type", row.ContentType.ToString());
|
||||
command.Parameters.AddWithValue("@ciphertext", row.Ciphertext);
|
||||
command.Parameters.AddWithValue("@nonce", row.Nonce);
|
||||
command.Parameters.AddWithValue("@tag", row.Tag);
|
||||
command.Parameters.AddWithValue("@wrapped_dek", row.WrappedDek);
|
||||
command.Parameters.AddWithValue("@wrap_nonce", row.WrapNonce);
|
||||
command.Parameters.AddWithValue("@wrap_tag", row.WrapTag);
|
||||
command.Parameters.AddWithValue("@kek_id", row.KekId);
|
||||
}
|
||||
|
||||
private static StoredSecret ReadStoredSecret(SqlDataReader reader) => new()
|
||||
{
|
||||
Name = new SecretName(reader.GetString(0)),
|
||||
Description = reader.IsDBNull(1) ? null : reader.GetString(1),
|
||||
ContentType = Enum.Parse<SecretContentType>(reader.GetString(2)),
|
||||
Ciphertext = reader.GetFieldValue<byte[]>(3),
|
||||
Nonce = reader.GetFieldValue<byte[]>(4),
|
||||
Tag = reader.GetFieldValue<byte[]>(5),
|
||||
WrappedDek = reader.GetFieldValue<byte[]>(6),
|
||||
WrapNonce = reader.GetFieldValue<byte[]>(7),
|
||||
WrapTag = reader.GetFieldValue<byte[]>(8),
|
||||
KekId = reader.GetString(9),
|
||||
Revision = reader.GetInt64(10),
|
||||
IsDeleted = reader.GetBoolean(11),
|
||||
DeletedUtc = reader.IsDBNull(12) ? null : ParseUtc(reader.GetString(12)),
|
||||
CreatedUtc = ParseUtc(reader.GetString(13)),
|
||||
UpdatedUtc = ParseUtc(reader.GetString(14)),
|
||||
CreatedBy = reader.IsDBNull(15) ? null : reader.GetString(15),
|
||||
UpdatedBy = reader.IsDBNull(16) ? null : reader.GetString(16),
|
||||
};
|
||||
|
||||
private static DateTimeOffset ParseUtc(string value) =>
|
||||
DateTimeOffset.Parse(value, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind);
|
||||
}
|
||||
+158
@@ -0,0 +1,158 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Keeps a node's local secret store converged with the shared SQL-Server hub, in hub-replication
|
||||
/// mode. Runs a bidirectional anti-entropy sweep on an interval: pulls rows the hub holds newer,
|
||||
/// pushes rows the node holds newer.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is the durability guarantee behind best-effort publishing. Live publishes make a write
|
||||
/// visible in milliseconds; this sweep is what makes it <em>certain</em>. A node that was offline
|
||||
/// while a peer wrote catches up on its next sweep, and a write made while the hub was unreachable
|
||||
/// is pushed on the next sweep instead of being lost.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The sweep never throws out of <see cref="ExecuteAsync"/>: a hub outage must degrade the node to
|
||||
/// "serving its last-known-good local secrets" rather than crash the host. Failures are logged and
|
||||
/// retried on the next tick.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed partial class SqlServerSecretSyncService : BackgroundService
|
||||
{
|
||||
private readonly SecretReplicationReconciler _reconciler;
|
||||
private readonly ISecretStore _local;
|
||||
private readonly ISecretReplicationHub _hub;
|
||||
private readonly SqlServerSecretsOptions _options;
|
||||
private readonly ILogger<SqlServerSecretSyncService> _logger;
|
||||
private readonly TimeProvider _timeProvider;
|
||||
|
||||
/// <summary>Creates the sync service.</summary>
|
||||
/// <param name="local">The node's local store.</param>
|
||||
/// <param name="hub">The shared hub store.</param>
|
||||
/// <param name="cacheInvalidator">Resolver-cache seam, evicted for every applied row.</param>
|
||||
/// <param name="options">SQL-Server replication options.</param>
|
||||
/// <param name="logger">Logger for sweep outcomes and failures.</param>
|
||||
/// <param name="timeProvider">Clock, injectable so tests need not wait a real interval.</param>
|
||||
public SqlServerSecretSyncService(
|
||||
ISecretStore local,
|
||||
ISecretReplicationHub hub,
|
||||
ISecretCacheInvalidator? cacheInvalidator,
|
||||
IOptions<SqlServerSecretsOptions> options,
|
||||
ILogger<SqlServerSecretSyncService> logger,
|
||||
TimeProvider? timeProvider = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
_local = local ?? throw new ArgumentNullException(nameof(local));
|
||||
_hub = hub ?? throw new ArgumentNullException(nameof(hub));
|
||||
_options = options.Value;
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||
_reconciler = new SecretReplicationReconciler(
|
||||
_local,
|
||||
cacheInvalidator,
|
||||
onRowFailed: (name, ex) => LogRowDiscarded(ex, name.Value));
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
if (_options.SyncOnStartup)
|
||||
{
|
||||
await SweepSafelyAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
using PeriodicTimer timer = new(_options.SyncInterval, _timeProvider);
|
||||
|
||||
while (await timer.WaitForNextTickAsync(stoppingToken).ConfigureAwait(false))
|
||||
{
|
||||
await SweepSafelyAsync(stoppingToken).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Runs one bidirectional reconcile against the hub.
|
||||
/// </summary>
|
||||
/// <param name="ct">A token to cancel the sweep.</param>
|
||||
/// <returns>How many rows were pulled from and pushed to the hub.</returns>
|
||||
internal async Task<(int Pulled, int Pushed)> SweepAsync(CancellationToken ct)
|
||||
{
|
||||
IReadOnlyList<SecretManifestEntry> hubManifest =
|
||||
await _hub.GetManifestAsync(ct).ConfigureAwait(false);
|
||||
|
||||
// Pull first: adopting the hub's newer rows before computing the push set means a row the hub
|
||||
// already had newer is not then pushed straight back at it.
|
||||
int pulled = await _reconciler
|
||||
.ReconcileAsync(hubManifest, (names, token) => _hub.GetManyAsync(names, token), ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
IReadOnlyList<SecretManifestEntry> localManifest =
|
||||
await _local.GetManifestAsync(ct).ConfigureAwait(false);
|
||||
|
||||
IReadOnlyList<SecretName> push =
|
||||
SecretReplicationReconciler.ComputePushSet(localManifest, hubManifest);
|
||||
|
||||
int pushed = 0;
|
||||
|
||||
if (push.Count > 0)
|
||||
{
|
||||
IReadOnlyList<StoredSecret> rows =
|
||||
await _reconciler.ReadLocalAsync(push, ct).ConfigureAwait(false);
|
||||
|
||||
foreach (StoredSecret row in rows)
|
||||
{
|
||||
// Verbatim, like the replicator — the hub records the originating node's revision.
|
||||
await _hub.ApplyReplicatedAsync(row, ct).ConfigureAwait(false);
|
||||
pushed++;
|
||||
}
|
||||
}
|
||||
|
||||
return (pulled, pushed);
|
||||
}
|
||||
|
||||
private async Task SweepSafelyAsync(CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
(int pulled, int pushed) = await SweepAsync(ct).ConfigureAwait(false);
|
||||
|
||||
if (pulled > 0 || pushed > 0)
|
||||
{
|
||||
LogSweepConverged(pulled, pushed);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// Host shutdown — not a failure.
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
LogSweepFailed(ex);
|
||||
}
|
||||
}
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 1,
|
||||
Level = LogLevel.Information,
|
||||
Message = "Secret hub sync converged: pulled {Pulled} row(s) from the hub, pushed {Pushed} row(s) to it.")]
|
||||
private partial void LogSweepConverged(int pulled, int pushed);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 2,
|
||||
Level = LogLevel.Warning,
|
||||
Message = "Secret hub sync failed; the node continues serving its local store and will retry on the next interval.")]
|
||||
private partial void LogSweepFailed(Exception exception);
|
||||
|
||||
[LoggerMessage(
|
||||
EventId = 3,
|
||||
Level = LogLevel.Warning,
|
||||
Message = "Discarding replicated secret row {SecretName} that could not be applied locally; the rest of the batch continues.")]
|
||||
private partial void LogRowDiscarded(Exception exception, string secretName);
|
||||
}
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Configuration for the shared SQL-Server secret store and its hub replication. Bound from a
|
||||
/// configuration section by the <c>AddZbSecretsSqlServer*</c> extensions.
|
||||
/// </summary>
|
||||
public sealed class SqlServerSecretsOptions
|
||||
{
|
||||
private string _schemaName = SqlServerSecretsSchema.DefaultSchemaName;
|
||||
|
||||
/// <summary>
|
||||
/// Connection string for the shared secret database. Required. Supply it through a
|
||||
/// <c>${secret:}</c> / <c>secret:</c> reference like every other connection string in the family
|
||||
/// — the shared store holds ciphertext only, but its credentials are still credentials.
|
||||
/// </summary>
|
||||
public string ConnectionString { get; set; } = string.Empty;
|
||||
|
||||
/// <summary>
|
||||
/// SQL schema the secret tables live in. Defaults to <c>zbsecrets</c> so the secret tables never
|
||||
/// collide with an application's own <c>dbo</c> objects when they share a database.
|
||||
/// </summary>
|
||||
/// <exception cref="ArgumentException">
|
||||
/// The value is not a plain identifier (letters, digits, and underscore, not starting with a
|
||||
/// digit, at most 128 characters).
|
||||
/// </exception>
|
||||
/// <remarks>
|
||||
/// Validated on assignment rather than at use: the schema name is interpolated into DDL and
|
||||
/// query text (T-SQL cannot parameterize an object name), so the allow-list is the injection
|
||||
/// boundary and it must be impossible to get an unvalidated value into the property at all.
|
||||
/// </remarks>
|
||||
public string SchemaName
|
||||
{
|
||||
get => _schemaName;
|
||||
set => _schemaName = ValidateIdentifier(value);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// How often the sync service reconciles the local store against the hub, in hub-replication
|
||||
/// mode. Defaults to 30 seconds. Ignored in shared-store mode (there is nothing to reconcile).
|
||||
/// </summary>
|
||||
public TimeSpan SyncInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// When <see langword="true"/> (the default), hub-replication mode runs a reconcile at startup
|
||||
/// before the first interval elapses, so a node that was down while a peer wrote does not serve
|
||||
/// stale secrets for a whole interval after it comes back.
|
||||
/// </summary>
|
||||
public bool SyncOnStartup { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Command timeout applied to every store operation. Defaults to 30 seconds — long enough for a
|
||||
/// contended shared database, short enough that a hung hub surfaces as an error rather than
|
||||
/// stalling a resolve indefinitely.
|
||||
/// </summary>
|
||||
public TimeSpan CommandTimeout { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// Throws if the options are unusable. Called by the DI extensions at registration time so a
|
||||
/// misconfigured node fails at startup rather than on the first secret resolve.
|
||||
/// </summary>
|
||||
/// <exception cref="InvalidOperationException">The connection string is missing, or an interval is not positive.</exception>
|
||||
public void Validate()
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(ConnectionString))
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
"SQL-Server secret store requires a ConnectionString (Secrets:SqlServer:ConnectionString).");
|
||||
}
|
||||
|
||||
if (SyncInterval <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"SQL-Server secret replication SyncInterval must be positive (was {SyncInterval}).");
|
||||
}
|
||||
|
||||
if (CommandTimeout <= TimeSpan.Zero)
|
||||
{
|
||||
throw new InvalidOperationException(
|
||||
$"SQL-Server secret store CommandTimeout must be positive (was {CommandTimeout}).");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Strict allow-list for a SQL identifier that will be interpolated into DDL or query text.
|
||||
/// Rejects everything that is not <c>[A-Za-z_][A-Za-z0-9_]{0,127}</c> — no brackets, quotes,
|
||||
/// dots, or spaces — so no bracket-escape (<c>]]</c>) trick can break out of the quoting applied
|
||||
/// at the use site. Uses the ASCII-only character tests deliberately, so homoglyph and
|
||||
/// full-width look-alikes are rejected rather than accepted as "letters".
|
||||
/// </summary>
|
||||
/// <param name="value">The candidate identifier.</param>
|
||||
/// <returns>The same value, once validated.</returns>
|
||||
/// <exception cref="ArgumentException">The value is not a plain identifier.</exception>
|
||||
internal static string ValidateIdentifier(string value)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(value);
|
||||
|
||||
if (value.Length > 128)
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"SQL schema name must be at most 128 characters (was {value.Length}).", nameof(value));
|
||||
}
|
||||
|
||||
if (!char.IsAsciiLetter(value[0]) && value[0] != '_')
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"SQL schema name must start with a letter or underscore (was '{value}').", nameof(value));
|
||||
}
|
||||
|
||||
foreach (char c in value)
|
||||
{
|
||||
if (!char.IsAsciiLetterOrDigit(c) && c != '_')
|
||||
{
|
||||
throw new ArgumentException(
|
||||
$"SQL schema name may contain only letters, digits, and underscore (was '{value}').",
|
||||
nameof(value));
|
||||
}
|
||||
}
|
||||
|
||||
return value;
|
||||
}
|
||||
}
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Schema constants and DDL for the shared SQL-Server secret store. Mirrors
|
||||
/// <c>ZB.MOM.WW.Secrets.Sqlite.SqliteSecretsSchema</c> column-for-column so a row means exactly the
|
||||
/// same thing in either store.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Timestamps are ISO-8601 <c>nvarchar</c>, not <c>datetimeoffset</c>.</b> This looks wrong for a
|
||||
/// SQL-Server schema and is deliberate: the same <see cref="ZB.MOM.WW.Secrets.Abstractions.StoredSecret"/>
|
||||
/// rows replicate between this store and SQLite (which stores <c>"O"</c>-format text), and
|
||||
/// last-writer-wins compares those timestamps for equality. <c>datetimeoffset</c> rounds to 100ns
|
||||
/// while .NET <c>DateTimeOffset</c> ticks are also 100ns but the round-trip through the driver is
|
||||
/// not guaranteed bit-identical across providers — a sub-tick difference would turn a tie into a
|
||||
/// spurious "newer", and two nodes would then overwrite each other forever. Storing the exact
|
||||
/// round-trippable text both providers already agree on removes the failure mode entirely.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The cost is that <c>ORDER BY updated_utc</c> is a lexicographic sort. That is still chronological
|
||||
/// for <c>"O"</c>-format UTC strings (fixed-width, zero-padded, always <c>+00:00</c>), which is what
|
||||
/// the store writes.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SqlServerSecretsSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// The schema version this build creates and supports. The migrator refuses a database whose
|
||||
/// recorded version is <em>newer</em> than this.
|
||||
/// </summary>
|
||||
public const int CurrentVersion = 1;
|
||||
|
||||
/// <summary>Default schema the secret tables live in.</summary>
|
||||
public const string DefaultSchemaName = "zbsecrets";
|
||||
|
||||
/// <summary>Name of the single-row table tracking the applied schema version.</summary>
|
||||
public const string SchemaVersionTable = "schema_version";
|
||||
|
||||
/// <summary>Name of the table storing envelope-encrypted secret records.</summary>
|
||||
public const string SecretTable = "secret";
|
||||
|
||||
/// <summary>
|
||||
/// Returns the idempotent DDL that provisions the schema, both tables, and the supporting index
|
||||
/// in <paramref name="schemaName"/>.
|
||||
/// </summary>
|
||||
/// <param name="schemaName">The already-validated SQL schema name to create the objects in.</param>
|
||||
/// <returns>The DDL batch.</returns>
|
||||
/// <remarks>
|
||||
/// <paramref name="schemaName"/> is interpolated, not parameterized — T-SQL has no parameter form
|
||||
/// for an object name. It is re-validated here against the same strict identifier allow-list
|
||||
/// <see cref="SqlServerSecretsOptions.SchemaName"/> enforces, and bracket-quoted below. The
|
||||
/// duplicate check is deliberate: this method is <c>public</c>, so it must defend itself rather
|
||||
/// than trust that every caller routed through the options type.
|
||||
/// </remarks>
|
||||
/// <exception cref="ArgumentException"><paramref name="schemaName"/> is not a plain identifier.</exception>
|
||||
public static string CreateSchemaDdl(string schemaName)
|
||||
{
|
||||
SqlServerSecretsOptions.ValidateIdentifier(schemaName);
|
||||
|
||||
return $"""
|
||||
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}')
|
||||
EXEC(N'CREATE SCHEMA [{schemaName}]');
|
||||
|
||||
IF OBJECT_ID(N'[{schemaName}].[{SchemaVersionTable}]', N'U') IS NULL
|
||||
CREATE TABLE [{schemaName}].[{SchemaVersionTable}] (
|
||||
id INT NOT NULL PRIMARY KEY CHECK (id = 1),
|
||||
version INT NOT NULL,
|
||||
applied_utc NVARCHAR(33) NOT NULL
|
||||
);
|
||||
|
||||
IF OBJECT_ID(N'[{schemaName}].[{SecretTable}]', N'U') IS NULL
|
||||
CREATE TABLE [{schemaName}].[{SecretTable}] (
|
||||
name NVARCHAR(450) NOT NULL PRIMARY KEY,
|
||||
description NVARCHAR(MAX) NULL,
|
||||
content_type NVARCHAR(64) NOT NULL,
|
||||
ciphertext VARBINARY(MAX) NOT NULL,
|
||||
nonce VARBINARY(MAX) NOT NULL,
|
||||
tag VARBINARY(MAX) NOT NULL,
|
||||
wrapped_dek VARBINARY(MAX) NOT NULL,
|
||||
wrap_nonce VARBINARY(MAX) NOT NULL,
|
||||
wrap_tag VARBINARY(MAX) NOT NULL,
|
||||
kek_id NVARCHAR(256) NOT NULL,
|
||||
revision BIGINT NOT NULL CONSTRAINT DF_secret_revision DEFAULT 0,
|
||||
is_deleted BIT NOT NULL CONSTRAINT DF_secret_is_deleted DEFAULT 0,
|
||||
deleted_utc NVARCHAR(33) NULL,
|
||||
created_utc NVARCHAR(33) NOT NULL,
|
||||
updated_utc NVARCHAR(33) NOT NULL,
|
||||
created_by NVARCHAR(256) NULL,
|
||||
updated_by NVARCHAR(256) NULL
|
||||
);
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM sys.indexes
|
||||
WHERE name = N'ix_secret_updated_utc'
|
||||
AND object_id = OBJECT_ID(N'[{schemaName}].[{SecretTable}]'))
|
||||
CREATE INDEX ix_secret_updated_utc
|
||||
ON [{schemaName}].[{SecretTable}] (updated_utc);
|
||||
""";
|
||||
}
|
||||
}
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
using System.Data;
|
||||
using System.Globalization;
|
||||
using Microsoft.Data.SqlClient;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
|
||||
/// <summary>
|
||||
/// Creates the shared SQL-Server secret schema and records the applied version. Idempotent: safe to
|
||||
/// run on every node at every startup, including concurrently. Refuses to run against a database
|
||||
/// whose recorded version is newer than this build supports.
|
||||
/// </summary>
|
||||
/// <param name="connectionFactory">Factory for connections to the shared secret database.</param>
|
||||
public sealed class SqlServerSecretsStoreMigrator(SecretsSqlServerConnectionFactory connectionFactory)
|
||||
: ISecretsStoreMigrator
|
||||
{
|
||||
// Concurrent Serializable DDL against the system catalogs can pick a deadlock victim (error
|
||||
// 1205) even though the IF NOT EXISTS guards make the logic itself safe. Every node runs this at
|
||||
// every startup, so a virgin database being provisioned by N nodes at once is the NORMAL case,
|
||||
// not a rare race — and an unretried victim aborts that node's whole host at startup. Retrying
|
||||
// is correct precisely because the migration is idempotent.
|
||||
private const int MaxDeadlockRetries = 5;
|
||||
private const int SqlDeadlockVictimErrorNumber = 1205;
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <exception cref="SecretStoreMigrationException">
|
||||
/// The recorded schema version is newer than <see cref="SqlServerSecretsSchema.CurrentVersion"/>.
|
||||
/// </exception>
|
||||
public async Task MigrateAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
for (int attempt = 1; ; attempt++)
|
||||
{
|
||||
try
|
||||
{
|
||||
await MigrateOnceAsync(cancellationToken).ConfigureAwait(false);
|
||||
return;
|
||||
}
|
||||
catch (SqlException ex)
|
||||
when (ex.Number == SqlDeadlockVictimErrorNumber && attempt < MaxDeadlockRetries)
|
||||
{
|
||||
// Staggered back-off so two victims do not retry in lockstep and deadlock again.
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(100 * attempt), cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async Task MigrateOnceAsync(CancellationToken cancellationToken)
|
||||
{
|
||||
string schema = connectionFactory.SchemaName;
|
||||
|
||||
await using SqlConnection connection =
|
||||
await connectionFactory.OpenConnectionAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
// Serializable: unlike the single-writer SQLite store, EVERY node in the cluster runs this
|
||||
// migration at startup, so concurrent first-boots are the normal case rather than a rare
|
||||
// race. The isolation level plus the IF NOT EXISTS guards make a simultaneous provision by
|
||||
// two nodes converge on one schema instead of one of them failing with "object exists".
|
||||
await using SqlTransaction transaction = (SqlTransaction)
|
||||
await connection.BeginTransactionAsync(IsolationLevel.Serializable, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
int existingVersion =
|
||||
await ReadExistingSchemaVersionAsync(connection, transaction, schema, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
if (existingVersion > SqlServerSecretsSchema.CurrentVersion)
|
||||
{
|
||||
throw new SecretStoreMigrationException(
|
||||
$"Shared secret database schema version {existingVersion} is newer than supported version " +
|
||||
$"{SqlServerSecretsSchema.CurrentVersion}. Upgrade this node before it writes to the shared store.");
|
||||
}
|
||||
|
||||
await using (SqlCommand ddl = connectionFactory.CreateCommand(
|
||||
connection, SqlServerSecretsSchema.CreateSchemaDdl(schema), transaction))
|
||||
{
|
||||
await ddl.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
await using (SqlCommand version = connectionFactory.CreateCommand(
|
||||
connection,
|
||||
$"""
|
||||
MERGE [{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}] AS target
|
||||
USING (SELECT 1 AS id, @version AS version, @applied_utc AS applied_utc) AS source
|
||||
ON target.id = source.id
|
||||
WHEN MATCHED THEN
|
||||
UPDATE SET version = source.version, applied_utc = source.applied_utc
|
||||
WHEN NOT MATCHED THEN
|
||||
INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
|
||||
""",
|
||||
transaction))
|
||||
{
|
||||
version.Parameters.AddWithValue("@version", SqlServerSecretsSchema.CurrentVersion);
|
||||
version.Parameters.AddWithValue("@applied_utc", DateTimeOffset.UtcNow.ToString("O"));
|
||||
await version.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
await transaction.CommitAsync(cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
private static async Task<int> ReadExistingSchemaVersionAsync(
|
||||
SqlConnection connection,
|
||||
SqlTransaction transaction,
|
||||
string schema,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
await using SqlCommand command = connection.CreateCommand();
|
||||
command.Transaction = transaction;
|
||||
|
||||
// OBJECT_ID returns NULL for an absent table, so this reads 0 on a virgin database without
|
||||
// needing the table to exist first.
|
||||
command.CommandText = $"""
|
||||
IF OBJECT_ID(N'[{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}]', N'U') IS NULL
|
||||
SELECT 0;
|
||||
ELSE
|
||||
SELECT ISNULL((SELECT version FROM [{schema}].[{SqlServerSecretsSchema.SchemaVersionTable}] WHERE id = 1), 0);
|
||||
""";
|
||||
|
||||
object? version = await command.ExecuteScalarAsync(cancellationToken).ConfigureAwait(false);
|
||||
|
||||
return version is null || version == DBNull.Value
|
||||
? 0
|
||||
: Convert.ToInt32(version, CultureInfo.InvariantCulture);
|
||||
}
|
||||
}
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net10.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
|
||||
<GenerateDocumentationFile>true</GenerateDocumentationFile>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>true</IsPackable>
|
||||
<PackageId>ZB.MOM.WW.Secrets.Replicator.SqlServer</PackageId>
|
||||
<Authors>ZB.MOM.WW</Authors>
|
||||
<Description>Shared SQL-Server secret store and hub replication for the ZB.MOM.WW SCADA family — cluster-wide, ciphertext-only secrets.</Description>
|
||||
<PackageProjectUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets</PackageProjectUrl>
|
||||
<RepositoryUrl>https://gitea.dohertylan.com/dohertj2/zb-mom-ww-secrets</RepositoryUrl>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.Secrets\ZB.MOM.WW.Secrets.csproj" />
|
||||
<ProjectReference Include="..\ZB.MOM.WW.Secrets.Abstractions\ZB.MOM.WW.Secrets.Abstractions.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Data.SqlClient" />
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -4,8 +4,9 @@ namespace ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// The single-node default <see cref="ISecretReplicator"/>: publishing a row is a no-op because
|
||||
/// there are no peers to broadcast to. The deferred <c>ZB.MOM.WW.Secrets.Akka</c> package replaces
|
||||
/// this registration with a real cluster replicator for clustered deployments.
|
||||
/// there are no peers to broadcast to. The <c>ZB.MOM.WW.Secrets.Replication.Akka</c> and
|
||||
/// <c>ZB.MOM.WW.Secrets.Replication.SqlServer</c> packages replace this registration with a real
|
||||
/// replicator for clustered deployments.
|
||||
/// </summary>
|
||||
public sealed class NoOpSecretReplicator : ISecretReplicator
|
||||
{
|
||||
|
||||
+8
-3
@@ -1,16 +1,21 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Runs the secrets SQLite schema migration at application startup when
|
||||
/// Runs the secret-store schema migration at application startup when
|
||||
/// <see cref="SecretsOptions.RunMigrationsOnStartup"/> is <see langword="true"/>. The migration is
|
||||
/// idempotent, so repeated restarts are safe.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Depends on the <see cref="ISecretsStoreMigrator"/> seam rather than a concrete migrator, so the
|
||||
/// same hosted service provisions the local SQLite store or the shared SQL-Server store depending
|
||||
/// only on which migrator the application registered.
|
||||
/// </remarks>
|
||||
internal sealed class SecretsMigrationHostedService(
|
||||
SqliteSecretsStoreMigrator migrator,
|
||||
ISecretsStoreMigrator migrator,
|
||||
IOptions<SecretsOptions> options) : IHostedService
|
||||
{
|
||||
/// <inheritdoc />
|
||||
|
||||
+14
-2
@@ -51,9 +51,15 @@ public static class SecretsServiceCollectionExtensions
|
||||
services.TryAddSingleton<ISecretCipher>(sp =>
|
||||
new AesGcmEnvelopeCipher(sp.GetRequiredService<IMasterKeyProvider>()));
|
||||
|
||||
services.TryAddSingleton<ISecretStore>(sp =>
|
||||
// Registered BOTH concretely and behind the interface, forwarding to the one singleton.
|
||||
// The concrete registration is what lets a replication package resolve the *undecorated*
|
||||
// local store when it replaces ISecretStore with a decorator — asking the container for
|
||||
// ISecretStore there would resolve the decorator itself and recurse forever.
|
||||
services.TryAddSingleton(sp =>
|
||||
new SqliteSecretStore(sp.GetRequiredService<SecretsSqliteConnectionFactory>()));
|
||||
|
||||
services.TryAddSingleton<ISecretStore>(sp => sp.GetRequiredService<SqliteSecretStore>());
|
||||
|
||||
services.TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>();
|
||||
|
||||
// ONE shared resolver instance backs both the read seam (ISecretResolver) and the
|
||||
@@ -81,10 +87,16 @@ public static class SecretsServiceCollectionExtensions
|
||||
services.TryAddSingleton<ISecretCacheInvalidator>(sp => sp.GetRequiredService<DefaultSecretResolver>());
|
||||
|
||||
// Migrator: singleton, constructed from the already-registered connection factory. Needed
|
||||
// before any store operations so the schema exists.
|
||||
// before any store operations so the schema exists. Registered BOTH concretely (for callers
|
||||
// that want the SQLite migrator specifically) and behind the ISecretsStoreMigrator seam that
|
||||
// the hosted service and CLI consume — a replication package that swaps the store registers
|
||||
// its own ISecretsStoreMigrator BEFORE calling AddZbSecrets, and TryAdd leaves it in place.
|
||||
services.TryAddSingleton(sp =>
|
||||
new SqliteSecretsStoreMigrator(sp.GetRequiredService<SecretsSqliteConnectionFactory>()));
|
||||
|
||||
services.TryAddSingleton<ISecretsStoreMigrator>(sp =>
|
||||
sp.GetRequiredService<SqliteSecretsStoreMigrator>());
|
||||
|
||||
// Hosted service that runs migrations on startup when SecretsOptions.RunMigrationsOnStartup.
|
||||
services.AddHostedService<SecretsMigrationHostedService>();
|
||||
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replication;
|
||||
|
||||
/// <summary>
|
||||
/// Decorates a local <see cref="ISecretStore"/> so every successful local write is published to
|
||||
/// peers through <see cref="ISecretReplicator"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is what turns the <see cref="ISecretReplicator"/> seam from a declaration into behaviour.
|
||||
/// Decorating the store — rather than calling the replicator from each write site — means the CLI,
|
||||
/// the Blazor UI, and any future caller all replicate automatically, and no new write path can
|
||||
/// forget to.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Publishing is best-effort.</b> A transport failure is logged and swallowed: the local write is
|
||||
/// already durable, and refusing it because a peer was unreachable would make every node a single
|
||||
/// point of failure for every other. Propagation is then guaranteed by the transport's periodic
|
||||
/// anti-entropy sweep instead, which is why every transport in this library reconciles
|
||||
/// <em>bidirectionally</em> — a dropped publish is repaired by the next sweep rather than lost.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <see cref="ApplyReplicatedAsync"/> deliberately does not publish (that would echo a peer's row
|
||||
/// straight back and, across three or more nodes, never stop). Neither does
|
||||
/// <see cref="ApplyRewrapAsync"/>: a re-wrap does not bump the revision, so it is invisible to
|
||||
/// last-writer-wins by design and a peer would discard it — KEK rotation is run once per independent
|
||||
/// store instead.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="inner">The local store doing the actual persistence.</param>
|
||||
/// <param name="replicator">The transport that publishes rows to peers.</param>
|
||||
/// <param name="logger">Logger for publish failures.</param>
|
||||
public sealed class ReplicatingSecretStore(
|
||||
ISecretStore inner,
|
||||
ISecretReplicator replicator,
|
||||
ILogger<ReplicatingSecretStore> logger) : ISecretStore
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task<StoredSecret?> GetAsync(SecretName name, CancellationToken ct) =>
|
||||
inner.GetAsync(name, ct);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<SecretMetadata>> ListAsync(bool includeDeleted, CancellationToken ct) =>
|
||||
inner.ListAsync(includeDeleted, ct);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
inner.GetManifestAsync(ct);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) =>
|
||||
inner.ApplyReplicatedAsync(row, ct);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<bool> ApplyRewrapAsync(
|
||||
StoredSecret rewrappedRow, byte[] expectedCurrentWrappedDek, CancellationToken ct) =>
|
||||
inner.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek, ct);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task UpsertAsync(StoredSecret row, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(row);
|
||||
|
||||
await inner.UpsertAsync(row, ct).ConfigureAwait(false);
|
||||
await PublishCurrentAsync(row.Name, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> DeleteAsync(SecretName name, string? actor, CancellationToken ct)
|
||||
{
|
||||
bool deleted = await inner.DeleteAsync(name, actor, ct).ConfigureAwait(false);
|
||||
|
||||
if (deleted)
|
||||
{
|
||||
// The tombstone is the thing being replicated — a delete propagates as a newer row, not
|
||||
// as an absence, so peers converge on "deleted" instead of resurrecting it.
|
||||
await PublishCurrentAsync(name, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return deleted;
|
||||
}
|
||||
|
||||
// Re-reads the row the store actually persisted and publishes THAT, never the caller's input:
|
||||
// the store assigns the revision and updated_utc, and those two fields are exactly what peers
|
||||
// order by. Publishing the caller's copy would broadcast a row whose ordering keys are wrong.
|
||||
private async Task PublishCurrentAsync(SecretName name, CancellationToken ct)
|
||||
{
|
||||
StoredSecret? persisted = await inner.GetAsync(name, ct).ConfigureAwait(false);
|
||||
|
||||
if (persisted is null)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await replicator.PublishAsync(persisted, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Deliberately swallowed — see the class remarks. Anti-entropy repairs it.
|
||||
logger.LogWarning(
|
||||
ex,
|
||||
"Failed to publish secret {SecretName} to peers; the local write succeeded and " +
|
||||
"anti-entropy will propagate it on the next sweep.",
|
||||
name.Value);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replication;
|
||||
|
||||
/// <summary>
|
||||
/// Transport-agnostic anti-entropy: compares a peer's manifest against the local store, pulls only
|
||||
/// the rows the peer holds a strictly newer copy of, and applies them last-writer-wins.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// This is the half of replication that is pure logic, deliberately separated from <em>how</em> the
|
||||
/// peer is reached. The SQL-Server package points it at a shared hub database; the Akka package
|
||||
/// points it at a cluster peer over pub/sub. Both get identical convergence behaviour, and the
|
||||
/// interesting edge cases (ties, tombstones, a peer that is behind) are testable offline without a
|
||||
/// database or a cluster.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Reconciliation is a pull, never a push: it only ever writes rows the peer says are newer, so
|
||||
/// running it against a peer that is behind is a no-op rather than a regression. It is also safe to
|
||||
/// run concurrently with live replication — <see cref="ISecretStore.ApplyReplicatedAsync"/> re-checks
|
||||
/// the same predicate under its own transaction, so a row that a broadcast delivered first is simply
|
||||
/// ignored the second time.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="local">The local store to bring up to date.</param>
|
||||
/// <param name="cacheInvalidator">
|
||||
/// Optional resolver-cache seam. When supplied, every applied name is evicted so a node picks up a
|
||||
/// peer's write within the reconcile interval instead of serving a stale plaintext for the rest of
|
||||
/// the cache TTL.
|
||||
/// </param>
|
||||
/// <param name="onRowFailed">
|
||||
/// Optional callback invoked when a single replicated row could not be applied. The row is skipped
|
||||
/// and the rest of the batch proceeds; without a callback the failure is silent, so transports
|
||||
/// should supply one that logs.
|
||||
/// </param>
|
||||
public sealed class SecretReplicationReconciler(
|
||||
ISecretStore local,
|
||||
ISecretCacheInvalidator? cacheInvalidator = null,
|
||||
Action<SecretName, Exception>? onRowFailed = null)
|
||||
{
|
||||
/// <summary>
|
||||
/// Returns the names for which <paramref name="remote"/> holds a strictly newer row than
|
||||
/// <paramref name="localManifest"/> — including names absent locally, and including tombstones
|
||||
/// (a delete propagates as a newer row, not as an absence).
|
||||
/// </summary>
|
||||
/// <param name="localManifest">The local store's manifest.</param>
|
||||
/// <param name="remote">The peer's manifest.</param>
|
||||
/// <returns>The names to pull from the peer, in manifest order.</returns>
|
||||
public static IReadOnlyList<SecretName> ComputePullSet(
|
||||
IReadOnlyList<SecretManifestEntry> localManifest,
|
||||
IReadOnlyList<SecretManifestEntry> remote)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(localManifest);
|
||||
ArgumentNullException.ThrowIfNull(remote);
|
||||
|
||||
Dictionary<string, SecretManifestEntry> localByName =
|
||||
localManifest.ToDictionary(entry => entry.Name.Value, StringComparer.Ordinal);
|
||||
|
||||
var pull = new List<SecretName>();
|
||||
|
||||
foreach (SecretManifestEntry remoteEntry in remote)
|
||||
{
|
||||
// Absent locally, or the peer's copy wins LWW. A name we hold and the peer does not is
|
||||
// deliberately NOT handled here: this side pulls, the peer's own reconcile pushes.
|
||||
if (!localByName.TryGetValue(remoteEntry.Name.Value, out SecretManifestEntry? localEntry) ||
|
||||
SecretLastWriterWins.IsNewer(remoteEntry, localEntry))
|
||||
{
|
||||
pull.Add(remoteEntry.Name);
|
||||
}
|
||||
}
|
||||
|
||||
return pull;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the names the local store holds a strictly newer row for than <paramref name="remote"/>
|
||||
/// — the mirror of <see cref="ComputePullSet"/>, used to push local writes a peer is missing.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every transport here reconciles in <b>both</b> directions, and this is why: publishing a write
|
||||
/// is best-effort (see <c>ReplicatingSecretStore</c>), so without a push sweep a write made while
|
||||
/// a peer was unreachable would sit on one node forever — the peer never learns the name exists,
|
||||
/// so its own pull can never ask for it.
|
||||
/// </remarks>
|
||||
/// <param name="localManifest">The local store's manifest.</param>
|
||||
/// <param name="remote">The peer's manifest.</param>
|
||||
/// <returns>The names to push to the peer.</returns>
|
||||
public static IReadOnlyList<SecretName> ComputePushSet(
|
||||
IReadOnlyList<SecretManifestEntry> localManifest,
|
||||
IReadOnlyList<SecretManifestEntry> remote) =>
|
||||
// Exactly the pull computation with the two sides swapped: "what does the peer need from me"
|
||||
// is "what would I need, if I were the peer".
|
||||
ComputePullSet(remote, localManifest);
|
||||
|
||||
/// <summary>
|
||||
/// Reconciles the local store against <paramref name="remoteManifest"/>, fetching and applying
|
||||
/// every row the peer holds a newer copy of.
|
||||
/// </summary>
|
||||
/// <param name="remoteManifest">The peer's manifest.</param>
|
||||
/// <param name="fetchAsync">
|
||||
/// Fetches the encrypted rows for the requested names from the peer. May return fewer rows than
|
||||
/// requested (a row deleted between manifest and fetch); extra rows are applied normally.
|
||||
/// </param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The number of rows applied locally.</returns>
|
||||
public async Task<int> ReconcileAsync(
|
||||
IReadOnlyList<SecretManifestEntry> remoteManifest,
|
||||
Func<IReadOnlyList<SecretName>, CancellationToken, Task<IReadOnlyList<StoredSecret>>> fetchAsync,
|
||||
CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(remoteManifest);
|
||||
ArgumentNullException.ThrowIfNull(fetchAsync);
|
||||
|
||||
IReadOnlyList<SecretManifestEntry> localManifest =
|
||||
await local.GetManifestAsync(ct).ConfigureAwait(false);
|
||||
|
||||
IReadOnlyList<SecretName> pull = ComputePullSet(localManifest, remoteManifest);
|
||||
|
||||
if (pull.Count == 0)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
int applied = 0;
|
||||
|
||||
// Chunked, not fetched in one shot. A cold-starting node's pull set is the peer's ENTIRE
|
||||
// inventory, and an unbounded fetch breaks on real backends — SQL Server caps a command at
|
||||
// 2100 parameters, so a hub with more than that many secrets would fail the same way every
|
||||
// interval and the node would never receive a single row. Chunking also bounds the size of
|
||||
// one replication message on the wire.
|
||||
foreach (IReadOnlyList<SecretName> batch in Chunk(pull, FetchBatchSize))
|
||||
{
|
||||
IReadOnlyList<StoredSecret> rows = await fetchAsync(batch, ct).ConfigureAwait(false);
|
||||
applied += await ApplyAsync(rows, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return applied;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Names fetched per round trip. Well under SQL Server's 2100-parameter command limit, and small
|
||||
/// enough that one replication message stays a sane size.
|
||||
/// </summary>
|
||||
internal const int FetchBatchSize = 500;
|
||||
|
||||
private static IEnumerable<IReadOnlyList<SecretName>> Chunk(
|
||||
IReadOnlyList<SecretName> names, int size)
|
||||
{
|
||||
for (int offset = 0; offset < names.Count; offset += size)
|
||||
{
|
||||
yield return [.. names.Skip(offset).Take(size)];
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the local rows for <paramref name="names"/>, skipping any that vanished between the
|
||||
/// manifest and the read. Used by transports to assemble a push batch.
|
||||
/// </summary>
|
||||
/// <param name="names">The names to read.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The encrypted rows that still exist locally.</returns>
|
||||
public async Task<IReadOnlyList<StoredSecret>> ReadLocalAsync(
|
||||
IReadOnlyList<SecretName> names, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(names);
|
||||
|
||||
var rows = new List<StoredSecret>(names.Count);
|
||||
|
||||
foreach (SecretName name in names)
|
||||
{
|
||||
StoredSecret? row = await local.GetAsync(name, ct).ConfigureAwait(false);
|
||||
|
||||
if (row is not null)
|
||||
{
|
||||
rows.Add(row);
|
||||
}
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies encrypted rows received from a peer, last-writer-wins, evicting the resolver cache for
|
||||
/// each. Used both by <see cref="ReconcileAsync"/> and by transports that receive live broadcasts.
|
||||
/// </summary>
|
||||
/// <param name="rows">The encrypted rows to apply.</param>
|
||||
/// <param name="ct">A token to cancel the operation.</param>
|
||||
/// <returns>The number of rows passed to the store.</returns>
|
||||
public async Task<int> ApplyAsync(IReadOnlyList<StoredSecret> rows, CancellationToken ct)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(rows);
|
||||
|
||||
int applied = 0;
|
||||
|
||||
foreach (StoredSecret row in rows)
|
||||
{
|
||||
try
|
||||
{
|
||||
await local.ApplyReplicatedAsync(row, ct).ConfigureAwait(false);
|
||||
applied++;
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Isolated per row, because a batch is not a transaction and must not be all-or-
|
||||
// nothing. One row a peer cannot supply cleanly (a hostile or buggy peer, a
|
||||
// constraint violation) would otherwise abandon every row after it — and since the
|
||||
// pull set is recomputed identically each sweep, in the same manifest order, the
|
||||
// SAME row would poison the SAME batch forever and every name behind it would never
|
||||
// converge. Drop the row, keep the batch.
|
||||
onRowFailed?.Invoke(row.Name, ex);
|
||||
continue;
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Evicted unconditionally: the store may have ignored the row as stale, but an
|
||||
// eviction only costs the next resolve a re-read, whereas skipping one when the row
|
||||
// DID land serves a stale plaintext until the TTL expires.
|
||||
cacheInvalidator?.Invalidate(row.Name);
|
||||
}
|
||||
}
|
||||
|
||||
return applied;
|
||||
}
|
||||
}
|
||||
@@ -200,12 +200,9 @@ public sealed class SqliteSecretStore(SecretsSqliteConnectionFactory connectionF
|
||||
DateTimeOffset localUpdated = ParseUtc(reader.GetString(0));
|
||||
long localRevision = reader.GetInt64(1);
|
||||
|
||||
// Incoming wins only if strictly newer by (updated_utc, then revision).
|
||||
bool incomingIsNewer =
|
||||
row.UpdatedUtc > localUpdated ||
|
||||
(row.UpdatedUtc == localUpdated && row.Revision > localRevision);
|
||||
|
||||
if (!incomingIsNewer)
|
||||
// Incoming wins only if strictly newer by (updated_utc, then revision). Routed through
|
||||
// the shared predicate so every store agrees on ties — see SecretLastWriterWins.
|
||||
if (!SecretLastWriterWins.IsNewer(row.UpdatedUtc, row.Revision, localUpdated, localRevision))
|
||||
{
|
||||
await transaction.CommitAsync(ct).ConfigureAwait(false);
|
||||
return;
|
||||
|
||||
@@ -10,6 +10,7 @@ namespace ZB.MOM.WW.Secrets.Sqlite;
|
||||
/// newer than this build supports.
|
||||
/// </summary>
|
||||
public sealed class SqliteSecretsStoreMigrator(SecretsSqliteConnectionFactory connectionFactory)
|
||||
: ISecretsStoreMigrator
|
||||
{
|
||||
/// <summary>Applies the schema migration to the secret store.</summary>
|
||||
/// <param name="cancellationToken">Cancellation token.</param>
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration" />
|
||||
<PackageReference Include="Microsoft.Extensions.Hosting.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
<PackageReference Include="ZB.MOM.WW.Audit" />
|
||||
</ItemGroup>
|
||||
|
||||
|
||||
+81
@@ -0,0 +1,81 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The DTO is the trust boundary for anything a peer sends. Every rejection here must surface as an
|
||||
/// <see cref="ArgumentException"/>, because that is the exception type the receiving actor filters
|
||||
/// on — anything else escapes its handler, restarts the actor, and (since the row is redelivered)
|
||||
/// loops.
|
||||
/// </summary>
|
||||
public sealed class HostileWireInputTests
|
||||
{
|
||||
private static SecretRowDto Valid() => new()
|
||||
{
|
||||
Name = "app/ok",
|
||||
ContentType = "Text",
|
||||
Ciphertext = [1],
|
||||
Nonce = [2],
|
||||
Tag = [3],
|
||||
WrappedDek = [4],
|
||||
WrapNonce = [5],
|
||||
WrapTag = [6],
|
||||
KekId = "sha256:x",
|
||||
Revision = 0,
|
||||
CreatedUtc = "2026-01-01T00:00:00.0000000+00:00",
|
||||
UpdatedUtc = "2026-01-01T00:00:00.0000000+00:00",
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void A_valid_row_materializes()
|
||||
{
|
||||
StoredSecret row = Valid().ToStoredSecret();
|
||||
|
||||
Assert.Equal("app/ok", row.Name.Value);
|
||||
Assert.Equal(SecretContentType.Text, row.ContentType);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("../../etc/passwd")]
|
||||
[InlineData("/rooted")]
|
||||
[InlineData("has space")]
|
||||
[InlineData("")]
|
||||
public void A_malformed_name_is_rejected(string name)
|
||||
{
|
||||
Assert.Throws<ArgumentException>(() => (Valid() with { Name = name }).ToStoredSecret());
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// An in-range number names no member — Enum.Parse would happily produce (SecretContentType)4096
|
||||
// and the store would persist it for later readers that assume a defined value.
|
||||
[InlineData("4096")]
|
||||
// Overflows the underlying type: Enum.Parse throws OverflowException, which is NOT an
|
||||
// ArgumentException, so it would escape the actor's filter and restart it.
|
||||
[InlineData("99999999999999999999")]
|
||||
[InlineData("NotAContentType")]
|
||||
[InlineData("")]
|
||||
public void An_unrecognized_content_type_is_rejected_as_an_ArgumentException(string contentType)
|
||||
{
|
||||
Assert.Throws<ArgumentException>(
|
||||
() => (Valid() with { ContentType = contentType }).ToStoredSecret());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_null_crypto_blob_is_rejected_at_the_boundary()
|
||||
{
|
||||
// `required byte[]` means "present in the payload", not "non-null" — an explicit JSON null
|
||||
// satisfies it. Caught here, or it blows up deep in the store's parameter binding instead.
|
||||
Assert.Throws<ArgumentException>(
|
||||
() => (Valid() with { Ciphertext = null! }).ToStoredSecret());
|
||||
Assert.Throws<ArgumentException>(
|
||||
() => (Valid() with { WrappedDek = null! }).ToStoredSecret());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_malformed_timestamp_is_rejected()
|
||||
{
|
||||
Assert.ThrowsAny<FormatException>(
|
||||
() => (Valid() with { UpdatedUtc = "not-a-date" }).ToStoredSecret());
|
||||
}
|
||||
}
|
||||
+160
@@ -0,0 +1,160 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Serialization;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Protocol;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Wire-contract tests. Secret ciphertext crosses the network as these bytes, so the round-trip is
|
||||
/// asserted field-by-field rather than by equality alone — a silently dropped crypto field would
|
||||
/// produce a row that stores fine and fails to decrypt much later, on another node.
|
||||
/// </summary>
|
||||
public sealed class SecretReplicationSerializerTests : TestKit
|
||||
{
|
||||
public SecretReplicationSerializerTests()
|
||||
: base(AkkaSecretsReplication.SerializationConfig)
|
||||
{
|
||||
}
|
||||
|
||||
private static StoredSecret Row(string name) => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
Description = "a description",
|
||||
ContentType = SecretContentType.ConnectionString,
|
||||
Ciphertext = [1, 2, 3, 250],
|
||||
Nonce = [4, 5, 6],
|
||||
Tag = [7, 8, 9],
|
||||
WrappedDek = [10, 11, 12],
|
||||
WrapNonce = [13, 14, 15],
|
||||
WrapTag = [16, 17, 18],
|
||||
KekId = "sha256:abc",
|
||||
Revision = 42,
|
||||
IsDeleted = true,
|
||||
DeletedUtc = new DateTimeOffset(2026, 3, 2, 1, 0, 0, TimeSpan.Zero),
|
||||
CreatedUtc = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero),
|
||||
UpdatedUtc = new DateTimeOffset(2026, 2, 1, 0, 0, 0, TimeSpan.Zero),
|
||||
CreatedBy = "alice",
|
||||
UpdatedBy = "bob",
|
||||
};
|
||||
|
||||
private object RoundTrip(object message)
|
||||
{
|
||||
Serializer serializer = Sys.Serialization.FindSerializerFor(message);
|
||||
Assert.IsType<SecretReplicationSerializer>(serializer);
|
||||
|
||||
byte[] bytes = serializer.ToBinary(message);
|
||||
string manifest = ((SerializerWithStringManifest)serializer).Manifest(message);
|
||||
|
||||
return ((SerializerWithStringManifest)serializer).FromBinary(bytes, manifest);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SecretRowsMessage_round_trips_every_field()
|
||||
{
|
||||
StoredSecret original = Row("app/db-conn");
|
||||
var message = new SecretRowsMessage([SecretRowDto.FromStoredSecret(original)]);
|
||||
|
||||
var restored = (SecretRowsMessage)RoundTrip(message);
|
||||
StoredSecret result = restored.Rows.Single().ToStoredSecret();
|
||||
|
||||
Assert.Equal(original.Name.Value, result.Name.Value);
|
||||
Assert.Equal(original.Description, result.Description);
|
||||
Assert.Equal(original.ContentType, result.ContentType);
|
||||
Assert.Equal(original.Ciphertext, result.Ciphertext);
|
||||
Assert.Equal(original.Nonce, result.Nonce);
|
||||
Assert.Equal(original.Tag, result.Tag);
|
||||
Assert.Equal(original.WrappedDek, result.WrappedDek);
|
||||
Assert.Equal(original.WrapNonce, result.WrapNonce);
|
||||
Assert.Equal(original.WrapTag, result.WrapTag);
|
||||
Assert.Equal(original.KekId, result.KekId);
|
||||
Assert.Equal(original.Revision, result.Revision);
|
||||
Assert.Equal(original.IsDeleted, result.IsDeleted);
|
||||
Assert.Equal(original.DeletedUtc, result.DeletedUtc);
|
||||
Assert.Equal(original.CreatedUtc, result.CreatedUtc);
|
||||
Assert.Equal(original.UpdatedUtc, result.UpdatedUtc);
|
||||
Assert.Equal(original.CreatedBy, result.CreatedBy);
|
||||
Assert.Equal(original.UpdatedBy, result.UpdatedBy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Timestamps_survive_the_wire_exactly()
|
||||
{
|
||||
// Last-writer-wins compares UpdatedUtc for EQUALITY to break revision ties. A serializer that
|
||||
// rounded to the millisecond would turn a tie into a spurious "newer" and two nodes would
|
||||
// then overwrite each other forever, so tick-exactness is a correctness requirement.
|
||||
StoredSecret original = Row("app/precise") with
|
||||
{
|
||||
UpdatedUtc = new DateTimeOffset(2026, 2, 1, 12, 34, 56, TimeSpan.Zero).AddTicks(1234567),
|
||||
};
|
||||
|
||||
var restored = (SecretRowsMessage)RoundTrip(
|
||||
new SecretRowsMessage([SecretRowDto.FromStoredSecret(original)]));
|
||||
|
||||
Assert.Equal(original.UpdatedUtc, restored.Rows.Single().ToStoredSecret().UpdatedUtc);
|
||||
Assert.Equal(original.UpdatedUtc.Ticks, restored.Rows.Single().ToStoredSecret().UpdatedUtc.Ticks);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ManifestAnnounce_round_trips()
|
||||
{
|
||||
var message = new SecretManifestAnnounce(
|
||||
[
|
||||
SecretManifestEntryDto.FromEntry(new SecretManifestEntry
|
||||
{
|
||||
Name = new SecretName("a/one"),
|
||||
Revision = 3,
|
||||
UpdatedUtc = new DateTimeOffset(2026, 5, 5, 5, 5, 5, TimeSpan.Zero),
|
||||
IsDeleted = true,
|
||||
}),
|
||||
]);
|
||||
|
||||
var restored = (SecretManifestAnnounce)RoundTrip(message);
|
||||
SecretManifestEntry entry = restored.Entries.Single().ToEntry();
|
||||
|
||||
Assert.Equal("a/one", entry.Name.Value);
|
||||
Assert.Equal(3, entry.Revision);
|
||||
Assert.True(entry.IsDeleted);
|
||||
Assert.Equal(new DateTimeOffset(2026, 5, 5, 5, 5, 5, TimeSpan.Zero), entry.UpdatedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PullRequest_round_trips()
|
||||
{
|
||||
var restored = (SecretPullRequest)RoundTrip(new SecretPullRequest(["a/one", "b/two"]));
|
||||
|
||||
Assert.Equal(["a/one", "b/two"], restored.Names);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void An_unknown_manifest_fails_loudly_rather_than_guessing()
|
||||
{
|
||||
// A mixed-version cluster must fail on an unrecognized payload, not mis-deserialize a secret.
|
||||
var serializer = (SerializerWithStringManifest)Sys.Serialization
|
||||
.FindSerializerFor(new SecretPullRequest([]));
|
||||
|
||||
Assert.Throws<System.Runtime.Serialization.SerializationException>(
|
||||
() => serializer.FromBinary([1, 2, 3], "zbs-from-the-future-v9"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_wire_contract_carries_no_field_that_could_hold_key_material()
|
||||
{
|
||||
// Structural guard on the trust boundary: the DTO's surface is fixed and reviewed. If someone
|
||||
// adds a property to SecretRowDto, this test fails and forces a deliberate look at whether
|
||||
// the new field is safe to put on the wire.
|
||||
string[] properties = [.. typeof(SecretRowDto)
|
||||
.GetProperties()
|
||||
.Select(p => p.Name)
|
||||
.Order(StringComparer.Ordinal)];
|
||||
|
||||
Assert.Equal(
|
||||
[
|
||||
"Ciphertext", "ContentType", "CreatedBy", "CreatedUtc", "DeletedUtc", "Description",
|
||||
"IsDeleted", "KekId", "Name", "Nonce", "Revision", "Tag", "UpdatedBy", "UpdatedUtc",
|
||||
"WrapNonce", "WrapTag", "WrappedDek",
|
||||
],
|
||||
properties);
|
||||
}
|
||||
}
|
||||
+354
@@ -0,0 +1,354 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Two real cluster nodes, two real SQLite stores, real remoting over loopback — the gate the G-7
|
||||
/// design called for, run in-process so it belongs to the ordinary offline suite instead of needing
|
||||
/// a deployed rig.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Nothing here is faked: each node runs its own <see cref="ActorSystem"/> with Akka.Remote on its
|
||||
/// own port, they form a cluster, and rows travel through the actual serializer and the actual
|
||||
/// distributed pub/sub mediator. A mock-based test of this behaviour would prove almost nothing —
|
||||
/// the failure modes worth catching (self-echo, sender loss across the mediator, tombstones that do
|
||||
/// not propagate, a partition that never re-converges) only appear when messages really cross nodes.
|
||||
/// </remarks>
|
||||
public sealed class TwoNodeClusterReplicationTests : IAsyncLifetime
|
||||
{
|
||||
private readonly List<string> _dbPaths = [];
|
||||
private ActorSystem _systemA = null!;
|
||||
private ActorSystem _systemB = null!;
|
||||
private SqliteSecretStore _storeA = null!;
|
||||
private SqliteSecretStore _storeB = null!;
|
||||
private ISecretStore _writableA = null!;
|
||||
private IActorRef _replicatorB = null!;
|
||||
|
||||
// Short so anti-entropy assertions do not dominate the suite's runtime.
|
||||
private static readonly TimeSpan AnnounceInterval = TimeSpan.FromMilliseconds(300);
|
||||
private static readonly TimeSpan Patience = TimeSpan.FromSeconds(20);
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
_storeA = CreateStore();
|
||||
_storeB = CreateStore();
|
||||
|
||||
// Port 0 lets the OS assign free ports, so parallel test runs cannot collide.
|
||||
_systemA = ActorSystem.Create("zb-secrets-cluster", ClusterConfig(port: 0));
|
||||
int portA = ClusterPort(_systemA);
|
||||
|
||||
// Node A seeds itself; node B joins it.
|
||||
_systemA.Dispose();
|
||||
_systemA = ActorSystem.Create("zb-secrets-cluster", ClusterConfig(portA, seedPort: portA));
|
||||
_systemB = ActorSystem.Create("zb-secrets-cluster", ClusterConfig(port: 0, seedPort: portA));
|
||||
|
||||
await AwaitClusterUpAsync(_systemA, expectedMembers: 2);
|
||||
await AwaitClusterUpAsync(_systemB, expectedMembers: 2);
|
||||
|
||||
IActorRef replicatorA = _systemA.ActorOf(
|
||||
SecretReplicationActor.Props(_storeA, null, AnnounceInterval), "zb-secret-replication");
|
||||
_replicatorB = _systemB.ActorOf(
|
||||
SecretReplicationActor.Props(_storeB, null, AnnounceInterval), "zb-secret-replication");
|
||||
|
||||
_writableA = new ReplicatingSecretStore(
|
||||
_storeA, new AkkaSecretReplicator(replicatorA), NullLogger<ReplicatingSecretStore>.Instance);
|
||||
|
||||
// The mediator gossips subscriptions between nodes; publishing before both sides are
|
||||
// registered would silently drop the message.
|
||||
await AwaitBothSubscribedAsync();
|
||||
}
|
||||
|
||||
private SqliteSecretStore CreateStore()
|
||||
{
|
||||
string path = Path.Combine(Path.GetTempPath(), $"zb-akka-{Guid.NewGuid():N}.db");
|
||||
_dbPaths.Add(path);
|
||||
var factory = new SecretsSqliteConnectionFactory(path);
|
||||
new SqliteSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None).GetAwaiter().GetResult();
|
||||
return new SqliteSecretStore(factory);
|
||||
}
|
||||
|
||||
private static Config ClusterConfig(int port, int? seedPort = null)
|
||||
{
|
||||
string seeds = seedPort is null
|
||||
? "[]"
|
||||
: $"[\"akka.tcp://zb-secrets-cluster@127.0.0.1:{seedPort}\"]";
|
||||
|
||||
return ConfigurationFactory.ParseString($$"""
|
||||
akka {
|
||||
loglevel = WARNING
|
||||
actor.provider = cluster
|
||||
remote.dot-netty.tcp {
|
||||
hostname = "127.0.0.1"
|
||||
public-hostname = "127.0.0.1"
|
||||
port = {{port}}
|
||||
}
|
||||
cluster {
|
||||
seed-nodes = {{seeds}}
|
||||
downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider"
|
||||
}
|
||||
}
|
||||
""").WithFallback(AkkaSecretsReplication.SerializationConfig);
|
||||
}
|
||||
|
||||
private static int ClusterPort(ActorSystem system) =>
|
||||
Cluster.Get(system).SelfAddress.Port
|
||||
?? throw new InvalidOperationException("Cluster address has no port.");
|
||||
|
||||
private static async Task AwaitClusterUpAsync(ActorSystem system, int expectedMembers)
|
||||
{
|
||||
Cluster cluster = Cluster.Get(system);
|
||||
DateTime deadline = DateTime.UtcNow + Patience;
|
||||
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expectedMembers)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await Task.Delay(100);
|
||||
}
|
||||
|
||||
throw new TimeoutException(
|
||||
$"Cluster did not reach {expectedMembers} Up members within {Patience}.");
|
||||
}
|
||||
|
||||
// Writes a throwaway secret on A and waits for it to land on B. That round trip is the only
|
||||
// reliable signal that pub/sub subscriptions have gossiped across both nodes.
|
||||
private async Task AwaitBothSubscribedAsync()
|
||||
{
|
||||
var probe = new SecretName("warmup/probe");
|
||||
DateTime deadline = DateTime.UtcNow + Patience;
|
||||
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
await _writableA.UpsertAsync(Row("warmup/probe", 0x01), CancellationToken.None);
|
||||
|
||||
if (await WaitForAsync(_storeB, probe, _ => true, TimeSpan.FromMilliseconds(500)))
|
||||
{
|
||||
await _storeA.DeleteAsync(probe, "warmup", CancellationToken.None);
|
||||
await _storeB.DeleteAsync(probe, "warmup", CancellationToken.None);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
throw new TimeoutException("Nodes did not establish pub/sub subscriptions in time.");
|
||||
}
|
||||
|
||||
private static StoredSecret Row(string name, byte marker) => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
ContentType = SecretContentType.Text,
|
||||
Ciphertext = [marker],
|
||||
Nonce = [1],
|
||||
Tag = [2],
|
||||
WrappedDek = [3],
|
||||
WrapNonce = [4],
|
||||
WrapTag = [5],
|
||||
KekId = "sha256:shared-kek",
|
||||
Revision = 0,
|
||||
CreatedUtc = DateTimeOffset.UtcNow,
|
||||
UpdatedUtc = DateTimeOffset.UtcNow,
|
||||
};
|
||||
|
||||
private static async Task<bool> WaitForAsync(
|
||||
ISecretStore store, SecretName name, Func<StoredSecret, bool> predicate, TimeSpan? timeout = null)
|
||||
{
|
||||
DateTime deadline = DateTime.UtcNow + (timeout ?? Patience);
|
||||
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
StoredSecret? row = await store.GetAsync(name, CancellationToken.None);
|
||||
|
||||
if (row is not null && predicate(row))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
await Task.Delay(50);
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_secret_written_on_node_A_becomes_resolvable_on_node_B()
|
||||
{
|
||||
var name = new SecretName("db/password");
|
||||
await _writableA.UpsertAsync(Row("db/password", 0xAA), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0xAA),
|
||||
"Node B never received the secret written on node A.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_ciphertext_arrives_byte_identical()
|
||||
{
|
||||
var name = new SecretName("app/exact");
|
||||
await _writableA.UpsertAsync(Row("app/exact", 0x7F), CancellationToken.None);
|
||||
|
||||
Assert.True(await WaitForAsync(_storeB, name, _ => true));
|
||||
|
||||
StoredSecret onA = (await _storeA.GetAsync(name, CancellationToken.None))!;
|
||||
StoredSecret onB = (await _storeB.GetAsync(name, CancellationToken.None))!;
|
||||
|
||||
Assert.Equal(onA.Ciphertext, onB.Ciphertext);
|
||||
Assert.Equal(onA.WrappedDek, onB.WrappedDek);
|
||||
Assert.Equal(onA.KekId, onB.KekId);
|
||||
// Verbatim: the peer keeps the originating node's revision and timestamp, or the row would
|
||||
// look newer on B than on A and bounce back.
|
||||
Assert.Equal(onA.Revision, onB.Revision);
|
||||
Assert.Equal(onA.UpdatedUtc, onB.UpdatedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_delete_on_node_A_tombstones_the_row_on_node_B()
|
||||
{
|
||||
var name = new SecretName("app/doomed");
|
||||
await _writableA.UpsertAsync(Row("app/doomed", 0xBB), CancellationToken.None);
|
||||
Assert.True(await WaitForAsync(_storeB, name, _ => true));
|
||||
|
||||
await _writableA.DeleteAsync(name, "operator", CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeB, name, r => r.IsDeleted),
|
||||
"The delete never propagated to node B.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_update_overwrites_the_earlier_value_on_the_peer()
|
||||
{
|
||||
var name = new SecretName("app/rotating");
|
||||
await _writableA.UpsertAsync(Row("app/rotating", 0x01), CancellationToken.None);
|
||||
Assert.True(await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0x01));
|
||||
|
||||
await _writableA.UpsertAsync(Row("app/rotating", 0x02), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0x02 && r.Revision == 1),
|
||||
"Node B did not converge on the updated value.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Anti_entropy_repairs_a_write_that_never_got_broadcast()
|
||||
{
|
||||
// Simulates a dropped publish — the write goes straight to node A's store, bypassing the
|
||||
// replicator entirely, so the ONLY route to node B is the periodic manifest exchange. This is
|
||||
// the case that makes best-effort publishing safe.
|
||||
var name = new SecretName("app/missed-broadcast");
|
||||
await _storeA.UpsertAsync(Row("app/missed-broadcast", 0xCC), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0xCC),
|
||||
"Anti-entropy failed to repair a write that was never broadcast.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Anti_entropy_pulls_in_the_other_direction_too()
|
||||
{
|
||||
// The mirror case: node B holds a row node A has never heard of. A cannot request a name it
|
||||
// does not know exists, so this only converges if the exchange pushes as well as pulls.
|
||||
var name = new SecretName("app/born-on-b");
|
||||
await _storeB.UpsertAsync(Row("app/born-on-b", 0xDD), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeA, name, r => r.Ciphertext[0] == 0xDD),
|
||||
"Node A never learned about a secret that originated on node B.");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_node_that_joins_late_catches_up_on_everything()
|
||||
{
|
||||
// Partition-heal / cold-start: the whole reason to choose this transport over a shared store.
|
||||
await _writableA.UpsertAsync(Row("bulk/one", 0x11), CancellationToken.None);
|
||||
await _writableA.UpsertAsync(Row("bulk/two", 0x22), CancellationToken.None);
|
||||
await _writableA.UpsertAsync(Row("bulk/three", 0x33), CancellationToken.None);
|
||||
|
||||
SqliteSecretStore lateStore = CreateStore();
|
||||
_systemB.ActorOf(
|
||||
SecretReplicationActor.Props(lateStore, null, AnnounceInterval), "late-joiner");
|
||||
|
||||
Assert.True(await WaitForAsync(lateStore, new SecretName("bulk/one"), r => r.Ciphertext[0] == 0x11));
|
||||
Assert.True(await WaitForAsync(lateStore, new SecretName("bulk/two"), r => r.Ciphertext[0] == 0x22));
|
||||
Assert.True(await WaitForAsync(lateStore, new SecretName("bulk/three"), r => r.Ciphertext[0] == 0x33));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_broadcast_round_trip_does_not_drift_the_originating_nodes_revision()
|
||||
{
|
||||
// Note on what this does and does NOT prove: the revision would also stay put if the self-echo
|
||||
// filter were removed, because re-applying an identical row ties on (updated_utc, revision)
|
||||
// and last-writer-wins rejects a tie. So this covers the drift symptom via defence in depth,
|
||||
// not the filter itself. The filter is what stops the echo being forwarded onward, which
|
||||
// needs three nodes to observe — worth adding if a third node ever joins this rig.
|
||||
var name = new SecretName("app/no-echo");
|
||||
await _writableA.UpsertAsync(Row("app/no-echo", 0xEE), CancellationToken.None);
|
||||
Assert.True(await WaitForAsync(_storeB, name, _ => true));
|
||||
|
||||
await Task.Delay(AnnounceInterval * 4);
|
||||
|
||||
StoredSecret onA = (await _storeA.GetAsync(name, CancellationToken.None))!;
|
||||
Assert.Equal(0, onA.Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_malformed_row_from_a_peer_is_discarded_without_stalling_replication()
|
||||
{
|
||||
// Peer input is untrusted: a path-traversing name must be dropped, and — critically — the
|
||||
// actor must keep serving everything else afterwards.
|
||||
_replicatorB.Tell(new Protocol.SecretRowsMessage(
|
||||
[
|
||||
new Protocol.SecretRowDto
|
||||
{
|
||||
Name = "../../etc/passwd",
|
||||
ContentType = "Text",
|
||||
Ciphertext = [1], Nonce = [1], Tag = [1],
|
||||
WrappedDek = [1], WrapNonce = [1], WrapTag = [1],
|
||||
KekId = "sha256:x",
|
||||
Revision = 0,
|
||||
CreatedUtc = DateTimeOffset.UtcNow.ToString("O"),
|
||||
UpdatedUtc = DateTimeOffset.UtcNow.ToString("O"),
|
||||
},
|
||||
]));
|
||||
|
||||
var name = new SecretName("app/after-malformed");
|
||||
await _writableA.UpsertAsync(Row("app/after-malformed", 0x5A), CancellationToken.None);
|
||||
|
||||
Assert.True(
|
||||
await WaitForAsync(_storeB, name, r => r.Ciphertext[0] == 0x5A),
|
||||
"Replication stalled after a malformed row was received.");
|
||||
}
|
||||
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
await CoordinatedShutdown.Get(_systemA).Run(CoordinatedShutdown.ClrExitReason.Instance);
|
||||
await CoordinatedShutdown.Get(_systemB).Run(CoordinatedShutdown.ClrExitReason.Instance);
|
||||
|
||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||
|
||||
foreach (string db in _dbPaths)
|
||||
{
|
||||
foreach (string path in new[] { db, db + "-wal", db + "-shm" })
|
||||
{
|
||||
try
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
// Best-effort temp cleanup.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="coverlet.collector" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" />
|
||||
<PackageReference Include="xunit" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" />
|
||||
<PackageReference Include="Akka.TestKit.Xunit2" />
|
||||
<PackageReference Include="Akka.Remote" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\src\ZB.MOM.WW.Secrets.Replicator.AkkaDotNet\ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
+111
@@ -0,0 +1,111 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer.DependencyInjection;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.DependencyInjection;
|
||||
|
||||
/// <summary>
|
||||
/// Container-resolution tests for both SQL-Server topologies.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// These exist because a code review caught that neither mode could resolve at all: the decorators
|
||||
/// ask for the concrete <see cref="SqliteSecretStore"/>, which the core package only registered
|
||||
/// behind <see cref="ISecretStore"/>. Every unit test passed and both modes were dead on arrival,
|
||||
/// because nothing built a provider. Resolving the graph is the assertion that matters.
|
||||
/// </remarks>
|
||||
public sealed class AddZbSecretsSqlServerTests
|
||||
{
|
||||
private static IConfiguration Config(string sqlitePath) =>
|
||||
new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["Secrets:SqlitePath"] = sqlitePath,
|
||||
["Secrets:RunMigrationsOnStartup"] = "false",
|
||||
["Secrets:MasterKey:Source"] = "Environment",
|
||||
["Secrets:MasterKey:EnvVarName"] = "ZB_SECRETS_TEST_KEY",
|
||||
["Secrets:SqlServer:ConnectionString"] = "Server=unused;Database=x;",
|
||||
}).Build();
|
||||
|
||||
private static string TempDb() => Path.Combine(Path.GetTempPath(), $"zb-di-{Guid.NewGuid():N}.db");
|
||||
|
||||
[Fact]
|
||||
public void SharedStore_mode_resolves_the_SqlServer_store_as_ISecretStore()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddZbSecretsSqlServerStore(Config(TempDb()));
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider();
|
||||
|
||||
// The SQL-Server store displaces SQLite — no decorator, because there is nothing to replicate.
|
||||
Assert.IsType<SqlServerSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
Assert.IsType<SqlServerSecretsStoreMigrator>(provider.GetRequiredService<ISecretsStoreMigrator>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HubReplication_mode_resolves_a_decorated_local_store()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddZbSecretsSqlServerReplication(Config(TempDb()));
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider();
|
||||
|
||||
Assert.IsType<ReplicatingSecretStore>(provider.GetRequiredService<ISecretStore>());
|
||||
Assert.IsType<SqlServerSecretReplicator>(provider.GetRequiredService<ISecretReplicator>());
|
||||
// The local store is still provisioned by the core SQLite migrator, not the hub's.
|
||||
Assert.IsType<SqliteSecretsStoreMigrator>(provider.GetRequiredService<ISecretsStoreMigrator>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HubReplication_mode_registers_both_the_hub_migration_and_the_sync_service()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddZbSecretsSqlServerReplication(Config(TempDb()));
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider();
|
||||
IHostedService[] hosted = [.. provider.GetServices<IHostedService>()];
|
||||
|
||||
// The hub schema is nobody's "own" store, so every node must provision it at startup too.
|
||||
Assert.Contains(hosted, h => h is SqlServerSecretSyncService);
|
||||
Assert.Single(hosted, h => h.GetType().Name == "SqlServerHubMigrationHostedService");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_decorator_wraps_the_undecorated_local_store_rather_than_itself()
|
||||
{
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
services.AddZbSecretsSqlServerReplication(Config(TempDb()));
|
||||
|
||||
using ServiceProvider provider = services.BuildServiceProvider();
|
||||
|
||||
// Guards against the self-referential registration that would stack-overflow on first use.
|
||||
var decorated = (ReplicatingSecretStore)provider.GetRequiredService<ISecretStore>();
|
||||
Assert.NotNull(decorated);
|
||||
Assert.NotSame(decorated, provider.GetRequiredService<SqliteSecretStore>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_missing_connection_string_fails_at_registration_not_at_first_resolve()
|
||||
{
|
||||
IConfiguration config = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["Secrets:SqlitePath"] = TempDb(),
|
||||
})
|
||||
.Build();
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddLogging();
|
||||
|
||||
InvalidOperationException ex = Assert.Throws<InvalidOperationException>(
|
||||
() => services.AddZbSecretsSqlServerStore(config));
|
||||
|
||||
Assert.Contains("ConnectionString", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Fakes;
|
||||
|
||||
/// <summary>
|
||||
/// An <see cref="ISecretReplicationHub"/> backed by a real SQLite store, standing in for the shared
|
||||
/// SQL-Server hub in offline convergence tests.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Deliberately backed by a real store rather than a dictionary: the behaviour under test is
|
||||
/// last-writer-wins convergence, which lives in <c>ApplyReplicatedAsync</c>. A hand-rolled fake would
|
||||
/// be re-implementing the very logic the tests exist to check, and would happily agree with a broken
|
||||
/// reconciler. The live SQL-Server tests then confirm the T-SQL store behaves the same way.
|
||||
/// </remarks>
|
||||
/// <param name="store">The SQLite store standing in for the hub database.</param>
|
||||
public sealed class SqliteBackedHub(SqliteSecretStore store) : ISecretReplicationHub
|
||||
{
|
||||
/// <summary>Number of times the sweep asked this hub for rows — asserts the fetch was skipped.</summary>
|
||||
public int GetManyCallCount { get; private set; }
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<SecretManifestEntry>> GetManifestAsync(CancellationToken ct) =>
|
||||
store.GetManifestAsync(ct);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<IReadOnlyList<StoredSecret>> GetManyAsync(
|
||||
IReadOnlyList<SecretName> names, CancellationToken ct)
|
||||
{
|
||||
GetManyCallCount++;
|
||||
|
||||
var rows = new List<StoredSecret>(names.Count);
|
||||
|
||||
foreach (SecretName name in names)
|
||||
{
|
||||
StoredSecret? row = await store.GetAsync(name, ct);
|
||||
|
||||
if (row is not null)
|
||||
{
|
||||
rows.Add(row);
|
||||
}
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task ApplyReplicatedAsync(StoredSecret row, CancellationToken ct) =>
|
||||
store.ApplyReplicatedAsync(row, ct);
|
||||
}
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
using Microsoft.Data.SqlClient;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// Shared connection details for the env-gated live SQL-Server suite.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Gated on <c>SECRETS_SQLSERVER_CONNSTR</c> and skips cleanly when it is unset, matching the family
|
||||
/// live-test idiom — the offline suite must stay runnable on any machine. Set it to a database the
|
||||
/// test is allowed to create and drop a schema in, for example:
|
||||
/// <c>Server=10.100.0.35,31433;Database=ZbSecretsTest;User Id=sa;Password=...;TrustServerCertificate=True</c>.
|
||||
/// </remarks>
|
||||
public static class LiveSqlServer
|
||||
{
|
||||
/// <summary>Environment variable holding the live connection string.</summary>
|
||||
public const string ConnectionStringVariable = "SECRETS_SQLSERVER_CONNSTR";
|
||||
|
||||
/// <summary>The configured connection string, or <see langword="null"/> when the suite is not enabled.</summary>
|
||||
public static string? ConnectionString =>
|
||||
Environment.GetEnvironmentVariable(ConnectionStringVariable);
|
||||
|
||||
/// <summary>Whether the live suite should run.</summary>
|
||||
public static bool IsEnabled => !string.IsNullOrWhiteSpace(ConnectionString);
|
||||
|
||||
/// <summary>Skip reason shown when the suite is not enabled.</summary>
|
||||
public const string SkipReason =
|
||||
"Live SQL-Server suite disabled; set SECRETS_SQLSERVER_CONNSTR to enable.";
|
||||
|
||||
/// <summary>
|
||||
/// Builds options against a throwaway schema so parallel runs and repeat runs cannot collide,
|
||||
/// and so a failed run leaves no trace in a shared database.
|
||||
/// </summary>
|
||||
/// <param name="schemaName">The unique schema name for this test.</param>
|
||||
/// <returns>Options targeting that schema.</returns>
|
||||
public static SqlServerSecretsOptions OptionsFor(string schemaName) => new()
|
||||
{
|
||||
ConnectionString = ConnectionString!,
|
||||
SchemaName = schemaName,
|
||||
};
|
||||
|
||||
/// <summary>Generates a unique, allow-list-legal schema name for one test class.</summary>
|
||||
/// <returns>A fresh schema name.</returns>
|
||||
public static string NewSchemaName() => $"zbtest_{Guid.NewGuid():N}"[..32];
|
||||
|
||||
/// <summary>Drops the throwaway schema and its tables.</summary>
|
||||
/// <param name="schemaName">The schema to drop.</param>
|
||||
/// <returns>A task that completes when the schema is gone.</returns>
|
||||
public static async Task DropSchemaAsync(string schemaName)
|
||||
{
|
||||
if (!IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await using var connection = new SqlConnection(ConnectionString);
|
||||
await connection.OpenAsync();
|
||||
|
||||
await using SqlCommand command = connection.CreateCommand();
|
||||
command.CommandText = $"""
|
||||
IF OBJECT_ID(N'[{schemaName}].[secret]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[secret];
|
||||
IF OBJECT_ID(N'[{schemaName}].[schema_version]', N'U') IS NOT NULL DROP TABLE [{schemaName}].[schema_version];
|
||||
IF EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'{schemaName}') EXEC(N'DROP SCHEMA [{schemaName}]');
|
||||
""";
|
||||
|
||||
await command.ExecuteNonQueryAsync();
|
||||
}
|
||||
}
|
||||
+365
@@ -0,0 +1,365 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Live;
|
||||
|
||||
/// <summary>
|
||||
/// The SQLite store's test suite, ported case-for-case onto the SQL-Server store.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Ported rather than newly written on purpose. The two stores must be behaviourally identical —
|
||||
/// a cluster can hold both at once — so the strongest available check is that the assertions
|
||||
/// written against SQLite pass unchanged here. Anywhere the T-SQL diverges in observable behaviour,
|
||||
/// one of these fails.
|
||||
/// </remarks>
|
||||
[Collection("live-sqlserver")]
|
||||
public sealed class SqlServerSecretStoreLiveTests : IAsyncLifetime
|
||||
{
|
||||
private readonly string _schema = LiveSqlServer.NewSchemaName();
|
||||
private SqlServerSecretStore _store = null!;
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
if (!LiveSqlServer.IsEnabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var factory = new SecretsSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
|
||||
await new SqlServerSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None);
|
||||
_store = new SqlServerSecretStore(factory);
|
||||
}
|
||||
|
||||
public Task DisposeAsync() => LiveSqlServer.DropSchemaAsync(_schema);
|
||||
|
||||
private static StoredSecret MakeSecret(
|
||||
string name,
|
||||
long revision = 0,
|
||||
byte[]? ciphertext = null,
|
||||
DateTimeOffset? updatedUtc = null,
|
||||
DateTimeOffset? createdUtc = null,
|
||||
bool isDeleted = false,
|
||||
DateTimeOffset? deletedUtc = null,
|
||||
string? createdBy = "alice",
|
||||
string? updatedBy = "alice") => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
Description = "desc",
|
||||
ContentType = SecretContentType.ConnectionString,
|
||||
Ciphertext = ciphertext ?? [1, 2, 3],
|
||||
Nonce = [4, 5, 6],
|
||||
Tag = [7, 8, 9],
|
||||
WrappedDek = [10, 11, 12],
|
||||
WrapNonce = [13, 14, 15],
|
||||
WrapTag = [16, 17, 18],
|
||||
KekId = "kek-1",
|
||||
Revision = revision,
|
||||
IsDeleted = isDeleted,
|
||||
DeletedUtc = deletedUtc,
|
||||
CreatedUtc = createdUtc ?? DateTimeOffset.UtcNow,
|
||||
UpdatedUtc = updatedUtc ?? DateTimeOffset.UtcNow,
|
||||
CreatedBy = createdBy,
|
||||
UpdatedBy = updatedBy,
|
||||
};
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Upsert_Then_Get_RoundTrips()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("app/db-conn"), CancellationToken.None);
|
||||
|
||||
StoredSecret? got = await _store.GetAsync(new SecretName("app/db-conn"), CancellationToken.None);
|
||||
|
||||
Assert.NotNull(got);
|
||||
Assert.Equal("app/db-conn", got!.Name.Value);
|
||||
Assert.Equal("desc", got.Description);
|
||||
Assert.Equal(SecretContentType.ConnectionString, got.ContentType);
|
||||
Assert.Equal(new byte[] { 1, 2, 3 }, got.Ciphertext);
|
||||
Assert.Equal(new byte[] { 4, 5, 6 }, got.Nonce);
|
||||
Assert.Equal(new byte[] { 7, 8, 9 }, got.Tag);
|
||||
Assert.Equal(new byte[] { 10, 11, 12 }, got.WrappedDek);
|
||||
Assert.Equal(new byte[] { 13, 14, 15 }, got.WrapNonce);
|
||||
Assert.Equal(new byte[] { 16, 17, 18 }, got.WrapTag);
|
||||
Assert.Equal("kek-1", got.KekId);
|
||||
Assert.Equal(0, got.Revision);
|
||||
Assert.False(got.IsDeleted);
|
||||
Assert.Null(got.DeletedUtc);
|
||||
Assert.Equal("alice", got.CreatedBy);
|
||||
Assert.Equal("alice", got.UpdatedBy);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Get_ReturnsNull_WhenAbsent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
Assert.Null(await _store.GetAsync(new SecretName("nope"), CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Upsert_Existing_OverwritesInPlace_BumpsRevision()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(
|
||||
MakeSecret("app/rotating", ciphertext: [1, 1, 1], createdBy: "alice", updatedBy: "alice"),
|
||||
CancellationToken.None);
|
||||
|
||||
StoredSecret afterFirst =
|
||||
(await _store.GetAsync(new SecretName("app/rotating"), CancellationToken.None))!;
|
||||
Assert.Equal(0, afterFirst.Revision);
|
||||
DateTimeOffset originalCreatedUtc = afterFirst.CreatedUtc;
|
||||
|
||||
await _store.UpsertAsync(
|
||||
MakeSecret("app/rotating", ciphertext: [9, 9, 9], createdBy: "bob", updatedBy: "bob"),
|
||||
CancellationToken.None);
|
||||
|
||||
StoredSecret afterSecond =
|
||||
(await _store.GetAsync(new SecretName("app/rotating"), CancellationToken.None))!;
|
||||
Assert.Equal(1, afterSecond.Revision);
|
||||
Assert.Equal(new byte[] { 9, 9, 9 }, afterSecond.Ciphertext);
|
||||
Assert.Equal(originalCreatedUtc, afterSecond.CreatedUtc);
|
||||
Assert.Equal("alice", afterSecond.CreatedBy);
|
||||
Assert.Equal("bob", afterSecond.UpdatedBy);
|
||||
Assert.False(afterSecond.IsDeleted);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task List_ExcludesTombstoned_ByDefault()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("keep"), CancellationToken.None);
|
||||
await _store.UpsertAsync(MakeSecret("gone"), CancellationToken.None);
|
||||
await _store.DeleteAsync(new SecretName("gone"), "carol", CancellationToken.None);
|
||||
|
||||
IReadOnlyList<SecretMetadata> visible =
|
||||
await _store.ListAsync(includeDeleted: false, CancellationToken.None);
|
||||
Assert.DoesNotContain(visible, m => m.Name.Value == "gone");
|
||||
Assert.Contains(visible, m => m.Name.Value == "keep");
|
||||
|
||||
IReadOnlyList<SecretMetadata> all =
|
||||
await _store.ListAsync(includeDeleted: true, CancellationToken.None);
|
||||
Assert.Contains(all, m => m.Name.Value == "gone");
|
||||
Assert.Contains(all, m => m.Name.Value == "keep");
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Delete_SetsTombstone_BumpsRevision_ReturnsFalseWhenAbsent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("app/secret"), CancellationToken.None);
|
||||
|
||||
Assert.True(await _store.DeleteAsync(new SecretName("app/secret"), "carol", CancellationToken.None));
|
||||
|
||||
StoredSecret tombstoned =
|
||||
(await _store.GetAsync(new SecretName("app/secret"), CancellationToken.None))!;
|
||||
Assert.True(tombstoned.IsDeleted);
|
||||
Assert.NotNull(tombstoned.DeletedUtc);
|
||||
Assert.Equal(1, tombstoned.Revision);
|
||||
Assert.Equal("carol", tombstoned.UpdatedBy);
|
||||
|
||||
Assert.False(await _store.DeleteAsync(new SecretName("app/secret"), "carol", CancellationToken.None));
|
||||
Assert.False(await _store.DeleteAsync(new SecretName("never-existed"), "carol", CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task GetManifest_ReturnsAllRows()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("a"), CancellationToken.None);
|
||||
await _store.UpsertAsync(MakeSecret("b"), CancellationToken.None);
|
||||
await _store.DeleteAsync(new SecretName("b"), "carol", CancellationToken.None);
|
||||
|
||||
IReadOnlyList<SecretManifestEntry> manifest = await _store.GetManifestAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(2, manifest.Count);
|
||||
SecretManifestEntry a = manifest.Single(e => e.Name.Value == "a");
|
||||
SecretManifestEntry b = manifest.Single(e => e.Name.Value == "b");
|
||||
Assert.False(a.IsDeleted);
|
||||
Assert.Equal(0, a.Revision);
|
||||
Assert.True(b.IsDeleted);
|
||||
Assert.Equal(1, b.Revision);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task ApplyReplicated_AppliesNewer_IgnoresStale()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
DateTimeOffset t1 = new(2026, 1, 1, 0, 0, 0, TimeSpan.Zero);
|
||||
DateTimeOffset t2 = new(2026, 1, 2, 0, 0, 0, TimeSpan.Zero);
|
||||
DateTimeOffset t3 = new(2026, 1, 3, 0, 0, 0, TimeSpan.Zero);
|
||||
|
||||
await _store.ApplyReplicatedAsync(
|
||||
MakeSecret("repl", revision: 5, ciphertext: [5, 5, 5], updatedUtc: t2, createdUtc: t1),
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.Equal(5, (await _store.GetAsync(new SecretName("repl"), CancellationToken.None))!.Revision);
|
||||
|
||||
await _store.ApplyReplicatedAsync(
|
||||
MakeSecret("repl", revision: 6, ciphertext: [6, 6, 6], updatedUtc: t3, createdUtc: t1),
|
||||
CancellationToken.None);
|
||||
|
||||
StoredSecret afterNewer = (await _store.GetAsync(new SecretName("repl"), CancellationToken.None))!;
|
||||
Assert.Equal(6, afterNewer.Revision);
|
||||
Assert.Equal(new byte[] { 6, 6, 6 }, afterNewer.Ciphertext);
|
||||
Assert.Equal(t3, afterNewer.UpdatedUtc);
|
||||
|
||||
await _store.ApplyReplicatedAsync(
|
||||
MakeSecret("repl", revision: 4, ciphertext: [4, 4, 4], updatedUtc: t1, createdUtc: t1),
|
||||
CancellationToken.None);
|
||||
|
||||
StoredSecret afterStale = (await _store.GetAsync(new SecretName("repl"), CancellationToken.None))!;
|
||||
Assert.Equal(6, afterStale.Revision);
|
||||
Assert.Equal(new byte[] { 6, 6, 6 }, afterStale.Ciphertext);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task ApplyRewrap_ChangesOnlyWrapEnvelopeAndKekId_PreservingRevisionBodyAndTimestamps()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(
|
||||
MakeSecret("app/rewrap", ciphertext: [1, 2, 3], createdUtc: new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero)),
|
||||
CancellationToken.None);
|
||||
await _store.UpsertAsync(MakeSecret("app/rewrap", ciphertext: [1, 2, 3]), CancellationToken.None);
|
||||
|
||||
StoredSecret before = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!;
|
||||
Assert.Equal(1, before.Revision);
|
||||
|
||||
bool applied = await _store.ApplyRewrapAsync(
|
||||
before with
|
||||
{
|
||||
WrappedDek = [90, 91, 92],
|
||||
WrapNonce = [93, 94, 95],
|
||||
WrapTag = [96, 97, 98],
|
||||
KekId = "kek-2",
|
||||
},
|
||||
before.WrappedDek,
|
||||
CancellationToken.None);
|
||||
Assert.True(applied);
|
||||
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("app/rewrap"), CancellationToken.None))!;
|
||||
Assert.Equal(new byte[] { 90, 91, 92 }, after.WrappedDek);
|
||||
Assert.Equal(new byte[] { 93, 94, 95 }, after.WrapNonce);
|
||||
Assert.Equal(new byte[] { 96, 97, 98 }, after.WrapTag);
|
||||
Assert.Equal("kek-2", after.KekId);
|
||||
Assert.Equal(before.Revision, after.Revision);
|
||||
Assert.Equal(before.UpdatedUtc, after.UpdatedUtc);
|
||||
Assert.Equal(before.CreatedUtc, after.CreatedUtc);
|
||||
Assert.Equal(before.UpdatedBy, after.UpdatedBy);
|
||||
Assert.Equal(new byte[] { 1, 2, 3 }, after.Ciphertext);
|
||||
Assert.Equal(before.Nonce, after.Nonce);
|
||||
Assert.Equal(before.Tag, after.Tag);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task ApplyRewrap_RewrapsTombstonedRows()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("app/dead"), CancellationToken.None);
|
||||
await _store.DeleteAsync(new SecretName("app/dead"), "carol", CancellationToken.None);
|
||||
|
||||
StoredSecret tomb = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!;
|
||||
Assert.True(tomb.IsDeleted);
|
||||
|
||||
Assert.True(await _store.ApplyRewrapAsync(
|
||||
tomb with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" },
|
||||
tomb.WrappedDek,
|
||||
CancellationToken.None));
|
||||
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("app/dead"), CancellationToken.None))!;
|
||||
Assert.Equal("kek-2", after.KekId);
|
||||
Assert.True(after.IsDeleted);
|
||||
Assert.Equal(tomb.Revision, after.Revision);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task ApplyRewrap_ReturnsFalse_WhenRowAbsent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
StoredSecret ghost = MakeSecret("never-existed") with { KekId = "kek-2" };
|
||||
|
||||
Assert.False(await _store.ApplyRewrapAsync(ghost, ghost.WrappedDek, CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task ApplyRewrap_ReturnsFalse_WhenCurrentWrapChanged_UnderConcurrentWrite()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("app/cas", ciphertext: [1, 2, 3]), CancellationToken.None);
|
||||
StoredSecret before = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!;
|
||||
|
||||
await _store.UpsertAsync(
|
||||
MakeSecret("app/cas", ciphertext: [9, 9, 9]) with { WrappedDek = [77, 78, 79] },
|
||||
CancellationToken.None);
|
||||
|
||||
bool applied = await _store.ApplyRewrapAsync(
|
||||
before with { WrappedDek = [1], WrapNonce = [2], WrapTag = [3], KekId = "kek-2" },
|
||||
before.WrappedDek,
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.False(applied);
|
||||
StoredSecret after = (await _store.GetAsync(new SecretName("app/cas"), CancellationToken.None))!;
|
||||
Assert.Equal(new byte[] { 77, 78, 79 }, after.WrappedDek);
|
||||
Assert.Equal("kek-1", after.KekId);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task GetMany_returns_only_the_requested_rows()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("a/one"), CancellationToken.None);
|
||||
await _store.UpsertAsync(MakeSecret("b/two"), CancellationToken.None);
|
||||
await _store.UpsertAsync(MakeSecret("c/three"), CancellationToken.None);
|
||||
|
||||
IReadOnlyList<StoredSecret> rows = await _store.GetManyAsync(
|
||||
[new SecretName("a/one"), new SecretName("c/three"), new SecretName("absent/name")],
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.Equal(2, rows.Count);
|
||||
Assert.Contains(rows, r => r.Name.Value == "a/one");
|
||||
Assert.Contains(rows, r => r.Name.Value == "c/three");
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task GetMany_short_circuits_on_an_empty_request()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
// Guards the IN () clause: an empty list would otherwise produce invalid T-SQL.
|
||||
Assert.Empty(await _store.GetManyAsync([], CancellationToken.None));
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task Migrator_is_idempotent()
|
||||
{
|
||||
Skip.IfNot(LiveSqlServer.IsEnabled, LiveSqlServer.SkipReason);
|
||||
|
||||
var factory = new SecretsSqlServerConnectionFactory(LiveSqlServer.OptionsFor(_schema));
|
||||
var migrator = new SqlServerSecretsStoreMigrator(factory);
|
||||
|
||||
// Every node runs this at every startup — re-running must be a no-op, not an error.
|
||||
await migrator.MigrateAsync(CancellationToken.None);
|
||||
await migrator.MigrateAsync(CancellationToken.None);
|
||||
|
||||
await _store.UpsertAsync(MakeSecret("survives/migration"), CancellationToken.None);
|
||||
await migrator.MigrateAsync(CancellationToken.None);
|
||||
|
||||
Assert.NotNull(await _store.GetAsync(new SecretName("survives/migration"), CancellationToken.None));
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Serializes the live suite: the tests share one database server and each provisions its own
|
||||
/// schema, so running them in parallel would multiply connection load for no benefit.
|
||||
/// </summary>
|
||||
[CollectionDefinition("live-sqlserver", DisableParallelization = true)]
|
||||
public sealed class LiveSqlServerCollection;
|
||||
+205
@@ -0,0 +1,205 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests.Fakes;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Convergence behaviour of the bidirectional hub sweep, exercised against real stores on both
|
||||
/// sides so the assertions are about actual persisted state rather than mock interactions.
|
||||
/// </summary>
|
||||
public sealed class SqlServerSecretSyncServiceTests : IDisposable
|
||||
{
|
||||
private readonly List<string> _dbPaths = [];
|
||||
private readonly SqliteSecretStore _local;
|
||||
private readonly SqliteSecretStore _hubStore;
|
||||
private readonly SqliteBackedHub _hub;
|
||||
|
||||
public SqlServerSecretSyncServiceTests()
|
||||
{
|
||||
_local = CreateStore();
|
||||
_hubStore = CreateStore();
|
||||
_hub = new SqliteBackedHub(_hubStore);
|
||||
}
|
||||
|
||||
private SqliteSecretStore CreateStore()
|
||||
{
|
||||
string path = Path.Combine(Path.GetTempPath(), $"zb-sync-{Guid.NewGuid():N}.db");
|
||||
_dbPaths.Add(path);
|
||||
var factory = new SecretsSqliteConnectionFactory(path);
|
||||
new SqliteSecretsStoreMigrator(factory).MigrateAsync(CancellationToken.None).GetAwaiter().GetResult();
|
||||
return new SqliteSecretStore(factory);
|
||||
}
|
||||
|
||||
private SqlServerSecretSyncService CreateService() => new(
|
||||
_local,
|
||||
_hub,
|
||||
cacheInvalidator: null,
|
||||
Options.Create(new SqlServerSecretsOptions { ConnectionString = "Server=unused;" }),
|
||||
NullLogger<SqlServerSecretSyncService>.Instance);
|
||||
|
||||
private static StoredSecret Row(string name, byte marker) => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
ContentType = SecretContentType.Text,
|
||||
Ciphertext = [marker],
|
||||
Nonce = [1],
|
||||
Tag = [2],
|
||||
WrappedDek = [3],
|
||||
WrapNonce = [4],
|
||||
WrapTag = [5],
|
||||
KekId = "sha256:test",
|
||||
Revision = 0,
|
||||
CreatedUtc = DateTimeOffset.UtcNow,
|
||||
UpdatedUtc = DateTimeOffset.UtcNow,
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_pulls_a_secret_the_node_has_never_seen()
|
||||
{
|
||||
await _hubStore.UpsertAsync(Row("db/password", 0xAA), CancellationToken.None);
|
||||
|
||||
(int pulled, int pushed) = await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, pulled);
|
||||
Assert.Equal(0, pushed);
|
||||
|
||||
StoredSecret? local = await _local.GetAsync(new SecretName("db/password"), CancellationToken.None);
|
||||
Assert.NotNull(local);
|
||||
Assert.Equal(0xAA, local!.Ciphertext[0]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_pushes_a_local_write_the_hub_never_received()
|
||||
{
|
||||
// The case that makes best-effort publishing safe: the publish failed (hub was down), so the
|
||||
// hub has never heard of this name and can never ask for it. Only a push sweep repairs this.
|
||||
await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None);
|
||||
|
||||
(int pulled, int pushed) = await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, pulled);
|
||||
Assert.Equal(1, pushed);
|
||||
|
||||
StoredSecret? hub = await _hubStore.GetAsync(new SecretName("api/key"), CancellationToken.None);
|
||||
Assert.NotNull(hub);
|
||||
Assert.Equal(0xBB, hub!.Ciphertext[0]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_preserves_the_originating_revision_when_pushing()
|
||||
{
|
||||
// Pushed verbatim, not upserted: if the hub stamped its own revision the row would look newer
|
||||
// than the writer's copy and bounce straight back on the next sweep.
|
||||
await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None);
|
||||
await _local.UpsertAsync(Row("api/key", 0xCC), CancellationToken.None);
|
||||
|
||||
StoredSecret localRow = (await _local.GetAsync(new SecretName("api/key"), CancellationToken.None))!;
|
||||
Assert.Equal(1, localRow.Revision);
|
||||
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
StoredSecret hubRow = (await _hubStore.GetAsync(new SecretName("api/key"), CancellationToken.None))!;
|
||||
Assert.Equal(localRow.Revision, hubRow.Revision);
|
||||
Assert.Equal(localRow.UpdatedUtc, hubRow.UpdatedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_converges_both_directions_in_one_pass()
|
||||
{
|
||||
await _local.UpsertAsync(Row("only/local", 0x11), CancellationToken.None);
|
||||
await _hubStore.UpsertAsync(Row("only/hub", 0x22), CancellationToken.None);
|
||||
|
||||
(int pulled, int pushed) = await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, pulled);
|
||||
Assert.Equal(1, pushed);
|
||||
|
||||
Assert.NotNull(await _local.GetAsync(new SecretName("only/hub"), CancellationToken.None));
|
||||
Assert.NotNull(await _hubStore.GetAsync(new SecretName("only/local"), CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_is_idempotent_so_a_converged_pair_stays_quiet()
|
||||
{
|
||||
await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None);
|
||||
|
||||
SqlServerSecretSyncService service = CreateService();
|
||||
await service.SweepAsync(CancellationToken.None);
|
||||
|
||||
(int pulled, int pushed) = await service.SweepAsync(CancellationToken.None);
|
||||
|
||||
// A steady-state cluster must not write on every tick — that would churn the hub forever.
|
||||
Assert.Equal(0, pulled);
|
||||
Assert.Equal(0, pushed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_propagates_a_delete_as_a_tombstone()
|
||||
{
|
||||
await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None);
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
await _local.DeleteAsync(new SecretName("api/key"), "tester", CancellationToken.None);
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
StoredSecret hubRow = (await _hubStore.GetAsync(new SecretName("api/key"), CancellationToken.None))!;
|
||||
Assert.True(hubRow.IsDeleted);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_does_not_resurrect_a_secret_deleted_on_the_hub()
|
||||
{
|
||||
await _local.UpsertAsync(Row("api/key", 0xBB), CancellationToken.None);
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
await _hubStore.DeleteAsync(new SecretName("api/key"), "admin", CancellationToken.None);
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
StoredSecret localRow = (await _local.GetAsync(new SecretName("api/key"), CancellationToken.None))!;
|
||||
Assert.True(localRow.IsDeleted);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_skips_the_hub_fetch_when_nothing_is_newer()
|
||||
{
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, _hub.GetManyCallCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Sweep_lets_the_newer_side_win_a_conflicting_write()
|
||||
{
|
||||
await _local.UpsertAsync(Row("api/key", 0x11), CancellationToken.None);
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
// Both sides then write independently; the hub's write is later.
|
||||
await _local.UpsertAsync(Row("api/key", 0x22), CancellationToken.None);
|
||||
await Task.Delay(10);
|
||||
await _hubStore.UpsertAsync(Row("api/key", 0x33), CancellationToken.None);
|
||||
|
||||
await CreateService().SweepAsync(CancellationToken.None);
|
||||
|
||||
StoredSecret localRow = (await _local.GetAsync(new SecretName("api/key"), CancellationToken.None))!;
|
||||
Assert.Equal(0x33, localRow.Ciphertext[0]);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||
|
||||
foreach (string db in _dbPaths)
|
||||
{
|
||||
foreach (string path in new[] { db, db + "-wal", db + "-shm" })
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
namespace ZB.MOM.WW.Secrets.Replicator.SqlServer.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// The schema-name allow-list is the injection boundary for the one value that cannot be
|
||||
/// parameterized (a T-SQL object name), so it gets adversarial coverage rather than a happy path.
|
||||
/// </summary>
|
||||
public sealed class SqlServerSecretsOptionsTests
|
||||
{
|
||||
[Theory]
|
||||
[InlineData("zbsecrets")]
|
||||
[InlineData("ZbSecrets")]
|
||||
[InlineData("_private")]
|
||||
[InlineData("schema_1")]
|
||||
public void SchemaName_accepts_plain_identifiers(string value)
|
||||
{
|
||||
var options = new SqlServerSecretsOptions { SchemaName = value };
|
||||
|
||||
Assert.Equal(value, options.SchemaName);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// The bracket-escape trick that would otherwise break out of the [quoting] at the use site.
|
||||
[InlineData("evil]]; DROP TABLE secret; --")]
|
||||
[InlineData("dbo.secret")]
|
||||
[InlineData("with space")]
|
||||
[InlineData("quote'name")]
|
||||
[InlineData("semi;colon")]
|
||||
[InlineData("1leading_digit")]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
public void SchemaName_rejects_anything_that_is_not_a_plain_identifier(string value)
|
||||
{
|
||||
Assert.Throws<ArgumentException>(() => new SqlServerSecretsOptions { SchemaName = value });
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SchemaName_rejects_an_over_long_identifier()
|
||||
{
|
||||
Assert.Throws<ArgumentException>(() =>
|
||||
new SqlServerSecretsOptions { SchemaName = new string('a', 129) });
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_rejects_a_missing_connection_string()
|
||||
{
|
||||
InvalidOperationException ex =
|
||||
Assert.Throws<InvalidOperationException>(() => new SqlServerSecretsOptions().Validate());
|
||||
|
||||
Assert.Contains("ConnectionString", ex.Message, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_rejects_a_non_positive_sync_interval()
|
||||
{
|
||||
var options = new SqlServerSecretsOptions
|
||||
{
|
||||
ConnectionString = "Server=x;",
|
||||
SyncInterval = TimeSpan.Zero,
|
||||
};
|
||||
|
||||
Assert.Throws<InvalidOperationException>(options.Validate);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_rejects_a_non_positive_command_timeout()
|
||||
{
|
||||
var options = new SqlServerSecretsOptions
|
||||
{
|
||||
ConnectionString = "Server=x;",
|
||||
CommandTimeout = TimeSpan.FromSeconds(-1),
|
||||
};
|
||||
|
||||
Assert.Throws<InvalidOperationException>(options.Validate);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_accepts_a_fully_configured_instance()
|
||||
{
|
||||
var options = new SqlServerSecretsOptions { ConnectionString = "Server=x;Database=y;" };
|
||||
|
||||
options.Validate();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CreateSchemaDdl_quotes_the_schema_name_everywhere_it_appears()
|
||||
{
|
||||
string ddl = SqlServerSecretsSchema.CreateSchemaDdl("zbsecrets");
|
||||
|
||||
Assert.Contains("[zbsecrets].[secret]", ddl, StringComparison.Ordinal);
|
||||
Assert.Contains("[zbsecrets].[schema_version]", ddl, StringComparison.Ordinal);
|
||||
// Every DDL statement is guarded, so re-running the migration on a provisioned database is
|
||||
// a no-op rather than an error.
|
||||
Assert.Contains("IF NOT EXISTS", ddl, StringComparison.Ordinal);
|
||||
Assert.Contains("IS NULL", ddl, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="coverlet.collector" />
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" />
|
||||
<PackageReference Include="xunit" />
|
||||
<PackageReference Include="xunit.runner.visualstudio" />
|
||||
<PackageReference Include="Xunit.SkippableFact" />
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection" />
|
||||
<PackageReference Include="Microsoft.Extensions.Configuration" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Using Include="Xunit" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\src\ZB.MOM.WW.Secrets.Replicator.SqlServer\ZB.MOM.WW.Secrets.Replicator.SqlServer.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,15 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Tests.Fakes;
|
||||
|
||||
/// <summary>Records every eviction so tests can assert the resolver cache was cleared.</summary>
|
||||
public sealed class RecordingCacheInvalidator : ISecretCacheInvalidator
|
||||
{
|
||||
private readonly List<SecretName> _invalidated = [];
|
||||
|
||||
/// <summary>The names evicted, in call order.</summary>
|
||||
public IReadOnlyList<SecretName> Invalidated => _invalidated;
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Invalidate(SecretName name) => _invalidated.Add(name);
|
||||
}
|
||||
+265
@@ -0,0 +1,265 @@
|
||||
using ZB.MOM.WW.Secrets.Abstractions;
|
||||
using ZB.MOM.WW.Secrets.Replication;
|
||||
using ZB.MOM.WW.Secrets.Sqlite;
|
||||
using ZB.MOM.WW.Secrets.Tests.Fakes;
|
||||
|
||||
namespace ZB.MOM.WW.Secrets.Tests.Replication;
|
||||
|
||||
/// <summary>
|
||||
/// Convergence rules for the transport-agnostic reconciler. These are the cases both replication
|
||||
/// packages inherit, so they are pinned here once against a real SQLite store rather than a fake —
|
||||
/// a reconciler that agrees with a mock but not with the store would be worthless.
|
||||
/// </summary>
|
||||
public sealed class SecretReplicationReconcilerTests : IDisposable
|
||||
{
|
||||
private readonly string _dbPath = Path.Combine(Path.GetTempPath(), $"zb-recon-{Guid.NewGuid():N}.db");
|
||||
private readonly SecretsSqliteConnectionFactory _factory;
|
||||
private readonly SqliteSecretStore _store;
|
||||
|
||||
public SecretReplicationReconcilerTests()
|
||||
{
|
||||
_factory = new SecretsSqliteConnectionFactory(_dbPath);
|
||||
new SqliteSecretsStoreMigrator(_factory).MigrateAsync(CancellationToken.None).GetAwaiter().GetResult();
|
||||
_store = new SqliteSecretStore(_factory);
|
||||
}
|
||||
|
||||
private static readonly DateTimeOffset T0 = new(2026, 7, 18, 12, 0, 0, TimeSpan.Zero);
|
||||
|
||||
private static SecretManifestEntry Entry(string name, long revision, DateTimeOffset updated, bool deleted = false) =>
|
||||
new() { Name = new SecretName(name), Revision = revision, UpdatedUtc = updated, IsDeleted = deleted };
|
||||
|
||||
private static StoredSecret Row(string name, long revision, DateTimeOffset updated, bool deleted = false) => new()
|
||||
{
|
||||
Name = new SecretName(name),
|
||||
ContentType = SecretContentType.Text,
|
||||
Ciphertext = [1, 2, 3],
|
||||
Nonce = [4],
|
||||
Tag = [5],
|
||||
WrappedDek = [6],
|
||||
WrapNonce = [7],
|
||||
WrapTag = [8],
|
||||
KekId = "sha256:test",
|
||||
Revision = revision,
|
||||
IsDeleted = deleted,
|
||||
DeletedUtc = deleted ? updated : null,
|
||||
CreatedUtc = T0,
|
||||
UpdatedUtc = updated,
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_pulls_names_absent_locally()
|
||||
{
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [],
|
||||
remote: [Entry("db/password", 0, T0)]);
|
||||
|
||||
Assert.Equal(["db/password"], pull.Select(n => n.Value));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_pulls_a_newer_remote_row()
|
||||
{
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [Entry("api/key", 1, T0)],
|
||||
remote: [Entry("api/key", 2, T0.AddSeconds(1))]);
|
||||
|
||||
Assert.Equal(["api/key"], pull.Select(n => n.Value));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_breaks_a_timestamp_tie_on_revision()
|
||||
{
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [Entry("api/key", 1, T0)],
|
||||
remote: [Entry("api/key", 2, T0)]);
|
||||
|
||||
Assert.Equal(["api/key"], pull.Select(n => n.Value));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_ignores_an_identical_row_so_redelivery_is_free()
|
||||
{
|
||||
// At-least-once transports redeliver. A tie must not be "newer", or two nodes would
|
||||
// ping-pong the same row forever.
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [Entry("api/key", 2, T0)],
|
||||
remote: [Entry("api/key", 2, T0)]);
|
||||
|
||||
Assert.Empty(pull);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_ignores_a_peer_that_is_behind()
|
||||
{
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [Entry("api/key", 5, T0.AddMinutes(1))],
|
||||
remote: [Entry("api/key", 2, T0)]);
|
||||
|
||||
Assert.Empty(pull);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_pulls_a_tombstone_so_deletes_propagate()
|
||||
{
|
||||
// A delete is a newer row, not an absence — otherwise a deleted secret would be resurrected
|
||||
// by the next anti-entropy sweep from a node that still has it.
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [Entry("api/key", 1, T0)],
|
||||
remote: [Entry("api/key", 2, T0.AddSeconds(1), deleted: true)]);
|
||||
|
||||
Assert.Equal(["api/key"], pull.Select(n => n.Value));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputePullSet_does_not_pull_a_name_only_the_local_side_has()
|
||||
{
|
||||
// The local side only ever pulls. Pushing local-only names is the peer's own reconcile.
|
||||
IReadOnlyList<SecretName> pull = SecretReplicationReconciler.ComputePullSet(
|
||||
localManifest: [Entry("local/only", 0, T0)],
|
||||
remote: []);
|
||||
|
||||
Assert.Empty(pull);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReconcileAsync_applies_only_the_rows_the_peer_holds_newer()
|
||||
{
|
||||
await _store.UpsertAsync(Row("shared/a", 0, T0), CancellationToken.None);
|
||||
|
||||
StoredSecret localA = (await _store.GetAsync(new SecretName("shared/a"), CancellationToken.None))!;
|
||||
|
||||
var reconciler = new SecretReplicationReconciler(_store);
|
||||
List<SecretName>? requested = null;
|
||||
|
||||
int applied = await reconciler.ReconcileAsync(
|
||||
remoteManifest:
|
||||
[
|
||||
// Behind the local copy — must not be fetched.
|
||||
Entry("shared/a", localA.Revision, localA.UpdatedUtc.AddMinutes(-1)),
|
||||
// Absent locally — must be fetched.
|
||||
Entry("shared/b", 3, T0.AddHours(1)),
|
||||
],
|
||||
fetchAsync: (names, _) =>
|
||||
{
|
||||
requested = [.. names];
|
||||
return Task.FromResult<IReadOnlyList<StoredSecret>>([Row("shared/b", 3, T0.AddHours(1))]);
|
||||
},
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, applied);
|
||||
Assert.Equal(["shared/b"], requested!.Select(n => n.Value));
|
||||
|
||||
StoredSecret? b = await _store.GetAsync(new SecretName("shared/b"), CancellationToken.None);
|
||||
Assert.NotNull(b);
|
||||
// Applied VERBATIM — the peer's revision, not a locally bumped one.
|
||||
Assert.Equal(3, b!.Revision);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReconcileAsync_skips_the_fetch_entirely_when_nothing_is_newer()
|
||||
{
|
||||
var reconciler = new SecretReplicationReconciler(_store);
|
||||
bool fetched = false;
|
||||
|
||||
int applied = await reconciler.ReconcileAsync(
|
||||
remoteManifest: [],
|
||||
fetchAsync: (_, _) =>
|
||||
{
|
||||
fetched = true;
|
||||
return Task.FromResult<IReadOnlyList<StoredSecret>>([]);
|
||||
},
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, applied);
|
||||
Assert.False(fetched);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyAsync_evicts_the_resolver_cache_for_every_row()
|
||||
{
|
||||
var invalidator = new RecordingCacheInvalidator();
|
||||
var reconciler = new SecretReplicationReconciler(_store, invalidator);
|
||||
|
||||
await reconciler.ApplyAsync([Row("a/one", 1, T0), Row("b/two", 1, T0)], CancellationToken.None);
|
||||
|
||||
Assert.Equal(["a/one", "b/two"], invalidator.Invalidated.Select(n => n.Value));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyAsync_evicts_even_when_the_store_rejected_the_row_as_stale()
|
||||
{
|
||||
// Serving a stale plaintext for the rest of the TTL is worse than a redundant re-read.
|
||||
await _store.UpsertAsync(Row("api/key", 0, T0), CancellationToken.None);
|
||||
|
||||
var invalidator = new RecordingCacheInvalidator();
|
||||
var reconciler = new SecretReplicationReconciler(_store, invalidator);
|
||||
|
||||
await reconciler.ApplyAsync([Row("api/key", 0, T0.AddYears(-1))], CancellationToken.None);
|
||||
|
||||
Assert.Equal(["api/key"], invalidator.Invalidated.Select(n => n.Value));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplyAsync_skips_a_poison_row_and_keeps_applying_the_rest()
|
||||
{
|
||||
// A batch is not a transaction. Without per-row isolation, one row a peer cannot supply
|
||||
// cleanly abandons every row after it — and since the pull set is recomputed identically
|
||||
// each sweep, in the same manifest order, the same row poisons the same batch forever.
|
||||
var failures = new List<SecretName>();
|
||||
var reconciler = new SecretReplicationReconciler(
|
||||
_store, cacheInvalidator: null, onRowFailed: (name, _) => failures.Add(name));
|
||||
|
||||
// A null ciphertext cannot be persisted (NOT NULL column) — stands in for any bad row.
|
||||
StoredSecret poison = Row("bad/row", 1, T0) with { Ciphertext = null! };
|
||||
|
||||
int applied = await reconciler.ApplyAsync(
|
||||
[Row("good/first", 1, T0), poison, Row("good/second", 1, T0)],
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.Equal(2, applied);
|
||||
Assert.Equal(["bad/row"], failures.Select(n => n.Value));
|
||||
Assert.NotNull(await _store.GetAsync(new SecretName("good/first"), CancellationToken.None));
|
||||
Assert.NotNull(await _store.GetAsync(new SecretName("good/second"), CancellationToken.None));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ReconcileAsync_fetches_in_bounded_batches()
|
||||
{
|
||||
// An unbounded fetch breaks on real backends: SQL Server caps a command at 2100 parameters,
|
||||
// so a cold-starting node pulling a large hub's entire inventory would fail identically
|
||||
// every interval and never receive a single row.
|
||||
const int total = SecretReplicationReconciler.FetchBatchSize + 20;
|
||||
|
||||
List<SecretManifestEntry> remote = [.. Enumerable.Range(0, total)
|
||||
.Select(i => Entry($"bulk/{i}", 1, T0))];
|
||||
|
||||
var batchSizes = new List<int>();
|
||||
var reconciler = new SecretReplicationReconciler(_store);
|
||||
|
||||
int applied = await reconciler.ReconcileAsync(
|
||||
remote,
|
||||
fetchAsync: (names, _) =>
|
||||
{
|
||||
batchSizes.Add(names.Count);
|
||||
return Task.FromResult<IReadOnlyList<StoredSecret>>(
|
||||
[.. names.Select(n => Row(n.Value, 1, T0))]);
|
||||
},
|
||||
CancellationToken.None);
|
||||
|
||||
Assert.Equal(total, applied);
|
||||
Assert.Equal(2, batchSizes.Count);
|
||||
Assert.All(batchSizes, size => Assert.True(size <= SecretReplicationReconciler.FetchBatchSize));
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||
foreach (string path in new[] { _dbPath, _dbPath + "-wal", _dbPath + "-shm" })
|
||||
{
|
||||
if (File.Exists(path))
|
||||
{
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+73
-15
@@ -23,16 +23,63 @@ construction).
|
||||
fix until bumped.
|
||||
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
|
||||
`Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
|
||||
- **G-7 clustered replication is designed + planned, not built** — plan at
|
||||
`docs/plans/2026-07-17-secrets-g7-*` (resolved to build the shared SQL-Server `ISecretStore`;
|
||||
Akka replicator deferred to phase 2). No `ISecretStore` SQL-Server implementation exists yet.
|
||||
- Library suite: **97 passing, 1 skipped**, green 2026-07-18.
|
||||
- **G-7 clustered replication is BUILT (2026-07-18, lib `0.2.0`) — both options, not just one.**
|
||||
The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build
|
||||
covers **both**, as two new packages:
|
||||
- **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** — `SqlServerSecretStore : ISecretStore` (all 7
|
||||
members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and *two*
|
||||
topologies: `AddZbSecretsSqlServerStore` (one shared store, Option A) and
|
||||
`AddZbSecretsSqlServerReplication` (local store + hub, with a bidirectional sweep).
|
||||
- **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** — peer-to-peer over distributed pub/sub
|
||||
(Option B): live broadcast + periodic manifest anti-entropy, an explicit
|
||||
`SerializerWithStringManifest`, ciphertext-only wire DTOs.
|
||||
- **Core gained the pieces both need:** `ISecretsStoreMigrator` (so the hosted service + CLI are
|
||||
store-agnostic), `SecretLastWriterWins` (one definition of the LWW tie-break, shared by every
|
||||
store and reconciler), `SecretReplicationReconciler`, and `ReplicatingSecretStore`.
|
||||
- **The `ISecretReplicator` seam had no caller** — nothing invoked `PublishAsync`, so local writes
|
||||
would never have propagated. `ReplicatingSecretStore` decorates the store, which fixes it for
|
||||
the UI, the CLI, and any future write path at once.
|
||||
- Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
|
||||
`AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
|
||||
- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
|
||||
against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
|
||||
divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
|
||||
(real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
|
||||
late-joiner catch-up, and malformed-peer-input handling.
|
||||
|
||||
### Open
|
||||
|
||||
- ⬜ Bump the four apps `0.1.2` → `0.1.3` to pick up KEK rotation + the SQLitePCLRaw security fix.
|
||||
- ⬜ Execute the G-7 plan (shared SQL-Server `ISecretStore`).
|
||||
- ⬜ Bump the four apps `0.1.2` → `0.1.3`/`0.2.0` to pick up KEK rotation + the SQLitePCLRaw security fix.
|
||||
- ⬜ Publish `0.2.0` (5 packages) to the Gitea feed — packed and scanned clean, **not yet pushed**.
|
||||
- ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed).
|
||||
- ⬜ Per-app live wonder gates for the G-8 rotation path.
|
||||
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
|
||||
drift symptom, not the forwarding filter itself (noted in the test).
|
||||
|
||||
### Post-build code review (2026-07-18) — 6 defects found and fixed
|
||||
|
||||
An independent review caught real bugs that a green suite had not:
|
||||
|
||||
- **CRITICAL — both replication modes were dead on arrival.** The decorators resolve the *undecorated*
|
||||
local store by concrete type, but core registered `SqliteSecretStore` only behind `ISecretStore`.
|
||||
Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with
|
||||
`No service for type 'SqliteSecretStore'`), fixed, and covered by new DI-resolution tests.
|
||||
- **Unbounded fetch.** A cold-starting node pulls the peer's entire inventory in one call; past
|
||||
~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the
|
||||
node never converges. Now chunked at 500.
|
||||
- **Poison row aborted the batch.** One unapplyable row abandoned every row after it, and since the
|
||||
pull set recomputes in the same order each sweep, it would poison the same batch forever. Now
|
||||
isolated per row and logged.
|
||||
- **`Enum.Parse` on peer input.** An overflowing numeric string throws `OverflowException`, which the
|
||||
actor's `ArgumentException` filter did not catch → actor restart loop on redelivery; and any
|
||||
in-range number was accepted as an undefined enum value. Now `TryParse` + `IsDefined`.
|
||||
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
|
||||
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.
|
||||
|
||||
One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's
|
||||
`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and
|
||||
three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as
|
||||
written.
|
||||
|
||||
## Execution status (2026-07-16)
|
||||
|
||||
@@ -146,15 +193,26 @@ mounted key)** for clustered pairs (hard requirement — same KEK on every node)
|
||||
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
|
||||
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.
|
||||
|
||||
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)
|
||||
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A —
|
||||
a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared
|
||||
Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure
|
||||
modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred
|
||||
phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the
|
||||
`ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable
|
||||
plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
|
||||
+ [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.**
|
||||
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib `0.2.0`)
|
||||
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork was designed as *either/or* with
|
||||
Option A recommended and Option B deferred. **Both were built** — the fork is now a deployment
|
||||
choice rather than a library one, and the two packages are independent (an app takes whichever
|
||||
matches its availability requirement, or neither).
|
||||
|
||||
| Package | Topologies | When |
|
||||
|---|---|---|
|
||||
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | shared store (Option A) · local + hub | every node can reach a shared DB |
|
||||
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | peer-to-peer over the cluster (Option B) | a node must resolve while **partitioned** |
|
||||
|
||||
Three constraints hold across all of them, and each is documented at the API and in the runbook:
|
||||
**same KEK on every node** (a mismatch fails closed and looks like corruption); **ciphertext only
|
||||
crosses trust boundaries**; **re-wraps do not replicate**, so `rewrap-all` runs once per independent
|
||||
store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no
|
||||
merge — stated plainly in the README rather than left for an operator to discover.
|
||||
|
||||
Design + plan: [`…-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
|
||||
+ [`…-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md).
|
||||
Operator runbook: [`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md).
|
||||
|
||||
### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
|
||||
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
|
||||
|
||||
@@ -1,5 +1,12 @@
|
||||
# G-7 — Clustered secret replication: design & fork resolution
|
||||
|
||||
> **✅ SUPERSEDED BY THE BUILD, 2026-07-18.** This document resolved the fork to Option A and
|
||||
> deferred Option B. On the user's instruction **both were built**, as
|
||||
> `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared-store *and* hub modes) and
|
||||
> `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`. The fork analysis below is still the right way to
|
||||
> *choose* between them — it is now a deployment decision, not a build one. The "Decision" section's
|
||||
> YAGNI reasoning for deferring Option B no longer applies.
|
||||
|
||||
> **Status:** design-only (G-7 in [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md)).
|
||||
> Companion executable plan: [`2026-07-17-secrets-g7-sqlserver-store.md`](2026-07-17-secrets-g7-sqlserver-store.md).
|
||||
> Prerequisite G-2…G-6 have landed for both clustered apps (ScadaBridge, OtOpcUa); G-8 (KEK
|
||||
|
||||
@@ -1,5 +1,28 @@
|
||||
# G-7 (Option A) — Shared SQL-Server `ISecretStore` Implementation Plan
|
||||
|
||||
> **✅ EXECUTED 2026-07-18 — and the scope was widened on the user's instruction.** This plan covered
|
||||
> Option A only, with Option B (Akka) deferred. The user directed that **both** be built, packaged as
|
||||
> `ZB.MOM.WW.Secrets.Replicator.SqlServer` and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` — so the
|
||||
> "Deferred phase-2" section below is **also built**, and the naming here (`SqlServer/` inside the
|
||||
> core package; `ZB.MOM.WW.Secrets.Akka`) is superseded by two standalone packages.
|
||||
>
|
||||
> Other deltas from what was planned, all deliberate:
|
||||
> - **Two SQL-Server topologies, not one.** Shared-store (as planned) *plus* a local-store-with-hub
|
||||
> mode, so the package's `Replicator` name is honest and partition tolerance is available without
|
||||
> Akka.
|
||||
> - **`SecretLastWriterWins` extracted to Abstractions.** The plan flagged Task 2 high-risk because
|
||||
> the LWW semantics "MUST match SQLite exactly"; a shared predicate makes that structural instead
|
||||
> of a review obligation.
|
||||
> - **`ReplicatingSecretStore` added.** Discovered during the build that nothing ever called
|
||||
> `ISecretReplicator.PublishAsync` — the seam was inert. A store decorator fixes every write path.
|
||||
> - **`ISecretReplicationHub` added**, so the hub sweep is testable offline rather than requiring a
|
||||
> live SQL Server for every convergence case.
|
||||
> - **Anti-entropy is bidirectional.** Pull-only would strand a write made while a peer was down:
|
||||
> the peer never learns the name exists, so it can never ask for it.
|
||||
>
|
||||
> Verified: 164 pass / 1 skip / 0 warnings, including 15 live tests against a real SQL Server 2022
|
||||
> and a 9-test in-process 2-node Akka cluster. See [`components/secrets/GAPS.md`](../../components/secrets/GAPS.md) §G-7.
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans (or
|
||||
> subagent-driven-development) to implement this plan task-by-task.
|
||||
|
||||
|
||||
@@ -1,11 +1,39 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-17-secrets-g7-sqlserver-store.md",
|
||||
"status": "executed 2026-07-18 with widened scope (both options built as .Replicator.* packages)",
|
||||
"tasks": [
|
||||
{"id": 0, "subject": "Task 1: SQL-Server schema + migrator", "status": "pending"},
|
||||
{"id": 1, "subject": "Task 2: SqlServerSecretStore : ISecretStore", "status": "pending", "blockedBy": [0]},
|
||||
{"id": 2, "subject": "Task 3: DI store-selection wiring", "status": "pending", "blockedBy": [1]},
|
||||
{"id": 3, "subject": "Task 4: Docs + adoption notes", "status": "pending", "blockedBy": [1]}
|
||||
{
|
||||
"id": 0,
|
||||
"subject": "Task 1: SQL-Server schema + migrator",
|
||||
"status": "completed"
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 2: SqlServerSecretStore : ISecretStore",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
0
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 3: DI store-selection wiring",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
1
|
||||
],
|
||||
"note": "Delivered as AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication in a standalone package rather than a SecretsOptions.Store enum in core."
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 4: Docs + adoption notes",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
1
|
||||
],
|
||||
"note": "docs/operations/clustered-secrets.md covers all three topologies."
|
||||
}
|
||||
],
|
||||
"deferredPhase2": "Option B — ZB.MOM.WW.Secrets.Akka replicator (build only on a stated partition-tolerance requirement); see plan doc.",
|
||||
"lastUpdated": "2026-07-17"
|
||||
"deferredPhase2": "BUILT 2026-07-18 as ZB.MOM.WW.Secrets.Replicator.AkkaDotNet (no longer deferred).",
|
||||
"lastUpdated": "2026-07-18"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user