dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(secrets): cluster replication via SQL Server and Akka.NET (G-7, 0.2.0)

Browse Source 
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.

Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.

Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.

Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-18 04:08:23 -04:00
parent e46060fada
commit dd0a846b64
55 changed files with 4848 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
components/secrets/GAPS.md
+73 -15
View File
@@ -23,16 +23,63 @@ construction).
fix until bumped.
- **All four apps are pinned to `0.1.2`** — OtOpcUa, MxAccessGateway, ScadaBridge take
`Secrets` + `.Abstractions` + `.Ui`; HistorianGateway takes `Secrets` + `.Ui`.
- **G-7 clustered replication is designed + planned, not built** — plan at
`docs/plans/2026-07-17-secrets-g7-*` (resolved to build the shared SQL-Server `ISecretStore`;
Akka replicator deferred to phase 2). No `ISecretStore` SQL-Server implementation exists yet.
- Library suite: **97 passing, 1 skipped**, green 2026-07-18.
- **G-7 clustered replication is BUILT (2026-07-18, lib `0.2.0`) — both options, not just one.**
The plan scoped Option A (shared SQL store) and deferred Option B (Akka) to phase 2; the build
covers **both**, as two new packages:
- **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** — `SqlServerSecretStore : ISecretStore` (all 7
members in T-SQL, semantics identical to SQLite), schema + idempotent migrator, and *two*
topologies: `AddZbSecretsSqlServerStore` (one shared store, Option A) and
`AddZbSecretsSqlServerReplication` (local store + hub, with a bidirectional sweep).
- **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** — peer-to-peer over distributed pub/sub
(Option B): live broadcast + periodic manifest anti-entropy, an explicit
`SerializerWithStringManifest`, ciphertext-only wire DTOs.
- **Core gained the pieces both need:** `ISecretsStoreMigrator` (so the hosted service + CLI are
store-agnostic), `SecretLastWriterWins` (one definition of the LWW tie-break, shared by every
store and reconciler), `SecretReplicationReconciler`, and `ReplicatingSecretStore`.
- **The `ISecretReplicator` seam had no caller** — nothing invoked `PublishAsync`, so local writes
would never have propagated. `ReplicatingSecretStore` decorates the store, which fixes it for
the UI, the CLI, and any future write path at once.
- Naming: the user chose `.Replicator.*` (not the plan's `.Replication.*` / `ZB.MOM.WW.Secrets.Akka`);
`AkkaDotNet` rather than `Akka` also avoids a namespace collision with the `Akka` root namespace.
- Library suite: **182 passing, 1 skipped**, green 2026-07-18 — including **15 live SQL-Server tests
against a real SQL Server 2022** (the SQLite store's suite ported case-for-case, so any behavioural
divergence between the two stores fails a test) and a **9-test in-process 2-node Akka cluster**
(real remoting, real stores) covering write→peer, delete propagation, both anti-entropy directions,
late-joiner catch-up, and malformed-peer-input handling.
### Open
- ⬜ Bump the four apps `0.1.2` → `0.1.3` to pick up KEK rotation + the SQLitePCLRaw security fix.
- ⬜ Execute the G-7 plan (shared SQL-Server `ISecretStore`).
- ⬜ Bump the four apps `0.1.2` → `0.1.3`/`0.2.0` to pick up KEK rotation + the SQLitePCLRaw security fix.
- ⬜ Publish `0.2.0` (5 packages) to the Gitea feed — packed and scanned clean, **not yet pushed**.
- ⬜ Adopt a topology in ScadaBridge / OtOpcUa (config + shared KEK; no code change needed).
- ⬜ Per-app live wonder gates for the G-8 rotation path.
- ⬜ Akka self-echo forwarding is only observable with **three** nodes; the 2-node rig covers the
drift symptom, not the forwarding filter itself (noted in the test).
### Post-build code review (2026-07-18) — 6 defects found and fixed
An independent review caught real bugs that a green suite had not:
- **CRITICAL — both replication modes were dead on arrival.** The decorators resolve the *undecorated*
local store by concrete type, but core registered `SqliteSecretStore` only behind `ISecretStore`.
Every unit test passed because nothing ever built a container. Reproduced (3 tests fail with
`No service for type 'SqliteSecretStore'`), fixed, and covered by new DI-resolution tests.
- **Unbounded fetch.** A cold-starting node pulls the peer's entire inventory in one call; past
~2100 names that exceeds SQL Server's parameter cap and fails identically every interval, so the
node never converges. Now chunked at 500.
- **Poison row aborted the batch.** One unapplyable row abandoned every row after it, and since the
pull set recomputes in the same order each sweep, it would poison the same batch forever. Now
isolated per row and logged.
- **`Enum.Parse` on peer input.** An overflowing numeric string throws `OverflowException`, which the
actor's `ArgumentException` filter did not catch → actor restart loop on redelivery; and any
in-range number was accepted as an undefined enum value. Now `TryParse` + `IsDefined`.
- **Null crypto blobs passed the boundary** (`required byte[]` means *present*, not *non-null*).
- **A failed pull-read replied with silence** — the peer waited forever with no diagnostic.
One reported "critical" was a **false positive**: `Self` used after `await` in the actor. Akka's
`ActorBase` caches `Self` in an instance field (unlike `Context`, which is `[ThreadStatic]`), and
three anti-entropy tests that traverse exactly that path pass — they could not if it threw. Left as
written.
## Execution status (2026-07-16)
@@ -146,15 +193,26 @@ mounted key)** for clustered pairs (hard requirement — same KEK on every node)
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A —
a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared
Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure
modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred
phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the
`ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable
plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.**
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) ✅ BUILT (2026-07-18, lib `0.2.0`)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork was designed as *either/or* with
Option A recommended and Option B deferred. **Both were built** — the fork is now a deployment
choice rather than a library one, and the two packages are independent (an app takes whichever
matches its availability requirement, or neither).
| Package | Topologies | When |
|---|---|---|
| `ZB.MOM.WW.Secrets.Replicator.SqlServer` | shared store (Option A) · local + hub | every node can reach a shared DB |
| `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` | peer-to-peer over the cluster (Option B) | a node must resolve while **partitioned** |
Three constraints hold across all of them, and each is documented at the API and in the runbook:
**same KEK on every node** (a mismatch fails closed and looks like corruption); **ciphertext only
crosses trust boundaries**; **re-wraps do not replicate**, so `rewrap-all` runs once per independent
store. Replicated topologies additionally accept eventual consistency and last-writer-wins with no
merge — stated plainly in the README rather than left for an operator to discover.
Design + plan: [`…-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`…-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md).
Operator runbook: [`ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`](../../ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md).
### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in