dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(secrets): build G-8 KEK rotation (RewrapAll); design+plan G-7 clustered replication

Browse Source 
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
  (bodies never re-encrypted; revision/timestamps preserved -> invisible to
  cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
  wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
  updates only the 4 wrap columns + kek_id, compare-and-swap on the current
  wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
  review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
  tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
  fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
  JSON counts report; README section + operator runbook.

Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).

G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
  replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
  executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.

Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
This commit is contained in:
Joseph Doherty
2026-07-17 02:55:43 -04:00
parent b009507e10
commit d82d3451e7
24 changed files with 1280 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
components/secrets/GAPS.md
+37 -8
View File
@@ -39,6 +39,14 @@ construction).
parallelization for Host.Tests).
- **OtOpcUa — G-2 + G-4 + G-5 + G-6 DONE (the LAST app — all three now adopted).** Executed 2026-07-16 via the plan below (10 tasks, 2 slices, subagent-driven, classification-driven reviews), **FF-merged + pushed to `origin/master` (lmxopcua @ `872cf7e3`, `ec6598ce..872cf7e3`)**. Clean build (0 errors). The full offline suite caught **one real regression** — `AddOtOpcUaDriverFactories`' registration now resolves `ISecretResolver` (Galaxy/OpcUaClient secret: refs), so two `ResilienceInvokerFactoryRegistrationTests` that built a minimal container threw `No service for ISecretResolver`; fixed test-only by registering a stub resolver (production always has it via the unconditional `AddZbSecrets`, and `GetRequiredService` correctly fails-fast for a genuinely misconfigured host). Touched-area suites green (Galaxy 338, OpcUaClient 142, AdminUI 662); remaining non-passes are unrelated (flaky Akka/Roslyn timing tests pass in isolation; the environmental `AbCip_Green_AgainstSim` sim-fixture test — zero AbCip code touched). **Three deliberate deviations from the plan's literal text** (verified corrections, same class as the CPM miss the plan itself caught): (1) **G-4 does NOT commit a `${secret:}` token into the default appsettings** — the expander is fail-closed AND section-agnostic and the only committed secret-shaped value is `ServerHistorian:ApiKey=""` (empty, disabled-by-default), so a committed token would break every dev/CI/`TwoNodeClusterHarness` boot for zero at-rest benefit; instead documented the token-delivery convention (`docs/security.md`) + updated the ServerHistorian comments, and proved fail-closed live (unseeded → `SecretNotFoundException` at pre-host expansion; seeded → resolves). (2) **G-5 shipped docs-only** (`docs/operations/2026-07-16-secrets-clustered-master-key.md`) — hardcoding `Source=File`+shared paths into the committed role-overlay appsettings (also consumed by dev + integration) would break those boots; base stays `Source=Environment`. Mirrors the ScadaBridge G-5 resolution. (3) **Task 8 resolves in the factory/driver only, NOT the probe** — the v3 `OpcUaClientDriverProbe` is unauthenticated `GetEndpoints`-only and never reads `Password`/`UserCertificatePassword`, so probe resolution would be dead code. Layer-B resolution mirrors Galaxy: the sync driver-registry factory means resolution happens lazily at the driver's async session-open (`GalaxyDriver.BuildClientOptionsAsync` / `OpcUaClientDriver.InitializeAsync`), threaded via ctor + `DriverFactoryBootstrap` (real resolver from DI; `NullSecretResolver` null-object backs test/parse paths only, fail-closed on `secret:`). `OpcUaClientDriverOptions` was converted `sealed class` → `sealed record` for the `with`-expression (no positional params → identical JSON; verified no reference-equality/dict-key/ToString-log usages → no leak/behavior change). AdminUI G-2c is structurally leak-proof: `DriverConfig` persists as an opaque JSON string via `RawTreeService`, which has NO resolver dependency. **Live wonder gate PROVEN 2026-07-16 (VPN up):** a throwaway harness drove OtOpcUa's real `GalaxyDriverBrowser` (resolves the `secret:` `ApiKeySecretRef` via `ISecretResolver` → `GalaxySecretRef.ResolveApiKeyAsync` → connects) against the REAL production MxGateway `wonder-app-vd03:5120`. **Dummy `secret:` ref → `MxGatewayAuthenticationException` (proves the resolved value is transmitted, not fail-closed-first); real `metadata:read` key → CONNECTED + browsed the real Galaxy root (10 nodes: ControlSystem/EUR/KPI_CNC/NAM/_TestArea/MESReceiver…).** Temp key minted via the gateway dashboard, then **revoked (functionally proven — the same key that browsed 10 nodes then returned Unauthenticated) + deleted** (row gone; no prod key touched; the dashboard key-list is stale-cached so revoke/delete were verified FUNCTIONALLY, not by the list UI). Harness + scratch stores destroyed. **All of G-2/G-4/G-5/G-6 now DONE + live-proven across all three apps + mxaccessgw.**
**Plan re-verified 2026-07-16 against `origin/master` @ `ec6598ce`** (the merged **v3.0 dual-namespace** rewrite, PR #472). The v3.0 rewrite did NOT touch the driver-secret flow, so the approach held; the plan was corrected for: OtOpcUa **is** on Central Package Management (was mis-stated as inline-versioned), `OpcUaClientDriverOptions` lives in `…Driver.OpcUaClient.Contracts` + `GalaxyDriverBrowser` in `…Driver.Galaxy.Browser`, and `DriverHostActor.cs` line shifts (1704/1756/1790 → 1766/1818/1852). Program.cs / Security / AdminUI anchors survived essentially unchanged.
- **G-8 (KEK rotation) — BUILT; G-7 (clustered replication) — DESIGNED + PLANNED (2026-07-17).**
G-8 shipped into `ZB.MOM.WW.Secrets` (lib `0.1.3`): the `Rewrap` DEK primitive, CAS-guarded
`ApplyRewrapAsync`, `KekRotationService.RewrapAllAsync`, the `secret rewrap-all` CLI verb, and an
operator runbook — full suite green (82+15, 0 regressions) + end-to-end CLI smoke, adversarial
crypto review PASS with a review-caught TOCTOU closed via compare-and-swap. G-7's SQL-store-vs-Akka
fork is resolved (build the shared SQL-Server `ISecretStore`; Akka replicator deferred phase-2)
with a design doc + executable plan + `.tasks.json`, **no code built**. See the G-7/G-8 backlog
entries below. Not yet committed/pushed (awaiting go-ahead).
## Design + implementation plans (2026-07-16)
@@ -55,7 +63,7 @@ code-verified anchors, co-located `.tasks.json`):
- [`docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md`](../../docs/plans/2026-07-16-secrets-adoption-mxaccessgw.md)
— **G-4 + G-5 + G-6** (8 tasks; single-box, no Layer B).
G-7/G-8 remain design-only/deferred (below).
**G-8 (KEK rotation) is now BUILT; G-7 (clustered replication) is now DESIGNED + PLANNED** (2026-07-17, below).
## Cross-cutting observations
@@ -109,14 +117,35 @@ mounted key)** for clustered pairs (hard requirement — same KEK on every node)
Add the RCL page to each dashboard (OtOpcUa AdminUI, MxGateway Server, ScadaBridge CentralUI)
and map each app's admin role onto `secrets:manage` / `secrets:reveal`.
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — build `ZB.MOM.WW.Secrets.Akka`
Deferred until G-2…G-6 land for a clustered app. Options per SPEC: shared SQL-Server
`ISecretStore` (simplest — mirrors the shared Data-Protection key ring) **or** the Akka
replicator (LWW, anti-entropy resync, tombstones). Requires a shared KEK.
### G-7 — Clustered replication (ScadaBridge, OtOpcUa) — DESIGNED + PLANNED (2026-07-17)
The SPEC's "shared SQL-Server store **vs** Akka replicator" fork is **resolved: build Option A —
a shared SQL-Server `ISecretStore`** (both clustered apps already run shared SQL + a shared
Data-Protection key ring, so it delivers cluster-wide secrets with zero distributed-systems failure
modes); the Akka replicator (`ZB.MOM.WW.Secrets.Akka`, LWW/anti-entropy/tombstones) is a **deferred
phase-2** to build only on a stated partition-tolerance requirement. Both require a shared KEK; the
`ISecretReplicator` seam stays in place so Option B remains a drop-in later. Design + executable
plan (+ `.tasks.json`): [`docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md`](../../docs/plans/2026-07-17-secrets-g7-clustered-replication-design.md)
+ [`docs/plans/2026-07-17-secrets-g7-sqlserver-store.md`](../../docs/plans/2026-07-17-secrets-g7-sqlserver-store.md). **No code built yet — plan is ready to execute.**
### G-8 — KEK-rotation runbook + `RewrapAll`
The `RewrapAll(oldKek, newKek)` admin primitive + an operator runbook. Deferred (design-only
in this cut).
### G-8 — KEK-rotation `RewrapAll` + runbook ✅ BUILT (2026-07-17, lib `0.1.3`)
The `RewrapAll(oldKek, newKek)` admin primitive + operator runbook are **built + fully tested** in
`ZB.MOM.WW.Secrets`. Because the envelope wraps a per-secret DEK under the KEK, rotation **re-wraps
DEKs only** — bodies are never re-encrypted, revision/timestamps are preserved (invisible to cluster
LWW). Shipped: `ISecretCipher.Rewrap(row, oldKek, newKek)` (fail-closed on wrong old KEK; DEK zeroed
on all paths; also catches malformed-length wraps → `SecretDecryptionException`),
`ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek)` (updates only the 4 wrap
columns + `kek_id`, **compare-and-swap on the current wrapped DEK** so a concurrent write can't
corrupt a row — review-caught TOCTOU), `KekRotationService.RewrapAllAsync` + `RewrapReport`
(enumerates all rows incl. tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK), the `secret rewrap-all` CLI verb (key material only via
env-var-name/file-path — never a literal), README section, and the operator runbook
[`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`](../../ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md).
**Verified:** full offline suite green (82 core + 15 UI, 0 regressions) + an end-to-end CLI smoke
(migrate → decrypts under new KEK, fail-closed under old → idempotent re-run → identical-KEK guard).
Version bumped `0.1.2`→`0.1.3` (additive interface members); **republish to the Gitea feed is a
separate opt-in step (the apps don't need rewrap — it's a library/CLI admin primitive).** Reviewed
by an adversarial crypto/correctness pass (all 7 categories PASS); the one real finding (concurrent-
write TOCTOU) is fixed by the CAS above, plus two low/nit fixes folded in.
## Out of scope (this component)