diff --git a/CLAUDE.md b/CLAUDE.md index 90f01c2..89f8708 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -149,7 +149,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | | Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.2** (4-arg content-root call site included) and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); **merged to `main` + pushed 2026-08-13 (`e42ff31`)**. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of ``. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `