diff --git a/CLAUDE.md b/CLAUDE.md index 7638374..78d942f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-th |---|---|---|---|---| | **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree address space (v3.0, `master` since 2026-07-16)** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal); every value has one source fanned to both NodeIds. **This is the wire contract ScadaBridge binds against** (ScadaBridge consumes v3 end-to-end; its #14 is closed). Drivers: Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient, `Sql` (SQL Server / PostgreSQL / Oracle, read-only poll), `Mqtt` (Plain + Sparkplug B, writable where a tag declares a destination), `MTConnect` (read-only), Calculation, and AVEVA Galaxy as a standard Equipment-kind driver via in-process `GalaxyDriver` → gRPC → **mxaccessgw**. Sole historian backend is **HistorianGateway**. Address-space availability guarantee: an unreadable deployment artifact is *no answer*, never an empty configuration — nodes hold last-known-good, and a driver-only node boots and serves from its LocalDb cache with central SQL down (ServiceLevel stays 240/250). Redundancy: one independent 2-node Akka mesh per application `Cluster`; Primary = oldest Up `driver` member (ServiceLevel 250) + one Secondary (240) **per `Cluster`**; cross-mesh commands ride gRPC with per-cluster PSK auth (Akka `ClusterClient` fully deleted 2026-08-06). Data plane is machine-to-machine: OPC UA UserName `apikey` + `otua__` minted on AdminUI `/api-keys`; `OpcUa:AllowAnonymous` defaults `true` (anonymous is a normal grantable principal); LDAP is AdminUI-login only. Node ACLs are enforced **only** behind `Security:Acl:Enabled` (default `false`); with it on, scoped `NodeAcl` grants decide **every** data-plane operation incl. browse/read/subscribe, fail closed, and reach open sessions. Every driver's authoring path now consults the driver's own runtime parse, so what authoring accepts is what the driver serves (tracker zeroed 2026-08-10, origin `d06e92c6`). **Full change history, live-gate records and the per-issue consumer-impact record for ScadaBridge: `.claude/skills/sister-project-history/otopcua.md` (skill `sister-project-history`).** | | **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** | -| **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | **Current `main`: `23a139a4` (2026-09-28: site secrets ride the artifact deploy and the gRPC secrets hub is retired — see below). Previous tip `5cc09b08` (2026-09-28: site scripts gain a read-only, allow-listed `Files` global — `ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots`, off by default — used by the in-memory MES/Delmia/alarm-API test machines SBTEST01/02 (`docker/seed-mes-test-machine.sh`) to parse downloaded recipe files; DEV vd03 and QA vq03/vq04 on `b3021e7c` since 2026-09-28, migrations 57; register row 126 closed) — adds site artifact staleness (`Site.ArtifactRevisionHash` stamped on every successful artifact deploy, migration `AddSiteArtifactRevision`; Sites page *Artifacts current / out of date / never deployed* badge), a rail-footer **Pending updates** indicator over a process-wide `PendingUpdatesCache` (stale instances + stale-artifact sites, Deployer/Admin), and a scrollable rail nav; design `docs/plans/2026-09-14-pending-updates-design.md`. Previous tip `9d529b82` (2026-09-11) — adds the Gitea #36 reset-aware handshake (`WriteBatchAndWaitAsync` reset-echo overload), the computed-alarm `IsMesReportable` flag (template checkbox + nullable instance override, `Alarms.MesReportableAsync()`, migration `AddAlarmMesReportable`; native alarms stay on the 900–999 band) and a nav fix nesting Flow Debugger in the Deployment section. DEPLOYED: `b3021e7c` on DEV vd03 and the QA pair vq03/vq04 (2026-09-28, migrations 55→57; records `docs/deployment/deployment-records/2026-09-28-wonder-app-{vd03-fe89363b,vq03-vq04-b3021e7c}.md`).** **Merged `23a139a4` (2026-09-28): site secrets now ride the artifact deploy and the pull-only gRPC secrets hub is retired in ScadaBridge** (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; central shares one secret store via `Secrets:Store:Provider=SqlServer`; rig live gate PASS 8/8, vd03/QA NOT deployed — rollout tracked as ScadaBridge register row 127 — see `ZB.MOM.WW.Secrets/CLAUDE.md` for the unpublished `0.6.3` hub-logging work this superseded for ScadaBridge). Management/API/CLI parity complete (`docs/plans/2026-09-10-management-api-cli-parity-review.md`): all 149 eligible contracts exposed in 20 CLI groups, 19 missing operations added, deadline/registry/outcome gaps fixed; CLI documentation checked against all 159 leaf commands; 9554 tests verified and primary 8-node rig rebuilt with read-only smoke checks. Previously completed 2026-09-09 whole-system arch review remediation (`docs/plans/2026-09-09-arch-review.md`: 92 findings; all 13 High + 33 Medium fixed; all 46 Low findings also committed and pushed (§11); 9478 solution tests and deployed browser smoke pass, primary 8-node rig rebuilt). Adds the `ZB.MOM.WW.ScadaBridge.CentralUI.SandboxWorker` child-process project and a coordinated central-pair migration that encrypts DataConnection protocol config at rest (old binaries cannot read protected rows). Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. **Inbound API keys live on the shared SQL Server store on a central pair** (design 2026-09-02, ScadaBridge register row 108): ScadaBridge is Auth's first consumer of the new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package at `0.3.0`, pointing both central nodes at one `zbauth` schema in the configuration database instead of each node's own per-node SQLite file. **Flow Debugger (including the instance picker and v1.4 step logs, 2026-09-09):** Central UI + CLI trace of one inbound-API method or value-change flow across Central / Site / Tags / External swimlanes, Live or Replay, built on `ScriptTrace` audit rows and the outbound-call audit rows; arch-reviewed and remediated the same day (F1–F13 — redacted + byte-capped inline trace copy, route-time site scope on Live runs, Replay open to `OperationalAudit` roles under `RequireFlowDebugger`; `docs/plans/2026-09-06-flow-debugger-arch-review.md` + `-remediation.md`); script entry (site → instance → script, 2026-09-08). | +| **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | **Current `main`: `2650f794` (2026-09-28: site-secrets follow-ups — a central secret-store outage fails only the sites that reference a secret, removal-only deploys are audited, each site node's mirror failures (standby included) ride the health report via a 12th replicated table `secret_mirror_status`, artifact-deploy CLI commands exit 1 on a failed site; rig gate 9/9). Previous tip `23a139a4` (2026-09-28: site secrets ride the artifact deploy and the gRPC secrets hub is retired — see below). Previous tip `5cc09b08` (2026-09-28: site scripts gain a read-only, allow-listed `Files` global — `ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots`, off by default — used by the in-memory MES/Delmia/alarm-API test machines SBTEST01/02 (`docker/seed-mes-test-machine.sh`) to parse downloaded recipe files; DEV vd03 and QA vq03/vq04 on `b3021e7c` since 2026-09-28, migrations 57; register row 126 closed) — adds site artifact staleness (`Site.ArtifactRevisionHash` stamped on every successful artifact deploy, migration `AddSiteArtifactRevision`; Sites page *Artifacts current / out of date / never deployed* badge), a rail-footer **Pending updates** indicator over a process-wide `PendingUpdatesCache` (stale instances + stale-artifact sites, Deployer/Admin), and a scrollable rail nav; design `docs/plans/2026-09-14-pending-updates-design.md`. Previous tip `9d529b82` (2026-09-11) — adds the Gitea #36 reset-aware handshake (`WriteBatchAndWaitAsync` reset-echo overload), the computed-alarm `IsMesReportable` flag (template checkbox + nullable instance override, `Alarms.MesReportableAsync()`, migration `AddAlarmMesReportable`; native alarms stay on the 900–999 band) and a nav fix nesting Flow Debugger in the Deployment section. DEPLOYED: `b3021e7c` on DEV vd03 and the QA pair vq03/vq04 (2026-09-28, migrations 55→57; records `docs/deployment/deployment-records/2026-09-28-wonder-app-{vd03-fe89363b,vq03-vq04-b3021e7c}.md`).** **Merged `23a139a4` (2026-09-28): site secrets now ride the artifact deploy and the pull-only gRPC secrets hub is retired in ScadaBridge** (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; central shares one secret store via `Secrets:Store:Provider=SqlServer`; rig live gate PASS 8/8, vd03/QA NOT deployed — rollout tracked as ScadaBridge register row 127 — see `ZB.MOM.WW.Secrets/CLAUDE.md` for the unpublished `0.6.3` hub-logging work this superseded for ScadaBridge). Management/API/CLI parity complete (`docs/plans/2026-09-10-management-api-cli-parity-review.md`): all 149 eligible contracts exposed in 20 CLI groups, 19 missing operations added, deadline/registry/outcome gaps fixed; CLI documentation checked against all 159 leaf commands; 9554 tests verified and primary 8-node rig rebuilt with read-only smoke checks. Previously completed 2026-09-09 whole-system arch review remediation (`docs/plans/2026-09-09-arch-review.md`: 92 findings; all 13 High + 33 Medium fixed; all 46 Low findings also committed and pushed (§11); 9478 solution tests and deployed browser smoke pass, primary 8-node rig rebuilt). Adds the `ZB.MOM.WW.ScadaBridge.CentralUI.SandboxWorker` child-process project and a coordinated central-pair migration that encrypts DataConnection protocol config at rest (old binaries cannot read protected rows). Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. **Inbound API keys live on the shared SQL Server store on a central pair** (design 2026-09-02, ScadaBridge register row 108): ScadaBridge is Auth's first consumer of the new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package at `0.3.0`, pointing both central nodes at one `zbauth` schema in the configuration database instead of each node's own per-node SQLite file. **Flow Debugger (including the instance picker and v1.4 step logs, 2026-09-09):** Central UI + CLI trace of one inbound-API method or value-change flow across Central / Site / Tags / External swimlanes, Live or Replay, built on `ScriptTrace` audit rows and the outbound-call audit rows; arch-reviewed and remediated the same day (F1–F13 — redacted + byte-capped inline trace copy, route-time site scope on Live runs, Replay open to `OperationalAudit` roles under `RequireFlowDebugger`; `docs/plans/2026-09-06-flow-debugger-arch-review.md` + `-remediation.md`); script entry (site → instance → script, 2026-09-08). | | **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) a full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib. No COM, no x86 worker. **Owns `AVEVA.Historian.Client`** at `histsdk/` (ordinary owned source, edited in place — the re-vendor model is retired). Dev: dashboard `:5220` (HTTP/1.1) + gRPC h2c `:5221`; production: one `Kestrel:Endpoints:Https` with `Protocols: Http1AndHttp2` (ALPN). Default-on leased session pools amortize the historian handshake (v6 pool + a separate v8 event-session pool). Client lib `ZB.MOM.WW.HistorianGateway.Client` + Contracts are on the Gitea feed at `0.3.0`; **OtOpcUa is the first consumer** (its sole historian backend since PR #423); ScadaBridge has no coupling. Known 2023 R2 server-side gates: native event reads return 0 rows (C2, won't-fix — SQL-path `ReadEvents` via `Runtime.dbo.Events` is the shipped workaround), `Int1`/`UInt1` analog-tag creation is degenerate (OtOpcUa maps Boolean → `Int2`), string historical writes non-convergent (follow-ups #18–#25). Only remaining archreview item: the secure-by-default TLS flip, deferred to 1.0.0. A Java client, `com.zb.mom.ww.historiangw:zb-mom-ww-historiangw-client:0.1.0` (Gitea Maven feed, `clients/java/`), is consumed by the ignitionoee repo's `modules/historian` Ignition module. **Full change history (ArchReview cycles 1–2, pending.md items, PR list): `.claude/skills/sister-project-history/historiangateway.md`.** | | **BTDB** (Batch Tracker) | `~/Desktop/btdb` — local `master` @ `695192d` (2026-09-11), **~198 commits ahead of `origin/master` (`d147c10`, 2023) and unpushed**; no branch in flight (`feat/transfer-machine-batch` and `fix/transfer-lot-walk` — DB-6 reopened: forward loop guard, whole-lot chain walks incl. merges — both merged 2026-09-11; `feat/manager-transfers` — the Manager edits, shows and audits transfers — merged `695192d` the same day). **Claude artifacts are gitignored + untracked there** (`CLAUDE.md`, `.claude/`, `docs/plans/`, `docs/reviews/`, `*.tasks.json`) — local only, never to the corporate remote. | SQL Server SSDT project + SQL CLR (net472; **Windows-only build**, the docker rig rebuilds it cross-platform from the same sources); **BTDB Manager = Blazor Web App net10.0** on the family libs (Auth 0.3.0, Theme 0.4.1, Health 0.3.0, Telemetry + Configuration 0.1.0, Secrets 0.6.2), Dapper + embedded queries, self-contained win-x64 NSSM service `ZB.MOM.WW.BTDB.Manager` on `:9877` (→ `:9876` after cut-over) | **`ZimmerBiomet/BTDB`** on the corporate Gitea `wonder-app-vd03:3000` (not on `gitea.dohertylan.com`) | The `BT` SQL Server database that is the plant's **batch / cycle / work-order system of record for data collection**. Galaxy machine objects write through `StartMachineBatch` / `StartMachineCycle` / `End*` / `AddMachineBatchWorkOrder` / `AddOtherData*` / `AddMachineBatchTransfer` / **`TransferMachineBatch`** (new 2026-09-11: end source if open + start destination by `Code` + copy not-moved-out work orders + link, one owned transaction) (own-or-join transaction template, `THROW 50000+`, all `GETUTCDATE()`); reports pull per-batch time-series **out of the AVEVA Historian through CLR sprocs** (`OPENQUERY(INSQL, …)` + `Runtime.dbo.*`), so **BT must be co-tenant with the Historian's `Runtime` DB** (`wonder-sql-*`). MESAPI and **ScadaBridge's inbound API** resolve `SAPID → Machine.Code`; no checked-in ScadaBridge script calls a BT sproc. **State 2026-09-11 (all merged on local `master`):** the Blazor conversion phases 1–3 (legacy net6.0 MVC app, Telerik reports and the mis-named `install.bat` deleted), the 2026-09-04 full review remediation (12 work packages, 60 findings fixed, 0 open; static invariant tests over the SQL + CLR sources, opt-in rig integration tests behind `BTDB_TEST_CONNECTION`, Gitea Actions gate `./build.sh`), and the DARS sprint-1 schema (`MachineBatchTransfer`, `ContainerID`/`MoveOutTimestamp` on `MachineBatchWO`, nestable `MachineGroup.ParentMachineGroupID`, Gen1 reactor load script) — the old `DARS/WW_ARCH/BatchTracker_DB/` fork and the `feat/manager-blazor` worktree are superseded. **Phase-4 cut-over on the DEV box** (`docs/operations/cut-over.md`) is the open step and is **blocked by the 2026-09-11 remote-access hold**. Repo orientation lives in btdb's own (gitignored) `CLAUDE.md`. **Full schema/contract notes, family edges and the gap analysis: `.claude/skills/sister-project-history/btdb.md`.** **Local Docker rig:** `btdb/docker/` — runtime-only image on `:9877` joined to `scadabridge-net` + the redundant glauth pair, registered in Overview as **BTDB Manager**; `docker/deploy-db.sh` publishes `BT` (dacpac + CLR via two side projects under `docker/db/`, plus a `Runtime` stand-in) into `scadabridge-mssql`; `smoke-queries.sh`, `validate-config-import.sh`, `db/validate-clr.sql` and `Database/Scripts/Validation/*.sql` are the rig gates. | @@ -166,7 +166,7 @@ each project's **code-verified current state**, and the **gaps** between. See | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | -| Secrets (encrypted store + `${secret:}` resolution) | Built; family version **`0.6.2`**, published. Pins: **all four apps `0.6.2`** (verified against each repo's checked-in pins AND its origin default branch 2026-09-03 — the earlier "OtOpcUa unpushed / mxgw 0.6.0 / ScadaBridge 0.5.1" notes were stale). **Target `0.6.1`+, never `0.6.0`, and use the 4-arg `AddZbSecrets(..., contentRootPath)` overload at every pre-host `${secret:}` expander call site — the pin alone does not close the hole.** `Secrets:SqlitePath` must be absolute and outside the content root (fail-closed, no opt-out; the 2026-08-09 mxgw outage). Production topologies decided (scadaproj#3/#4 closed): OtOpcUa = Akka pair-local (still on the hub for its own topology); **ScadaBridge retired the gRPC secrets hub for itself (2026-09-28, ScadaBridge `main` @ `23a139a4`) — sites now receive secrets via Deploy Artifacts instead of the hub, central still on ONE shared SQL store (`Secrets:Store:Provider=SqlServer`, `0.5.1` minimum)**; both default-off elsewhere. **`0.6.3` (hub sweep outcome logging + public `ISecretsHubSyncStatus`) is built + tested on scadaproj `main` but NOT published to the feed** — ScadaBridge, its original motivation, no longer needs it once the hub is retired for that app; it remains available for the library's other hub consumers (OtOpcUa) if published later. Full narrative (`0.1.x`→`0.6.3`, incidents, live gates, container caveat, `NU1102` note): [`ZB.MOM.WW.Secrets/CLAUDE.md`](ZB.MOM.WW.Secrets/CLAUDE.md). | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | +| Secrets (encrypted store + `${secret:}` resolution) | Built; family version **`0.6.3`** (published 2026-09-28; apps still pin `0.6.2`). Pins: **all four apps `0.6.2`** (verified against each repo's checked-in pins AND its origin default branch 2026-09-03 — the earlier "OtOpcUa unpushed / mxgw 0.6.0 / ScadaBridge 0.5.1" notes were stale). **Target `0.6.1`+, never `0.6.0`, and use the 4-arg `AddZbSecrets(..., contentRootPath)` overload at every pre-host `${secret:}` expander call site — the pin alone does not close the hole.** `Secrets:SqlitePath` must be absolute and outside the content root (fail-closed, no opt-out; the 2026-08-09 mxgw outage). Production topologies decided (scadaproj#3/#4 closed): OtOpcUa = Akka pair-local (still on the hub for its own topology); **ScadaBridge retired the gRPC secrets hub for itself (2026-09-28, ScadaBridge `main` @ `23a139a4`, follow-ups `2650f794`) — sites now receive secrets via Deploy Artifacts instead of the hub, central still on ONE shared SQL store (`Secrets:Store:Provider=SqlServer`, `0.5.1` minimum)**; both default-off elsewhere. **`0.6.3` (hub sweep outcome logging + public `ISecretsHubSyncStatus`) is PUBLISHED to the feed (2026-09-28) but pinned by no app** — ScadaBridge, its original motivation, retired the hub for itself; it is there for the other hub consumer (OtOpcUa) to adopt. Full narrative (`0.1.x`→`0.6.3`, incidents, live gates, container caveat, `NU1102` note): [`ZB.MOM.WW.Secrets/CLAUDE.md`](ZB.MOM.WW.Secrets/CLAUDE.md). | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) | | LocalDb (embedded cache + 2-node sync) | **Adopted** by ScadaBridge (Phases 1+2 merged; pins **`0.3.4`**) and OtOpcUa (Phases 1+2 merged; pins **`0.3.4`** since 2026-09-03, `b8b4c18f`) — no version skew remains (every release is wire-compatible, so one would be supported). Replication is **default-OFF**; enabled on the docker rigs' site-a pairs and on the **QA pair vq03/vq04** (ScadaBridge site pair + OtOpcUa pair — OtOpcUa's sync only actually CONNECTED after the 2026-09-10 `b8b4c18f` upgrade; the old build answered the sync POST with the login 302); DEV vd03 is single-node. Operational constraints: a pair stops/starts together (no rolling upgrade), a node offline > `TombstoneRetention` (7 d) can resurrect deleted rows. Full release narrative (`0.1.1`→`0.3.4`) + gates: [`ZB.MOM.WW.LocalDb/CLAUDE.md`](ZB.MOM.WW.LocalDb/CLAUDE.md). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + the two adoption designs ([ScadaBridge](docs/plans/2026-07-19-scadabridge-localdb-design.md), [OtOpcUa](docs/plans/2026-07-20-otopcua-localdb-design.md)) | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) | Each hosted library directory carries its own `CLAUDE.md` with the full version narrative, consumer matrix and diff --git a/ZB.MOM.WW.Secrets/CLAUDE.md b/ZB.MOM.WW.Secrets/CLAUDE.md index 9f923eb..9fa8a30 100644 --- a/ZB.MOM.WW.Secrets/CLAUDE.md +++ b/ZB.MOM.WW.Secrets/CLAUDE.md @@ -10,4 +10,4 @@ Shared encrypted secret store + `${secret:}` resolution for the **ZB.MOM.WW SCAD | Component | Status | Goal | Design | Implementation | |---|---|---|---|---| -| Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.2** (4-arg content-root call site included) and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); **merged to `main` + pushed 2026-08-13 (`e42ff31`)**. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of ``. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `