From 129d37f9e253211feb56d88e1e695cbe49aee647 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 10 Nov 2026 09:39:57 -0500 Subject: [PATCH] HistorianGateway: Contracts/Client 0.4.0 released (historiangw#79) --- .claude/skills/sister-project-history/historiangateway.md | 2 ++ CLAUDE.md | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.claude/skills/sister-project-history/historiangateway.md b/.claude/skills/sister-project-history/historiangateway.md index 6462ddd..22fd335 100644 --- a/.claude/skills/sister-project-history/historiangateway.md +++ b/.claude/skills/sister-project-history/historiangateway.md @@ -67,3 +67,5 @@ fallback/filter expansion or SQL readback gate is pursued. Existing shipped SQL features were not removed. Targeted Release verification: 24 passed, 6 live skips, zero failed. Native gRPC StartQuery timeout remains an unresolved blocker for persistence evidence. + +- **2026-11-10 — Contracts + Client 0.4.0 released (historiangw#79).** PR #118 (`dde0c28`, tag `v0.4.0`) cut the Unreleased CHANGELOGs (incl. the previously-missing #61 StoreForwardStatus fields 8/9 entry) and pushed both packages to `dohertj2-gitea`. Breaking: `ReadBlocks`/`ReadBlocksAsync` removed (#62, wire-equivalent). OtOpcUa remains on 0.3.0; its bump is the next cross-repo step. diff --git a/CLAUDE.md b/CLAUDE.md index bc488c3..8c8fb02 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,7 @@ own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-th | **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree address space (v3.0, `master` since 2026-07-16)** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal); every value has one source fanned to both NodeIds. **This is the wire contract ScadaBridge binds against** (ScadaBridge consumes v3 end-to-end; its #14 is closed). Drivers: Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient, `Sql` (SQL Server / PostgreSQL / Oracle, read-only poll), `Mqtt` (Plain + Sparkplug B, writable where a tag declares a destination), `MTConnect` (read-only), Calculation, and AVEVA Galaxy as a standard Equipment-kind driver via in-process `GalaxyDriver` → gRPC → **mxaccessgw**. Historian backend is pluggable (`ServerHistorian:Backend`): **HistorianGateway** (default), **SqlServer** or **QuestDb** — the latter two add typed values with real status and OPC UA HistoryUpdate (QuestDB: soft delete + TTL retention). Address-space availability guarantee: an unreadable deployment artifact is *no answer*, never an empty configuration — nodes hold last-known-good, and a driver-only node boots and serves from its LocalDb cache with central SQL down (ServiceLevel stays 240/250). Redundancy: one independent 2-node Akka mesh per application `Cluster`; Primary = oldest Up `driver` member (ServiceLevel 250) + one Secondary (240) **per `Cluster`**; cross-mesh commands ride gRPC with per-cluster PSK auth (Akka `ClusterClient` fully deleted 2026-08-06). Data plane is machine-to-machine: OPC UA UserName `apikey` + `otua__` minted on AdminUI `/api-keys`; `OpcUa:AllowAnonymous` defaults `true` (anonymous is a normal grantable principal); LDAP is AdminUI-login only. Node ACLs are enforced **only** behind `Security:Acl:Enabled` (default `false`); with it on, scoped `NodeAcl` grants decide **every** data-plane operation incl. browse/read/subscribe, fail closed, and reach open sessions. Every driver's authoring path now consults the driver's own runtime parse, so what authoring accepts is what the driver serves (tracker zeroed 2026-08-10, origin `d06e92c6`). **`master` @ `11a5c1d4` (2026-11-10, not deployed):** opt-in out-of-process script runner (`Scripting:Runner`, #780 — sends `BadResourceUnavailable`, ScadaBridge#112 handles it), VT→VT dependencies now flow + `VirtualTagDependencyCycle` deploy error (#791), disabled VT keeps its node as `BadOutOfService` (#787), Sql write block Insert/Upsert/Procedure (#781), OPC SDK 1.5.378.182 (stricter CA KeyUsage — check DEV/QA certs, #782). **Full change history, live-gate records and the per-issue consumer-impact record for ScadaBridge: `.claude/skills/sister-project-history/otopcua.md` (skill `sister-project-history`).** | | **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** Alarm ack (main `488649c`, 2026-10-20) authenticates as the operator and writes `.AckMsg` so the Historian records the real operator; `AcknowledgeAlarmRequest` gained `operator_password`/`operator_domain` (proto change, published as Contracts/Client **`0.3.0`** on the feed 2026-11-06, mxaccessgw#214; OtOpcUa's `GatewayGalaxyAlarmAcknowledger` must send the password before it deploys against this — lmxopcua#749). Main `e5e7f18` (2026-11-06) also locks the worker pipe to the gateway's account and redacts secured bulk writes in logs. | | **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | **Current `main`: `c810bd0f` (2026-11-10 issue waves, NOT deployed: AuditLog clustered-key flip + off-box chain-head export + maintenance login + archive retention — four migrations ship together with both centrals stopped (#107); opt-in `leader-auto-down` downing provider (#106, drill #116); derived templates survive bundles (#118); bundle-signing policy + runbook (#110); Flow Debugger Node log + Akka→Serilog (#56/#120); OPC UA Good-subcode + BadResourceUnavailable handling (#117/#112)). Previous tip `3808b49f` (2026-10-22: 2026-10-21 eight-slice system review remediated — ~107 findings, bearer-token CLI auth, `MachineDataDb` removed, config-audit summaries + retention, migration `ReviewRemediationIndexes` (QA/prod not applied); Topology area rename fix, first-layer preload and area-subtree **Deploy all** / CLI `deploy area` (Gitea #38–#40); record `docs/plans/2026-10-21-system-review-remediation.md`). Previous tip `3a217b94` (2026-09-29: stale per-node secret-mirror status rows are now pruned; 2026-09-28: site-secrets follow-ups — a central secret-store outage fails only the sites that reference a secret, removal-only deploys are audited, each site node's mirror failures (standby included) ride the health report via a 12th replicated table `secret_mirror_status`, artifact-deploy CLI commands exit 1 on a failed site; rig gate 9/9). Previous tip `23a139a4` (2026-09-28: site secrets ride the artifact deploy and the gRPC secrets hub is retired — see below). Previous tip `5cc09b08` (2026-09-28: site scripts gain a read-only, allow-listed `Files` global — `ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots`, off by default — used by the in-memory MES/Delmia/alarm-API test machines SBTEST01/02 (`docker/seed-mes-test-machine.sh`) to parse downloaded recipe files; DEV vd03 and QA vq03/vq04 on `b3021e7c` since 2026-09-28, migrations 57; register row 126 closed) — adds site artifact staleness (`Site.ArtifactRevisionHash` stamped on every successful artifact deploy, migration `AddSiteArtifactRevision`; Sites page *Artifacts current / out of date / never deployed* badge), a rail-footer **Pending updates** indicator over a process-wide `PendingUpdatesCache` (stale instances + stale-artifact sites, Deployer/Admin), and a scrollable rail nav; design `docs/plans/2026-09-14-pending-updates-design.md`. Previous tip `9d529b82` (2026-09-11) — adds the Gitea #36 reset-aware handshake (`WriteBatchAndWaitAsync` reset-echo overload), the computed-alarm `IsMesReportable` flag (template checkbox + nullable instance override, `Alarms.MesReportableAsync()`, migration `AddAlarmMesReportable`; native alarms stay on the 900–999 band) and a nav fix nesting Flow Debugger in the Deployment section. DEPLOYED: `b3021e7c` on DEV vd03 and the QA pair vq03/vq04 (2026-09-28, migrations 55→57; records `docs/deployment/deployment-records/2026-09-28-wonder-app-{vd03-fe89363b,vq03-vq04-b3021e7c}.md`).** **Merged `23a139a4` (2026-09-28): site secrets now ride the artifact deploy and the pull-only gRPC secrets hub is retired in ScadaBridge** (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; central shares one secret store via `Secrets:Store:Provider=SqlServer`; rig live gate PASS 8/8, vd03/QA NOT deployed — rollout tracked as ScadaBridge register row 127 — see `ZB.MOM.WW.Secrets/CLAUDE.md` for the unpublished `0.6.3` hub-logging work this superseded for ScadaBridge). Management/API/CLI parity complete (`docs/plans/2026-09-10-management-api-cli-parity-review.md`): all 149 eligible contracts exposed in 20 CLI groups, 19 missing operations added, deadline/registry/outcome gaps fixed; CLI documentation checked against all 159 leaf commands; 9554 tests verified and primary 8-node rig rebuilt with read-only smoke checks. Previously completed 2026-09-09 whole-system arch review remediation (`docs/plans/2026-09-09-arch-review.md`: 92 findings; all 13 High + 33 Medium fixed; all 46 Low findings also committed and pushed (§11); 9478 solution tests and deployed browser smoke pass, primary 8-node rig rebuilt). Adds the `ZB.MOM.WW.ScadaBridge.CentralUI.SandboxWorker` child-process project and a coordinated central-pair migration that encrypts DataConnection protocol config at rest (old binaries cannot read protected rows). Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. **Inbound API keys live on the shared SQL Server store on a central pair** (design 2026-09-02, ScadaBridge register row 108): ScadaBridge is Auth's first consumer of the new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package at `0.3.0`, pointing both central nodes at one `zbauth` schema in the configuration database instead of each node's own per-node SQLite file. **Flow Debugger (including the instance picker and v1.4 step logs, 2026-09-09):** Central UI + CLI trace of one inbound-API method or value-change flow across Central / Site / Tags / External swimlanes, Live or Replay, built on `ScriptTrace` audit rows and the outbound-call audit rows; arch-reviewed and remediated the same day (F1–F13 — redacted + byte-capped inline trace copy, route-time site scope on Live runs, Replay open to `OperationalAudit` roles under `RequireFlowDebugger`; `docs/plans/2026-09-06-flow-debugger-arch-review.md` + `-remediation.md`); script entry (site → instance → script, 2026-09-08). | -| **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) a full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib. No COM, no x86 worker. **`main` @ `860bf99` (2026-11-10, not deployed): redundancy + store-forward compose (F1b, #72 — keep both off on boxes until live gate #115), deterministic server-op faults (#112), versioned outbox frames + size limits + expiry cap (#113), stream keepalive guard on by default (#59), ReadEvents EventCount pass-down (#81). Earlier: `99b4a7d` (2026-11-06, historiangw#55): gRPC-only Historian client at `src/Historian.Client`** — `histsdk/` and the whole SQL path (RuntimeDb, SQL live-value writer, SQL `ReadEvents`) are removed; `Transport` must be `RemoteGrpc`, ReadBlocks removed from the contract (#62, 2026-11-10); RenameTags restored over StartJob (#56); UInt1 tags created as UInt2, Int8/UInt8/FileTime/Guid/Structure/Event rejected (lossless-cast-or-reject); empty event ids get a gateway-minted GUID (#28); AddHistoricalValues backfills string tags via `string_value` (#19). merged to main 2026-11-10 (PRs #116, #117 → `733eb25`); Contracts/Client changes unreleased (#79). Deployed boxes still run `d8d890c` with the old config (redeploy needs a config change). Dev: dashboard `:5220` (HTTP/1.1) + gRPC h2c `:5221`; production: one `Kestrel:Endpoints:Https` with `Protocols: Http1AndHttp2` (ALPN). Default-on leased session pools amortize the historian handshake (v6 pool + a separate v8 event-session pool). Client lib `ZB.MOM.WW.HistorianGateway.Client` + Contracts are on the Gitea feed at `0.3.0`; **OtOpcUa is the first consumer** (its sole historian backend since PR #423); ScadaBridge has no coupling. Native event reads now return rows through the gRPC-only client (2026-10-16 live run; the old 0-row C2 gate and its SQL `ReadEvents` workaround are gone — historiangw#24 closed 2026-11-09; the deployed `d8d890c` boxes still use the SQL reader until #78). Known 2023 R2 server-side gates: SysTimeSec raw reads still time out (#29, blocks lmxopcua#694/#700), `Int1`/`UInt1` analog-tag creation is degenerate (OtOpcUa maps Boolean → `Int2`), string historical writes non-convergent (follow-ups #18–#25). Only remaining archreview item: the secure-by-default TLS flip, deferred to 1.0.0. A Java client, `com.zb.mom.ww.historiangw:zb-mom-ww-historiangw-client:0.1.0` (Gitea Maven feed, `clients/java/`), is consumed by the ignitionoee repo's `modules/historian` Ignition module. **Full change history (ArchReview cycles 1–2, pending.md items, PR list): `.claude/skills/sister-project-history/historiangateway.md`.** `HistorianStatus.GetOperationalDiagnostics` (22 RPCs) and the historical Int1 serializer are **merged to `main`** (Int1 persisted readback still unverified, #22/#29). Offline suite on `main`: 1311 passed / 54 skipped / 0 failed. New gap work targets native Historian gRPC only. | +| **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) a full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib. No COM, no x86 worker. **`main` @ `860bf99` (2026-11-10, not deployed): redundancy + store-forward compose (F1b, #72 — keep both off on boxes until live gate #115), deterministic server-op faults (#112), versioned outbox frames + size limits + expiry cap (#113), stream keepalive guard on by default (#59), ReadEvents EventCount pass-down (#81). Earlier: `99b4a7d` (2026-11-06, historiangw#55): gRPC-only Historian client at `src/Historian.Client`** — `histsdk/` and the whole SQL path (RuntimeDb, SQL live-value writer, SQL `ReadEvents`) are removed; `Transport` must be `RemoteGrpc`, ReadBlocks removed from the contract (#62, 2026-11-10); RenameTags restored over StartJob (#56); UInt1 tags created as UInt2, Int8/UInt8/FileTime/Guid/Structure/Event rejected (lossless-cast-or-reject); empty event ids get a gateway-minted GUID (#28); AddHistoricalValues backfills string tags via `string_value` (#19). merged to main 2026-11-10 (PRs #116, #117 → `733eb25`); released as **Contracts/Client 0.4.0** (2026-11-10, PR #118 → `dde0c28`, tag `v0.4.0`, #79 closed — breaking: ReadBlocks removed; OtOpcUa still on 0.3.0, bump pending). Deployed boxes still run `d8d890c` with the old config (redeploy needs a config change). Dev: dashboard `:5220` (HTTP/1.1) + gRPC h2c `:5221`; production: one `Kestrel:Endpoints:Https` with `Protocols: Http1AndHttp2` (ALPN). Default-on leased session pools amortize the historian handshake (v6 pool + a separate v8 event-session pool). Client lib `ZB.MOM.WW.HistorianGateway.Client` + Contracts are on the Gitea feed at `0.4.0` (OtOpcUa pinned to `0.3.0`); **OtOpcUa is the first consumer** (its sole historian backend since PR #423); ScadaBridge has no coupling. Native event reads now return rows through the gRPC-only client (2026-10-16 live run; the old 0-row C2 gate and its SQL `ReadEvents` workaround are gone — historiangw#24 closed 2026-11-09; the deployed `d8d890c` boxes still use the SQL reader until #78). Known 2023 R2 server-side gates: SysTimeSec raw reads still time out (#29, blocks lmxopcua#694/#700), `Int1`/`UInt1` analog-tag creation is degenerate (OtOpcUa maps Boolean → `Int2`), string historical writes non-convergent (follow-ups #18–#25). Only remaining archreview item: the secure-by-default TLS flip, deferred to 1.0.0. A Java client, `com.zb.mom.ww.historiangw:zb-mom-ww-historiangw-client:0.1.0` (Gitea Maven feed, `clients/java/`), is consumed by the ignitionoee repo's `modules/historian` Ignition module. **Full change history (ArchReview cycles 1–2, pending.md items, PR list): `.claude/skills/sister-project-history/historiangateway.md`.** `HistorianStatus.GetOperationalDiagnostics` (22 RPCs) and the historical Int1 serializer are **merged to `main`** (Int1 persisted readback still unverified, #22/#29). Offline suite on `main`: 1311 passed / 54 skipped / 0 failed. New gap work targets native Historian gRPC only. | | **BTDB** (Batch Tracker) | `~/Desktop/btdb` — local `master` @ `a81a480` (2026-10-01), **~211 commits ahead of `origin/master` (`d147c10`, 2023) and unpushed**; no branch in flight (`feat/transfer-machine-batch` and `fix/transfer-lot-walk` — DB-6 reopened: forward loop guard, whole-lot chain walks incl. merges — both merged 2026-09-11; `feat/manager-transfers` — the Manager edits, shows and audits transfers — merged `695192d` the same day). **Claude artifacts are gitignored + untracked there** (`CLAUDE.md`, `.claude/`, `docs/plans/`, `docs/reviews/`, `*.tasks.json`) — local only, never to the corporate remote. | SQL Server SSDT project + SQL CLR (net472; **Windows-only build**, the docker rig rebuilds it cross-platform from the same sources); **BTDB Manager = Blazor Web App net10.0** on the family libs (Auth 0.4.1 — **sign-in by user bind, `BindMode=User`, no LDAP service account**, Theme 0.4.1, Health 0.3.0, Telemetry + Configuration 0.1.0, Secrets 0.6.2), Dapper + embedded queries, self-contained win-x64 NSSM service `ZB.MOM.WW.BTDB.Manager` on `:9877` (→ `:9876` after cut-over) | **`ZimmerBiomet/BTDB`** on the corporate Gitea `wonder-app-vd03:3000` (not on `gitea.dohertylan.com`) | The `BT` SQL Server database that is the plant's **batch / cycle / work-order system of record for data collection**. Galaxy machine objects write through `StartMachineBatch` / `StartMachineCycle` / `End*` / `AddMachineBatchWorkOrder` / `AddOtherData*` / `AddMachineBatchTransfer` / **`TransferMachineBatch`** (new 2026-09-11: end source if open + start destination by `Code` + copy not-moved-out work orders + link, one owned transaction) (own-or-join transaction template, `THROW 50000+`, all `GETUTCDATE()`); reports pull per-batch time-series **out of the AVEVA Historian through CLR sprocs** (`OPENQUERY(INSQL, …)` + `Runtime.dbo.*`), so **BT must be co-tenant with the Historian's `Runtime` DB** (`wonder-sql-*`). MESAPI and **ScadaBridge's inbound API** resolve `SAPID → Machine.Code`; no checked-in ScadaBridge script calls a BT sproc. **State 2026-09-11 (all merged on local `master`):** the Blazor conversion phases 1–3 (legacy net6.0 MVC app, Telerik reports and the mis-named `install.bat` deleted), the 2026-09-04 full review remediation (12 work packages, 60 findings fixed, 0 open; static invariant tests over the SQL + CLR sources, opt-in rig integration tests behind `BTDB_TEST_CONNECTION`, Gitea Actions gate `./build.sh`), and the DARS sprint-1 schema (`MachineBatchTransfer`, `ContainerID`/`MoveOutTimestamp` on `MachineBatchWO`, nestable `MachineGroup.ParentMachineGroupID`, Gen1 reactor load script) — the old `DARS/WW_ARCH/BatchTracker_DB/` fork and the `feat/manager-blazor` worktree are superseded. **Phase-4 cut-over on the DEV box** (`docs/operations/cut-over.md`) is the open step and is **blocked by the 2026-09-11 remote-access hold**. Repo orientation lives in btdb's own (gitignored) `CLAUDE.md`. **Full schema/contract notes, family edges and the gap analysis: `.claude/skills/sister-project-history/btdb.md`.** **Local Docker rig:** `btdb/docker/` — runtime-only image on `:9877` joined to `scadabridge-net` + the redundant glauth pair, registered in Overview as **BTDB Manager**; `docker/deploy-db.sh` publishes `BT` (dacpac + CLR via two side projects under `docker/db/`, plus a `Runtime` stand-in) into `scadabridge-mssql`; `smoke-queries.sh`, `validate-config-import.sh`, `db/validate-clr.sql` and `Database/Scripts/Validation/*.sql` are the rig gates. | ## Cross-project relationships