# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## What this repository is `scadaproj` is primarily an umbrella/index workspace that aggregates a family of related SCADA / OT / Wonderware / OPC UA "sister projects" that live as **sibling directories under `~/Desktop/`**. It now also **hosts nine shared libraries itself** (verified 2026-07-18) — [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) (login/identity), [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) (UI kit), [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) (health checks), [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) (observability), [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) (config validation), [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) (audit event model), [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) (encrypted secret store), [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) (Galaxy browse), and [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) (embedded cache + 2-node sync) — all the realized output of their respective component normalizations (see [Component normalization](#component-normalization)). (A tenth directory, `ZB.MOM.WW.SPHistorianClient/`, is **retired** — 0 tracked files, only gitignored build residue left on disk.) The point of this file is to give a high-level scan of each sister project — its purpose, location, stack, and primary commands — so a fresh Claude Code session can orient across the whole family without opening each repo first. Each sister project keeps its own authoritative `CLAUDE.md`. This index is a **summary of those files**; when you actually work in a project, open that project's own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-this-index). > The project list below is **curated manually**. Add or remove entries as the family > changes — do not assume every directory under `~/Desktop/` belongs here. ## Sister projects (core SCADA/OT family) ### Runtime / implementation (active code) | Project | Location | Stack | Repo | Summary | |---|---|---|---|---| | **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient) **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings; ScadaBridge is already namespace-URI-durable via `OpcUaNodeReference` and its #14 phase 2 shipped 2026-07-23 (main @ `266f001a`): `OpcUaReferenceForm.IsDurable` + a browse-picker nudge to the durable `nsu=;s=…` form + doc sweep. The binding re-author (legacy `ns=2` collides with v3 `raw`), native-alarm dedup across the raw+uns fan-out, and UNS-bound→RawPath alarm routing are DEFERRED to a live v3 rig — ScadaBridge #14 stays OPEN. Scope: ScadaBridge `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=///`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. **Redundancy corrections (2026-07-21, `master @ 0de1352e`) — both consumer-visible:** (1) the downing strategy moved from SBR `keep-oldest` to Akka `AutoDowning` (`Cluster:SplitBrainResolverStrategy`, default `auto-down`, 15s window), because a two-node pair on `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node** — `KeepOldest.OldestDecision` only rescues a side holding >=2 members, so the 1-vs-1 survivor downs *itself* and exits. Same fix and same rationale as ScadaBridge `cf3bd52f`; accepted trade is dual-active during a real partition. **Its live gate is still open** (the pathology needs 1-vs-1; docker-dev is one six-node mesh) and is deferred to the per-cluster mesh work. (2) The redundancy **Primary is now the oldest Up `driver` member, not `RoleLeader` (lowest address)** — the two diverge after any node restart, so the Primary-gated data plane could enable on a node not hosting the cluster singletons; this is the node whose `ServiceLevel` reads 250 vs 240, so **a consumer selecting by ServiceLevel may now prefer a different node than before after a restart**. `NodeRedundancyState.IsRoleLeaderForDriver` renamed `IsDriverPrimary`. The election is still scoped **per Akka cluster, not per application `Cluster`** — the per-cluster mesh design (`docs/plans/2026-07-21-per-cluster-mesh-design.md`) closes that; **Phases 0a/0b/1/2/3/4/5/6 done + merged to `master` (Phase 6 pushed to origin 2026-07-24, tip `2839deb5`; Phase 5 was `35552a96`), 7 not started.** **Phase 2 (merge `6eaa81ca`)** added a `ClusterClient` command transport behind the `MeshTransport:Mode` dark switch (`Dps` default), the prerequisite for splitting the fleet into one mesh per application `Cluster` (central and node no longer need shared gossip membership). **Phase 3 (merge `d01b0695`)** added config **fetch-and-cache**: behind the `ConfigSource:Mode` dark switch (`Direct` default = today's behaviour), a driver node fetches its deployed-configuration artifact **from central over a gRPC stream** (SHA-256-verified against the dispatch's `RevisionHash`), caches it in `ZB.MOM.WW.LocalDb`, and reads config **only** from that cache — never central SQL. **Consumer-relevant:** this extends the #485 availability guarantee — a `FetchAndCache` driver node **boots and serves its last-known-good address space with central SQL Server completely down** (restoring from the LocalDb pointer, no central dependency at boot), and a fetch that fails is treated as *no answer* (apply fails, last-known-good kept), never an empty configuration. The Phase 3 live gate caught + fixed a serve-gate deadlock (central had gated the artifact serve on `Status==Sealed`, but a deployment seals only after every node acks and a fetching node can't ack until it fetches). **Phase 4 (merge `26fad75c`, live-gated on docker-dev; Task-9 table drop `3a590a0c`)** cut the driver-side ConfigDb connection entirely: a **driver-only** node (`driver`, not `admin`) now boots with **no ConfigDb connection string at all** — config via FetchAndCache into LocalDb, scripted-alarm Part 9 condition state in a replicated LocalDb table, and central persists its deploy acks. **Consumer-visible ServiceLevel change (matters to any ServiceLevel-selecting OPC UA client, i.e. ScadaBridge):** DB-reachability is *retired* from the ServiceLevel calc on such DB-less nodes, so a healthy site node now publishes **240/250 even with central SQL Server completely down** (it has no DB connection to lose — survive-alone), where a DB-backed node whose ConfigDb went unreachable would drop to 0/100. So during a central-SQL outage a consumer keeps seeing the site nodes as fully authoritative rather than watching their ServiceLevel collapse. The old single-Akka-mesh election scope was unchanged as of Phase 4 (one Primary fleet-wide) — **Phase 6 (below) closes that.** **Phase 5 (merge `35552a96`, live gate PASSED 2026-07-23)** replaced the DPS fan-out of the four live-telemetry observability channels (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`) with a gRPC server-streaming contract — each **driver node hosts** the telemetry server and **central dials in** — behind the `Telemetry:Mode`/`TelemetryDial:Mode` dark switch (`Dps` default), fail-closed shared-bearer auth; `redundancy-state`/`fleet-status`/`deployment-acks` deferred with rationale. **Not consumer-facing to ScadaBridge** — this is central↔node AdminUI observability, not the OPC UA address space, with **no wire-contract change**. Also pinned `System.Security.Cryptography.Xml` 10.0.10 — the same four NU1903 advisories that broke ScadaBridge's fresh restores (`dced0d27`) were breaking OtOpcUa's too (204 errors on a clean restore, invisible locally because NuGet audit data is cached). **Phase 6 (MESH PARTITION — code done+reviewed, merged to `master` + pushed 2026-07-24, tip `2839deb5`; its live gate DEFERRED by owner choice "merge now, gate after")** split OtOpcUa's single fleet-wide Akka mesh into **one independent 2-node mesh per application `Cluster`** (central pair + one pair per site), the same shape ScadaBridge already runs — same `otopcua` ActorSystem name, separated purely by per-pair self-first seed partitioning, with central reaching each site mesh over the explicit ClusterClient/gRPC/ConfigServe transports (Phases 2/3/5) instead of shared gossip. **Consumer-visible (ServiceLevel-selecting clients, i.e. ScadaBridge):** the redundancy election is now scoped **per application `Cluster`, not per Akka mesh** — so there are now **two-or-more Primaries fleet-wide, one per `Cluster`, by design** (`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton on every driver node). A consumer selecting by ServiceLevel therefore correctly sees one Primary (250) + one Secondary (240) **per application `Cluster`**, resolving the "one Primary fleet-wide" scope caveat noted under Phases 0b/4. **No OPC UA wire-contract change** — this is internal Akka topology + AdminUI/redundancy, not the address space. Also new: a fail-closed `SplitTopologyTransportValidator` refuses to boot a cluster-role node left on a DPS transport; secrets replication is now pair-local (no SQL-hub — sites have no SQL); the compiled transport-mode defaults stay `Dps` (the validator, not a default flip, is the guardrail). Plan `OtOpcUa/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`; the split now makes the auto-down 1-vs-1 crash-the-oldest live gate (open since Phase 0a) finally testable, deferred to Phase 7. | | **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** | | **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. | | **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib (consumed as a Gitea-feed package). No COM, no x86 worker. **Dev:** two plaintext endpoints from `appsettings.Development.json` — dashboard on `:5220` (HTTP/1.1), gRPC h2c on `:5221`. **Production:** single `Kestrel:Endpoints:Https` endpoint with `Protocols: Http1AndHttp2` multiplexes dashboard + gRPC over one TLS port (ALPN); warn-only if no TLS endpoint configured (valid behind a reverse proxy / Kubernetes ingress; the warn predicate covers any non-Development environment, i.e. Production + Staging). In a non-Development environment the gateway also logs warn-only **production-readiness** checks (pending.md D2/D3) — relative runtime-artifact paths + secret hygiene (`ApiKeys:Mode=Disabled`, empty/dev-placeholder pepper, dev-placeholder LDAP password). **Owns `AVEVA.Historian.Client`** — imported into the repo at `histsdk/` (subtree, folded in with history 2026-06-28; the old "vendored / re-vendor from upstream" model is **retired** — `histsdk/**` is ordinary owned source, edited in place; the gateway builds it directly). Store-forward uses a crash-safe FasterLog append-only outbox (`Microsoft.FASTER.Core` 2.6.5; `CommitMode` PerEntry/Periodic), not SQLite. **Handshake amortization (pending.md A1) done + live-validated** — a default-on leased-session pool (`Historian:SessionPool`) reuses pre-authenticated sessions across reads/writes/status ops/tag-browse/metadata (~4.7× measured; probe and blocks stay per-call), with a `<~15 s` keepalive + reactive re-auth, surfaced via a `PooledHistorianClient` facade so services are unchanged; the `HistorianSession` primitive lives in the owned `AVEVA.Historian.Client` under `histsdk/`; browse/metadata + SendEvent broadening merged to `main` + pushed. **`SendEvent` is also amortized** via a **separate, parallel event-session pool** (`Historian:EventSessionPool`, default-on; v8/ECDH auth — kept distinct from the v6 pool), warranted by a GREEN v8 Event-session reuse spike (~10–16×); `ReadEvents` stays per-call / gated (C2). The full offline suite is green on macOS (0 warnings); the env-gated live historian + Galaxy integration suite exercises the amortized path and otherwise skips without a live server. **Part C minimal (pending.md C1/C3a) merged to `main`:** `Int8`/`UInt8` live write types un-gated + live-proven against `wonder-sql-vd03` (in the owned `histsdk/` SDK); the 2023 R2 gRPC interface-version integers recorded as evidence (C3a); `UInt1` attempted but **server-blocked** (the historian accepts `EnsureTags(UInt1)` yet stores a degenerate analog tag) → re-gated fail-closed. **C2 closed won't-fix (2026-06-26):** event reads are server-gated on the 2023 R2 historian over **both** transports — gRPC retrieval-server-gated (0 rows scoped to a managed connection), and the WCF certificate transport + auth DO reach the historian cross-platform (CM_EVENT registers on the `0x501` event connection) but the query still returns 0 rows: the same server-side per-connection row gate. (An earlier note saying "WCF not served on 2023 R2" was a test error vs the reverse tunnel.) Not client-fixable. Reusable SDK wins: `ConnectViaAddress` (WCF Via for tunneled access), `EventReadConnectionModeOverride`. **SQL-path `ReadEvents` (merged 2026-06-26) — the practical workaround for the C2 gate:** event reads now **ship** via the historian's `Runtime.dbo.Events` SQL view (config-gated `RuntimeDb:EventReadsEnabled` + `EventReadMaxRows`), mirroring the SQL live-write path (no COM/native); live-proven streaming real events (incl. an INSQL NOT-NULL `CAST` fix the live test caught) while the native event-query stays gated — so **event reads work** despite the native dead-end. **All of Part C + C2 + SQL-ReadEvents are merged to `origin/main`** (gateway @ `fabab1a`, histsdk @ `f0a1b04`). **Follow-ups merged to `main` (2026-06-27):** `historiangw` **PR #6** — `WriteLiveValues` UTC→server-local timestamp fix (live-validated exact) + a documented `SendEvent` `Source_Object` protocol limitation (`pending.md` C4); the Plan-1 .NET **client lib + packable Contracts** (PR #5); and a C2 cross-ref docs PR (#4). **First consumer (merged):** OtOpcUa **PR #423** (→ `master`) adopts `ZB.MOM.WW.HistorianGateway.Client` as its sole historian read/write backend, retiring its bespoke Wonderware historian driver. **Local Docker (2026-06-29):** `HistorianGateway/docker/` runs the sidecar in a container against the REAL wonder datasources (historian `wonder-sql-vd03:32565` TLS + Galaxy SQL `wonder-app-vd03`/`ZB`), dashboard login + gRPC API-key auth disabled — verified `/health/ready` Healthy. Runtime-only image: publish framework-dependent on the host (authed Gitea feed → no Docker restore), COPY into `aspnet:10.0`; secrets via gitignored `env_file`; isolated compose project `zb-historiangw`; container egress needs the host VPN up. **Client surface (`clients/dotnet/`):** the published `ZB.MOM.WW.HistorianGateway.Client` lib + a `ZB.MOM.WW.HistorianGateway.Client.Cli` smoke CLI (`probe`/`read-raw`/`browse`, JSON out; in the `.slnx`, not packed). **Current: `origin/main` @ `56a70df`** — **AHC-002 string READS landed via native RE from the Mac (PR #17 merged 2026-07-16):** the SDK now parses string historian result-buffer rows into `HistorianSample.StringValue` (a per-row `valueType` discriminator — `1`=numeric→`double`, `3`=string; value = `u32 charLen` + UTF-16LE), and the gateway forwards `string_value` through `HistorianRead.ReadRaw` — **no contract change, no package bump** (the proto already carried the field; mapper already forwarded it); live-proven against 2023 R2. String **tag-create** is now doable via SQL `aaStringTagInsert` (`SqlStringTagProvisioner`, test tooling — single + double byte). String **historical writes** were probed (bounded 6-candidate native blob probe) but **non-convergent** — tag-info resolves a **real GUID + SingleByteString** (unlike the Int1/UInt1 degenerate stub) so writes aren't blocked at resolution, but `BSuccess` is **not an oracle** (all framings optimistically accepted) and none round-tripped → **deferred, needs a native-client write wire capture** (blind guessing can't finish). Eight follow-up issues filed (`historiangw` #18–#25): datatype gaps (native string tag-create #18, string writes #19, DateTime/FileTime #20, Guid/Structure/Event #21, Int1/UInt1 discrete-create #22 [follow-on to #11], DoubleByteString read-verify #23) + **SQL-fallback trackers** (event reads via `Runtime.dbo.Events` #24, live writes via `History`/`v_StringHistory` #25). Also folded a doc-truth reconcile of the "Key structural decisions" bullets (GalaxyRepository `PackageReference`, Contracts/Client 0.3.0 published, OtOpcUa on 0.3.0). Prior: **ArchReview #2 (cycle-2) remediation MERGED 2026-07-14 (PR #12):** all 8 workstreams A–H — store-forward drain honesty (`IsRetryableDeliveryFault`, typed `HistorianOutboxFullException`→`ResourceExhausted`), streaming-lease caps (per-key cap 8→4, `MaxStreamLeaseHold`, bounded `EndQuery` teardown, client `StreamTimeout` default null→10min), a **live idle-out capture (GREEN-A, native `(type 4, code 51)`) that narrowed the session-expiry classifier and closed the stability-F5 tail**, histsdk residue (typed event-send `FromNativeError`, `HistorianProtocolIntegrityException`, IP/host scrub), read-path caps (`MaxAggregateIntervals`/`MaxAtTimeTimestamps`, 5M→`ResourceExhausted`, verify-once-per-connection version gate, batched audit drain), release hygiene (csproj `` sole authority + packed-README/AnyCPU/FQDN/CHANGELOG guards), security/ops (opt-in `ForwardedHeaders` trust list, k8s key-bootstrap runbook, dashboard pool/dead-letter signals), and doc-truth #2 (`Historian:VerifyServerInterfaceVersion` now a real fail-closed key). Verified 0-warn, offline 1388 pass + live suite green vs `wonder-sql-vd03`. **Contracts+Client 0.3.0 PUBLISHED** to the `dohertj2-gitea` feed (2026-07-14; carries the `StreamTimeout` default change) and **OtOpcUa bumped to 0.3.0** (`lmxopcua master @ 7f79cd59`, PR #440 — only consumer delta is the StreamTimeout default; driver tests 103/0). **Boolean historization remapped Int1→Int2 (2026-07-14, `lmxopcua master @ 38e21df2`, PR #442, closes #441):** the R2-06 live gate surfaced that the historian's `Int1` analog-tag creation path is server-degenerate (`EnsureTags(Int1)` stores an unusable stub) — filed gateway-side as `historiangw` #11, documented there as an explicit evidence-based exclusion (PR #16), and routed back to OtOpcUa as issue #441. OtOpcUa's `HistorianTypeMapper` now maps `Boolean → Int2` (0/1 integer); the value-write path already sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips cleanly and a pre-existing Float Boolean tag retypes cleanly (both are supported analog types). Closes the last R2-06 red test → live gate **6/6**; gateway driver unit 103/103. **archreview round-2 deferred gates closed 2026-07-14** (`lmxopcua master @ 30672888`): **R2-04 T13+T15 LIVE-PASSED** (PR #444) — the behavior-affecting S4 primary-gate default-deny verified on a 2-node docker-dev rig (ServiceLevel 250/240; boot-window write rejected `not primary (role unknown)` + denial meter `reason=role-unknown`; steady-state secondary rejected + node reverted; primary write reaches device) plus the in-process `PrimaryGateFailoverTests` (delivered-snapshot drives the gate); and the pre-existing red S7.Cli phrase-scan test fixed (PR #443, restored a stripped rationale comment). Round-2 remainder is now only infra-gated: R2-08 live-GLAuth outage, R2-01 S7 SYN-blackhole. **The full `*.IntegrationTests` sweep is now COMPLETE (2026-07-15, `lmxopcua master @ 152a5645`):** ran all 10 driver + server integration suites serially (on the Mac against native fixtures on the `10.100.0.35` Docker host), then closed all 7 resulting follow-ups (PRs #446–#452) — **every driver + server integration suite is verified GREEN with ZERO OtOpcUa production regressions;** every original "failure" was fixture/harness/test rot, not product. Highlights of cross-repo relevance: (a) **the Host integration harness's LDAP swapped from the retired `bitnami/openldap:2.6` to GLAuth** (PR #451) — the harness now runs its own ephemeral GLAuth container on `:3894` (seeded by `tests/.../Host.IntegrationTests/glauth/config.toml`, distinct from the shared dev GLAuth on `:3893`), unifying the whole family's opt-in real-LDAP path on GLAuth and removing the lone OpenLDAP outlier; (b) the deploy-E2E "amd64-emulated-SQL timing" complaint was **misdiagnosed** — those tests had only ever run against the EF in-memory provider (no FK enforcement) and lacked the `ServerCluster`/`ClusterNode` seed the real-SQL `NodeDeploymentState` FK requires; fixed with a harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST`/`_LDAP_HOST` overrides so the fixtures run on the native-amd64 Docker host (PR #452). See `archreview/plans/INTEGRATION-SWEEP-STATUS.md` for the full record. Cycle-2 gated tail: CI first-green-run (needs a Gitea act_runner), the histsdk protobuf/Grpc.Tools major bump (03-CC-8, blocked on migrating off obsolete `GrpcWebHandler.HttpVersion`), and the 1.0.0 TLS-default flip. The prior cycle-1 **ArchReview remediation is fully merged** (Now + Next tiers 2026-07-09 @ `508b9c3`; **Later tier 2026-07-13** via a coordinated two-PR merge, `12385c8` + `6a24a30`). Off a 2026-07-08 six-report architecture review (`archreview/`) turned into 8 workstream plans (`docs/plans/2026-07-09-archreview-*`; tracker `...-remediation-tracking.md`), shipped clean under `TreatWarningsAsErrors`: a hard-fail `ResilienceComposition` validator (StoreForward+Redundancy both-on), corruption-tolerant outbox + dead-letter/overflow knobs, `/auth/login` rate-limit, server-side `max_values` cap, a headless `apikey` CLI, histsdk parser hardening, Gitea Actions CI + warnings-as-errors on all trees, pool observability (lease-wait/waiter/counters) + session-pool health probe, and a typed histsdk fault taxonomy with type-based classification + session churn-guard. **Contracts+Client 0.2.0 published** to the `dohertj2-gitea` feed (2026-07-09; Int1 proto3-optional, opc_quality doc, CA-chain pin). **Cross-repo 0.2.0/G-2 chain COMPLETE:** OtOpcUa bumped to Client+Contracts `0.2.0` (2026-07-10, `master @ f6eaa267`; M3 `opc_quality` gate clear), and the shared `ZB.MOM.WW.Auth.ApiKeys` gained `ExpiresUtc` at `0.1.4` — consumed by HistorianGateway (`apikey create --expires` + dashboard Expires column) and mxaccessgw (verifier enforcement). **Later tier (roadmap 16–21) MERGED to `main` 2026-07-13** (coordinated two-PR merge; both branches now deleted): **PR #8** (`archreview-later-21-and-maxsessions`, merge `12385c8`) = Later-21 (audit off the mutating-RPC critical path + `WriteLiveValues` SQL batching + composed-failure/shutdown tests) + the **MaxSessions 4→8 raise** (live concurrent-session spike GREEN); **PR #9** (`later-16-page-through-streaming`, merge `6a24a30`) = Later-16 **ReadRaw page-through streaming** (streams per server result buffer; pool holds the lease across the enumeration; new `Historian:ReadLimits:AllowStreamingBeyondCeiling`), Later-17 **redundancy honest-hardening** (exposes the `Redundancy:WriteAcknowledgement`/`WriteFanout`/`FailureThreshold` ack policy the docs already claimed), Later-18 **per-key concurrency cap** (`ApiKeys:MaxConcurrentCallsPerKey` → gRPC `ResourceExhausted`) + api-key-rotation runbook, Later-19 CHANGELOG-discipline guard test, Later-20 **K8s manifests + hardened prod Dockerfile + operator runbooks** (`deploy/k8s/`, `docker/Dockerfile.production`, `docs/runbooks/`). The two PRs conflicted on `CLAUDE.md`/tracking/`ValidatorTests`, resolved as the union (MaxSessions lands at **8**). Live-proven vs `wonder-sql-vd03`; offline gateway 729 pass / client 140 pass / 0 warnings. **Only remaining archreview item: the secure-by-default TLS-default flip, deferred to 1.0.0** (pre-announced in the client 0.2.0 CHANGELOG). (Prior tip `93c6051` = the 2026-07-07 CommentChecker XML-doc sweep.) | ## Cross-project relationships The three indexed projects are **separate repos and separate processes**, coupled at **runtime over wire protocols (gRPC + OPC UA)** — not by project/compile references. They share the `ZB.MOM.WW.*` product namespace (`ZB.MOM.WW.OtOpcUa.*`, `ZB.MOM.WW.ScadaBridge.*`; the gateway uses `MxGateway.*`). The common subject is **AVEVA System Platform (Wonderware) "Galaxy"** data, and `mxaccessgw` is the linchpin that the other two connect through. ### Data flow ``` AVEVA System Platform — Wonderware "Galaxy" (OT source of truth: runtime tags + Galaxy Repository SQL DB) ▲ │ MXAccess COM (32-bit, STA message pump) │ ┌────────────────┴─────────────────┐ │ MxAccessGateway (mxaccessgw) │ gateway x64 .NET10 + worker x86 net48 │ gRPC service; OWNS the 32-bit │ protos: mxaccess_gateway / mxaccess_worker │ COM bitness + STA pump │ / galaxy_repository └──────┬─────────────────────┬──────┘ gRPC (MxCommand/MxEvent + │ gRPC (ScadaBridge "MxGateway" adapter: GalaxyRepository browse) │ native MxAccess data + A&C alarms) │ │ ┌─────┴──────┐ │ │ OtOpcUa │ GalaxyDriver maps │ │ OPC UA srv │ Galaxy hierarchy → │ │ (.NET 10) │ OPC UA addr space │ └─────┬──────┘ │ │ OPC UA (opc.tcp; data + A&C alarms) ▼ ▼ ┌──────────────────────────────────────────────┐ │ ScadaBridge — Data Connection Layer (DCL) │ │ OPC UA adapter │ MxGateway adapter │ custom │ └─────────────────────────┬────────────────────┘ ▼ Instance Actors → site clusters → central cluster / UI ``` ### Edge-by-edge - **MxAccessGateway is the foundation.** It is the *only* component that loads 32-bit MXAccess COM (its x86 net48 worker owns the COM apartment + STA pump). It exposes that to modern x64/.NET-10 callers over gRPC, and also serves Galaxy Repository SQL browse RPCs. This is *why* the other two exist as .NET 10 / x64 and never touch COM directly. - **OtOpcUa → MxAccessGateway** (gRPC client). OtOpcUa's in-process `GalaxyDriver` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) uses two gateway channels: the `GalaxyRepositoryClient` for the static hierarchy, and an MXAccess session (`MxCommand`/`MxEvent` protos) for live read/write/subscribe. A `DeployWatcher` polls the gateway's deploy-event signal to rebuild the OPC UA address space on Galaxy redeploy. OtOpcUa's job is a **protocol bridge**: it republishes Galaxy — now bound as a *standard Equipment-kind driver* alongside its native-protocol drivers, not a special SystemPlatform mirror — as an OPC UA address space (live values, Part 9 alarms, HistoryRead) for *any* OPC UA client. - **ScadaBridge → OPC UA** (OPC UA client). ScadaBridge's DCL has an OPC UA adapter that collects data and mirrors native OPC UA Alarms & Conditions. OtOpcUa is exactly such a server, so ScadaBridge can ingest Wonderware data **indirectly via OtOpcUa**. - **ScadaBridge → MxAccessGateway** (gRPC client). The DCL *also* has a dedicated **MxGateway adapter** that talks to `mxaccessgw` directly for native MxAccess data and alarms — so ScadaBridge can reach Wonderware data **directly, bypassing OtOpcUa**. Both adapters implement the same `IAlarmSubscribableConnection` seam, and a read-only `NativeAlarmActor` unifies OPC-UA-A&C and MxAccess alarms onto one condition model. ### Net effect - `mxaccessgw` is a shared dependency of **both** OtOpcUa and ScadaBridge. - ScadaBridge has **two paths** to the same Wonderware data: (1) OPC UA → OtOpcUa → gateway, or (2) MxGateway adapter → gateway directly. Path 1 gives standards-based OPC UA decoupling; path 2 gives a more direct/native feed. - **HistorianGateway is a near-independent sidecar.** It reaches the Historian via its owned gRPC client (`AVEVA.Historian.Client`, imported into the repo at `histsdk/`) and the Galaxy Repository SQL DB directly, not through `mxaccessgw`. It consumes the shared `ZB.MOM.WW.GalaxyRepository` lib (Gitea-feed package). Any client that needs Historian data or Galaxy browse can target it independently. **As of 2026-06-27 it is a runtime dependency of OtOpcUa** (ScadaBridge still has no coupling): OtOpcUa adopted HistorianGateway as its **sole historian read/write backend** (continuous historization + alarms via the published `ZB.MOM.WW.HistorianGateway.Client`), retiring its bespoke Wonderware historian driver — **merged** to OtOpcUa `master` (PR #423). The gateway-side follow-ups also **merged** to `historiangw` `main`: the Plan-1 client lib + packable Contracts (PR #5) and the `WriteLiveValues` UTC→server-local timestamp fix + a documented `SendEvent` `Source_Object` protocol limitation (PR #6, `pending.md` C4). OtOpcUa pre-existing test failure tracked as `lmxopcua` issue #424. - Coupling is loose: each repo references the others only as **sibling context** (the `## Sister Projects` note in ScadaBridge's own `CLAUDE.md` lists `MxAccessGateway` and `OtOpcUa` with their Gitea URLs but states they are *not part of its solution*). - **The break surface is the wire contracts, not code.** Because coupling is by network protocol, the things that break across repo boundaries are: the gateway's `.proto` files (`mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto`), the `historian_gateway.v1` proto (HistorianGateway's own contract), and the OPC UA address-space shape OtOpcUa publishes (browse paths, node IDs, A&C alarm model). Changes to any of these must be coordinated across the affected repos — a green build in one repo does not prove the others still interoperate. ## Component normalization Because the sister repos re-implement the same cross-cutting concerns separately and drift apart, [`components/`](components/) normalizes them: per component, the **one target** spec, each project's **code-verified current state**, and the **gaps** between. See [`components/README.md`](components/README.md) for the convention and workflow. | Component | Status | Goal | Design | Implementation | |---|---|---|---|---| | Auth (login / identity / authz) | Adopted (lib `0.1.4`; all 3 apps, merged to **local default** main/master + **pushed to origin** (gitea)) | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) | | UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.3.1`; **all 4 apps pinned `0.3.1`**, pushed to origin) | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) | | Health (readiness / liveness / active-node) | **Adopted** (lib `0.1.0`, on the feed; **all 4 apps** reference it + wire `MapZbHealth` — verified 2026-07-18; the prior "not yet adopted" claim was stale) | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) | | Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) | | Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) | | Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) | | Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) | | Secrets (encrypted store + `${secret:}` resolution) | Built (libs **`0.1.3`**, **published to the Gitea feed**; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; production enablement would need real KEK distribution (own decision/issue). Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; only mxgw is Bootstrap-free) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `