scadaproj/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## What this repository is

`scadaproj` is primarily an umbrella/index workspace that aggregates a family of
related SCADA / OT / Wonderware / OPC UA "sister projects" that live as **sibling
directories under `~/Desktop/`**. It now also **hosts nine shared libraries itself** (verified
2026-07-18) — [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) (login/identity),
[`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) (UI kit),
[`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) (health checks),
[`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) (observability),
[`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) (config validation),
[`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) (audit event model),
[`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) (encrypted secret store),
[`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) (Galaxy browse), and
[`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) (embedded cache + 2-node sync) — all the realized output of their
respective component normalizations (see [Component normalization](#component-normalization)).
(A tenth directory, `ZB.MOM.WW.SPHistorianClient/`, is **retired** — 0 tracked files, only
gitignored build residue left on disk.)
The point of this file is to give a high-level scan of each sister project — its purpose,
location, stack, and primary commands — so a fresh Claude Code session can orient across
the whole family without opening each repo first.

Each sister project keeps its own authoritative `CLAUDE.md`. This index is a
**summary of those files**; when you actually work in a project, open that project's
own `CLAUDE.md` for the full picture. See [Refreshing this index](#refreshing-this-index).

> The project list below is **curated manually**. Add or remove entries as the family
> changes — do not assume every directory under `~/Desktop/` belongs here.

## Sister projects (core SCADA/OT family)

### Runtime / implementation (active code)

| Project | Location | Stack | Repo | Summary |
|---|---|---|---|---|
| **OtOpcUa** | `~/Desktop/OtOpcUa` | .NET 10, OPC UA, gRPC | `gitea.dohertylan.com/dohertj2/lmxopcua` | OPC UA server that exposes industrial data sources under a **two-subtree (dual-namespace) address space (v3.0, merged to `master` 2026-07-16, PR #472, merge `ec6598ce`)** — native-protocol drivers (Modbus, S7, AB CIP/Legacy, TwinCAT, FOCAS, OpcUaClient), a **read-only `Sql` DB-poll driver** (polls DB tables/views on an interval → OPC UA variables; `connectionStringRef` env resolution, schema-walk address picker, live-gated on docker-dev — merged to `master` 2026-07-24. **Three providers ship as of 2026-08-05** (merge `d54f3c83`): **SQL Server**, **PostgreSQL** (managed `Npgsql`) and **Oracle** (managed `Oracle.ManagedDataAccess.Core`) — no native/thin-client installs; `MySql`/`Odbc` remain reserved enum names that fail at construction. Each dialect owns its own parameter markers/bind names + `ConfigureCommand` (Oracle sets `BindByName`, and its binds carry a `p_` prefix because Oracle rejects a bind named after a reserved word — ORA-01745). Live-gated against dedicated fixtures on `10.100.0.35` (postgres `:15432`, oracle-free `:11521`): env-gated integration suites 11/11, plus a chrome/AdminUI gate on docker-dev (provider dropdown authoring, Test-connect, `/raw` browse + browse-commit) that found and fixed a SqlServer-only gate left in the Test-Connect probe, merge `4e461951`), an **`Mqtt` broker-subscribe driver with Sparkplug B** (one MQTTnet-5 client per driver, TLS+auth, hand-rolled reconnect; **Plain** mode binds a tag to a concrete topic, **SparkplugB** mode decodes vendored Eclipse-Tahu protobuf under one `spBv1.0/{groupId}/#` subscription and binds by the `group/edgeNode/device?/metric` tuple — birth/alias/seq-gap/rebirth state machine, death→STALE, a birth-driven browse picker with a scoped Request-rebirth NCMD; live-gated on docker-dev against a Mosquitto TLS+auth broker + a project-owned C# Sparkplug edge-node simulator on `10.100.0.35:8883` — **merged to `master` 2026-07-27, merge `c3a2b0f7`, pushed to origin**; three follow-ups since, all live-gated and merged to `master`: **(1) write-through is no longer deferred** (Gitea #508, merge `2a7d1167`, 2026-07-31, 21/21 live) — MQTT nodes are **writable** where the tag declares a destination: a Plain tag publishes to a new `writeTopic` TagConfig key, a Sparkplug tag encodes an **NCMD/DCMD** command. **Consumer-visible to any OPC UA client, i.e. ScadaBridge:** an MQTT node now materializes `SecurityClassification.Operate` instead of `ViewOnly` when its tag declares a write destination, and `BadNotWritable` when it does not — browse-time writability is computed by the same encoder the write path uses, so a node can never advertise writability it would then refuse. `Good` means **published**, not applied: MQTT has no application-level ack, so the write posture is optimistic-Good with the standard write-outcome self-correction behind it. **(2) Sparkplug primary-host STATE publishing** (Gitea #511, merge `857a10b4`, 2026-07-31, 5/5 live) — `actAsPrimaryHost` was accepted by config and did nothing; it now publishes a retained `ONLINE` on `spBv1.0/STATE/{hostId}`, arms the matching `OFFLINE` as the MQTT **Last Will**, and publishes an explicit `OFFLINE` on clean shutdown. The claim is bound to its **own dedicated broker connection** because a Will can only be registered in the CONNECT packet, and it is **Primary-gated** — only the node holding the redundancy Primary role claims, since an ungated pair would let a *Secondary's* death publish `OFFLINE` over a healthy Primary and stop every edge node in the group. New driver-facing seam `IDriverPrimaryRoleSource` in `Core.Abstractions`, published by `DriverHostActor` off the data-plane gate verdict. Accepted trade: a brief edge-visible `OFFLINE` window during failover. **Not an OPC UA wire-contract change** — this is what OtOpcUa asserts to *other applications on the MQTT broker*. **(3) structured Sparkplug metrics** (Gitea #515, merge `857a10b4`, 3/3 live) — `Template` (UDT) and `DataSet` metrics were skipped with a warning; they now **flatten to sibling tags** (`Motor` → `Motor/Speed`, `Motor/Stator/Temp`; `Recipe` → `Recipe/Step/0`, `Recipe/Target/1`, row index always present so a growing table never renames existing tags), and a metric's `PropertySet` projects to node **metadata**, never tags. Flattening was chosen over OPC UA Structures precisely so the result is consumable by any client with no type dictionary — **relevant to ScadaBridge: a Sparkplug UDT now surfaces as ordinary scalar nodes** rather than being invisible. Skip-and-warn survives for everything still unflattenable (template definitions, extension values, malformed datasets, payloads past the depth/member bounds), and refusal is **whole** — never a partially-flattened UDT), a **read-only `MTConnect` Agent driver** (one hand-rolled `System.Xml.Linq` HTTP/XML client per driver against a vendor-neutral MTConnect Agent — `/probe` builds the device model, `/current` primes, and a `multipart/x-mixed-replace` `/sample` long-poll pump feeds `ISubscribable.OnDataChange` with ring-buffer sequence-gap re-baselining and a heartbeat watchdog; `UNAVAILABLE` → `BadNoCommunication`; the TrakHound MTConnect.NET dependency was evaluated and **dropped** — neither pinned package can parse an MTConnect document or frame the sample stream socket-free; browse comes free from the Wave-0 universal discovery browser, so the driver ships no browser project; **write-back is deferred, every MTConnect node is read-only**; live-gated on an isolated docker-dev stack against a real `mtconnect/agent:2.7.0.12` + SHDR-adapter fixture on `10.100.0.35:5000` — **merged to `master` 2026-07-27, PR #506, merge `90bdaa44`, pushed to origin**), **and AVEVA System Platform (Wonderware) Galaxy, now a standard Equipment-kind driver** (the old SystemPlatform mirror / alias-tag model was retired ~2026-06-12). **v3.0 NodeId/namespace scheme (this is the wire contract ScadaBridge binds against — a CUTOVER for its Data-Connection-Layer bindings; ScadaBridge is already namespace-URI-durable via `OpcUaNodeReference` and its #14 phase 2 shipped 2026-07-23 (main @ `266f001a`): `OpcUaReferenceForm.IsDurable` + a browse-picker nudge to the durable `nsu=<uri>;s=…` form + doc sweep. **ScadaBridge #14 is CLOSED (2026-07-24) — ScadaBridge consumes v3 end-to-end.** The live gates that were deferred to a real v3 rig all ran against the local `otopcua-dev` v3 cluster and PASSED: phase-2 `nsu=` hardening (`266f001a`), a DCL host-rewrite fix (`86d129de`), and the raw-path + native-alarm live gates (items B/C). Item C's premise was obsoleted by OtOpcUa #17; the item-A binding re-author (legacy `ns=2` collides with v3 `raw`) is downstream data-only work, not a code gap. Closing comment `#issuecomment-5223`. Scope record: ScadaBridge `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md`):** the single `https://zb.com/otopcua/ns` namespace is replaced by **two** — `https://zb.com/otopcua/raw` (device tree Folder→Driver→Device→TagGroup→Tag, NodeId `s=<RawPath>`) and `https://zb.com/otopcua/uns` (equipment tree Area→Line→Equipment→signal, NodeId `s=<Area>/<Line>/<Equipment>/<EffectiveName>`); every value has ONE source (the raw tag) fanned to both NodeIds (identical value/quality/timestamp), each UNS variable `Organizes`-references its raw node, writes route through either NodeId, HistoryRead works via both under one historian tagname, and native alarms fan (one `ReportEvent`) to the raw device folder + every referencing equipment folder. The old `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme is **retired**. (v3 program = Batches 1–4: greenfield Raw-tree schema + `/raw` authoring UI + Calculation driver + UNS reference-only Equipment + `{{equip}}` reference-relative scripts, all merged; Batch 4 lit up the address space.) Galaxy access flows through the in-process `GalaxyDriver` → gRPC → the **mxaccessgw** gateway. Surfaces live read + authorized write, native OPC UA Part 9 alarms, and server-side HistoryRead. **Address-space availability guarantee (matters to any OPC UA consumer, i.e. ScadaBridge — issues #485/#486, merged `master` 2026-07-21, live-gated on docker-dev):** a deployment artifact the node cannot read is treated as *no answer*, never as *an empty configuration*. Previously a transient ConfigDb error mid-deploy emptied the served address space (a `PureRemove` of every node — observed live, and it stayed empty), stopped every driver and cleared every subscription, while still reporting the deploy Applied; a consumer would have seen all bindings go bad. The node now holds its last-known-good address space, drivers and subscriptions, and reports the deploy **Failed** without advancing its revision so the retry lands. **Historian backend (merged 2026-06-27, PR #423):** OtOpcUa's sole historian read/write backend is **HistorianGateway** (via `ZB.MOM.WW.HistorianGateway.Client` — continuous historization + alarms), replacing the retired bespoke Wonderware historian driver. **Redundancy corrections (2026-07-21, `master @ 0de1352e`) — both consumer-visible:** (1) the downing strategy moved from SBR `keep-oldest` to Akka `AutoDowning` (`Cluster:SplitBrainResolverStrategy`, default `auto-down`, 15s window), because a two-node pair on `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node** — `KeepOldest.OldestDecision` only rescues a side holding >=2 members, so the 1-vs-1 survivor downs *itself* and exits. Same fix and same rationale as ScadaBridge `cf3bd52f`; accepted trade is dual-active during a real partition. **Its live gate is now CLOSED** — Phase 7 drill D4 (2026-07-24, `c50ebcf7`) ran the 1-vs-1 crash-the-oldest drill on the three-mesh rig and every survivor stayed Up without self-downing. (2) The redundancy **Primary is now the oldest Up `driver` member, not `RoleLeader` (lowest address)** — the two diverge after any node restart, so the Primary-gated data plane could enable on a node not hosting the cluster singletons; this is the node whose `ServiceLevel` reads 250 vs 240, so **a consumer selecting by ServiceLevel may now prefer a different node than before after a restart**. `NodeRedundancyState.IsRoleLeaderForDriver` renamed `IsDriverPrimary`. The election is still scoped **per Akka cluster, not per application `Cluster`** — the per-cluster mesh design (`docs/plans/2026-07-21-per-cluster-mesh-design.md`) closes that; **PROGRAM COMPLETE — Phases 0a/0b/1/2/3/4/5/6/7 all done + merged to `master` and pushed (Phase 7 `c50ebcf7`, bootstrap guard `d1dac87f`, 2026-07-24; Phase 6 was `2839deb5`, Phase 5 `35552a96`).** **Phase 2 (merge `6eaa81ca`)** added a second command transport behind the `MeshTransport:Mode` dark switch (`Dps` default), the prerequisite for splitting the fleet into one mesh per application `Cluster` (central and node no longer need shared gossip membership) — originally an Akka `ClusterClient`. **That transport has since been migrated off Akka entirely (ClusterClient→gRPC migration, 2026-08-06): Phases 0/1A/1B/2 are MERGED to `master` and pushed** (merges `10b6d209`/`b360239e`/`7203d135`; Phase 2 cutover `caf2eec6` flipped the docker-dev rig default to `Grpc`, live gate PASS 8/8 + 135-min soak clean, full suite green on the cutover SHA) — cross-mesh commands now ride gRPC: central→node `NodeCommandService` on the `:4056` telemetry listener, node→central deploy acks `CentralCommandService/ReportApply` on central's `:4055`, per-cluster PSK bearer auth, fail-closed, fire-and-forget with the deploy-deadline + boot-replay backstops unchanged. **Phase 3 (full `ClusterClient` deletion — code, the HOCON receptionist/client block, `CentralContactPoints`, the `ClusterClient` mode value; `SplitTopologyTransportValidator` now requires `Grpc`) is MERGED to `master` @ `2fa880b6` and pushed** (2026-08-06; branch carried 7 deletion steps + reviews + a docs sweep; DoD full suite exit 0, 50/50 suites, 9,326 passed / 0 failed). **Phase 4 (terminal live gate) PASSED 8/8 the same day — the ClusterClient→gRPC program is CLOSED** (gate record `d0ac6fc5`, run on `master @ 0b194003`, provenance-guarded post-deletion image): zero cross-mesh Akka associations (exactly 3 established `:4053` connections rig-wide, all pair-internal), zero receptionist/ClusterClient log lines across every drill, PSK negatives fail-closed both services incl. unset-key boot refusal, exactly-once ack matrix, central-kill/site-kill/mid-deploy-coordinator-kill drills all honest (one loud WRN per drop, +2:00 deadlines, boot-replay, no dupes — a coordinator killed in the dispatch second had its deadline fired by the recovered coordinator on the surviving central at exactly +2:00), terminal full suite RC=0 direct capture 50/50 suites 9,333/0. Sole re-carry: the I-4 cold-PSK-cache latency measurement, structurally unmeasurable on docker-dev (`ClusterPsks` answers ahead of the store), deferred to the first store-backed deployment. One non-mesh observation: a same-second restart of the central pair reproduced the documented guard-off cold-start split-brain (Phase 7 `Cluster:BootstrapGuard` is the opt-in mitigation, default OFF) — healed by a single-node rejoin; both site pairs re-formed clean. Not an OPC UA wire-contract change — internal mesh transport only, no ScadaBridge impact. **Phase 3 (merge `d01b0695`)** added config **fetch-and-cache**: behind the `ConfigSource:Mode` dark switch (`Direct` default = today's behaviour), a driver node fetches its deployed-configuration artifact **from central over a gRPC stream** (SHA-256-verified against the dispatch's `RevisionHash`), caches it in `ZB.MOM.WW.LocalDb`, and reads config **only** from that cache — never central SQL. **Consumer-relevant:** this extends the #485 availability guarantee — a `FetchAndCache` driver node **boots and serves its last-known-good address space with central SQL Server completely down** (restoring from the LocalDb pointer, no central dependency at boot), and a fetch that fails is treated as *no answer* (apply fails, last-known-good kept), never an empty configuration. The Phase 3 live gate caught + fixed a serve-gate deadlock (central had gated the artifact serve on `Status==Sealed`, but a deployment seals only after every node acks and a fetching node can't ack until it fetches). **Phase 4 (merge `26fad75c`, live-gated on docker-dev; Task-9 table drop `3a590a0c`)** cut the driver-side ConfigDb connection entirely: a **driver-only** node (`driver`, not `admin`) now boots with **no ConfigDb connection string at all** — config via FetchAndCache into LocalDb, scripted-alarm Part 9 condition state in a replicated LocalDb table, and central persists its deploy acks. **Consumer-visible ServiceLevel change (matters to any ServiceLevel-selecting OPC UA client, i.e. ScadaBridge):** DB-reachability is *retired* from the ServiceLevel calc on such DB-less nodes, so a healthy site node now publishes **240/250 even with central SQL Server completely down** (it has no DB connection to lose — survive-alone), where a DB-backed node whose ConfigDb went unreachable would drop to 0/100. So during a central-SQL outage a consumer keeps seeing the site nodes as fully authoritative rather than watching their ServiceLevel collapse. The old single-Akka-mesh election scope was unchanged as of Phase 4 (one Primary fleet-wide) — **Phase 6 (below) closes that.** **Phase 5 (merge `35552a96`, live gate PASSED 2026-07-23)** replaced the DPS fan-out of the four live-telemetry observability channels (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`) with a gRPC server-streaming contract — each **driver node hosts** the telemetry server and **central dials in** — behind the `Telemetry:Mode`/`TelemetryDial:Mode` dark switch (`Dps` default), fail-closed shared-bearer auth; `redundancy-state`/`fleet-status`/`deployment-acks` deferred with rationale. **Not consumer-facing to ScadaBridge** — this is central↔node AdminUI observability, not the OPC UA address space, with **no wire-contract change**. Also pinned `System.Security.Cryptography.Xml` 10.0.10 — the same four NU1903 advisories that broke ScadaBridge's fresh restores (`dced0d27`) were breaking OtOpcUa's too (204 errors on a clean restore, invisible locally because NuGet audit data is cached). **Phase 6 (MESH PARTITION — merged to `master` + pushed 2026-07-24, tip `2839deb5`; live gate RAN and caught three fix-forward defects — a boot-crash `StackOverflow` on every node from resolving `IClusterRoleInfo` inside the `AddAkka` configurator while the `ActorSystem` was still building, plus rig LDAP-missing on sites and a site-pair split-brain — all fixed, `792f28ec`)** split OtOpcUa's single fleet-wide Akka mesh into **one independent 2-node mesh per application `Cluster`** (central pair + one pair per site), the same shape ScadaBridge already runs — same `otopcua` ActorSystem name, separated purely by per-pair self-first seed partitioning, with central reaching each site mesh over the explicit ClusterClient/gRPC/ConfigServe transports (Phases 2/3/5) instead of shared gossip. **Consumer-visible (ServiceLevel-selecting clients, i.e. ScadaBridge):** the redundancy election is now scoped **per application `Cluster`, not per Akka mesh** — so there are now **two-or-more Primaries fleet-wide, one per `Cluster`, by design** (`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton on every driver node). A consumer selecting by ServiceLevel therefore correctly sees one Primary (250) + one Secondary (240) **per application `Cluster`**, resolving the "one Primary fleet-wide" scope caveat noted under Phases 0b/4. **No OPC UA wire-contract change** — this is internal Akka topology + AdminUI/redundancy, not the address space. Also new: a fail-closed `SplitTopologyTransportValidator` refuses to boot a cluster-role node left on a DPS transport; secrets replication is now pair-local (no SQL-hub — sites have no SQL); the compiled transport-mode defaults stay `Dps` (the validator, not a default flip, is the guardrail). Plan `OtOpcUa/docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. **Phase 7 (`c50ebcf7`, 2026-07-24) closed the program:** 6/6 failover drills PASSED across all three pair types (graceful manual failover, auto-down both directions, crash-the-oldest, cold-start-alone, rejoin-as-Secondary), closing the auto-down gate open since Phase 0a. The manual-failover button is **MAIN-only by construction** (it acts on the local node's `Cluster.State` and the AdminUI runs only on the central pair) — site pairs fail over via auto-down only. A follow-on **bootstrap guard** (`d1dac87f`, opt-in `Cluster:BootstrapGuard:Enabled`, **default OFF**) closes simultaneous-cold-start split-brain: the lexicographically lower `host:port` node is the preferred founder, the higher node TCP-probes its partner and joins if reachable / self-forms if not — the order is chosen **before** the single `JoinSeedNodes`, so it can never re-form mid-handshake. **Deferment register (2026-07-27, `e08855fb`): `OtOpcUa/deferment.md`** — a source-verified inventory of unfinished work. **Two entries are consumer-relevant to ScadaBridge:** (1) **node-level ACLs are authored, deployed and NOT ENFORCED** — the `IPermissionEvaluator`/`TriePermissionEvaluator` subsystem has zero production consumers and `OtOpcUaNodeManager` never references it, yet operators can author `NodeAcl` rows that ship in every artifact; actual enforcement is a realm-qualified **write** gate plus an alarm-ack gate, so **reads are ungated and authored deny rules have no effect** (do not assume OtOpcUa filters an address space per-user). **⚠️ Changed 2026-08-01 (Gitea #520, merged to `master` + pushed, merge `fbe7403c`): the OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine, so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an `otua_<keyId>_<secret>` credential as the password, minted on the AdminUI `/api-keys` page (shown once). `LdapOpcUaUserAuthenticator` is deleted; **LDAP survives for AdminUI login only.** **This does not break ScadaBridge today** — `OpcUa:AllowAnonymous` still defaults to `true`, and an anonymous session is now a normal grantable principal (`anonymous`) rather than a bypass, arriving at the same role gates a key does; set it `false` to make keys mandatory. A key's grants still collapse into the two server-wide gates and **only `Cluster`-scoped grants count** — honouring a narrower grant's flags while unable to honour its scope would turn "WriteOperate on Equipment X" into server-wide write authority. Keys reach a driver-only node inside the deployment artifact (the peppered HMAC only, never a secret), so **revocation is asymmetric: immediate on a node with a ConfigDb, next-deployment on a driver-only one.** (2) **a driver-reported remote change is an operator PROMPT, never an address-space rebuild** (`feat/deferment-remediation`, 2026-07-27, Gitea #518) — `IRediscoverable` was inert fleet-wide and is now consumed: a Galaxy redeploy / TwinCAT symbol-version bump / MTConnect Agent restart / Sparkplug rebirth raises a **"re-browse" chip** on the AdminUI `/hosts` page (carried on `DriverHealthChanged.RediscoveryNeededUtc`/`.RediscoveryReason`). It is **advisory**: the served address space is deliberately NOT changed, because v3 authors raw tags through the `/raw` browse-commit flow — so **the served tree still only changes on a config redeploy**, which makes it more stable than the driver docs imply, not less. Gitea #507 (discovered-node injection) is **closed as superseded** and its code deleted — it read two structures that are structurally empty in v3, so it could never have injected anything. **`IHostConnectivityProbe` is now consumed** (Gitea #521, merged `master` `abda6f9e` 2026-07-30, live-gated on docker-dev): eleven drivers computed per-host connectivity that nothing read. It now rides `DriverHealthChanged.HostStatuses` to a **Hosts column on the AdminUI `/hosts` page**, showing the case the driver-level state chip structurally cannot — a multi-device driver staying aggregate-Healthy while one of its PLCs is unreachable. Resolved *neither* way the issue framed it: the `DriverHostStatus` table's intended publisher (driver nodes upserting rows) was **unbuildable** after Phase 4, since a driver-only node has no ConfigDb connection — so the capability was kept, the transport moved to the Phase 5 telemetry stream, and the permanently-empty table was **dropped**. **Not consumer-facing to ScadaBridge** — AdminUI observability, no OPC UA wire-contract change. **That register's remediation is MERGED to `master` and pushed** (2026-07-28, merge `914d70bf`) — all six ranked items plus two deferred follow-ups: ACL non-enforcement made explicit in docs + UI warnings (wire-up = Gitea #520), the rediscovery signal above, **#516 driver config edits no longer silently discarded** (a changed `DriverConfig` now stop+respawns the driver instead of an in-place delta that five drivers ignored while the deploy sealed green — **operationally visible: a config edit now costs a driver reconnect**), AdminUI dispatch-map parity guards, plan bookkeeping, and the driver **stability tiers documented as inert** (every driver runs Tier A), and that machinery has since been **DELETED** (Gitea #522, merge `abda6f9e` 2026-07-30) — `MemoryTracking`, `MemoryRecycle`, `ScheduledRecycleScheduler`, `IDriverSupervisor`, the vestigial `DriverTypeRegistry`, and the operator-authorable `RecycleIntervalSeconds` knob are gone. Deleted rather than activated because the premise was gone, not merely unused: "Tier C" meant an out-of-process driver Host a supervisor could restart, and none exists anywhere (Galaxy is gRPC-to-mxaccessgw since PR 7.2, FOCAS in-process since 2026-04-24), so `IDriverSupervisor` had zero implementations. The inertness was also broader than filed — nothing constructed the trackers outside their own unit tests, so "pass real tiers" was never a flag flip. **`DriverTier` itself survives and is load-bearing** (`GetTierDefaults` supplies the real per-capability timeout/retry/breaker policies). Two follow-ups the first pass had deliberately left unbuilt also landed: Sql/FOCAS now re-parse config in place too (each derives its connection/backend from config, so the parse was made **atomic** — options + dialect + connection string, or options + backend, adopted together — rather than half-applying a change), and tag-set drift detection (FOCAS/TwinCAT/MTConnect/AbCip re-browsed every 5 min and raised the same `/hosts` prompt when the device's tag set differed from what was authored) — **which was REMOVED two days later and is NOT in `master`** (Gitea #523, merge `2a0461a1`, 2026-07-30): the check was recurring, unconfigurable (interval was a constructor parameter with no appsettings key) and never live-gated outbound traffic to real PLCs/CNCs/Agents, and how often to browse someone's plant equipment is an operational decision rather than an implementation detail. **Consequence to carry forward:** AbCip and FOCAS browse a live backend but implement no `IRediscoverable`, so they now have **no tag-change signal at all** — a PLC re-download is invisible until an operator re-browses by hand; Galaxy/TwinCAT/MTConnect/Mqtt raise natively and are unaffected. **Not consumer-facing to ScadaBridge either way** — no OPC UA wire-contract change, and the `IRediscoverable` prompt above is untouched. That same round (merge `2a0461a1`) also retired 13 stale skipped tests (6 revived onto the v3 raw/UNS shape, 3 folded into a widened parity corpus, 4 deleted as retired), added a reflection guard over the deploy-time TagConfig gate after finding MQTT shipped an `Inspect()` nobody called, fixed three OpcUaClient integration tests that had been silently failing on `master` (v3 made the driver resolve references through its authored `RawTags` table, so the tests' bare `ns=3;s=…` node ids failed locally at the resolver and never reached the wire), and wrote the two missing driver guides (`docs/drivers/Sql.md`, `docs/drivers/Calculation.md` — Sql had shipped with no user-facing doc at all). A third tracker round merged **2026-07-31 (`9fa701b1`)**: Gitea **#526** split `ISqlDialect`/`SqlServerDialect` into a `Driver.Sql.Dialect` leaf so the Sql **browser** no longer drags the runtime driver — and, through it, `ZB.MOM.WW.OtOpcUa.Core` — into the AdminUI's build/deploy graph (measured absent before/after; this was the remaining cause #512 had wrongly attributed to MQTT), guarded now by a csproj-graph test over **every** `*.Browser` project; **#525** deleted the unconsumed `IDriver.GetMemoryFootprint` / `FlushOptionalCachesAsync` across all 12 drivers plus the support code that only fed them; **#481** bridged driver comms-loss into scripted-alarm input quality and made the engine distinguish a never-pushed input from a driver reporting Bad. **Consumer-visible to ScadaBridge, with a caveat:** a scripted-alarm condition whose input reports Bad with no value now reports Bad `Quality` on the wire where it previously read Good, and virtual tags computed from such an input go Bad too — the *comms-loss* half was initially unreachable, because the #481 live gate found that `DisconnectObserved` had **no production sender**: a driver whose device died mid-run was never detected, never reported comms loss, and did not recover when the device returned (live-observed on Modbus; a node restart was needed). **That is now FIXED and the caveat is withdrawn** — Gitea **#527**, merged `master` `916f0f79` 2026-07-31, live-gated twice on docker-dev. A connected driver whose device dies is declared disconnected once it has gone 90 s without a successful read **while also** reporting itself non-Healthy, which emits `ConnectivityChanged(false)` and drives both quality paths (#477 native conditions, #481 scripted-alarm inputs), and it **re-initialises and recovers by itself when the device returns** (~8 s in the gate, no node restart). **So OtOpcUa condition Quality can now be relied on to signal that a PLC went away** — with the honest bound that detection is delayed by up to ~2 minutes by design, not instant. The trip condition is deliberately a conjunction rather than "non-Healthy for N polls": `Degraded` means *connected but some data is bad*, and drivers that write health per-tag inside their read loop latch `Degraded` against a perfectly healthy device when one authored address is unreadable, so a state-only rule would have reconnected working drivers every 90 s forever. Also fixed in the same change: the reconnect retry now tears the transport down before re-initialising, since Modbus and MQTT overwrite an `IAsyncDisposable` field in `InitializeAsync` and would otherwise leak a socket per retry. **AB CIP / AB Legacy honoured the wrong TCP port until 2026-08-01** (Gitea **#534**, merged `master` `94990990`): both libplctag drivers carried the authored EtherNet/IP `Port` on their tag-create params and consumed it nowhere, and libplctag does not fault on a portless gateway — it substitutes 44818. So a device authored on a non-default port was not merely unreachable; the connection was silently redirected to whatever answers on 44818, which on a plant network is a different PLC whose tags would then read `Good`. Anyone who concluded that AB devices must live on 44818 can stop working around it. **The discovery half of that defect survived until 2026-08-06 as Gitea #539, now FIXED** (merge `5b9ff9d0`): the AbCip “Browse controller” path still passed the bare host to libplctag, so browsing a device authored on a non-default port silently returned the symbol table of whatever answers on 44818 — an operator could commit tags against the wrong PLC. Both discovery sites (symbol-list enumerator + UDT template reader) now compose `host:port`, a source-scan guard test pins every libplctag Tag construction in both AB drivers to the composed form, and the fix was live-gated against the fixture host (browse on `:44818` returns the nine-tag table; browse on a dead `:44999` — the issue's exact repro — now fails instead of redirecting). Found only because the AbLegacy PCCC test fixture had never actually been deployed — it now runs one simulator per family on its own port, and that suite passes for the first time. **The FOCAS fixed tree became publishable for the first time on 2026-08-05** (Gitea **#540**, merged `master` `4e22f024`): browse-committing a fixed-tree leaf produced a tag the driver could never read, so the whole `Identity`/`Axes`/`Spindle`/`Program`/`OperationMode`/`Timers` surface could not reach the address space through any v3 authoring path — v2 auto-injected it and #507 removed that injection as superseded by a browse-commit route that did not work. A captured browse leaf's id **is** its driver `FullName`, and the fixed tree emitted its internal cache key `{deviceHost}/{path}` there; both authoring paths wrote that verbatim into the tag address, no FOCAS parser accepts it, and the driver skipped every such tag at init — so all 20 committed tags on the wonder Makino read `BadNodeIdUnknown` forever while the deploy sealed green and the driver reported Healthy with 0 errors. Fixed at the emitter rather than by stripping the host at commit, which repairs the `/raw` browse-commit and the manual picker together: a fourth address form `FIXED:<path>` (e.g. `FIXED:Identity/SeriesNumber`) served from the driver's own poll-loop snapshot, read-only, and carrying **no device** — the tag's Device row already routes it, and the duplicate routing channel was the root cause rather than the parse failure. **Consumer-visible to ScadaBridge: new FOCAS nodes can now appear** where the fixed tree is authored (they could not before), they are `ViewOnly`, and their quality is honest — values are dated from the last answered poll and go Bad after 8 poll cadences (floored at 30 s) instead of serving a dead CNC's last snapshot as Good indefinitely. Two authoring-side guardrails shipped with it (the tag modal and the deploy-time inspector now run the real address parser, so a bad address is rejected where it is authored rather than silently skipped at driver init), and the manual picker — which had offered `axis`/`spindle`/`program`/`status` "parameter groups" that are not FOCAS addressing spaces, defaulting to the unusable `axis:5` — now composes PMC/`PARAM:`/`MACRO:` through the parser itself. **The other half of #540 was split out as Gitea #541 and is now FIXED** (merged to `master` + pushed, merge `355e59aa`, 2026-08-06, live-gated on docker-dev over two rounds). Every driver skipped an unmappable tag at init with only a log line, so **a driver could report Healthy / 0 errors / a live-advancing Last read while ignoring most of its tags** — that is no longer invisible. A new optional `ISkippedTagReporter` capability over a shared `SkippedTagLedger` carries the authored count, the exact skipped count and a capped sample of RawPath + reason on `DriverHealthChanged`, surfacing as a **Tags column on the AdminUI `/hosts` page** (`20 / 22 skipped`; red when a driver is polling *nothing*, which is the state most likely to look healthy because it has no failing reads to report). Transport is the Phase 5 telemetry stream, **not a table** — the #521 lesson, since a driver-only node has no ConfigDb. **The issue's own count was wrong: it named eight drivers, and it is twelve** — Calculation, MTConnect, Galaxy and OpcUaClient were unlisted, and OpcUaClient's was the quietest of the twelve (a bare `continue` in `NamespaceMap` with not even a log line). A reflective fleet guard over every `IDriver` now pins the invariant so a thirteenth driver cannot re-introduce the silence. Sql reports the **union of its two drop points** — the blob that does not map *and* the §8.1 catalog gate — because schema drift (a renamed column, a dropped view) produces the second, and reporting only the first would have it claim `0 skipped` while polling half its tags. The signal is **additive to driver state, never `Degraded`**: latching non-Healthy for a config problem would satisfy both halves of #527's comms-loss conjunction and reconnect-loop a perfectly healthy device whose tags were all skipped. **Not consumer-facing to ScadaBridge** — AdminUI observability, no OPC UA wire-contract change; a skipped tag's node still materializes and still reads `BadNodeIdUnknown` exactly as before. What changes is that an operator can now *see* it. The live gate found one defect the unit tests could not (a driver that faulted before building a tag table reported `0 tags` against its real authored count — a positive claim made on behalf of a driver that never looked; `GetSkippedTags()` now returns null until the first committed build), and surfaced **three drivers on the dev rig that had been silently skipping their tags for an unknown length of time**, each reporting Healthy with zero errors. The duplicate-sibling-axis residual (the wonder Makino returns two axis folders both called `AA`, which the commit gate refused and whose name-keyed runtime cache let the second overwrite the first) is **FIXED as Gitea #548** (merge `8b5c478c`, 2026-08-06): duplicated sibling names — axes AND spindles — are occurrence-suffixed in slot order (`AA_1`/`AA_2`; unique names stay bare, so nothing renames on ordinary machines), the emitter/snapshot-keys/servo-loads/spindle-resolution all consume one shared segment list so they cannot disagree, and the bare name of a duplicate set deliberately reads `BadNodeIdUnknown` rather than one arbitrary axis's data. Red-first offline tests plus a live focas-mock wire gate; not yet deployed to wonder (the box runs `4e22f024`) — authoring the Makino's `Axes/` needs that deployment plus a re-browse. #540 itself is CLOSED as of 2026-08-06. **Deployed to `wonder-app-vd03` the same day** (box `4e461951` → `4e22f024`; no EF migration in the delta, so a plain binary swap): the 20 dead FOCAS tags had their addresses rewritten in place to the `FIXED:` form rather than being re-browsed — an exact 1:1 transform that preserves every RawPath/Name/DataType/TagGroup/UNS reference, so nothing downstream rebinds — and they now read **20/20 Good** over OPC UA with real machine data (Series `G431`, 7 axes, mode `AUTO`, program `//CNC_MEM/USER/LIBRARY/O1234`), where they had been 0/20 since the v3 cutover. Nothing else in that merge was consumer-visible: the `NodeAcl` scope model gained a variable-depth `NodeScope` plus the scope-path producer its trie builder had always required (Gitea #520 stage 3). **Node ACLs ARE now enforced, but only behind an opt-in flag (Gitea #520 stage 5, `master` 2026-08-01).** `Security:Acl:Enabled` defaults to **`false`**, and with it off nothing changes for a consumer: the two coarse server-wide gates (`WriteOperate` on Value writes, `AlarmAck` on Part 9 methods) remain the whole of data-plane authorization and HistoryRead stays ungated. **With it on, scoped `NodeAcl` grants decide EVERY data-plane operation** — read, browse, subscribe, write, alarm commands and HistoryRead (#520 stages 5 + 6, `master` 2026-08-01, live-gated on docker-dev) — and the scoped decision **replaces** the server-wide role rather than adding to it. **What this means for ScadaBridge — this changed on 2026-08-01 and the previous entry said the opposite:** reads, browses and subscriptions are ungated only while the flag is **off**. On a node where an operator turns it on, a read-only consumer is affected too: **every** operation ScadaBridge performs — browsing the address space, reading, subscribing, and its writes / alarm acks / HistoryRead — is denied unless the principal it authenticates as holds a matching grant — and since `OpcUa:AllowAnonymous` still defaults `true`, that principal is the reserved `anonymous` one unless ScadaBridge is given a key. Granting `anonymous` the permissions it needs, or issuing it an `otua_` key and granting that principal, are both first-class options. Two consequences of stage 6 specifically. **Browse is needed to navigate, and a grant deep in the tree automatically makes the folders above it browsable** — otherwise the grant could not be reached — so a narrow grant still exposes the names of its ancestors. And **revoking a grant now stops a running subscription**: the monitored item receives one `BadUserAccessDenied` and then goes silent, and a later deployment restoring the grant resumes it with no client action, so ScadaBridge does not need to reconnect to recover. Both were verified live on docker-dev, not only in tests. Two behaviours worth knowing: an ACL or key change **reaches sessions that are already open** (each decision stamps the current deployment generation and re-checks the key, so revoking a key stops its live session rather than waiting for a reconnect), and enforcement **fails closed on every unresolvable case** — a misconfigured node denies everything rather than allowing everything, which is the safe direction but presents as a total outage of the three gated operations rather than a subtle permission bug. **#520 is now COMPLETE (stage 7, `master` `3796c0bf`, 2026-08-01, live-gated on docker-dev):** grants are authored through a scope picker over the cluster's real raw + UNS trees, which supplies the scope kind and target together, and both the AdminUI and the deploy gate now refuse a grant whose scope resolves to nothing. **Not consumer-facing to ScadaBridge** — authoring-side only, no OPC UA wire-contract or enforcement change; it matters only in that a grant issued to ScadaBridge's principal can no longer be saved pointing at an entity that does not exist. **Tracker zeroed 2026-08-10 — everything below is merged to `master` and pushed (origin now at `d06e92c6`; the repo had run ~45 commits ahead of origin over 2026-08-07→10 and is back in sync).** Six subagent rounds closed Gitea **#549–#559**, completing the **authoring-vs-deploy-gate parity arc**: every driver's AdminUI tag editor, the CSV import/export merge, the `/raw` authoring service, browse-commit, and the deploy-time `Inspect` gate now all consult the **driver's own runtime parse** as the single verdict authority (`TagConfigParseGate`; accept-gate-first Inspects with rejection-vs-advisory classified structurally), so what any authoring path accepts is exactly what the driver serves — the "authored green, reads `BadNodeIdUnknown` forever" class is closed fleet-wide. **Two served-behaviour fixes are consumer-visible to any OPC UA client:** a leading-whitespace MQTT TagConfig blob is now served instead of silently refused (#559 — MQTT was the one driver whose runtime mappers carried their own untrimmed framing gate; `JsonDocument.Parse` is now the fleet-wide framing authority), and an OpcUaClient tag authored with the legacy PascalCase `NodeId` key now survives an editor round-trip (#557 — the editor previously dropped the key on save, taking the tag dark on the next deploy). One deliberate authoring tightening: an untouched `{}` Modbus config is refused at save (#554 — it always read `BadNodeIdUnknown` under a green deploy). Also in that window: **connect-path resilience** — `IDriver.InitializeAsync` now runs through the per-capability invoker (timeout/breaker, `DriverCapability.Initialize`; R2-10, merge `1038c0e2`), closing the last uninvoked driver entry point, with the breaker threshold made cadence-reachable and a respawn invalidating the breaker tracker so a mid-outage deploy cannot strand a stale OPEN chip on `/hosts`. And a plan-task freshness sweep (2026-08-10, `d06e92c6`) live-verified Modbus-Int64 end-to-end authoring and the S7/AbCip Test-Connect probe E2E (12/12; the probe suite's skip-gating turned out to be pure TCP reachability — runnable from macOS, the "needs Windows-VM" note was stale) and closed the two plan tasks whose subjects v3 had deleted; open plan tasks are down to 3, all deliberate holds. None of the above changes the OPC UA wire contract beyond the two served-behaviour fixes named. |
| **MxAccessGateway** (`mxaccessgw`) | `~/Desktop/MxAccessGateway` | .NET 10 gateway (x64) + .NET 4.8 worker (**x86**), gRPC | `gitea.dohertylan.com/dohertj2/mxaccessgw` | gRPC gateway giving modern clients full MXAccess parity without loading 32-bit COM. Two-process: gateway (ASP.NET Core gRPC + Blazor dashboard) + per-session x86 worker that owns the MXAccess COM STA. **OtOpcUa depends on this.** |
| **ScadaBridge** | `~/Desktop/ScadaBridge` | .NET 10, Akka.NET, Docker | `gitea.dohertylan.com/dohertj2/ScadaBridge` | Full implementation of the distributed SCADA platform — hub-and-spoke (1 central cluster + N site clusters). Projects prefixed `ZB.MOM.WW.ScadaBridge.*`; solution `ZB.MOM.WW.ScadaBridge.slnx`. Ships `src/`, `tests/`, `docker/` topology, and the design docs that are the spec. |
| **HistorianGateway** | `~/Desktop/HistorianGateway` | .NET 10 x64, gRPC, Blazor | `gitea.dohertylan.com/dohertj2/historiangw` | Single-process gRPC sidecar exposing (1) full read/write API to the AVEVA Historian (5 gRPC services; 15 retrieval modes; historical/backfill writes; tag-config lifecycle; SQL live-value path; store-forward + redundancy resilience; all default-disabled) and (2) read-only Galaxy object-hierarchy browse via the shared `ZB.MOM.WW.GalaxyRepository` lib (consumed as a Gitea-feed package). No COM, no x86 worker. **Dev:** two plaintext endpoints from `appsettings.Development.json` — dashboard on `:5220` (HTTP/1.1), gRPC h2c on `:5221`. **Production:** single `Kestrel:Endpoints:Https` endpoint with `Protocols: Http1AndHttp2` multiplexes dashboard + gRPC over one TLS port (ALPN); warn-only if no TLS endpoint configured (valid behind a reverse proxy / Kubernetes ingress; the warn predicate covers any non-Development environment, i.e. Production + Staging). In a non-Development environment the gateway also logs warn-only **production-readiness** checks (pending.md D2/D3) — relative runtime-artifact paths + secret hygiene (`ApiKeys:Mode=Disabled`, empty/dev-placeholder pepper, dev-placeholder LDAP password). **Owns `AVEVA.Historian.Client`** — imported into the repo at `histsdk/` (subtree, folded in with history 2026-06-28; the old "vendored / re-vendor from upstream" model is **retired** — `histsdk/**` is ordinary owned source, edited in place; the gateway builds it directly). Store-forward uses a crash-safe FasterLog append-only outbox (`Microsoft.FASTER.Core` 2.6.5; `CommitMode` PerEntry/Periodic), not SQLite. **Handshake amortization (pending.md A1) done + live-validated** — a default-on leased-session pool (`Historian:SessionPool`) reuses pre-authenticated sessions across reads/writes/status ops/tag-browse/metadata (~4.7× measured; probe and blocks stay per-call), with a `<~15 s` keepalive + reactive re-auth, surfaced via a `PooledHistorianClient` facade so services are unchanged; the `HistorianSession` primitive lives in the owned `AVEVA.Historian.Client` under `histsdk/`; browse/metadata + SendEvent broadening merged to `main` + pushed. **`SendEvent` is also amortized** via a **separate, parallel event-session pool** (`Historian:EventSessionPool`, default-on; v8/ECDH auth — kept distinct from the v6 pool), warranted by a GREEN v8 Event-session reuse spike (~10–16×); `ReadEvents` stays per-call / gated (C2). The full offline suite is green on macOS (0 warnings); the env-gated live historian + Galaxy integration suite exercises the amortized path and otherwise skips without a live server. **Part C minimal (pending.md C1/C3a) merged to `main`:** `Int8`/`UInt8` live write types un-gated + live-proven against `wonder-sql-vd03` (in the owned `histsdk/` SDK); the 2023 R2 gRPC interface-version integers recorded as evidence (C3a); `UInt1` attempted but **server-blocked** (the historian accepts `EnsureTags(UInt1)` yet stores a degenerate analog tag) → re-gated fail-closed. **C2 closed won't-fix (2026-06-26):** event reads are server-gated on the 2023 R2 historian over **both** transports — gRPC retrieval-server-gated (0 rows scoped to a managed connection), and the WCF certificate transport + auth DO reach the historian cross-platform (CM_EVENT registers on the `0x501` event connection) but the query still returns 0 rows: the same server-side per-connection row gate. (An earlier note saying "WCF not served on 2023 R2" was a test error vs the reverse tunnel.) Not client-fixable. Reusable SDK wins: `ConnectViaAddress` (WCF Via for tunneled access), `EventReadConnectionModeOverride`. **SQL-path `ReadEvents` (merged 2026-06-26) — the practical workaround for the C2 gate:** event reads now **ship** via the historian's `Runtime.dbo.Events` SQL view (config-gated `RuntimeDb:EventReadsEnabled` + `EventReadMaxRows`), mirroring the SQL live-write path (no COM/native); live-proven streaming real events (incl. an INSQL NOT-NULL `CAST` fix the live test caught) while the native event-query stays gated — so **event reads work** despite the native dead-end. **All of Part C + C2 + SQL-ReadEvents are merged to `origin/main`** (gateway @ `fabab1a`, histsdk @ `f0a1b04`). **Follow-ups merged to `main` (2026-06-27):** `historiangw` **PR #6** — `WriteLiveValues` UTC→server-local timestamp fix (live-validated exact) + a documented `SendEvent` `Source_Object` protocol limitation (`pending.md` C4); the Plan-1 .NET **client lib + packable Contracts** (PR #5); and a C2 cross-ref docs PR (#4). **First consumer (merged):** OtOpcUa **PR #423** (→ `master`) adopts `ZB.MOM.WW.HistorianGateway.Client` as its sole historian read/write backend, retiring its bespoke Wonderware historian driver. **Local Docker (2026-06-29):** `HistorianGateway/docker/` runs the sidecar in a container against the REAL wonder datasources (historian `wonder-sql-vd03:32565` TLS + Galaxy SQL `wonder-app-vd03`/`ZB`), dashboard login + gRPC API-key auth disabled — verified `/health/ready` Healthy. Runtime-only image: publish framework-dependent on the host (authed Gitea feed → no Docker restore), COPY into `aspnet:10.0`; secrets via gitignored `env_file`; isolated compose project `zb-historiangw`; container egress needs the host VPN up. **Client surface (`clients/dotnet/`):** the published `ZB.MOM.WW.HistorianGateway.Client` lib + a `ZB.MOM.WW.HistorianGateway.Client.Cli` smoke CLI (`probe`/`read-raw`/`browse`, JSON out; in the `.slnx`, not packed). **Current: `origin/main` @ `56a70df`** — **AHC-002 string READS landed via native RE from the Mac (PR #17 merged 2026-07-16):** the SDK now parses string historian result-buffer rows into `HistorianSample.StringValue` (a per-row `valueType` discriminator — `1`=numeric→`double`, `3`=string; value = `u32 charLen` + UTF-16LE), and the gateway forwards `string_value` through `HistorianRead.ReadRaw` — **no contract change, no package bump** (the proto already carried the field; mapper already forwarded it); live-proven against 2023 R2. String **tag-create** is now doable via SQL `aaStringTagInsert` (`SqlStringTagProvisioner`, test tooling — single + double byte). String **historical writes** were probed (bounded 6-candidate native blob probe) but **non-convergent** — tag-info resolves a **real GUID + SingleByteString** (unlike the Int1/UInt1 degenerate stub) so writes aren't blocked at resolution, but `BSuccess` is **not an oracle** (all framings optimistically accepted) and none round-tripped → **deferred, needs a native-client write wire capture** (blind guessing can't finish). Eight follow-up issues filed (`historiangw` #18–#25): datatype gaps (native string tag-create #18, string writes #19, DateTime/FileTime #20, Guid/Structure/Event #21, Int1/UInt1 discrete-create #22 [follow-on to #11], DoubleByteString read-verify #23) + **SQL-fallback trackers** (event reads via `Runtime.dbo.Events` #24, live writes via `History`/`v_StringHistory` #25). Also folded a doc-truth reconcile of the "Key structural decisions" bullets (GalaxyRepository `PackageReference`, Contracts/Client 0.3.0 published, OtOpcUa on 0.3.0). Prior: **ArchReview #2 (cycle-2) remediation MERGED 2026-07-14 (PR #12):** all 8 workstreams A–H — store-forward drain honesty (`IsRetryableDeliveryFault`, typed `HistorianOutboxFullException`→`ResourceExhausted`), streaming-lease caps (per-key cap 8→4, `MaxStreamLeaseHold`, bounded `EndQuery` teardown, client `StreamTimeout` default null→10min), a **live idle-out capture (GREEN-A, native `(type 4, code 51)`) that narrowed the session-expiry classifier and closed the stability-F5 tail**, histsdk residue (typed event-send `FromNativeError`, `HistorianProtocolIntegrityException`, IP/host scrub), read-path caps (`MaxAggregateIntervals`/`MaxAtTimeTimestamps`, 5M→`ResourceExhausted`, verify-once-per-connection version gate, batched audit drain), release hygiene (csproj `<Version>` sole authority + packed-README/AnyCPU/FQDN/CHANGELOG guards), security/ops (opt-in `ForwardedHeaders` trust list, k8s key-bootstrap runbook, dashboard pool/dead-letter signals), and doc-truth #2 (`Historian:VerifyServerInterfaceVersion` now a real fail-closed key). Verified 0-warn, offline 1388 pass + live suite green vs `wonder-sql-vd03`. **Contracts+Client 0.3.0 PUBLISHED** to the `dohertj2-gitea` feed (2026-07-14; carries the `StreamTimeout` default change) and **OtOpcUa bumped to 0.3.0** (`lmxopcua master @ 7f79cd59`, PR #440 — only consumer delta is the StreamTimeout default; driver tests 103/0). **Boolean historization remapped Int1→Int2 (2026-07-14, `lmxopcua master @ 38e21df2`, PR #442, closes #441):** the R2-06 live gate surfaced that the historian's `Int1` analog-tag creation path is server-degenerate (`EnsureTags(Int1)` stores an unusable stub) — filed gateway-side as `historiangw` #11, documented there as an explicit evidence-based exclusion (PR #16), and routed back to OtOpcUa as issue #441. OtOpcUa's `HistorianTypeMapper` now maps `Boolean → Int2` (0/1 integer); the value-write path already sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips cleanly and a pre-existing Float Boolean tag retypes cleanly (both are supported analog types). Closes the last R2-06 red test → live gate **6/6**; gateway driver unit 103/103. **archreview round-2 deferred gates closed 2026-07-14** (`lmxopcua master @ 30672888`): **R2-04 T13+T15 LIVE-PASSED** (PR #444) — the behavior-affecting S4 primary-gate default-deny verified on a 2-node docker-dev rig (ServiceLevel 250/240; boot-window write rejected `not primary (role unknown)` + denial meter `reason=role-unknown`; steady-state secondary rejected + node reverted; primary write reaches device) plus the in-process `PrimaryGateFailoverTests` (delivered-snapshot drives the gate); and the pre-existing red S7.Cli phrase-scan test fixed (PR #443, restored a stripped rationale comment). Round-2 remainder is now only infra-gated: R2-08 live-GLAuth outage, R2-01 S7 SYN-blackhole. **The full `*.IntegrationTests` sweep is now COMPLETE (2026-07-15, `lmxopcua master @ 152a5645`):** ran all 10 driver + server integration suites serially (on the Mac against native fixtures on the `10.100.0.35` Docker host), then closed all 7 resulting follow-ups (PRs #446–#452) — **every driver + server integration suite is verified GREEN with ZERO OtOpcUa production regressions;** every original "failure" was fixture/harness/test rot, not product. Highlights of cross-repo relevance: (a) **the Host integration harness's LDAP swapped from the retired `bitnami/openldap:2.6` to GLAuth** (PR #451) — the harness now runs its own ephemeral GLAuth container on `:3894` (seeded by `tests/.../Host.IntegrationTests/glauth/config.toml`, distinct from the shared dev GLAuth on `:3893`), unifying the whole family's opt-in real-LDAP path on GLAuth and removing the lone OpenLDAP outlier; (b) the deploy-E2E "amd64-emulated-SQL timing" complaint was **misdiagnosed** — those tests had only ever run against the EF in-memory provider (no FK enforcement) and lacked the `ServerCluster`/`ClusterNode` seed the real-SQL `NodeDeploymentState` FK requires; fixed with a harness seeding helper + `OTOPCUA_HARNESS_SQL_HOST`/`_LDAP_HOST` overrides so the fixtures run on the native-amd64 Docker host (PR #452). See `archreview/plans/INTEGRATION-SWEEP-STATUS.md` for the full record. Cycle-2 gated tail: CI first-green-run (needs a Gitea act_runner), the histsdk protobuf/Grpc.Tools major bump (03-CC-8, blocked on migrating off obsolete `GrpcWebHandler.HttpVersion`), and the 1.0.0 TLS-default flip. The prior cycle-1 **ArchReview remediation is fully merged** (Now + Next tiers 2026-07-09 @ `508b9c3`; **Later tier 2026-07-13** via a coordinated two-PR merge, `12385c8` + `6a24a30`). Off a 2026-07-08 six-report architecture review (`archreview/`) turned into 8 workstream plans (`docs/plans/2026-07-09-archreview-*`; tracker `...-remediation-tracking.md`), shipped clean under `TreatWarningsAsErrors`: a hard-fail `ResilienceComposition` validator (StoreForward+Redundancy both-on), corruption-tolerant outbox + dead-letter/overflow knobs, `/auth/login` rate-limit, server-side `max_values` cap, a headless `apikey` CLI, histsdk parser hardening, Gitea Actions CI + warnings-as-errors on all trees, pool observability (lease-wait/waiter/counters) + session-pool health probe, and a typed histsdk fault taxonomy with type-based classification + session churn-guard. **Contracts+Client 0.2.0 published** to the `dohertj2-gitea` feed (2026-07-09; Int1 proto3-optional, opc_quality doc, CA-chain pin). **Cross-repo 0.2.0/G-2 chain COMPLETE:** OtOpcUa bumped to Client+Contracts `0.2.0` (2026-07-10, `master @ f6eaa267`; M3 `opc_quality` gate clear), and the shared `ZB.MOM.WW.Auth.ApiKeys` gained `ExpiresUtc` at `0.1.4` — consumed by HistorianGateway (`apikey create --expires` + dashboard Expires column) and mxaccessgw (verifier enforcement). **Later tier (roadmap 16–21) MERGED to `main` 2026-07-13** (coordinated two-PR merge; both branches now deleted): **PR #8** (`archreview-later-21-and-maxsessions`, merge `12385c8`) = Later-21 (audit off the mutating-RPC critical path + `WriteLiveValues` SQL batching + composed-failure/shutdown tests) + the **MaxSessions 4→8 raise** (live concurrent-session spike GREEN); **PR #9** (`later-16-page-through-streaming`, merge `6a24a30`) = Later-16 **ReadRaw page-through streaming** (streams per server result buffer; pool holds the lease across the enumeration; new `Historian:ReadLimits:AllowStreamingBeyondCeiling`), Later-17 **redundancy honest-hardening** (exposes the `Redundancy:WriteAcknowledgement`/`WriteFanout`/`FailureThreshold` ack policy the docs already claimed), Later-18 **per-key concurrency cap** (`ApiKeys:MaxConcurrentCallsPerKey` → gRPC `ResourceExhausted`) + api-key-rotation runbook, Later-19 CHANGELOG-discipline guard test, Later-20 **K8s manifests + hardened prod Dockerfile + operator runbooks** (`deploy/k8s/`, `docker/Dockerfile.production`, `docs/runbooks/`). The two PRs conflicted on `CLAUDE.md`/tracking/`ValidatorTests`, resolved as the union (MaxSessions lands at **8**). Live-proven vs `wonder-sql-vd03`; offline gateway 729 pass / client 140 pass / 0 warnings. **Only remaining archreview item: the secure-by-default TLS-default flip, deferred to 1.0.0** (pre-announced in the client 0.2.0 CHANGELOG). (Prior tip `93c6051` = the 2026-07-07 CommentChecker XML-doc sweep.) |

## Cross-project relationships

The three indexed projects are **separate repos and separate processes**, coupled at
**runtime over wire protocols (gRPC + OPC UA)** — not by project/compile references. They
share the `ZB.MOM.WW.*` product namespace (`ZB.MOM.WW.OtOpcUa.*`, `ZB.MOM.WW.ScadaBridge.*`;
the gateway uses `MxGateway.*`). The common subject is **AVEVA System Platform (Wonderware)
"Galaxy"** data, and `mxaccessgw` is the linchpin that the other two connect through.

### Data flow

```
                 AVEVA System Platform — Wonderware "Galaxy"
                 (OT source of truth: runtime tags + Galaxy Repository SQL DB)
                                  ▲
                                  │  MXAccess COM  (32-bit, STA message pump)
                                  │
                 ┌────────────────┴─────────────────┐
                 │   MxAccessGateway  (mxaccessgw)   │  gateway x64 .NET10 + worker x86 net48
                 │   gRPC service; OWNS the 32-bit   │  protos: mxaccess_gateway / mxaccess_worker
                 │   COM bitness + STA pump          │          / galaxy_repository
                 └──────┬─────────────────────┬──────┘
        gRPC (MxCommand/MxEvent +             │ gRPC  (ScadaBridge "MxGateway" adapter:
        GalaxyRepository browse)              │        native MxAccess data + A&C alarms)
                 │                            │
           ┌─────┴──────┐                     │
           │  OtOpcUa   │  GalaxyDriver maps  │
           │ OPC UA srv │  Galaxy hierarchy → │
           │  (.NET 10) │  OPC UA addr space  │
           └─────┬──────┘                     │
                 │  OPC UA (opc.tcp; data + A&C alarms)
                 ▼                            ▼
           ┌──────────────────────────────────────────────┐
           │  ScadaBridge — Data Connection Layer (DCL)    │
           │  OPC UA adapter │ MxGateway adapter │ custom   │
           └─────────────────────────┬────────────────────┘
                                      ▼
            Instance Actors → site clusters → central cluster / UI
```

### Edge-by-edge

- **MxAccessGateway is the foundation.** It is the *only* component that loads 32-bit
  MXAccess COM (its x86 net48 worker owns the COM apartment + STA pump). It exposes that to
  modern x64/.NET-10 callers over gRPC, and also serves Galaxy Repository SQL browse RPCs.
  This is *why* the other two exist as .NET 10 / x64 and never touch COM directly.
- **OtOpcUa → MxAccessGateway** (gRPC client). OtOpcUa's in-process `GalaxyDriver`
  (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) uses two gateway channels: the
  `GalaxyRepositoryClient` for the static hierarchy, and an MXAccess session
  (`MxCommand`/`MxEvent` protos) for live read/write/subscribe. A `DeployWatcher` polls the
  gateway's deploy-event signal to rebuild the OPC UA address space on Galaxy redeploy.
  OtOpcUa's job is a **protocol bridge**: it republishes Galaxy — now bound as a *standard
  Equipment-kind driver* alongside its native-protocol drivers, not a special SystemPlatform
  mirror — as an OPC UA address space (live values, Part 9 alarms, HistoryRead) for *any* OPC
  UA client.
- **ScadaBridge → OPC UA** (OPC UA client). ScadaBridge's DCL has an OPC UA adapter that
  collects data and mirrors native OPC UA Alarms & Conditions. OtOpcUa is exactly such a
  server, so ScadaBridge can ingest Wonderware data **indirectly via OtOpcUa**.
- **ScadaBridge → MxAccessGateway** (gRPC client). The DCL *also* has a dedicated
  **MxGateway adapter** that talks to `mxaccessgw` directly for native MxAccess data and
  alarms — so ScadaBridge can reach Wonderware data **directly, bypassing OtOpcUa**.
  Both adapters implement the same `IAlarmSubscribableConnection` seam, and a read-only
  `NativeAlarmActor` unifies OPC-UA-A&C and MxAccess alarms onto one condition model.

### Net effect

- `mxaccessgw` is a shared dependency of **both** OtOpcUa and ScadaBridge.
- ScadaBridge has **two paths** to the same Wonderware data: (1) OPC UA → OtOpcUa →
  gateway, or (2) MxGateway adapter → gateway directly. Path 1 gives standards-based OPC UA
  decoupling; path 2 gives a more direct/native feed.
- **HistorianGateway is a near-independent sidecar.** It reaches the Historian via its owned gRPC
  client (`AVEVA.Historian.Client`, imported into the repo at `histsdk/`) and the Galaxy Repository SQL DB directly, not through `mxaccessgw`. It consumes the shared
  `ZB.MOM.WW.GalaxyRepository` lib (Gitea-feed package). Any client that needs Historian data or Galaxy
  browse can target it independently.
  **As of 2026-06-27 it is a runtime dependency of OtOpcUa** (ScadaBridge still has no coupling): OtOpcUa
  adopted HistorianGateway as its **sole historian read/write backend** (continuous historization +
  alarms via the published `ZB.MOM.WW.HistorianGateway.Client`), retiring its bespoke Wonderware historian
  driver — **merged** to OtOpcUa `master` (PR #423). The gateway-side follow-ups also **merged** to
  `historiangw` `main`: the Plan-1 client lib + packable Contracts (PR #5) and the `WriteLiveValues`
  UTC→server-local timestamp fix + a documented `SendEvent` `Source_Object` protocol limitation
  (PR #6, `pending.md` C4). OtOpcUa pre-existing test failure tracked as `lmxopcua` issue #424.
- Coupling is loose: each repo references the others only as **sibling context** (the
  `## Sister Projects` note in ScadaBridge's own `CLAUDE.md` lists `MxAccessGateway` and
  `OtOpcUa` with their Gitea URLs but states they are *not part of its solution*).
- **The break surface is the wire contracts, not code.** Because coupling is by network
  protocol, the things that break across repo boundaries are: the gateway's `.proto` files
  (`mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto`), the
  `historian_gateway.v1` proto (HistorianGateway's own contract), and the OPC UA address-space
  shape OtOpcUa publishes (browse paths, node IDs, A&C alarm model). Changes to any of these
  must be coordinated across the affected repos — a green build in one repo does not prove the
  others still interoperate.

## Component normalization

Because the sister repos re-implement the same cross-cutting concerns separately and drift
apart, [`components/`](components/) normalizes them: per component, the **one target** spec,
each project's **code-verified current state**, and the **gaps** between. See
[`components/README.md`](components/README.md) for the convention and workflow.

| Component | Status | Goal | Design | Implementation |
|---|---|---|---|---|
| Auth (login / identity / authz) | Adopted (**feed carries `0.2.1`, and ALL FOUR apps pin `0.2.1` as of 2026-08-13**: OtOpcUa `master @ 08e33fc7`, ScadaBridge `main @ d28824d9`, MxAccessGateway `main @ 1cb14d2`, HistorianGateway `e42ff31` on its `dashboard-ui-sweep-theme-0.4.1` branch — each built + tested green, all PUSHED to origin 2026-08-13). **`0.2.0` (2026-08-10) adds LDAP backup-DC sticky failover**, **`0.2.1` (2026-08-12) fixes a defect that broke AD logins outright** — an AD continuation referral escaped `NovellLdapConnection.Search`, so any `SearchBase` at a domain root failed every login as `ServiceAccountBindFailed` against a perfectly healthy directory. Live-proven against `DC=nam,DC=zimmer,DC=com`; the app-level exposure is closed by the pin bumps. **`FallbackServers` is live-gated** via the local redundant GLAuth pair ([`infra/glauth-redundant/`](infra/glauth-redundant/), primary :3893 + backup :3894 on the dev Mac): HistorianGateway dashboard drill PASS 2026-08-13 (failover, stickiness, walk-back). | Shared `ZB.MOM.WW.Auth` lib | [`components/auth/`](components/auth/) | [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/) |
| UI Theme (layout / tokens / components) | Adopted (feed `0.2.0`–`0.4.1`). **`0.4.0` (2026-08-11) upstreamed the Bootstrap `.btn`/`.btn-sm` sizing block** that all **five** consumers had independently written into their own stylesheets (Bootstrap hard-codes `--bs-btn-font-size: 1rem`, oversized against the kit's 0.9rem body) — sized by `--bs-btn-*` variable override **only**, never by redeclaring box properties, because `.btn-group`'s seam/radius machinery reads those same variables and a literal `font-size` at equal specificity silently flattens `.btn-sm` into a padding-only difference (a real defect found in mxgw's local copy). Same release added the `rail-btn-block` full-width rail-foot modifier; **`0.4.1` fixes it** — 0.4.0 shipped `width: auto`, which fills for an `<a>` but shrink-wraps a `<button>` (measured 114px short in the same rail), now `calc(100% - 1.2rem)` to clear `.rail-btn`'s side margins. **Consumers are five, not four** — the ignitionoee router joined. **All five consumers are on `0.4.1` and all five are PUSHED** (2026-08-11): ignitionoee `main @ 38aa5c1`, OtOpcUa `master @ e82881c1`, mxgw `main @ 0a9715d`. The two whose sweeps were not deploy-ready pushed to **branches**, deliberately leaving their default branch untouched: HistorianGateway `dashboard-ui-sweep-theme-0.4.1 @ df7b879` (`main` stays `e9c31da`) and ScadaBridge `ui/central-ui-cleanup-sweep @ 9e243493` (`main` was then `b6f383a2`). **ScadaBridge's caveat is now CLOSED and its sweep is merged, deployed and live** (2026-08-11): the branch landed on `main`, and the two follow-ups `d14e0ee4` + `3eb7df74` fixed exactly the four detail-modal surfaces the caveat named — the modal is now gated on **user intent (the held id)**, never on the row still resolving, so a background refresh that drops the row leaves it mounted and says why the summary is missing instead of unmounting the subtree and disposing Close's own handler mid-click. That disposal *was* the intermittent `GetRequiredEventBindingEntry`, and `NotificationReportDetailModalTests.Modal_StaysOpen_WhenItsRowLeavesThePage` now pins it. **Deployed to wonder-app-vd03: OtOpcUa `e82881c1`, and ScadaBridge `3eb7df74` (2026-08-11 — theme.css/layout.css/nav-state.js served from `_content/ZB.MOM.WW.Theme` and referenced by the pages; all six changed routes 200). HistorianGateway is still `e9c31da` = 0.3.1 — its 0.4.1 remains on the branch.** **On bump, TRIM the local `.btn` block, don't delete it** — four of the five were pure sizing, but mxgw's also carried `border-radius`/`font-weight`/`white-space` that a blind delete would have dropped. | Shared `ZB.MOM.WW.Theme` RCL | [`components/ui-theme/`](components/ui-theme/) | [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/) |
| Health (readiness / liveness / active-node) | **Adopted** (feed carries `0.1.0`–`0.3.0`, current lib version **`0.3.0`**; **all 4 apps** reference it + wire `MapZbHealth`). **`0.2.0` (2026-07-24)** added an optional per-entry `data` object to the canonical health JSON and made `AkkaClusterHealthCheck` publish its cluster view (`leader`/`selfAddress`/`selfRoles`/`memberCount`/`unreachableCount`) — additive, so pre-0.2.0 payloads still parse. **`0.2.1` (2026-07-24) is a behaviour fix with a live consumer impact:** the active tier now returns **Unhealthy (503)** on a role-member-but-not-leader node instead of `Degraded`, which the spec maps to 200. The old value made `/health/active` answer 200 on *every* node, so it could not distinguish active from standby over HTTP at all — and OtOpcUa's Traefik routes the admin UI by exactly this probe, so its leader-pinning had silently never worked ([`lmxopcua#494`](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/494)). **`0.3.0` (2026-07-24) finished that fix and ENDED a family-wide duplication.** 0.2.1 made the tier a real 503 but still selected by `RoleLeader`; on a rebuilt rig the two orderings diverge live (akka leader `central-1`, oldest admin `central-2` — the singleton host), so it would have pinned Traefik to the node *not* hosting the work. **Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered.** They agree on a freshly-formed cluster — which is why single-node and happy-path tests never caught it — and diverge permanently after any restart. Both Akka apps had already discovered this independently and each written a private replacement rather than use the shared check (ScadaBridge `OldestNodeActiveHealthCheck`, OtOpcUa `ClusterPrimaryHealthCheck`), so the package's entire active-node surface had **zero consumers — not unused but avoided, twice, for the same reason.** 0.3.0 promotes those two into one `ClusterActiveNode` primitive (oldest `Up` member, optional role scope, plus an ordered role-preference so a fused node answers for the role its singletons are pinned to); `ActiveNodeHealthCheck` and `AkkaActiveNodeGate` both delegate to it, and results now carry `activeRole`/`selfAddress`/`activeNode` in the 0.2.0 `data` object so a **standby reports who IS active**. Two further behaviour changes: a node holding **none** of the preferred roles is now **Unhealthy**, not Healthy — the old "not applicable ⇒ Healthy" is exactly what made the tier answer 200 everywhere, and the case is reachable (OtOpcUa's `RoleParser` admits `dev`-only and `cluster-`only nodes); and identity compares `UniqueAddress`, so a node restarted on the same host:port is correctly a different member. **Both apps migrated and their private copies deleted** — OtOpcUa (`feat/health-0.3.0-active-node` @ `88adec04`) also routes `RedundancyStateActor.SelectOldestUpMemberOfRole` through the shared primitive, so the tier and the OPC UA `ServiceLevel` 250/240 split cannot drift; ScadaBridge (`feat/site-node-health` @ `5d4853a1`) routes `ActiveNodeEvaluator` through it, unifying the S&F delivery gate, the heartbeat `IsActive` stamp, the inbound-API gate and the health tier on one rule. ScadaBridge keeps `SitePairActiveNodeHealthCheck` deliberately — it is a thin adapter over the site's own role-scoped `IClusterNodeProvider` (itself now backed by the shared rule), not a second copy of it. mxgw/HistorianGateway have no Akka cluster and are unaffected. **Both app branches are MERGED and PUSHED** (verified 2026-08-07): OtOpcUa `88adec04` is on `origin/master`; ScadaBridge merged via `b3dc17a5` (2026-07-24) on `origin/main`, private-copy deletions included. | Shared `ZB.MOM.WW.Health` lib | [`components/health/`](components/health/) | [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/) |
| Observability (metrics / traces / logs) | **Adopted** (libs `0.1.0`, on the feed; **all 4 apps** call `AddZbTelemetry`; `AddZbSerilog` in all but ScadaBridge, which keeps its own factory by design) | Shared `ZB.MOM.WW.Telemetry` lib + `.Serilog` | [`components/observability/`](components/observability/) | [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/) |
| Config + validation (options / startup validation) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed — the "local only / not yet pushed" caveat was stale) | Shared `ZB.MOM.WW.Configuration` lib | [`components/configuration/`](components/configuration/) | [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/) |
| Audit (event model + writer seam) | Adopted (lib `0.1.0`, on the feed; **all 4 apps**, pushed to origin) | Shared `ZB.MOM.WW.Audit` lib | [`components/audit/`](components/audit/) | [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/) |
| Galaxy Repository (object-hierarchy SQL browse + gRPC service) | **Adopted** (feed `0.1.0` + `0.2.0`; consumed at `0.2.0` by HistorianGateway **and mxaccessgw**, whose Server wires `AddZbGalaxyRepository` — verified 2026-07-18; the prior "mxaccessgw adoption is a follow-on" claim was stale) | Shared `ZB.MOM.WW.GalaxyRepository` lib | _(design in histsdk + design doc 2026-06-23)_ | [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/) |
| Secrets (encrypted store + `${secret:}` resolution) | Built (current lib version **`0.5.1`**, **published to the Gitea feed** — the `0.1.x`→`0.5.1` progression is narrated below; **ScadaBridge pins `0.5.1`** (it consumes the new Grpc replicator), the other three apps still pin the `0.2.3` libraries, and the `0.3.0` interactive-console delta ships in the CLI tool; **HistorianGateway adopted + live-proven** 2026-07-16; **mxaccessgw adopted G-4/G-5/G-6 + merged to `origin/main` @ `e088dfa`** 2026-07-16, box-verified fail-closed + encrypt-at-rest; **ScadaBridge adopted G-3/G-4/G-5/G-6 + merged to `origin/main` @ `128f1596`** 2026-07-16, **G-3 live-proven vs the real production MxGateway gateway** `wonder-app-vd03:5120` (secret-ref ApiKey → Connected + browsed real Galaxy); **OtOpcUa adopted G-2/G-4/G-5/G-6 + merged to `origin/master` @ `872cf7e3`** (lmxopcua) 2026-07-16 — the last app, so **all four now adopted**; Layer-B driver secrets (Galaxy API key + OpcUaClient `Password`/`UserCertificatePassword`) resolve fail-closed at driver session-open, `AddZbSecrets` registered unconditionally so driver-only nodes work; **G-2 live-proven vs the real production MxGateway** `wonder-app-vd03:5120` — a `secret:`-ref Galaxy ApiKey resolved through OtOpcUa's real `GalaxyDriverBrowser` → dummy key `MxGatewayAuthenticationException`, real key → browsed the real Galaxy root (10 nodes); temp key minted+revoked+deleted); **G-8 KEK-rotation BUILT (lib bumped `0.1.2`→`0.1.3`, 2026-07-17):** `Rewrap` DEK primitive + CAS-guarded `ApplyRewrapAsync` + `KekRotationService.RewrapAllAsync` + `secret rewrap-all` CLI + operator runbook (`ZB.MOM.WW.Secrets/docs/operations/kek-rotation.md`), full suite green + CLI smoke + adversarial crypto review PASS (TOCTOU closed via compare-and-swap). **G-7 clustered replication BUILT 2026-07-18 (lib `0.2.0`) — BOTH fork options, packaged as two new libs:** **`ZB.MOM.WW.Secrets.Replicator.SqlServer`** (shared SQL-Server `ISecretStore` = the plan's Option A, *plus* a local-store-with-hub mode) and **`ZB.MOM.WW.Secrets.Replicator.AkkaDotNet`** (peer-to-peer over distributed pub/sub = Option B, which the plan had deferred). Core gained `ISecretsStoreMigrator`, `SecretLastWriterWins` (one shared LWW tie-break so stores cannot diverge), `SecretReplicationReconciler`, and `ReplicatingSecretStore` — the last of which fixes a real gap: **nothing had ever called `ISecretReplicator.PublishAsync`**, so the seam was inert. Verified 170 offline / 184 with the live SQL suite / 1 skip / 0 warnings, including **15 live tests vs a real SQL Server 2022** (the SQLite suite ported case-for-case) and a **9-test in-process 2-node Akka cluster** (real remoting; write→peer, delete propagation, both anti-entropy directions, late-joiner catch-up). Hard constraints across all topologies: same KEK on every node, ciphertext-only across trust boundaries, and re-wraps deliberately do NOT replicate (`rewrap-all` runs once per independent store). Operator runbook `ZB.MOM.WW.Secrets/docs/operations/clustered-secrets.md`. **Corrected 2026-07-18:** G-8 + the G-7 plan **ARE committed and pushed** on `main` (`d82d345`) — the "not yet committed/pushed" note was stale. **`0.1.3` PUBLISHED to the feed 2026-07-18** (restore-verified), carrying a **transitive security pin** (`SQLitePCLRaw.lib.e_sqlite3` → `2.1.12`, advisory GHSA-2m69-gcr7-jv3q) that `0.1.2` lacked. **All 4 apps bumped to `0.2.1`** 2026-07-18 (local `feat/secrets-0.2.0` branches, NOT pushed). `0.2.1` fixes a critical defect: `0.2.0`'s Akka replicator was **silently inert** (TryAdd registration order — it published into a no-op sink and never spawned an actor). SQL-Server replication was never affected. Clustered topology is **wired but default-OFF and NOT live-validated** — OtOpcUa on Akka peer-to-peer, ScadaBridge on SQL-hub mode (its central and site clusters are separate Akka clusters, so pub/sub cannot cross them). Separately, a **live** SQLitePCLRaw vulnerability was found and fixed in ScadaBridge, where a NuGetAuditSuppress had been masking vulnerable 2.1.11 in 15 projects. **`0.2.0` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer; merged to `main` @ `15ef1f3`) — no app has adopted a clustered topology yet. **`0.2.2` (2026-07-18, [scadaproj#1](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/1) FIXED): the Akka replicator's hosted-process DI deadlock root-caused** — a circular singleton dependency hidden in factory lambdas (`ISecretStore` decorator → `ISecretReplicator` → actor provider → `ISecretCacheInvalidator` → `DefaultSecretResolver` → `ISecretStore`), which MS.DI's `StackGuard` turns into a silent cross-thread call-site-lock deadlock instead of a stack overflow; NOT `DistributedPubSub`/HOCON (both disproven — the actor was never constructed). Fixed by deferring the one cycle edge (`DeferredSecretCacheInvalidator`, resolves on first eviction); **Akka live gate now 6/6** (was: hang), regression coverage `HostedProcessResolutionTests` (container-built graph + hosted services; red on 0.2.1), suite 180/0/15. Fourth defect class only visible with the DI graph built in a real host. **`0.2.2` PUBLISHED to the feed 2026-07-18** (5 packages, restore-verified from a scratch consumer). **OtOpcUa bumped to `0.2.2` same day (`lmxopcua master @ 2254ae3d`): Akka gate re-run vs the FEED packages 6/6**, plus a new provider-based `The_startup_hook_actually_creates_the_replication_actor` test of OtOpcUa's own wiring (8/8); **ENABLED + live-proven on the OtOpcUa docker-dev rig same day (`lmxopcua master @ c878fbbd`, #482 CLOSED):** all six rig nodes on `Secrets:Replication:Enabled=true` with one shared dev KEK; seeded row on site-a-1 alone converged to all five peers ≤60 s, byte-identical ciphertext on all six + decrypt-verified; site-node restart clean. Flag remains default-false in the product; **production topologies DECIDED 2026-08-07 ([scadaproj#3](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/3), design `docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`):** OtOpcUa = Akka pair-local (post-Phase-6 meshes; seed each pair) with **one fleet-wide production KEK via appsettings/env, never committed**; ScadaBridge = a **new `ZB.MOM.WW.Secrets.Replicator.Grpc` central-hosted hub (pull-only — updates originate at central only, by design)**, because the SqlServer hub's site-side connection string violates ScadaBridge's sites-talk-to-central-not-the-DB rule — SQL-hub mode stays shelved. **That program is now COMPLETE and scadaproj#3 is CLOSED (2026-08-07).** The `Grpc` replicator is **BUILT and PUBLISHED** — family `0.4.0` (`scadaproj main @ 321ab22`) then `0.4.1` (`c86cead`, adding a rate-limited denial Warning the live gate asked for), 6 packages on the feed, restore-verified; pull-only is enforced **at the contract** (`secrets_hub.v1` has exactly two RPCs, `GetManifest`/`GetSecrets`, and no write RPC exists to call — pinned by a descriptor-reflection test plus a ScadaBridge route pin), auth is fail-closed constant-time bearer with uniform denial and a `GetSecrets` batch cap, and the site-side sweep is never-throw with per-row parse isolation so offline = last-known-good. **ScadaBridge is wired both ends and live-gated PASS 4/4** (`ScadaBridge main @ f6c3f7c5`, merged + pushed; gate `ScadaBridge/docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md`): the node role is a **method parameter** (`SecretsNodeRole.Central|Site`) so config cannot flip a site into hosting the hub, mode key `Secrets:Replication:Mode` ∈ {`SqlServer` (default, legacy-equivalent), `Grpc`}, hub mapped on central's existing h2c control-plane listener (`CentralGrpcPort`, default 8083, no new port), fail-closed startup on a missing bearer token or endpoint, 31 DI/endpoint pin tests; the gate showed 5 s convergence byte-identical and decrypt-verified on both site-a nodes, cold boot with central down serving last-known-good then resuming, tombstones ≤9 s with no resurrection, and uniform auth denial with zero credential material in any log. **OtOpcUa's pair-local topology is live-gated PASS 4/4** (`lmxopcua master @ 1bd79c8e`, gate `OtOpcUa/docs/plans/2026-08-07-secrets-pair-local-live-gate.md`): pair convergence ≤7 s, intra-pair delete <1 s, clean pair restart, and **no cross-pair propagation proven with seeded isolation controls on the other pairs** rather than by mere absence — superseding the 2026-07-18 six-node gate, which predates the mesh Phase 6 partition. **Both products remain default-off; production enablement is a separate operator action.** **[scadaproj#4](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/4) is CLOSED 2026-08-07** (plan `docs/plans/2026-08-07-secrets-central-shared-sql-store.md`) — ScadaBridge's central pair no longer diverges because it structurally cannot: in Grpc mode both central nodes' `ISecretStore` is **ONE shared SQL-Server store** (`AddZbSecretsSqlServerStore`; central now REQUIRES `Secrets:SqlServer:ConnectionString`, fail-closed — blank refused naming the key, `${secret:}` refused as bootstrap circularity — and the Layer-A pre-host expander follows the store swap), so both hubs serve identical manifests by construction and the live-observed empty-manifest-from-`central-b` defect is unreproducible. The site client gained ordered **`FallbackEndpoints`** with sticky preference (family `0.5.0`, `main @ 7a2980e`→`e62b4a5`): list the second central only — safe ONLY because both centrals serve one shared store (failover between independent stores is exactly the silent convergence stop #4 recorded). **Live-gated PASS 5/5 + re-drill** (`ScadaBridge/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md`): central-pair parity byte-identical (sha256-matched hub responses, 1 SQL row); failover both directions (18 s convergence with central-a down, recovered-primary wrap in one sweep tick, sticky/no flapping); the #4 delete-while-follower-offline residual **GATED** (tombstone on the returning node's first sweep, no resurrection); expander proven reading the SHARED store by three-way value discrimination against a stale-SQLite decoy; fail-closed negatives (boot refusal naming the key from the expander block; SQL down → hub **errors, never an empty manifest**; sites keep last-known-good). The gate found one library defect, fixed as **`0.5.1`** (`main @ 31ca940`): the boot migrator lost the virgin-DB concurrent bring-up race (error 2714 crashed a central when the pair first-deployed simultaneously against an empty DB — the retry filter only caught 1205); now retries 1205/2714/1913/2627, re-drilled clean (both centrals booting in the same second against a virgin DB), so **0.5.1 is the minimum version for central pairs**. **Family is now `0.6.2`** (2026-08-11, `main @ c5e894d`; 0.6.0 `944595a`, 0.6.1 `cbfe296` — all three published and restore-verified). **`0.6.2` is message-only**: the under-content-root rejection now teaches the container case and states that no override exists, so nobody hunts for a flag that was deliberately not added. **0.6.1 consumers need not re-pin for it**; ScadaBridge, which has not landed yet, targets 0.6.2 directly. — **fail-closed validation of the store path, and the default moves.** `Secrets:SqlitePath` is now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator` using two rules in the new public `SecretsStorePathRules`, and **neither implies the other**: it must be **absolute** (a relative value resolves against the process working directory — `C:\Windows\System32` for a Windows service; the source tree in a dev loop, where mxgw was observed writing its store into `src/` and tripping that repo's tree-hygiene test) **and not inside the content root** (an *absolute* path can still point into the directory every deployment here renames on upgrade — the 2026-08-09 mxgw incident, where `…\MxGateway\Server\gateway-auth.db` left with `Server\` and an empty store was created at the same path, so **no gRPC consumer could authenticate for two days, silently**). The rules **reject rather than relocate**, and are deliberately **not** exempted in Development — a guard that never runs in a dev loop or CI rots, and the config behind the outage looked entirely reasonable. **BREAKING for configuration, not API:** `SecretsOptions.SqlitePath` now defaults to `SecretsOptions.DefaultSqlitePath` (absolute, `ZB.MOM.WW/secrets.db` under **`LocalApplicationData`**), so **an app configuring a relative path will not boot** — remove the key or set an absolute one. Per-user, not `CommonApplicationData`, on purpose: that is `/usr/share` on macOS/Linux, which a normal user cannot write, so a machine-wide library default would fail every app's dev boot on the family's primary dev machine. **The migration edge was closed by evidence before the default moved** — a host relying on the OLD default would look somewhere new, find nothing, and create an empty store, i.e. this bug re-introduced by its own fix; all five deployed instances on wonder were checked and every one already sets an absolute path outside its content root (mxgw + OtOpcUa pinned in **service env**, which survives a missed config-preservation step; ScadaBridge central/site and HistorianGateway in config overlays, which do not). Also: `AddZbSecrets` validator registration is now idempotent — `TryAddEnumerable` de-duplicates on implementation *type*, which a factory descriptor does not carry, so it **threw** on the second call rather than skipping (mxgw calls `AddZbSecrets` twice; the Akka replicator calls it internally). 500 tests pass. **Rollout status:** pre-bump pins were **ScadaBridge `0.5.1`, the other three `0.2.3`** — so three of the four are four-minor jumps, not re-pins. **mxgw MERGED + PUSHED 0.6.0** (`mxaccessgw origin/main @ 9bc70d1`, three commits: the session health check, the config path rule, the re-pin; build 0/0, suite 890/890) — **but 0.6.0 carries the hole and mxgw has the affected shape, so it needs a 0.6.1 follow-up.** HistorianGateway has adopted **0.6.1** and re-verified on the exact probe that found the defect (in-content-root path → aborts with **no `.db`/`-wal`/`-shm` left behind**); its branch is pending its user. **OtOpcUa has adopted 0.6.2** (`b037cc7a`, superseding `835e35a8`; unpushed) with the heavy part discharged by **live evidence rather than changelog trust**: the full six-node rig booted on 0.6.1 under the new validation, a convergence smoke showed a secret written via AdminUI reaching both central nodes' independent stores, and an isolation control confirmed site-a holds only its own pair's secrets — the mesh Phase 6 posture intact. ScadaBridge is the last outstanding. OtOpcUa's jump also moves `Replicator.AkkaDotNet`, whose pair-local topology was live-gated on `0.2.3` — a re-gate may be the real cost there. **`0.6.1` closes a hole in the 0.6.0 guard, found by the HistorianGateway adoption — target `0.6.1`, never `0.6.0`.** The validator resolves the content root from `IHostEnvironment` and skipped the under-content-root rule when it was absent — but an app that expands `${secret:}` references **before building its host** composes the subsystem in a throwaway `ServiceCollection` with no `IHostEnvironment` in it and then **migrates the store from that container**. So pre-host an absolute in-content-root path passed the only rule still applying (rooted), the migrator **created the database**, and the boot failed a second later once the real host registered `IHostEnvironment`. Net effect: a rejected boot that still left an empty store with `-wal`/`-shm` **at the rejected path** — the exact artifact that made the MxGateway outage read as *the database is there, it's just empty*. A relative path was unaffected (the rooted rule fires either way), and that asymmetry is what exposed it. The mistake was conflating *no `IHostEnvironment` registered* with *no content root exists*. Fix is an additive overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`**; the 3-arg overload is unchanged. **THE PIN ALONE DOES NOT CLOSE IT — the 4-arg call site is the fix.** The exposure is **pre-host-expander-shaped, not app-shaped**: an app with a correctly configured path shows *no symptom* and is still unprotected, because the guard is not running at the moment the store is created — so a green re-pin reads as done while the gap is untouched. **mxgw and HistorianGateway are CONFIRMED affected** (mxgw `GatewayApplication.cs:82`, HG `Program.cs:90` — both `new ServiceCollection().AddZbSecrets(...)` followed by `SqliteSecretsStoreMigrator.MigrateAsync`); **OtOpcUa is CONFIRMED affected too** (`Program.cs`, byte-for-byte the same shape) and has adopted 0.6.1; ScadaBridge is expected to be. **Containers: the rule deliberately over-rejects.** A store on a volume mounted INSIDE the content root (`/app/data/…` with `/app` the content root) is durable across container replacement but still rejected — in a container the image is replaced on upgrade while the volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there. Not relaxed and **no opt-out flag** — the OtOpcUa session, which owns the only real containerized deployment here and hit the false positive, argued for keeping it strict on three grounds worth reusing: the workaround is an *improvement* rather than a concession (it separates the application from its durable state, the exact distinction the incident blurred); an override flag has the lifecycle of a knob set once to clear a symptom and never revisited, surviving the redeploy it was reasoned against; and mount detection would make the verdict depend on runtime platform state, so **the same path would pass on one host and fail on another** — strictly worse than a stable false positive with a cheap documented fix. Workaround is a second mount of the same volume outside the content root (the OtOpcUa rig used `/data`); the file sits at the volume root either way, so seeded stores carry over. **`NU1102` after a bump is a cached-index artifact, not the feed** — and it **recurs on every publish**, consistently stranding `Replicator.AkkaDotNet` at the previous version while its siblings resolve (each package carries its own cached registration index), which reads even more like a feed problem than the all-packages case. Seen three times; two independent scratch restores did not reproduce it. `dotnet nuget locals http-cache --clear`. Treat it as expected post-publish behaviour rather than diagnosing the feed. Both affected repos put a **do-not-simplify-back-to-the-3-arg-overload** comment on the call site, because it still compiles and silently restores the gap — the regression arrives disguised as removing an unnecessary argument. **Verify with positive controls, not a clean boot** — a clean boot is equally consistent with the validator never being wired, and probing the rejected paths is what found this. Upgrade note applying to both versions: a store seeded under the **old relative default is orphaned, not migrated, and nothing signals it** — deleting the key, rather than leaving a relative value, is what makes the new default apply. `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` `0.1.0` for the shared `OptionsValidatorBase`/`ValidationBuilder` vocabulary. **`Secrets.Ui` is byte-identical between 0.2.3 and 0.6.0** (zero commits since 2026-07-19; both assemblies 39,424 B differing only in version stamps), so the scadaproj#2 delete-modal browser gate still covers it — the version moved only because this family shares one version across every package. Prior: ScadaBridge pinned `0.5.1` (`main` `68f812ea`→`4d7f09d5`→`e697477c`, pushed; 472 offline tests). Rewrap fan-out on ScadaBridge collapses to **ONE run against central's shared SQL store** (the CLI auto-targets it via central's appsettings) + both nodes of every site pair — the "both central nodes" caveat is deleted from the runbook. The denial-warning alerting note stands (the Warning is rate-limited; alerting must sum per-window deltas). The gate's residual, **[ScadaBridge#34](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/34)**, is **FIXED+CLOSED 2026-08-08 (`ScadaBridge main @ e9c412e5`)**: the pre-existing container wedge on any unhandled boot exception (crash banner then ~100% CPU main-thread spin, container stays `running`, `restart: unless-stopped` never fires) was root-caused to **dotnet running as container PID 1** — Linux ignores the SIGABRT the runtime's crash path raises against PID 1, so `abort()` could never kill the process; not specific to any exception type or boot stage. Fixed two-layer: an `AppDomain.UnhandledException` handler registered before anything can throw exits `134` via the `exit()` syscall (covers every thread, inert under WebApplicationFactory so designed boot-refusal exceptions still propagate to tests), plus `init: true` on all 8 compose nodes for crash paths that bypass the managed event (`Environment.FailFast`, runtime-internal aborts). Live-verified: the deterministic wedge repro now exits 134 with `RestartCount` climbing under `unless-stopped`, trace intact, proven with and without init; rig redeployed healthy with `docker-init` as PID 1. The CoordinatedShutdown no-`Environment.Exit` source guard carries a precise carve-out (exactly one call, only inside the handler — which still fires Akka's `run-by-clr-shutdown-hook`). **Operator note: production overlays running this image under their own compose should also add `init: true`.** Side-find `lmxopcua#483` **FIXED+CLOSED 2026-07-19** (`lmxopcua @ 5f72ff85`): OtOpcUa's static per-page-rendermode router made the routed Secrets.Ui RCL page a dead SSR page; fixed with a host-side wrapper page carrying `@rendermode InteractiveServer` (RCL unchanged — the other three apps' globally-interactive routers would throw on a nested render mode), pin tests + Playwright-verified live. **Family-wide /admin/secrets sweep 2026-07-19:** ScadaBridge's page had a DIFFERENT defect — Secrets.Ui injects the shared `ZB.MOM.WW.Audit.IAuditWriter` seam, which ScadaBridge never registered (it has its own deliberately-distinct Commons `IAuditWriter`) → hard 500 on every render, hidden behind the login wall. **FIXED+CLOSED as [ScadaBridge#22](https://gitea.dohertylan.com/dohertj2/ScadaBridge/issues/22)** (`ScadaBridge main @ fc86e8bf`): `CentralSharedSeamAuditWriter` bridges the shared seam to `ICentralAuditWriter` (central direct-write → `dbo.AuditLog`, NOT the site SQLite chain), pinned by red-first WebApplicationFactory DI tests over the real Program.cs, live-proven on the local docker cluster (page interactive; `secret.add`/`secret.delete` audit rows landed in `dbo.AuditLog`). The sweep also surfaced a **Secrets.Ui library defect, [scadaproj#2](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/2) — FIXED+CLOSED same day via `0.2.3` (5 packages PUBLISHED to the feed; `scadaproj main @ 62ab8f6`)**: `ConfirmDeleteModal` used bare `class="modal"` and the RCL shipped no CSS, so Bootstrap 5's `.modal{display:none}` made the delete modal permanently invisible on every Bootstrap host (confirmed live on ScadaBridge AND OtOpcUa; HG links the same bundle; **all four apps are Bootstrap hosts** — the earlier claim that mxgw was Bootstrap-free was false, corrected 2026-08-11: `src/ZB.MOM.WW.MxGateway.Server/libman.json` pins `bootstrap@5.3.3` into `wwwroot/lib/bootstrap/` and `App.razor:7` links it ahead of `<ThemeHead />`. What mxgw's CLAUDE.md forbids is Blazor *component libraries* (MudBlazor/Radzen/FluentUI), not Bootstrap CSS/JS) — handlers worked, only CSS hid it. Fix entirely in the RCL: `zb-secrets-modal*` class rename + a **component-emitted `<style>` block** (Theme tokens w/ fallbacks; testids unchanged; bUnit pins). Scoped `.razor.css` was deliberately rejected — only ScadaBridge linked a scoped-CSS bundle **at the time (2026-07-19)**, so isolation CSS would silently not load on the other 3 hosts — still the right call, and note the premise has since changed: OtOpcUa's 2026-08-10 AdminUI sweep linked its own dead bundle (`App.razor:23`), so as of 2026-08-11 two of the four link one, while mxgw and HistorianGateway have zero `*.razor.css` and emit no bundle at all (the same silent-failure class). Live-verified on ScadaBridge (bumped to `0.2.3` @ `63510778`): modal visible `display=flex` with zero JS forcing, delete flows through the real path. **Rollout COMPLETE 2026-07-19 — all four apps on `0.2.3`, pushed:** OtOpcUa (`lmxopcua master @ be87ddeb`; AdminUI 665/665 + secrets integration 8/8; docker-dev rig rebuilt, modal gate PASS live), HistorianGateway (`main @ 7452eb4`, from `0.2.0`; plus `6bc005d` **fixing the pre-existing AngleSharp NU1902 warn-as-error that had made the whole HG test suite unbuildable** — test-only pin to 1.5.2, suite now runs: 854 pass / 1 pre-existing VPN-gated live failure, histsdk failures identical to the pre-bump baseline; docker container rebuilt, modal gate PASS live), and mxgw (`main @ 8f7ee49`, from `0.2.0`; 780 pass / 45 macOS-only NamedPipe-harness failures identical before and after the bump — zero delta). **mxgw's modal gate was originally SKIPPED on the false "Bootstrap-free so never affected" reasoning, and is now PASS (2026-08-11)** — run live against a local rig: modal root class is `zb-secrets-modal`, `document.querySelector('.modal')` is null so Bootstrap's `.modal{display:none}` has nothing to bite, computed `display:flex` / `position:fixed` / z-index 1081 over a full-viewport 1080 backdrop, and `elementFromPoint` at the confirm button's centre returns the button rather than the backdrop (visible ≠ hit-testable — that probe is the one that proves usable). Cancel disarmed cleanly with no delete performed. The exemption reached the right conclusion by luck: mxgw was a Bootstrap host and *was* affected, but had already taken the `0.2.3` fix independently of the sweep. **Lesson: an exemption from a family-wide gate must re-derive the exempting property from the repo at gate time** — see [`admin_ui_cleanup.md`](admin_ui_cleanup.md) **`0.3.0` (2026-07-19, merged to `main` @ `87ffe1b`) added a menu-driven Spectre.Console interactive operator console** to the `secret` CLI: bare `secret` with no verb on a real TTY (redirected I/O or any verb → unchanged headless behavior, no-args-redirected → usage/exit-2) opens a guided session that targets a deployment's store by reading the **target app's own appsettings.json** (base + env overlay + env vars → `Secrets` section, so it hits exactly the store the app reads; recents cached at `~/.zb-secrets/recent-targets.json`, names/paths only). Eight flows: list/set/reveal/delete plus the two headline ones — **Reference audit & seed** (classifies every `${secret:NAME}` ref Ok/PresentUnverified/Missing/Tombstoned/Undecryptable/InvalidName, then seeds the gaps — deployment setup) and **KEK doctor** (per-row Ok/WrongKek/Corrupt verdicts + guided rewrap-all or lost-KEK re-set — lockout recovery), plus ciphertext-only bundle export/import. Degraded (no-KEK) sessions open metadata-only and upgrade in place via masked paste/key-file; no login (local trust); key material never a literal arg. **5 packages published at `0.3.0` to the Gitea feed** (restore-verified); the four apps still pin the `0.2.3` libraries (the console ships in the CLI tool, not the app packages). Runbook `ZB.MOM.WW.Secrets/docs/operations/interactive-console.md` + quickstart `ZB.MOM.WW.Secrets/docs/interactive-console-quickstart.md`. | Shared `ZB.MOM.WW.Secrets` lib (5 packages + CLI) | [`components/secrets/`](components/secrets/) | [`ZB.MOM.WW.Secrets/`](ZB.MOM.WW.Secrets/) |
| LocalDb (embedded cache + 2-node sync) | **Adopted by ScadaBridge — Phases 1 AND 2 COMPLETE and MERGED** to `main` (2026-07-20, PR #23, merge `28ca04d7`, pushed to origin; both `feat/localdb-phase*` branches fully contained and kept only as history). Libs **`0.1.3`** on the Gitea feed (3 pkgs, restore-verified; `0.1.0`–`0.1.2` also present). **`0.1.2` (2026-07-21) fixes snapshot resync for a WIPED peer**: a converged pair prunes every oplog row on ack, and the handshake read an empty oplog as "no gap possible" — so the steady state of a healthy pair was the one state from which a rebuilt node (lost volume, re-imaged host) could never be back-filled; it came back empty and stayed empty until the next deploy. The gap is now measured against `last_acked_seq` when the oplog is empty. Found by the OtOpcUa Phase 1 live gate (check 4) as a documented limitation, then closed. **`0.1.3` (2026-07-21) closes the other half, which only became REACHABLE once back-fill worked**: the rebuilt peer's OWN writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq space and a rebuilt peer numbers from 1, so the healthy node's stale watermark made the sending side skip its whole oplog — the pair looked converged right up until the moment only the rebuilt node applied a deploy. Fixed by resetting the inbound watermark on a peer-node-id change (NOT last_acked_seq — that is what 0.1.2 keys off) plus rejecting a peer claim beyond anything we ever produced. Live-gated on the OtOpcUa docker-dev rig: `OtOpcUa/docs/plans/2026-07-21-localdb-followups-live-gate.md`. **`0.1.1` fixes a latent boot defect**: the library never created the parent directory of `LocalDb:Path` and `SqliteLocalDb` opens the file eagerly, so a missing directory was a hard boot failure (`SQLite Error 14`) — routine given the relative `./data/…` defaults, and masked on docker only by the volume mount. ScadaBridge had worked around it app-side; the fix moved into the library and the shim is deleted. **Phase 1** (merged in PR #23): `OperationTracking` + `site_events` consolidated into one `LocalDb:Path` database, fail-closed bearer interceptor on the sync endpoint, `ISyncStatus` health bridge, one-time legacy migrator. **Phase 2** (merged in the same PR): adds `sf_messages` + 7 site config tables (10 replicated in total) and **deletes** ScadaBridge's bespoke `SiteReplicationActor`, its messages, StoreAndForward's `ReplicationService`, and `ReplaceAllAsync` — including the whole **notify-and-fetch** exchange, so a standby now makes NO config fetch on deploy (`SiteReconciliationActor`'s startup fetch survives; different path). `notification_lists`/`smtp_configurations` are created but deliberately NEVER registered — permanently empty, and a channel for them would carry plaintext SMTP passwords. **Live gate PASS, all 10 checks** (`ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md`): byte-identical config rows with identical HLC + origin node id, standby zero-fetch, 3-table cascade delete converging with tombstones on both nodes, site failover with zero duplicate ids, clean both-nodes-together restart. **Replication remains default-OFF**, enabled on the docker rig's site-a pair only (site-b/site-c unreplicated as the default-OFF pin); **no production deployment has it enabled.** **Operational constraints:** a site pair must be stopped/started TOGETHER (rolling upgrade unsupported since the `SfBufferSnapshot` compat handler was deleted), a node offline beyond `TombstoneRetention` (7 d) can resurrect deleted rows, and `MaxBatchSize` batches by ROW COUNT against a 4 MB gRPC cap (rig pins **16**; the 500 default would allow ~35 MB). **Also adopted by OtOpcUa — Phase 1 COMPLETE and MERGED** to `master` 2026-07-21 (merge `c957db52`), and **Phase 2 (alarm store-and-forward on LocalDb) EXECUTED, live-gated and MERGED** to `master` (merge `06b5d81b`; the earlier "designed but NOT executed" claim was stale — corrected 2026-08-07; residual: live-gate checks 3+4 were never re-run on the post-mesh-Phase-6 topology): a pair-local consolidated SQLite cache of the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention) that lets a driver node **boot from cache when central SQL Server is unreachable**, registered in the `hasDriver` branch only (admin-only nodes never register it), with the same default-OFF fail-closed replication. Its live gate passed 8/8 and caught four defects offline tests missed. **Consumers: ScadaBridge (Phases 1+2) and OtOpcUa (Phases 1+2)**, both pinned to `0.1.3` — verified 2026-07-21 (OtOpcUa Phase 2 merge verified 2026-08-07). | Shared `ZB.MOM.WW.LocalDb` lib | [`docs/plans/2026-07-17-localdb-design.md`](docs/plans/2026-07-17-localdb-design.md) + [`docs/plans/2026-07-19-scadabridge-localdb-design.md`](docs/plans/2026-07-19-scadabridge-localdb-design.md) + [`docs/plans/2026-07-20-otopcua-localdb-design.md`](docs/plans/2026-07-20-otopcua-localdb-design.md) (both adoption designs live here); ScadaBridge holds the Phase 1 plan + Phase 2 gate under its own `docs/plans/`, OtOpcUa its two phase plans + two live gates | [`ZB.MOM.WW.LocalDb/`](ZB.MOM.WW.LocalDb/) |

The auth component is fully populated: a normalized [`spec`](components/auth/spec/SPEC.md), a
proposed [`shared-contract`](components/auth/shared-contract/ZB.MOM.WW.Auth.md), three
[`current-state`](components/auth/current-state/) docs, and an adoption [`GAPS`](components/auth/GAPS.md)
backlog. Common ground = LDAP/GLAuth identity + peppered-HMAC API keys; left per-project =
the authz vocabularies (OPC-UA permissions / gRPC scopes / roles + site-scoping).

The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Auth/`](ZB.MOM.WW.Auth/)
(a plain directory of this repo — no nested `.git`, verified 2026-08-07; .NET 10; 4 packages — `Abstractions`, `Ldap`, `ApiKeys`, `AspNetCore`;
250 tests passing + 2 skipped (verified 2026-08-10 at `0.2.0`; was 215+1 at `0.1.5`); feed carries `0.1.0`–`0.2.0`). The implementation plan is at
[`docs/plans/2026-06-01-zb-mom-ww-auth-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-auth-shared-library.md).
**Adopted across all three apps on 2026-06-02** (auth GAPS #1–#8) on each repo's `feat/adopt-zb-auth` branch —
committed + reviewed, then **fast-forward-merged into the repo's local default (main/master) and PUSHED to origin
(gitea) on 2026-06-03** (in sync; the `feat/*` branches kept locally as history). Cutover: shared `Auth.Ldap`,
`Auth.ApiKeys` (ScadaBridge inbound fully re-architected to the keyId/Bearer model), `IGroupRoleMapper<TRole>` seam,
`Transport`-enum config, canonical `ZbClaimTypes`/`ZbCookieDefaults`, unified dev base DN `dc=zb,dc=local`, and the
canonical-six role vocabulary (with ScadaBridge's accepted auditor/admin SoD collapse). Consumer pins: **all four apps pin `0.2.1` as of 2026-08-13** (commits listed in the table row above, all pushed to origin). Prior history — all four pinned `0.1.5` (verified 2026-08-07): MxGateway via `e107019` 2026-07-18 (earlier bumped `0.1.2`→`0.1.4` 2026-07-09 for the G-2 API-key-expiry verifier enforcement — commit `197731a`; also required implementing the 0.1.3 `IApiKeyAdminStore.SetScopes/SetEnabled` members on a test fake), ScadaBridge `50d79ed1` + HistorianGateway `df211ee` 2026-07-18, OtOpcUa `1adaa2fc` 2026-07-20 (PR #484, family alignment from `0.1.1`). Per-repo detail in [`components/auth/GAPS.md`](components/auth/GAPS.md) +
`docs/plans/2026-06-02-auth-audit-normalization*.md`.
Build/test from `ZB.MOM.WW.Auth/`: `dotnet test`. Consumer matrix: OtOpcUa → Abstractions+Ldap+AspNetCore;
MxAccessGateway & ScadaBridge → all four (ApiKeys not used by OtOpcUa).
**`0.1.4` (2026-07-09, archreview G-2):** optional `ApiKeyRecord.ExpiresUtc` (nullable = never-expires), `ApiKeyFailure.KeyExpired`, `ApiKeyVerifier` expiry enforcement (inclusive, injected clock), a `CreateKeyAsync(expiresUtc)` overload, and a SQLite **schema v3** additive `expires_utc` column (idempotent `ALTER`; donor v2 DBs upgrade in place, no key invalidated). **All four Auth pkgs published to the Gitea feed at `0.1.4`** (2026-07-09). **HistorianGateway adopted the expiry surface at `0.1.4`** — `apikey create --expires` + dashboard Expires column (Task-10 done). **`0.1.5` (2026-07-18)** is `0.1.4` plus the `SQLitePCLRaw.lib.e_sqlite3 2.1.12` transitive security pin (GHSA-2m69-gcr7-jv3q), no API change; all four apps now pin it (see consumer pins above). **`0.2.0` (2026-08-10) — LDAP backup-DC sticky failover** (plan `docs/plans/2026-08-10-ldap-backup-dc-failover.md`, design `…-design.md`; **4 pkgs published to the feed, restore-verified**; family `Directory.Build.props` now `0.2.0`): one additive `LdapOptions.FallbackServers` list (`"host"` or `"host:port"`, empty default = failover off, single-endpoint behaviour byte-unchanged) makes `LdapAuthService` walk an ordered endpoint list (primary + fallbacks) on **system-side failures only** — connect/TLS, service-account bind, or search. A user-side answer from a healthy directory (`Success`/`UserNotFound`/`AmbiguousUser`/`BadCredentials`/`GroupLookupFailed`) is **final and never retried elsewhere** (re-asking a second DC about a rejected password would multiply one bad-password event into several against the account lockout policy). Endpoint preference is **sticky** — the last endpoint that gave *any* authoritative answer (success **or** a user-side failure) is tried first next time — so a primary outage costs its connect timeout once, not per login; an exhausted walk leaves the preference untouched. Fail-closed contract preserved per-endpoint; boot validation rejects a malformed `FallbackServers` entry naming the field; optional `ILogger<LdapAuthService>` emits a Warning per failed hop (endpoint + stage, no secrets) and an Information on sticky change; wired via `AddZbLdapAuth` (`GetService`, so logging is optional). Motivation: the ZB production directory is NAM AD with two DCs. Unit suite covers the walk/stickiness (14 tests) + env-gated live GLAuth failover test (`ZB_LDAP_IT=1`). **No `Abstractions` contract change beyond the new option; no `LdapAuthFailure` enum change.** App adoption (pin bump to `0.2.0` + authoring `FallbackServers`) is a follow-up.
**`0.2.1` (2026-08-12) — AD continuation referrals no longer fail the search** (**4 pkgs published to the feed, restore-verified**; the assembly restored *from the feed* carries the fix and `0.2.0` does not — checked as a positive control, not by trusting the push). Active Directory answers **any search based at a domain root** with continuation references (to `CN=Configuration`, `DC=ForestDnsZones`, …). With referral following disabled — the default, and what we want, since chasing one would re-bind somewhere we never authenticated — Novell surfaces each by throwing `LdapReferralException` from `Next()`. That escaped `NovellLdapConnection.Search`, and `LdapAuthService` maps a failed search to `ServiceAccountBindFailed`, so **every login against `SearchBase = DC=nam,DC=zimmer,DC=com` failed system-side on both DCs while the directory, the credentials and the service account were all healthy** — ~546 ms, identical on each endpoint, so failover could not help either. Found by the new `tools/LdapProbe` (below) and live-proven fixed against the ZB NAM domain root. The search loop now skips referrals (`ReadEntries(ILdapSearchResults)`, extracted so it is unit-testable through the Novell seam without a live directory) and continues; a 1000-referral cap guards a client that throws without advancing, since that would spin forever inside a fail-closed auth path. **Non-referral faults still propagate** — swallowing those would turn "the directory is broken" into "no such user", which is the wrong answer to give about someone's credentials. 7 new tests (`NovellSearchReferralTests`), **6 verified red against the pre-fix code**, the seventh being the non-referral control that must stay green; suite 257 pass / 2 skip / 0 fail. **No `Abstractions` change; no config change.** **Consumer impact: an app on `0.1.5` cannot authenticate against an AD domain root at all**, so the pin bump is a functional fix, not housekeeping. **Closed 2026-08-13: all four apps adopted `0.2.1`** (see the table row for commits), and `FallbackServers` was live-gated the same day against the redundant local GLAuth pair (`infra/glauth-redundant/`).

**`tools/LdapProbe`** (`ldapprobe`, not packable, outside the Auth solution) is the diagnostic that found the above: it drives the **real** `LdapAuthService` so its verdict is what the four apps produce, then separately queries the directory for what the library cannot report — full group DNs, **nested** groups via `LDAP_MATCHING_RULE_IN_CHAIN`, the **primary group** (which AD omits from `memberOf`, so it can never be a role group), group type (security vs distribution), decoded `userAccountControl`, referral counts, and the TLS chain with key sizes. `--list-groups '<pat>'` enumerates every group in the directory matching a pattern, which a membership dump structurally cannot do. Findings recorded in [`ldap_setup.md`](ldap_setup.md), including the **`UG_WW_*` Wonderware group convention (39 groups)** that partly overturns that doc's earlier "no group→role convention exists" claim — it was true of Ignition, false of the domain.

The UI-theme component is fully populated: a normalized [`spec`](components/ui-theme/spec/SPEC.md),
a [`design-tokens`](components/ui-theme/spec/DESIGN-TOKENS.md) reference, a
[`shared-contract`](components/ui-theme/shared-contract/ZB.MOM.WW.Theme.md), three
[`current-state`](components/ui-theme/current-state/) docs, and an adoption [`GAPS`](components/ui-theme/GAPS.md)
backlog. Shared = Technical-Light tokens + IBM Plex fonts + side-rail shell + widgets; left
per-project = each app's `site.css` page layout, route content, scoped `.razor.css`.

The shared RCL is **built and lives in this repo** at [`ZB.MOM.WW.Theme/`](ZB.MOM.WW.Theme/)
(.NET 10 Razor Class Library; single package; 48 bUnit tests (re-verified 2026-08-11 at `0.4.1`); feed carries `0.2.0`–`0.4.1`,
**published to the Gitea feed**). The build plan is at
[`docs/plans/2026-06-01-zb-mom-ww-theme-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-theme-shared-library.md);
the adoption plan at [`docs/plans/2026-06-03-ui-theme-adoption.md`](docs/plans/2026-06-03-ui-theme-adoption.md).
**Adopted across all three apps on 2026-06-03** (full canonical cutover, SPEC §7) on each repo's
`feat/adopt-zb-theme` branch — committed + spec/code-reviewed, then **fast-forward-merged into each repo's local
default (master/main) and PUSHED to origin (gitea)** (in sync; `feat/*` kept locally as history): OtOpcUa
→`lmxopcua` `master`@`11de14d`, ScadaBridge `main`@`58352a6`, MxGateway→`mxaccessgw` `main`@`73e54e2`. The `0.1.0 → 0.2.0` bump first promoted nav-expand persistence
into the kit (`NavRailSection.Key`/`data-nav-key` + a localStorage `nav-state.js` enhancer emitted by a new
`<ThemeScripts/>`), so all three apps share one persistence mechanism (OtOpcUa's bespoke cookie/JS-interop nav
island retired); MxGateway additionally gained a net-new Blazor `<LoginCard>` `/login` page over its existing
hardened endpoint. Per-app result in [`components/ui-theme/GAPS.md`](components/ui-theme/GAPS.md).
Build/test from `ZB.MOM.WW.Theme/`: `dotnet test`. Consumer matrix: all three apps consume
the single `ZB.MOM.WW.Theme` package (OtOpcUa AdminUI, MxGateway Server, ScadaBridge Host + CentralUI).

The health component is fully populated: a normalized [`spec`](components/health/spec/SPEC.md), a
[`shared-contract`](components/health/shared-contract/ZB.MOM.WW.Health.md), three
[`current-state`](components/health/current-state/) docs, and an adoption [`GAPS`](components/health/GAPS.md)
backlog. Shared = three-tier endpoint convention (ready/active/healthz) + canonical JSON writer +
`IActiveNodeGate` seam + `GrpcDependencyHealthCheck` + `AkkaClusterHealthCheck` + `ActiveNodeHealthCheck`
+ `DatabaseHealthCheck<TContext>`; left per-project = which probes each app registers,
orchestrator wiring, and ScadaBridge's distributed health-monitoring pipeline.

The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Health/`](ZB.MOM.WW.Health/)
(.NET 10; 3 packages — `ZB.MOM.WW.Health`, `ZB.MOM.WW.Health.Akka`, `ZB.MOM.WW.Health.EntityFrameworkCore`;
64 tests (verified 2026-07-18); `dotnet pack` → 3 nupkgs @ 0.1.0).
**Adopted by all four apps** — verified 2026-07-18 (each references the package and wires `MapZbHealth`); the previous "not yet adopted" claim was stale. Per-app probe coverage vs the spec has NOT been re-verified — see [`components/health/GAPS.md`](components/health/GAPS.md).
Build/test from `ZB.MOM.WW.Health/`: `dotnet test`. Consumer matrix: MxAccessGateway → core only;
OtOpcUa & ScadaBridge → all three packages.

The observability component is fully populated: a normalized [`spec`](components/observability/spec/SPEC.md),
a [`metric-conventions`](components/observability/spec/METRIC-CONVENTIONS.md) reference, a
[`shared-contract`](components/observability/shared-contract/ZB.MOM.WW.Telemetry.md), three
[`current-state`](components/observability/current-state/) docs, and an adoption [`GAPS`](components/observability/GAPS.md)
backlog. Shared = OTel Resource (service.name/site.id/node.role identity triple) + standard instrumentation
(ASP.NET Core, HttpClient, gRPC client, runtime, process) + Prometheus always-on exporter + OTLP opt-in
+ Serilog two-stage bootstrap + SiteId/NodeRole/NodeHostname enrichers + TraceContextEnricher (trace_id/span_id)
+ ILogRedactor seam; left per-project = application Meters/ActivitySources, sink config, per-operation
enrichers, and redaction policies.

The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Telemetry/`](ZB.MOM.WW.Telemetry/)
(.NET 10; 2 packages — `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Telemetry.Serilog`; 34 tests (verified 2026-07-18);
`dotnet pack` → 2 nupkgs @ 0.1.0). **Adopted across all three apps on 2026-06-01** (branch
`feat/adopt-zb-telemetry` per repo, behaviour-preserving): `AddZbTelemetry` (Resource + standard
instrumentation + Prometheus `/metrics`) everywhere; OtOpcUa + MxGateway on `AddZbSerilog` (MxGateway's
MEL→Serilog migration + metrics export both landed in this pass — they were *not* actually done
beforehand despite an earlier claim); ScadaBridge keeps its `LoggerConfigurationFactory` (min-level
governance) and only adds the shared `TraceContextEnricher`. Deferred: MxGateway `ms`→`s` + Meter
rename, ScadaBridge app instruments + Site-node HTTP/1.1 metrics listener, OTLP wiring. Per-repo
result tracked in [`components/observability/GAPS.md`](components/observability/GAPS.md).
Build/test from `ZB.MOM.WW.Telemetry/`: `dotnet test`. Consumer matrix: all three apps consume both
packages after adoption (OtOpcUa, MxGateway Server, ScadaBridge Host + any instrumented project).

The configuration component is fully populated: a normalized [`spec`](components/configuration/spec/SPEC.md), a
[`shared-contract`](components/configuration/shared-contract/ZB.MOM.WW.Configuration.md), three
[`current-state`](components/configuration/current-state/) docs, and an adoption [`GAPS`](components/configuration/GAPS.md)
backlog. Shared = the `IValidateOptions<T>` failure-accumulation base (`OptionsValidatorBase<T>`) +
reusable rule primitives (`ValidationBuilder`: port / host:port / required / positive-duration / one-of /
min-count) + `AddValidatedOptions<TOptions,TValidator>()` (bind + validate + `ValidateOnStart`) + the
pre-host `ConfigPreflight` aggregator (generalizes ScadaBridge's `StartupValidator`, byte-compatible
message); left per-project = each app's options classes + domain rules, and OtOpcUa's
draft/generation-content validation (DB-side `sp_ValidateDraft`; its C# `DraftValidator` is dormant).

The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Configuration/`](ZB.MOM.WW.Configuration/)
(.NET 10; single package `ZB.MOM.WW.Configuration`; 42 tests (verified 2026-07-18); `dotnet pack` → 1 nupkg @ 0.1.0).
The implementation plan is at
[`docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md`](docs/plans/2026-06-01-zb-mom-ww-configuration-shared-library.md).
**Adopted across all three apps on 2026-06-01** (OtOpcUa, MxAccessGateway, ScadaBridge) on each repo's
local default branch (`main`/`master`) — merged, **not yet pushed** to remotes; the package was first
published to the Gitea feed. Behaviour-preserving onto `OptionsValidatorBase`/`AddValidatedOptions`
for MxGateway + ScadaBridge (validator messages byte-identical), `StartupValidator` → `ConfigPreflight`
for ScadaBridge, and net-new `Ldap`/`OpcUa` validators for OtOpcUa. Per-app result tracked in
[`components/configuration/GAPS.md`](components/configuration/GAPS.md).
Build/test from `ZB.MOM.WW.Configuration/`: `dotnet test`. Consumer matrix: all three apps consume the
single package; ScadaBridge is the heaviest adopter (per-module validators + `StartupValidator` →
`ConfigPreflight`); OtOpcUa adoption is additive (it has no `IValidateOptions` usage today).

The audit component is fully populated: a normalized [`spec`](components/audit/spec/SPEC.md), an
[`event-model`](components/audit/spec/EVENT-MODEL.md) reference, a
[`shared-contract`](components/audit/shared-contract/ZB.MOM.WW.Audit.md), three
[`current-state`](components/audit/current-state/) docs, and an adoption [`GAPS`](components/audit/GAPS.md)
backlog. Common ground = canonical `AuditEvent` record + `AuditOutcome` enum + `IAuditWriter` /
`IAuditRedactor` seams + helpers (`NullAuditRedactor`, `TruncatingAuditRedactor`, `NoOpAuditWriter`,
`CompositeAuditWriter`, `RedactingAuditWriter`) + `AddZbAudit` DI registration; left per-project =
transport/storage and domain vocabulary. Closes the loop on Auth — audit's `Actor` field = the Auth
principal. `IAuditRedactor` is aligned with Telemetry's `ILogRedactor` seam convention.

The shared library is **built and lives in this repo** at [`ZB.MOM.WW.Audit/`](ZB.MOM.WW.Audit/)
(.NET 10; 1 package — `ZB.MOM.WW.Audit`; only non-BCL dependency `Microsoft.Extensions.DependencyInjection.Abstractions`;
23 tests (verified 2026-07-18); `dotnet pack` → 1 nupkg @ 0.1.0). Repo: `https://gitea.dohertylan.com/dohertj2/zb-mom-ww-audit`.
**Adopted across all three apps on 2026-06-02** (audit GAPS #1–#6) on each repo's `feat/adopt-zb-audit` branch
(stacked on `feat/adopt-zb-auth`) — committed + reviewed, then **merged into the repo's local default (main/master)
and PUSHED to origin (gitea) on 2026-06-03** (in sync). Depth =
**DEEP adopt** (the canonical 9-field `AuditEvent` is the record everywhere; domain fields ride in `DetailsJson`).
OtOpcUa: canonical record + `AuditWriterActor : IAuditWriter` + `Outcome` column/migration + `ClusterAudit` fix.
MxGateway: new canonical SQLite `audit_event` store + `IAuditWriter` + `IApiKeyAuditStore`→canonical adapter.
**ScadaBridge: a full audit-subsystem re-architecture** (the program's largest task) — canonical record everywhere via a
deterministic codec; site SQLite split into `audit_event` + an `audit_forward_state` forwarding sidecar; central
partitioned `dbo.AuditLog` collapsed to 10 canonical cols + persisted computed cols (`CollapseAuditLogToCanonical`
migration, MSSQL-verified). Phase 3 wires `Actor` from the Auth principal at authenticated emit sites (per-app
`IAuditActorAccessor`). Per-repo detail in [`components/audit/GAPS.md`](components/audit/GAPS.md) +
`docs/plans/2026-06-02-auth-audit-normalization-phase2-deep.md` + `…-scadabridge-audit-rearch.md`.
Build/test from `ZB.MOM.WW.Audit/`: `dotnet test`. Consumer matrix: all three apps consume the single
`ZB.MOM.WW.Audit` package (OtOpcUa, MxAccessGateway, ScadaBridge — DEEP-adopted as the canonical record).

The Galaxy Repository component normalizes the **Galaxy object-hierarchy SQL browse + reusable gRPC service**
that was previously embedded in `mxaccessgw`. Shared = canonical `galaxy_repository.v1` proto (wire-compatible
with `mxaccessgw`'s existing contract so OtOpcUa's `GalaxyRepositoryClient` is unaffected), the SQL browse
provider (`HierarchySql` / `AttributesSql` validated reverse-engineered queries), in-memory hierarchy cache +
snapshot + deploy-poll refresh `BackgroundService`, `GalaxyHierarchyProjector`, and `AddZbGalaxyRepository` /
`MapZbGalaxyRepository` DI extension. Left per-consumer = section path, subtree auth filtering, and any
app-specific paging defaults.

The shared library is **built and lives in this repo** at [`ZB.MOM.WW.GalaxyRepository/`](ZB.MOM.WW.GalaxyRepository/)
(.NET 10; single package `ZB.MOM.WW.GalaxyRepository`; `dotnet pack` → 1 nupkg @ 0.1.0, **published to
the Gitea NuGet feed** `gitea.dohertylan.com/api/packages/dohertj2/nuget`). The design doc is at
[`docs/plans/2026-06-23-historian-gateway-design.md`](docs/plans/2026-06-23-historian-gateway-design.md) (§10, component 1).
**Consumed by HistorianGateway** as a `PackageReference` from that Gitea feed, pinned at `0.2.0` (originally a
cross-repo `ProjectReference` to this scadaproj tree; switched to the feed package 2026-06-24; bumped `0.1.0` → `0.2.0` 2026-06-25).
**mxaccessgw adoption is a tracked follow-on** — once adopted, mxaccessgw's inline Galaxy browse code is replaced
by the shared lib (the `galaxy_repository.v1` wire contract is already identical, so OtOpcUa and ScadaBridge
clients are unaffected). Build/test from `ZB.MOM.WW.GalaxyRepository/`: `dotnet test`.
Consumer matrix: HistorianGateway (initial); mxaccessgw (follow-on adoption).

## Hosted application: ZB.MOM.WW.Overview (family dashboard)

Besides the shared libraries, this repo hosts one **application**:
[`ZB.MOM.WW.Overview/`](ZB.MOM.WW.Overview/) — a Blazor Server single-pane dashboard that polls
every registered instance's anonymous `/health/ready` + `/health/active`, caches one snapshot
(pages never poll) and renders per-cluster state including the Akka **leader chip**, which is what
the `ZB.MOM.WW.Health` `0.2.0`/`0.3.0` per-entry `data` work exists to feed. **No login, strictly
read-only, holds no credentials** — a requirement, not an omission; it references no Auth/Audit/
Secrets package. Design + impl plan: `docs/plans/2026-07-22-overview-dashboard-{design,impl-plan}.md`
(ledger 19/20; the open task is Phase 4 live acceptance, recorded in `41013dc` as 7 of 9 design §9
checks passing, 2 blocked on an OtOpcUa rig redeploy).

The `Applications` registry ships **empty** on purpose: a deployment supplying none fails startup
naming the key rather than booting an empty dashboard. Per-deployment overlays are committed —
`appsettings.Docker.json` (the local rigs, joined via external networks) and
`appsettings.Wonder.json`.

**Deployed to `wonder-app-vd03` 2026-08-11** (`main @ 4f97e72`) as that box's **fifth** app:
service `ZbOverview` on **:5320** under the shared NSSM, `ASPNETCORE_ENVIRONMENT=Wonder`,
5/5 instances Up. Two constraints worth carrying to any future box:

- **Publish self-contained** (`-r win-x64 --self-contained true`). wonder has **no .NET 10 runtime**
  — every app there carries its own. An FDD publish installs fine and then fails at service start.
- **Registry `BaseUrl` must not put co-located instances on a shared host name.** Leader-address →
  instance resolution matches on **host only** (Akka and HTTP ports differ by design), which assumes
  distinct hosts per instance — true on the rig, false on a single box, where the fleet-wide scan
  lets the first matching entry win the chip for *every* group. On wonder that made ScadaBridge's
  **site** card claim `Leader · central`, a different Akka cluster. Fixed by using `127.0.0.1`,
  which matches nothing, so each chip falls back to the raw Akka address.

**A second instance runs on the QA box `wonder-app-vq03:5320`** (observed live 2026-08-13, watching
**8 instances across 4 applications**, all Up — it is the fastest way to read QA's whole topology).
Its registry uses **real FQDNs**, and that is correct rather than a divergence: the `127.0.0.1`
rule above exists only because VD03 co-locates every instance on one host. QA is a **2-node pair**
(`vq03`/`vq04`) with genuinely distinct hosts, so leader resolution works as designed and the chips
resolve to real names (`node-a (vq03)`). Carry the *reason*, not the literal `127.0.0.1`.

> ⚠️ **`wonder-app-vd03` is the DEV box, not production** (`vd` = DEV, `vq` = QA — confirmed by
> [`mes-delmia-integration-api.md`](mes-delmia-integration-api.md) §1.1). Its
> `ASPNETCORE_ENVIRONMENT` genuinely **is** `Production`, but that is a .NET config-overlay name and
> is orthogonal to the deployment tier. No production box has been identified. Full host/port
> inventory for both tiers: [`deployment.md`](deployment.md) §2.1 + §7.1.

## Per-project primary commands

Run these from inside each project directory (not from `scadaproj`).

```bash
# OtOpcUa
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test  ZB.MOM.WW.OtOpcUa.slnx
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod"   # single test
# Docker fixtures run on shared host 10.100.0.35; control via `lmxopcua-fix` (in ~/bin)

# MxAccessGateway (PowerShell on Windows)
dotnet build src/MxGateway.sln
dotnet build src/MxGateway.Worker/MxGateway.Worker.csproj -p:Platform=x86   # worker MUST be x86
dotnet test  src/MxGateway.Tests/MxGateway.Tests.csproj                     # no MXAccess needed (fake worker)
dotnet run   --project src/MxGateway.Server/MxGateway.Server.csproj

# ScadaBridge  (~/Desktop/ScadaBridge)
dotnet build ZB.MOM.WW.ScadaBridge.slnx
bash docker/deploy.sh        # rebuild + redeploy the 8-node cluster
cd infra && docker compose up -d   # local test services (SQL, OPC UA, SMTP, REST, Traefik) — LDAP is NOT here

# HistorianGateway  (~/Desktop/HistorianGateway)
dotnet build ZB.MOM.WW.HistorianGateway.slnx
dotnet test  ZB.MOM.WW.HistorianGateway.slnx   # unit + golden; live integration tests skip without env vars
dotnet run --project src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj
# Dev: dashboard on :5220 (HTTP/1.1), gRPC h2c on :5221 (from appsettings.Development.json)
# Production: single Kestrel:Endpoints:Https with Protocols=Http1AndHttp2 (ALPN, one TLS port)
# Live integration (need HISTORIAN_GRPC_HOST + HISTORIAN_GRPC_WRITE_SANDBOX_TAG + GALAXY_SQL_CONNSTR set)
dotnet test ZB.MOM.WW.HistorianGateway.slnx --filter "Category=LiveIntegration"
# Local Docker (HistorianGateway/docker/): publish FDD on the host (authed Gitea feed → no Docker restore),
# runtime-only aspnet:10.0 image; login + gRPC API-key auth disabled; points at the REAL wonder datasources.
dotnet publish src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj -c Release -o docker/publish -p:UseAppHost=false
cd docker && docker compose up -d --build   # dashboard/health/metrics :5220, gRPC h2c :5221 (needs host VPN for egress)

# ZB.MOM.WW.Overview  (~/Desktop/scadaproj/ZB.MOM.WW.Overview — the hosted family dashboard)
dotnet build ZB.MOM.WW.Overview.slnx
dotnet test  ZB.MOM.WW.Overview.slnx      # 177 tests, fully offline
cd docker && docker compose up -d --build # local rigs → http://localhost:5320 (start the rigs first)
# wonder deploy artifact — SELF-CONTAINED, the box has no .NET 10 runtime:
dotnet publish src/ZB.MOM.WW.Overview/ZB.MOM.WW.Overview.csproj -c Release -r win-x64 --self-contained true -o <out>

# ZB.MOM.WW.LocalDb  (~/Desktop/scadaproj/ZB.MOM.WW.LocalDb — a hosted shared lib)
dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test  ZB.MOM.WW.LocalDb.slnx   # 149 tests, fully offline (no live deps)
dotnet pack  ZB.MOM.WW.LocalDb.slnx -c Release -o artifacts   # 3 nupkgs @ 0.1.3
```

> **Shared GLAuth (all three apps + HistorianGateway):** LDAP auth for every local dev/test stack is provided by a
> single `zb-shared-glauth` container on the Linux fixture host **`10.100.0.35:3893`**
> (`baseDN dc=zb,dc=local`, Transport=None). Source of truth and deploy runbook:
> [`scadaproj/infra/glauth/`](infra/glauth/) (`config.toml` + `docker-compose.yml` + `README.md`).

## Refreshing this index

This file is meant to be re-scanned when `scadaproj` is opened in Claude Code:

1. List sibling SCADA/OT directories: `find ~/Desktop -maxdepth 2 -iname "claude.md"`.
2. For each project the user wants indexed, read the **top of its `CLAUDE.md`**
   (project overview + build/run sections) and update its row above.
3. Keep the project set **curated** — only the SCADA/OT/Wonderware/OPC-UA family belongs here.
4. Flag new duplicates/overlaps and namespace mismatches rather than silently merging them.

### Index edits go on `main`, and get pushed

Every sister project's `CLAUDE.md` tells it to update its entry here whenever a fact this
index records changes. That edit lands in **this** repo, which has its own checkout — so a
naive `git commit` writes it onto whatever branch `scadaproj` happens to be on, usually an
unrelated feature branch, sometimes one with no upstream. The index then disagrees with
reality for as long as that branch stays unmerged, which is the exact drift the rule exists
to prevent, and it is invisible unless someone goes looking.

**So: commit index edits on `main` and push them.** Stash or set aside unrelated work first
— never commit an index edit onto a dirty branch — and restore the original branch after:

```bash
git rev-parse --abbrev-ref HEAD    # note it
git checkout main && git pull
#   …edit the entry…
git commit -am "docs(index): <what changed about <project>>" && git push origin main
git checkout -                     # restore
```

**Observed 2026-07-27:** the OtOpcUa `Sql` poll driver's entry had sat unpushed on
`feat/overview-dashboard` since 2026-07-24 for exactly this reason, and the `Mqtt` driver's
entry was about to join it. Both were recovered by cherry-picking onto `main` (`6ab50fe`,
`591eaac`, `7dcd93c`) — **cherry-pick stranded index commits rather than merging the branch
they are stuck on**, since that branch usually carries unrelated in-progress work.

## Other workspace projects with a CLAUDE.md (not indexed — promote on request)

Listed so they can be pulled into the index above if you decide they belong.

**SCADA/OT, de-indexed (still have a `CLAUDE.md` under `~/Desktop/`):**

- **OpcUaTestServer** — dual-instance OPC UA test server (.NET 10) for testing OPC UA clients / simulating automation.
- **scada** (ScadaLink) — design docs + scaffolding for the distributed SCADA platform (`ZB.MOM.ScadaLink.*`).
- **scadalink-design-opcua-browser** — ScadaBridge variant focused on OPC UA browser / Data Connection Layer work.
- **DARS** — DARS → Wonderware SCADA migration (design phase, Q2 2026 go-live).
- **DARS_BU** — backup snapshot of DARS.
- **plan** — 3-year SCADA IT/OT modernization roadmap (markdown).

**Outside the SCADA family:**

- **delmia** — DELMIA / Apriso (Intercim Velocity) MES customization export (ASP.NET `.asmx`); MES-adjacent.
- **lightctrl** — Raspberry Pi Python I/O control (edge hardware).
- **codestats** — Rust CLI for code statistics on .NET solutions/dirs (dev tooling for the .NET repos).
- **servecli** — Rust portable SSH/SFTP server for Windows remote task management (greenfield).
- **JdeScopingTool** — JD Edwards "LotFinder" .NET 4.8 → .NET 10 migration (ERP, not SCADA).
- **chat** — local-first roleplay chat engine.
- **candy2** — candy.ai chat scraping / browser automation.
- **menardslist** — Menards.com cart → printable picklist PDF.