# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Project Goal Build an OPC UA server (.NET 10) that exposes industrial data sources — including AVEVA System Platform (Wonderware) Galaxy — under a unified Equipment-based address space. Galaxy access flows through the in-process `GalaxyDriver` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) talking gRPC to a separately installed **mxaccessgw** gateway process. The gateway owns the MXAccess COM bitness constraint (its worker is x86 net48); everything in this repo is .NET 10. PR 7.2 retired the legacy in-process `Galaxy.Host` / `Galaxy.Proxy` / `Galaxy.Shared` projects + the `OtOpcUaGalaxyHost` Windows service. See `docs/v2/Galaxy.Performance.md` for the runtime perf surface (tracing, metrics, soak harness). ## Related Projects (scadaproj umbrella) This repo is one of a family of related SCADA / OT / Wonderware / OPC UA **sister projects** that live as sibling directories under `~/Desktop/`. They are **separate repos and separate processes**, coupled at runtime over wire protocols (gRPC + OPC UA) — not by project/compile references — and share the `ZB.MOM.WW.*` product namespace. The common subject is AVEVA System Platform (Wonderware) "Galaxy" data, and `mxaccessgw` is the linchpin the others connect through. - **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) — the umbrella/index workspace that aggregates the whole family (a high-level scan of each sister project: purpose, location, stack, primary commands). It also hosts the shared `ZB.MOM.WW.*` libraries — `ZB.MOM.WW.Auth`, `ZB.MOM.WW.Theme`, `ZB.MOM.WW.Health`, `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Configuration`, and `ZB.MOM.WW.GalaxyRepository`. **This repo is indexed there** — see the OtOpcUa entry in `../scadaproj/CLAUDE.md`. - **`~/Desktop/MxAccessGateway`** (`mxaccessgw`, [`../MxAccessGateway/CLAUDE.md`](../MxAccessGateway/CLAUDE.md)) — the gRPC gateway that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session x86 net48 worker). **OtOpcUa's `GalaxyDriver` depends on this** for live Galaxy read/write/subscribe and Galaxy Repository browse. - **`~/Desktop/HistorianGateway`** ([`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse (via the shared `ZB.MOM.WW.GalaxyRepository` lib). Patterned on `mxaccessgw` but with no COM / no x86 worker. **OtOpcUa's historian backend** (`Driver.Historian.Gateway`) consumes it as the sole historian path. - **`~/Desktop/ScadaBridge`** ([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) — the distributed SCADA platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's OPC UA address space. **Propagate cross-repo changes to the umbrella index.** When a fact the index records about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow, shared-lib consumption, or per-project commands — update the **OtOpcUa entry in `../scadaproj/CLAUDE.md`** in the same change so the index never drifts from this repo. **Since 2026-09-03 that entry is a short summary row.** The detailed OtOpcUa narrative — change history, live-gate records, redundancy/auth/ACL behaviour, and the per-issue consumer-impact record for ScadaBridge — lives in `../scadaproj/.claude/skills/sister-project-history/otopcua.md` (the `sister-project-history` skill). Append detail **there**; keep the index row to a few lines and only touch it when the summary itself (purpose, wire contract, driver set, auth posture, commands) changes. **The index edit belongs on `scadaproj`'s `main`, and must be pushed.** `scadaproj` is a separate repo with its own checkout, so committing there inherits whatever branch it happens to be on — which is usually *not* `main` and is often an unrelated feature branch with no upstream. That silently drifts the index for as long as that branch stays unmerged, which is the exact failure the rule above exists to prevent. Do this explicitly: ```bash cd ~/Desktop/scadaproj git rev-parse --abbrev-ref HEAD # note it, to restore afterwards git stash list # never commit onto a dirty unrelated branch git checkout main && git pull # …edit the OtOpcUa row in CLAUDE.md and/or .claude/skills/sister-project-history/otopcua.md… git commit -am "docs(index): " git push origin main git checkout - # restore the original branch ``` Observed 2026-07-27: the `Sql` poll driver's index entry had sat unpushed on an unrelated feature branch since 2026-07-24 because it was committed onto the current checkout, so the index disagreed with reality for three days despite this rule. If you find index commits stranded on a feature branch, cherry-pick them onto `main` rather than merging that branch. > ⛔ **Wonder box remote access (2026-09-11): no `psbridge`, no `winrm`.** Both skills are retired/disabled — > the WinRM remote-exec channel set off the security software on the wonder boxes (vd03/vq03/vq04). Do not > run remote commands (service control, publish/swap, PowerShell) on those boxes and do not stand up a > replacement channel until the user clears one. Still allowed: UNC/SMB file copies and logging directly > into the SQL Servers. Deploy steps that need a command on the box are blocked; see the umbrella > [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md) for the current hold. ## Architecture Overview ### Data Flow 1. **Galaxy Repository DB (ZB)** — SQL Server database holding the deployed object hierarchy and attribute definitions. The mxaccessgw's `GalaxyRepositoryClient` queries it via gRPC; the driver consumes the materialised hierarchy through `IGalaxyHierarchySource`. 2. **MXAccess (via mxaccessgw)** — Live read/write/subscribe over a gRPC session. The gateway owns the COM apartment + STA pump server-side; the driver speaks `MxCommand` / `MxEvent` protos exclusively. 3. **OPC UA Server** — Exposes the deployed tree under **two OPC UA namespaces** (see below). Galaxy tags are bound by `TagConfig.FullName` (`tag_name.AttributeName`); reads/writes/subscriptions are translated to that reference for MXAccess. ### v3 OPC UA Address Space (Batch 4): dual namespace The server exposes **two OPC UA namespaces** (replacing the single `https://zb.com/otopcua/ns`), built by `AddressSpaceComposer` / `AddressSpaceApplier` and served by `OtOpcUaNodeManager` (which registers both URIs; `V3NodeIds` + `AddressSpaceRealm` are the identity authority): | Namespace URI | Realm | Subtree | NodeId `s=` scheme | |---|---|---|---| | `https://zb.com/otopcua/raw` | `AddressSpaceRealm.Raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) | | `https://zb.com/otopcua/uns` | `AddressSpaceRealm.Uns` | the UNS tree (Area → Line → Equipment → signal) | slash-joined **`Area/Line/Equipment/EffectiveName`** | - **Single source, fanned to both.** Every device value has exactly one source — the raw tag's node. A `UnsTagReference` projects that raw tag into an equipment as a UNS-namespace variable that carries an **`Organizes` reference to its raw node** and mirrors it. One driver publish for a RawPath fans (in `DriverHostActor`) to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp — the mux key stays single (RawPath). - **Writes via either NodeId** route to the same driver ref under the same `WriteOperate` gating (the node-manager write gate is realm-qualified and fails closed); a failed device write reverts both NodeIds via the shared fan-out (write-outcome self-correction). - **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and fan via SDK notifiers to the raw device folder AND every referencing equipment folder — one `ReportEvent`, one Server-object copy; ack/confirm/shelve route on `ConditionId = RawPath`. - **Historian dual-registration.** Both NodeIds register the **same** historian tagname; the mux `HistorizedTagRef` stays single (RawPath). HistoryRead works through either NodeId. The retired `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme and the single-namespace `EquipmentTags` materialization path are gone. See `docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` + `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`. **How a deploy reaches that address space (surgical-deploys, 2026-08-15).** The old blanket rule — ANY non-empty `ChangedRawTags` / `ChangedUnsReferenceVariables` routes to `Rebuild` — is gone, because it made a routine edit a fleet-wide outage: `RawTagPlan.ReferencingEquipmentPaths` is part of the record's equality, so adding or dropping **one** `UnsTagReference` flipped its backing raw tag to Changed and tore down every node on the server, killing every client subscription. `AddressSpaceChangeClassifier` is still default-closed; what changed is which deltas it whitelists. - **Still a full rebuild:** an **alarm-bearing** raw-tag delta (`Alarm != null` on either side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an **identity** change on a *paired* raw-tag delta (`TagId`/`DriverInstanceId`/`ParentNodeId`/`Realm` — pairing is by RawPath, so this guard catches a tag deleted and re-created under the same path; a plain **rename never pairs** and is served surgically as remove-one-node + add-one-node with any referencing UNS mirror re-pointed, live-gated 2026-08-15), a raw **container** (Folder/Driver/Device/TagGroup) rename or removal, `ChangedEquipment` / `ChangedAlarms`, a non-surgical equipment-tag delta, and anything else the `with`-equality whitelist does not name (including any FUTURE field). - **Now surgical:** a **non-alarm** raw-tag delta over the in-place-updatable attributes (`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` / `ArrayLength`) and/or `ReferencingEquipmentPaths` — applied via `ISurgicalAddressSpaceSink.UpdateTagAttributes` in the **Raw** realm — and **every** `UnsReferenceDelta` (add / remove / re-point / backing-edit). - ⚠️ **UNS mirrors repair through the RECORD, not the classifier.** A raw-side historian or shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape (`EnsureVariable` early-returns on an existing node). `UnsReferenceVariable` therefore carries the backing tag's `Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}`, so the same edit makes the reference record unequal and the planner emits a **coincident** `UnsReferenceDelta`. Adding a newly-inherited field needs no classifier change — **omitting one silently reopens the staleness.** Repair is **in place** (`UpdateTagAttributes`, Uns realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an identity change (re-point, effective-name edit) is remove+recreate — and only for that one node. - **A non-rebuild apply materialises the DELTA, not the composition.** `MaterialiseHierarchy` still runs full (parents must exist); the other passes take the plan's added/changed subsets. Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread. - ⚠️ **A degraded non-rebuild apply does NOT advance `_lastApplied`.** The full passes used to re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the delta path removed that. Holding the composition back is the replacement — the next deploy re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct, bounded resync). A **rebuild** still advances even with failures. - **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is what makes every whitelist above safe. - **Announcements:** `AnnounceAddedNodes` announces surgically-added **Raw** containers/tags (`ComputeRawAddAnnouncements`, #599 — the former "raw adds are undiscoverable until a re-browse" gap is closed) as well as added UNS reference variables. - **Array writability rides the same whitelist:** `ArrayWritable` joins the surgical raw-tag attributes and `UnsReferenceVariable` carries `BackingArrayWritable` (see "Equipment-node writes" under Authentication). Removing an equipment that has references removes the references before the equipment subtree, and `RemoveEquipmentSubtree` severs the inverse `Organizes` edges. See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`. ### Key Concept: Tag Name and FullName Galaxy objects have a **tag_name** — a globally unique system name used for MXAccess read/write — and attributes are referenced as `tag_name.AttributeName` (e.g. `DelmiaReceiver_001.DownloadPath`). This dot-separated reference is what `TagConfig.FullName` stores for a Galaxy equipment tag. The Galaxy address picker browses the live hierarchy using contained names for navigation, then resolves the selection to the `tag_name.AttributeName` form. **Galaxy is a standard Equipment-kind driver.** Galaxy points are ordinary equipment `Tag`s bound to `GalaxyMxGateway` via `TagConfig.FullName` (`tag_name.AttributeName`), authored through the standard Tag modal + Galaxy address picker on the equipment page's Tags tab — the same flow as Modbus or S7. There is no alias machinery, no `SystemPlatform` namespace kind, and no relay→alias converter. See `docs/plans/2026-06-12-galaxy-standard-driver-design.md` for the full design. ### Data Type Mapping Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`. ### Change Detection `DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys. **The signal is consumed as an operator prompt, not as an address-space rebuild** (§8.2, 2026-07-27). `DriverInstanceActor` subscribes in `PreStart` and unsubscribes in `PostStop` — the detach is load-bearing, because the `IDriver` instance outlives the actor when the host respawns a child around the same driver object. A raise is recorded and rides the driver-health snapshot (`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`) to the AdminUI `/hosts` page as a **"re-browse" chip**. ⚠️ **It is advisory: a Galaxy redeploy still does NOT change the served address space.** v3 authors raw tags explicitly through the `/raw` browse-commit flow, so an operator must re-browse the device and commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact records. Two gotchas if you touch this: - `PublishHealthSnapshot` dedups on a health fingerprint. The rediscovery timestamp **must** stay in that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the operator. Pinned by `DriverInstanceActorRediscoverySignalTests`. - The shared test `StubDriver.GetHealth()` returns `DateTime.UtcNow`, so its fingerprint differs on every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests use a stub with stable health for exactly this reason. **#507 is closed as superseded, and the injection path is deleted.** Its retained code read `EquipmentNode.DriverInstanceId` and `EquipmentTags`, both *structurally empty* in v3 (`AddressSpaceComposer.cs`, `DeploymentArtifact.cs`), so removing the guard would have changed a log line and injected nothing. Gone with it: `HandleDiscoveredNodes`, `PartitionDiscoveredByDeviceHost`, `ApplyDiscoveredPlansForDriver`, the redeploy re-inject tail, `DiscoveredNodeMapper`, `AddressSpaceApplier.MaterialiseDiscoveredNodes`, the connect-time discovery loop (`StartDiscovery`/`RediscoverTick`/`DiscoveredNodesReady`/`TriggerRediscovery`), and the 18 `DiscoveryInjectionDormantV3` tests — v2 characterization tests asserting an equipment-rooted graft (`EquipmentRootNodeId == "EQ-1"`) that v3 cannot produce. `ITagDiscovery` itself stays: the `/raw` browse picker drives it through `Commons/Browsing/DiscoveryDriverBrowser`, which never went through the actor. **`IRediscoverable` is the ONLY tag-change signal, and four drivers do not raise it.** A periodic tag-set drift check briefly existed (shipped `e77c8a35`, **removed** by Gitea **#523**): every `SupportsOnlineDiscovery` driver was re-browsed on a 5-minute timer and the result diffed against its authored tags. It was taken back out because it generated recurring, **unconfigurable** (interval was a constructor parameter with no appsettings key), **never-live-gated** outbound traffic to real PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the design treated it as an implementation detail. ⚠️ **Consequence to know before you rely on rediscovery:** **AbCip and FOCAS browse a live backend but implement no `IRediscoverable`, so they have no tag-change signal at all** — a PLC program re-download or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT, MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an event-driven signal, or one that runs when something *else* already indicates change (a reconnect, a deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`TagSetDriftDetector`, `DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` / `DiscoveryStreamIncludesAuthoredTags`. **`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes `OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is `DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip). ### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed) Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into `Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*. The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it: - ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.** `Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely. `LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any success. Each conjunct has a test that goes red when the *other* is deleted. - **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect supervisor. `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has no transport. Re-check this if a new driver lands. - **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.** `ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable` field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync` leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts` probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not* via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first, which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent — the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new `IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite. - The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` / `DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped reading. - **What comms loss now does downstream — one contract, six seams (archreview R3-02).** "A value consumer never reports a quality better than its source's" is stated once in `docs/DriverLifecycle.md` §"The quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three things changed with it, all client-visible: - ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes` stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**: the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see `Value = null` + the Bad status; the retained value is what the next Good re-publish and the `(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9 conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop the tag" (ScadaBridge) will see more Bads — that is the fix working. - **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's `forceRaise`, MQTT's new initial push). Instead the host **audits**: refs still latched 30 s after a reconnect log a Warning and increment `otopcua.driver.comms_latched_refs`. Non-zero on that counter means a driver reconnects without re-publishing. - **Every dedup on the recovery path is keyed on (value, status), not value.** A device that comes back usually comes back holding the same reading, so a value-only dedup swallows the recovery and the node stays Bad forever — the v3-gate trap, now closed in the Calculation driver, the virtual-tag actor and (via a connection-loss stale) MQTT's plain-mode suppression. **Any new consumer of driver values must repeat this**; a regression test for it must replay the IDENTICAL value or it proves nothing. - Consequences per seam: MQTT/Sparkplug stale on connection loss and push cached snapshots on every (re)subscribe (`docs/drivers/Mqtt.md`); the Calculation driver and virtual tags **hold** while any input is Bad and inherit the worst input's sub-code, evaluate through Uncertain, and hand scripts the input's real `StatusCode`/`SourceTimestampUtc` (`docs/drivers/Calculation.md`, `docs/ScriptEditor.md`) — so a script that relied on receiving `null` for a dead input no longer runs at all while it is Bad. See `docs/DriverLifecycle.md` §"The comms-loss watchdog" and §"The quality contract". ## mxaccessgw The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3. ## Build Commands ```bash dotnet restore ZB.MOM.WW.OtOpcUa.slnx dotnet build ZB.MOM.WW.OtOpcUa.slnx dotnet test ZB.MOM.WW.OtOpcUa.slnx # all tests dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single test project dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test ``` `Category=SlowUnit` (#606) tags the 16 wall-clock-bound Driver.Mqtt unit tests (blackhole connect-timeout / reconnect-backoff windows, ~1 min of the project's 73 s). CI runs them (its filter excludes only E2E + LiveIntegration); for a fast local loop add `--filter "Category!=SlowUnit"`. CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`) is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle shipped. `Core.AlarmHistorian` now carries a surgical direct pin of `SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls transitively. `dotnet restore`/`build` is clean with the audit active. Test projects live under `tests//` (Core, Server, Drivers, Drivers/Cli, Client, Tooling) — there is no single unit-test project. Unit suites are named `*.Tests`; integration suites are `*.IntegrationTests` and need their Docker fixture up (see Docker Workflow). DB-backed tests in `*.Configuration.Tests`, `*.Admin.Tests`, and `*.Server.Tests` require the central SQL Server. ## Docker Workflow (driver fixtures + central SQL Server) > **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing + > deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate** > recipes (S7 blackhole, HistorianGateway LiveIntegration), the integration-test > harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail. > **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28). > ⚠️ **The `project: lmxopcua` label is aspirational, not universal.** This note used to claim every fixture compose file carries it. As of 2026-07-24 only the **MQTT** fixture actually does — `grep -rn "labels:" tests --include=*.yml` matches `Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml` and nothing else. So `docker ps --filter label=project=lmxopcua` returns the MQTT fixture alone, not the fleet. Add the label when you touch an older fixture; until then enumerate by container name. Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`. **`docker -H ssh://...` does NOT work from the Windows VM.** Windows OpenSSH ↔ docker.exe stdio bridging hangs (`docker system dial-stdio` runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs `docker compose` server-side. > **On macOS there is no `lmxopcua-fix` — drive the host over SSH directly.** The helper is a Windows-VM > script; `~/bin` is empty on the Mac and the commands below will not resolve. Use passwordless SSH > (and `rsync` in place of `sync`), which is what `infra/README.md` §1 documents as the Mac path: > > ```bash > ssh dohertj2@10.100.0.35 'docker ps --format "{{.Names}}\t{{.Status}}"' > ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mqtt && docker compose up -d' > rsync -av tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/ \ > dohertj2@10.100.0.35:/opt/otopcua-mqtt/ # the `sync` step, by hand > ``` > > A local container runtime *does* exist on the Mac (OrbStack), so the `docker-dev/` rig itself runs > locally there — it is only the shared **fixtures** that live on `10.100.0.35`. **Use `lmxopcua-fix.ps1` (in `~/bin`) to control fixtures from the Windows VM:** ```powershell lmxopcua-fix ls # list all lmxopcua-tagged containers on the host lmxopcua-fix up modbus standard # bring a profile up lmxopcua-fix up abcip controllogix lmxopcua-fix up s7 s7_1500 lmxopcua-fix up opcuaclient # single-service stack, no profile arg lmxopcua-fix down modbus # tear stack down lmxopcua-fix logs modbus lmxopcua-fix sync modbus # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/ ``` **`sync` is the deployment step.** When you edit a fixture's compose file or Dockerfile under `tests/.../Docker/`, run `lmxopcua-fix sync ` to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; `/opt/otopcua-/` is a mirrored deployment. **Endpoints (defaults already point at the docker host):** - SQL Server (always-on): `10.100.0.35,14330` — used by `appsettings.json` for `ConfigDb`. - Modbus: `10.100.0.35:5020` (`MODBUS_SIM_ENDPOINT`) - AB CIP: `10.100.0.35:44818` (`AB_SERVER_ENDPOINT`) - S7: `10.100.0.35:1102` (`S7_SIM_ENDPOINT`) - OPC UA reference (opc-plc): `opc.tcp://10.100.0.35:50000` (`OPCUA_SIM_ENDPOINT`) - MQTT (Mosquitto, **auth on both listeners — no anonymous fallback**): `10.100.0.35:8883` TLS+auth (`MQTT_FIXTURE_ENDPOINT`) · `10.100.0.35:1883` plaintext-but-authenticated (`MQTT_FIXTURE_PLAIN_ENDPOINT`). Also needs `MQTT_FIXTURE_USERNAME` / `MQTT_FIXTURE_PASSWORD`, and `MQTT_FIXTURE_CA_CERT` to pin the fixture CA (`scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt`). A co-running publisher loops JSON under `otopcua/fixture/…`. - MQTT **Sparkplug B**: the same broker, plus the `sparkplug` compose profile's `otopcua-sparkplug-sim` (project-owned C# edge-node simulator) — group **`OtOpcUaSim`**, edge nodes **`EdgeA`**/**`EdgeB`**, `EdgeA` device **`Filler1`**, ~2 s cadence. It subscribes to `spBv1.0/OtOpcUaSim/NCMD/{node}` and re-births on request. **Births are never retained**, so `docker restart otopcua-sparkplug-sim` is how you force a fresh one while a browse window is open. Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at `opc.tcp://localhost:4840` — **that's not on the docker host**. **Local docker-dev rig — login is DISABLED, so do live `/run` verification yourself (don't wait for the user to sign in).** The local `docker-dev/docker-compose.yml` stack (AdminUI at `http://localhost:9200` via Traefik; OPC UA `opc.tcp://localhost:4840` central-1 / `:4841` central-2) runs the AdminUI with `Security__Auth__DisableLogin: "true"` — **no sign-in form; it's auto-authenticated as a full-access admin.** So AdminUI / Razor `/run` verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against `http://localhost:9200`) does **not** require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA *data-plane* auth is still real LDAP against the shared GLAuth on `10.100.0.35:3893` — that only gates Client.CLI read/write **role** operations, e.g. binding a `multi-role` / `opc-writeop` user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the `ZB.MOM.WW.HistorianGateway` sidecar — still need the user.) See `docs/v2/dev-environment.md` for the full inventory and rationale. ## Build & Runtime Constraints - Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint is owned by the mxaccessgw worker (x86 net48), not by anything in this repo. - The gateway's MXAccess worker requires a deployed ArchestrA Platform on the machine running the gateway. The OtOpcUa server itself does not. ## Transport Security The server supports configurable OPC UA transport security via the `OpcUa:EnabledSecurityProfiles` list in `appsettings.json`. Phase 1 profiles (the `OpcUaSecurityProfile` enum members): `None` (default), `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`. Security policies are built from the enabled profiles by `BuildSecurityPolicies` at startup. The server certificate is always created even for `None`-only deployments because `UserName` token encryption depends on it. See `docs/security.md` for the full guide. ## Redundancy The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide. **Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.** - **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`). - **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**. **A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". **Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard". Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. **RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below. **Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences: - **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`. - `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh. - `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it. - **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists. - **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip. - **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4). See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. ## Role authority (`EffectiveRoles`) **A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and 01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence — `Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix (`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one `EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and `AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs` read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`). - **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing `Cluster:Roles` would leave the bound options disagreeing with the registered instance. `AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op today since none of the four shipped overlays carries a `Cluster` section. - **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles` still wins outright) — the boot banner also names which source won (`source=Configured` / `Environment` / `None`). - **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through `EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:** `RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the library nor with a local case-insensitive pre-filter, which would split the Primary election from `ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)"). - **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s — leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException` (design-time tooling) passes through untouched. - **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement: a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads `OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal — with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI RBAC role). See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and `archreview/plans/R3-04-one-role-authority-plan.md`. ## Mesh command transport (`MeshTransport`) **The Akka `ClusterClient` transport described in this section's earlier history is DELETED** (branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md), Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension, `MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values) rather than silently falling back to `Dps`. What remains is exactly two transports, selected by `MeshTransport:Mode`: - **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**: it works only when central and the node share a gossip ring, which after Phase 6 means within one 2-node mesh. - **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a node on anything else. Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`, served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort` and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and no rotation: each command goes to the specific node dialled, which is what makes per-node delivery provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer applies to either direction. Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and stops at the first success. Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed on every unresolvable case — constant-time compare, no key material logged. Production keys are `${secret:OT-MESH-PSK-}`; the docker-dev rig supplies three literal dev keys via env (`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`). Things that carried over unchanged from the `ClusterClient` era, because they were never `ClusterClient`-specific to begin with: - **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery. The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't. - **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept a command from either transport identically. - **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()` replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient deletion — it was always about the deploy path, not the transport. `redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) / [plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) / [live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history — those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md` is the original Phase 2 design doc and is likewise left as a historical record of the transport this section now describes as deleted. **Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: - **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag. - **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out. - **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. - `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again. ## Config source (`ConfigSource` / `ConfigServe`) Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via `DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, fail-closed). Things worth knowing before touching it: - **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport). - **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment` stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch. **The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** — `ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and hashes the result, so the invariant holds unchanged and nothing between the composer and `DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be upgraded BEFORE central starts emitting compressed blobs.** - **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**. **Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does **not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches). - **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading SQL today; `StoreAsync` is idempotent and the cache replicates). - **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about). - **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState` ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts. On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is no `OTOPCUA_CONFIG_MODE` env var in the compose file. See `docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and `docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`. ## Live telemetry transport (`Telemetry` / `TelemetryDial`) Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and `/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode` (central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof` event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split (Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before touching it: - **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`), `driver-health` (`DriverHealthChanged`), `driver-resilience-status` (`DriverResilienceStatusChanged`) — feeding the exact same central sinks (`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in either mode. - **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542): `FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end** (no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its `DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above. - **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only `TelemetryDial:Mode` on central actually switches anything: which of the two always-available sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve). - **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` == `TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor` (`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern. - **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from `ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side stores are not evicted). The node-local hub replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1 node stream is up) — the same semantics the DPS pill has always had. See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md` §`Telemetry`/`TelemetryDial` for the config keys, and `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live gate procedure. ## LocalDb pair-local store (Phases 1 + 2) Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**: `deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2). **Phase 1** caches the **deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`, in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by `AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` + `LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes never register it.** The cache can optionally replicate to the node's redundant pair peer over the library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret exception). **Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db` and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm history no longer takes it to the grave. `SqliteStoreAndForwardSink` became `LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key `AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new `HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the same event accepted on both nodes — which happens in every boot window, since `HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue. **The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and `.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15). The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3 upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3) now throws `LocalDbRegistrationException` for a table that is the parent of a row-mutating ON DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire. 0.3.0 made the snapshot-resync flag **per-table** (see the ⚠️ below, now corrected); 0.3.2 added validated config `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, left unset here) and a public `LocalDbOplogGapException`, neither yet wired to a call site. **Its two new APIs — `DeregisterReplicated` and `RegisterReplicated(table, baselineExistingRows:)` — are now both used** (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two things are worth knowing: - **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching **byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps `16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" was corrected with the bump; do not restore that rationale. - **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).** `LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges` (via `LocalDbSetup.ReplicationIsConfigured`): **`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` + `LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only, silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has nothing to capture for. - ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no peer must still drain, so the two predicates are deliberately not the same. The **passive** half (no `PeerAddress`) cannot name its peer, so it stands down for its **mesh partner** (the other member of its `cluster-{ClusterId}` role); a legacy passive half with no mesh role still drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the published Primary verdict (and the live data-plane gate, which must stay the same decision) **allows** — for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the Sparkplug primary-host claim with its Primary peer. - **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the same load-bearing DDL→register order. Baselining is what makes enabling replication on a node that has been running *without* it actually converge: capture is CDC and the snapshot streamer pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in used to replicate only writes made after the restart. It seeds the ledger at the LWW floor (HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote write of the same key and wins only where the peer has no version at all. It is idempotent, so it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot flag is **per-table**, not per-database — baselining one table no longer streams every registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried until 2026-09-03, the flag was per-database and baselining anything streamed *every* registered table in full; the old advice to trim `alarm_sf_events` before enabling replication no longer applies at the current pin.) - **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and prunes that table's oplog + row-version rows. Merely declining to register would leave every pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row inside every write transaction — on precisely the node that has no peer to capture for. The call returns `true` when it removed something, so the Information line fires **once, and only when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where resolving one from DI would nest a resolution inside a singleton construction; `Program.cs` assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe the line rather than infer it. - ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake compares registered-table digests **fail-closed**, so a node that deregisters a table its peer still replicates stops syncing with a schema mismatch. Enabling replication on an established pair means setting the config and **restarting both nodes** — do it on both, or on neither. - **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was enabled from a fresh pair, so the impact of the old gap was low — central and site-b are default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests` pins both branches (registered-table set on a configured node, empty set + zero triggers + the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged. See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config cache), `docs/AlarmHistorian.md`, and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`. ## Phase 4 — cut the driver-side ConfigDb connection Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles` has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs` gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is unaffected (it keeps ConfigDb via its admin role). Consequences: - **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator` fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused node may stay `Direct`. - **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and `Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes **240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture — where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less (driver-only) nodes". - **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore` (replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces `EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed `ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration `20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted. - **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is the system of record. - **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty `NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so `OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only — the same rows a driver node never received via config either, so this is not a regression. Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and `docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4. ## Authentication: API keys on the data plane, LDAP on AdminUI (#520) ⚠️ **Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine, so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an `otua__` credential as the password. `ApiKeyOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/`) implements `IOpcUaUserAuthenticator`: one indexed lookup on `keyId` plus one peppered HMAC-SHA256, `FixedTimeEquals`, **no network I/O**. The key resolves to a **principal**; `NodeAcl.Principal` rows grant it permissions. `LdapOpcUaUserAuthenticator` is **deleted**, along with four `Security:Ldap` keys that existed only for it (`AuthTimeoutMs`, `MaxConcurrentAuthentications`, `OutageFailureThreshold`, `OutageCooldownSeconds`) — they bounded an LDAP bind running inline in the SDK's synchronous, `SessionManager`-lock-holding impersonation callback, and nothing on the remaining LDAP path has that property. **LDAP remains, for AdminUI login only** (`Security:Ldap`, `OtOpcUaLdapAuthService` → `ILdapAuthService`, group→role mapping via `GroupToRole` + `LdapGroupRoleMapping`). Do not "restore" the data-plane authenticator when you see LDAP wiring in `Program.cs` — it is there for the AdminUI. Two things to know before relying on any of this: - ⚠️ **Node ACLs are enforced only behind `Security:Acl:Enabled` (default `false`).** With the flag **off**, a key's grants collapse into the two server-wide role gates (`WriteOperate` on Value writes, `AlarmAck` on Part 9 methods), **only `Cluster`-scoped grants count** — `ApiKeyInterimRoleMapper` deliberately under-grants, because honouring a narrower grant's flags while unable to honour its *scope* would turn "WriteOperate on Equipment X" into server-wide write authority — and **read, browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous**. With the flag **on** (`DataPlaneAuthorizer`), scoped grants decide **every** data-plane operation: write and alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6). - **The two role gates re-check the session's KEY on every call, flag off or on (Gitea #563).** Roles are resolved once at activation, so an expired / deployment-disabled key used to keep `WriteOperate` / `AlarmAck` on its open session until the client disconnected. `DataPlaneAuthorizer.AuthorizeKeyUsable` runs the same `FindUsableKey` probe against the same snapshot + clock as the ACL path (one dictionary lookup, no I/O) and denies `BadUserAccessDenied` before the role check. **Only those two gates** — reads/subscriptions stay ungated with the flag off, by design. Anonymous carries no key and is unaffected. No flag: a dead key keeping write authority is a defect, not a rollout choice. ⚠️ On a fused node a key minted after the last deploy authenticates (ConfigDb) but cannot write/ack until a deployment carries it — the flag-on path always behaved this way. - ⚠️ **The ACL decision REPLACES the role check — the two are never chained.** Chained, a grant would have to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped grants confer a role, so every narrow grant would stay inert and the flag would appear to do nothing. The load-bearing test is therefore an **allow** — a session carrying *no roles* writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too. - **The trie is built inside `DataPlaneAuthorityState.Apply`**, not by a second wiring at the publish actor, and the state owns its `PermissionTrieCache` rather than being handed one. An uninstalled trie denies everything, which is indistinguishable from working enforcement until someone checks that an ALLOW is possible at all. - **A deploy reaches sessions that are already open**, in both directions: each decision stamps the *current* generation rather than the one the session activated under, and re-checks the authenticating key — so revoking a key or a grant stops an open session immediately instead of at the client's next reconnect. On a driver-only node the artifact is the only revocation channel. - `UserAuthorizationState` is built **per decision walk, not per session** — and since the hot-path change it is allocated only on a cache **miss**, so a cache hit never constructs one. Its LDAP-shaped freshness/staleness predicates are switched off EXPLICITLY (`AuthCacheMaxStaleness = MaxValue`). ⚠️ Relying on "the stamp is always fresh" was not safe: `MembershipResolvedUtc` comes from `DataPlaneAuthorizer`'s `TimeProvider` while `IsStale` is evaluated against `TriePermissionEvaluator`'s, so **any divergence between the two clocks denies every request** with a verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead. - **Stage 6's hot seams (read/browse/subscribe) go through a per-`(session, node, operation, realm)` decision cache**, cleared wholesale when the generation advances. It is **sharded per API key** (`ApiKeyId ?? Principal` — despite the type name, sessions sharing one key share one shard and one budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and not every other key's decisions. ⚠️ It caches the **grant walk only** — key revocation and expiry are re-checked **before** the cache lookup on every call, because expiry moves with the clock rather than with a deployment. - **Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors.** A grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward only, so `PermissionTrieNode.BrowseTransitPrincipals` marks the path. Planting a real `Browse` grant on each ancestor instead would be collected by the additive downward walk and leak `Browse` onto every sibling subtree beneath that ancestor. ⚠️ **The topmost ancestor is the single shared root folder** (`s=OtOpcUa`, raw namespace) that BOTH realm trees hang under — its identifier is a display name, not a path, so `DataPlaneAuthorizer.BuildScope` maps it onto the trie's **zero-segment realm-root scope** (`V3NodeIds.SharedRootFolderId`, realm-qualified: in the UNS namespace that identifier is an ordinary Area), and its Browse considers transit in **either** realm because it fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse gate denies the whole call when its source node is denied, so clients were stopped at the front door. - **Revocation reaches live subscriptions** (stage 6): every other operation re-asks per request, but a subscription is authorized once and then streams — and streaming is how machine consumers actually read this server. `DataPlaneAuthorityState.Changed` fires `RevalidateMonitoredItems`. ⚠️ A denied item is queued `BadUserAccessDenied` and then **detached from its `MonitoredNode2` fan-out — NOT set `MonitoringMode.Disabled`**: disabling discards the item's queued notifications, so the client is silenced without ever being told why (caught only by the over-the-wire test). Detaching is also reversible, so restoring the grant resumes the feed with no client action. ⚠️ **`Changed` is no longer the only trigger (arch-review 03/S19):** `AclRevalidationTimer`, armed by `OtOpcUaServerHostedService` at `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0` disables, inert while `Security:Acl:Enabled` is false), runs the same pass on the clock — because `ExpiresUtc` passes with the clock, not with a deployment. The pass's verdict memo is **pass-local and must stay that way**; hoisting it across ticks would serve an expiry that landed between two ticks the earlier tick's allow. **Event/alarm items ride both triggers too (Gitea #562)** — a second map (`_eventItems`), fed by the `SubscribeToEvents` override, holds every *direct* notifier subscription; a denied one is detached via the SDK's own protected `SubscribeToEvents(…, unsubscribe: true)` (what a client delete does) and re-attached with `unsubscribe: false` on restore — again never `Disabled`. Unlike a data item it is **not told** (no value slot for a status; the feed goes quiet), and a client delete of a stood-down item is absorbed by the override (the SDK would otherwise remove an item that is not attached). **Server-object (`i=2253`) event subscriptions are filtered per event at delivery (Gitea #586).** The `SubscribeToAllEvents` override attaches them to **nothing** (just records them in `_allEventsItems`); each condition's own `OnReportEvent` (`OnConditionReportEvent`, fires once per event ⇒ still one Server-object copy) queues the event to each recorded item whose session holds `Subscribe` on the condition (RawPath) **or** any notifier folder it reports through — the per-folder path's question. The root-notifier forward to the core Server object (`OnReportEvent` override) is **dropped**: it was the ungated path, and it is the only one that served an item created before the alarm was deployed (the SDK's `AddRootNotifier` subscribes existing all-events items with `unsubscribe: true`). Revalidation is **implicit** — every event re-asks the authorizer, so a deploy or expiry lands on the next event with no pass. `ConditionRefresh` on an `i=2253` item is filtered the same way. The dispatcher **must** keep the SDK's audit guard (`AuditEventState` only while `Server.Auditing` over SignAndEncrypt/https), because conditions report their Part 9 method audit events through themselves. `ReconcileAllEventsMonitoring` must keep `AreEventsMonitored` raised on every condition while any `i=2253` item exists, because `ConditionState` gates its method **audit** reports on `EventsMonitored()` (method-driven state-change events no longer depend on it — #587, below). The protected `SubscribeToEvents` override that no-ops all-events items is what stops the SDK's `AddRootNotifier` (`unsubscribe: true` bug) taking that unit back. Both review findings (review of `75e559f6`). ⚠️ The materialise path raises only the **new** condition's unit (`RaiseAllEventsMonitoring`); a full reconcile there made a deploy of N alarms O(N²) under `Lock` with ScadaBridge connected — keep the full reconcile to remove/rebuild/(un)subscribe. **Model-change and audit events are gated too (Gitea #588).** The manager's own server-wide events (`GeneralModelChangeEvent`, the rejected-write `AuditWriteUpdateEvent`) go through `ReportToServerObject`: flag **off** ⇒ `Server.ReportEvent` as before; flag **on** ⇒ delivered per item to `_allEventsItems` (same audience), a model change narrowed to the `Affected` nodes the session may **Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other event type fails closed. Auditing is off by default (never configured in this repo). - **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`, `NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node, event subscription and HistoryRead is still gated. `docs/security.md`. - **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure` (plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): a numeric array of another element type is **converted, checked, element by element** to the node's element type (ScadaBridge writes CLR `int[]`/`double[]` lists) — any out-of-range, fractional-to-integer or NaN/∞ element refuses the whole write `BadTypeMismatch` with nothing sent, so a `UInt32` > `Int32.MaxValue` can never reach a Galaxy Int32 attribute as reinterpreted bits; non-numeric mismatches are refused (`Byte[]` ≈ ByteString); wrong length ⇒ `BadTypeMismatch`; an array to a **scalar** node is refused only on Modbus/S7 (`EnsureVariable`'s `refuseArrayOnScalar` — pass-through drivers keep their browse-committed scalar-shaped array tags writable; a new sink wrapper must forward that argument, and the reflection guard checks only that the call arrives); `IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus bounded by `maxRegistersPerWrite`/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would resize the Galaxy attribute); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray || ArrayWritable)` is projected onto the raw node and every UNS mirror. - **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`, preserving today's behaviour) controls *admission*; when admitted the session authenticates as the reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as the same `RoleCarryingUserIdentity` a key produces. Set it `false` to make keys mandatory. A key may **not** use the `anonymous` principal, and a failure to read anonymous's grants **denies** rather than admitting with no roles. - **Keys reach a driver-only node inside the deployment artifact** (stage 4). `ConfigComposer` snapshots the `ApiKey` rows — the peppered HMAC only, never a secret, and the pepper is node config that is deliberately *not* in the artifact — and `DeploymentArtifact.ParseDataPlaneAuthority` decodes them node-side alongside the cluster's `NodeAcl` rows. `OpcUaPublishActor` installs both on `IDataPlaneAuthoritySource` on every apply, **before** its `plan.IsEmpty` early-return, so a deployment that only revokes a key still lands. A node with a ConfigDb keeps `ConfigDbApiKeyStore` (per-activation read, wrapped — see below); a driver-only node gets `ArtifactApiKeyStore`. ⚠️ **The asymmetry is ACTIVATION-scoped, not session-scoped — this line used to say "immediate on a ConfigDb node" and that was wrong.** Clearing `Enabled` in the ConfigDb stops the **next session activation** on an admin/fused node (bounded by `Security:ApiKeys:StoreCacheSeconds`, default 5 s) and takes until the next deployment on a driver-only one — but an **already-open** session re-checks its key against the in-memory *artifact snapshot* (`DataPlaneAuthorizer.Evaluate` → `snapshot.FindUsableKey`), never against `IApiKeyStore`, so on **both** node kinds an `Enabled` revocation reaches an open session only at the next deployment. **Expiry is the exception**: `FindUsableKey` compares `ExpiresUtc` against the authorizer's clock on every call, so it is immediate everywhere — and since arch-review 03/S19 it reaches live subscriptions too, on the host's `Security:Acl:RevalidationIntervalSeconds` timer (default 60 s; the session is **silenced with `BadUserAccessDenied`, not closed**, and recovers on re-activation). Direct event/alarm subscriptions are revalidated on both triggers as well (#562); with the flag **off**, expiry and revocation reach the two role gates (#563) but not reads or subscriptions. - ⚠️ **The activation lookup is BOUNDED (arch-review 03/S18).** The SDK raises impersonation synchronously under a `SessionManager`-wide lock, so a ConfigDb that is *slow* rather than *refusing* stalled **every** session activation on the server, Anonymous and X.509 included. `BoundedApiKeyStore` wraps the ConfigDb store — and only it, never `ArtifactApiKeyStore`, which is two in-memory dictionary probes — with `Security:ApiKeys:StoreTimeoutMs` (2000; exceeding it **denies**, via the authenticator's existing catch-all) plus a `StoreCacheSeconds` (5) read-through cache. Faults and timeouts are never cached, a cached hit whose `ExpiresUtc` has passed is re-read rather than served, and a timeout logs a distinct Warning + `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. Both knobs at `0` restore the old behaviour exactly. - ⚠️ **The auth generation is a node-local COUNTER driven by the `RevisionHash`, not derived from its bits.** `PermissionTrieCache` treats the highest `GenerationId` as current, so a generation taken from hash bytes would install a *lower* number about half the time and the cache would go on serving the previous deployment's ACLs with nothing logged. Pinned by `DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases`. - Note `OtOpcUaServerHostedService` takes `IOpcUaUserAuthenticator` **and** `IDataPlaneAuthoritySource` as **required** ctor parameters — the built-in DI container does not fall back to a default value for an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a node with neither role) or it does not boot. **Grants are authored through a scope picker, not free text (stage 7).** `/clusters/{id}/acls` opens `AclScopePickerModal` over the cluster's raw + UNS trees, and selecting a node supplies `ScopeKind` **and** `ScopeId` together — so `RawDevice` + a TagId, a cross-cluster scope, a `Cluster` grant carrying an id, and the retired `FolderSegment` kind are all unauthorable by construction rather than merely rejected later. ⚠️ **A `Cluster` row must produce a NULL `ScopeId`** — the deploy gate rejects one that carries an id (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would author rows that cannot be deployed. `IAclScopeService.CheckScopeAsync` mirrors `DraftValidator.ValidateNodeAclScopes` rule for rule and **blocks the save** when a scope resolves to nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches the picker); the list page marks such rows `unresolved`. **UNS tags are deliberately not offered** — a UNS signal is a `UnsTagReference`, and a `Tag` grant already covers the raw node *and* every UNS projection of it. The picker is `AclScopeTree`, a separate component from `RawTree`, which already carries two modes. **All eight stages are done** (`d4389f57` enforcement, stage 7 authoring after it), each live-gated on docker-dev. The rationale for every decision above is in `docs/plans/2026-07-31-apikey-acl-enforcement-design.md`. See `docs/security.md` for the full guide. Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.) ## AB drivers: native transport (`Transport`) AbCip and AbLegacy have a second, managed wire transport (`Driver.Ab.Transport`) beside libplctag, chosen by the driver-config key `transport` = `Native` | `Libplctag` (string; takes effect when the driver is **constructed** — every process start, or a deploy that recreates it, not `ReinitializeAsync`; AdminUI "Transport" dropdown; CLIs `--transport`). "Default" is encoded by omitting the key; the AdminUI must never re-emit a concrete default. The defaults differ per driver (final review, 2026-10-07): - **AbCip: an absent key resolves to the node default, `Native` unless pinned.** Precedence (one place, `AbCipTransportResolver`, shared by driver, probe and CLI): explicit per-driver key → node-level appsetting **`Drivers:AbCip:DefaultTransport`** (`Native` | `Libplctag`, env `Drivers__AbCip__DefaultTransport`, `AbCipDriverDefaultsOptions`; a bad value **fails host start**) → compiled `Native`. Bound unconditionally because the driver factory AND the admin-node Test Connect probe both read it. ⚠️ So **the upgrade restart alone moves no-key AbCip drivers to Native**; to stay on libplctag, set the node key — NOT a per-driver key through the old AdminUI, whose typed form drops unknown keys (#576). Evidence: AbCip reads proven on a real CompactLogix (Z-35162, read-only); AbCip writes are `ab_server`-only. - **AbLegacy: an absent key resolves to `Libplctag` for every family; `Native` is opt-in** (no node key). On a wire the native PCCC path is proven only for **N-file reads/writes and one F8 read**, all on `ab_server`; `S:0` (the default probe), B/T/C/R/S/L/ST/A/O/I and the T/C/R sub-element offsets are encoded from the PCCC table and **unverified natively**. Explicit Native on PLC-5/`LogixPccc` logs a once-per-instance Warning. AbLegacy native refuses status-bit sub-element writes (`T4:0.DN`…, `BadNotSupported`) and its ST byte order is unverified. Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an AbCip native **type mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn once per tag — only genuine transport failures evict and degrade. `DeviceSessionPool.Shared` logs through the host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by `LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport, `docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and `docs/plans/2026-10-07-native-ab-transport-gate-results.md`. ## Other driver notes - **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are refused (a one-byte-per-element encode would clobber neighbour bits). - **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types `BadNotSupported`). - **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown. - **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync` first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip/FOCAS alarm projections) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw "tag name collision" on `AdoptConfigDuringInit`); S7 swaps `Plc` under `_gate`. - **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME surface as Int32 ms); an undecodable buffer or unrepresentable write is `BadOutOfRange` with no health impact. Optional `localTsap`/`remoteTsap`/`connectionType` override the slot TSAP (`remoteTsap` ⊕ `connectionType`). Test Connect reads 2 bytes of `MB0` so a PUT/GET-disabled CPU fails the probe. The negotiated PDU size is logged and on `S7Driver.GetDiagnostics()` only — there is no driver-diagnostics surface on `/hosts`. - **TwinCAT (#637):** TIME/TOD publish Float64 ms, DATE/DT a UTC DateTime; a tag authored `UDInt` keeps the raw counter (and keeps historizing — DateTime is not historized). - **AbLegacy (#636):** `AbLegacyAddress` (in `AbLegacy.Contracts`) is family-aware: PLC-5 `I:`/`O:` indices are OCTAL and are emitted to libplctag as decimal (libplctag's `pccc.c` parses decimal only). MicroLogix function files and indirect `N7:[N7:0]` addresses parse but read `BadNotSupported`. One check (`AbLegacyAddress.Validate`) backs the editor (family-independent) and the deploy gate (family from the tag's `DeviceConfig.PlcFamily`); an unparseable address reads `BadNodeIdUnknown` for that tag only. - **OpcUaClient (#635):** OperationLimits are honoured by the SDK session (`SessionClientBatched`) — do not re-add driver-side chunking. Subscription tuning keys are range-checked at `ParseOptions`. `discoveryUrl` endpoints go AFTER configured ones; more than one discovered server without `discoveryServerUri` fails the connect. Session events are the OTel counter `otopcua.driver.opcuaclient.events`. `HistoryAggregateType` carries the full Part 13 set; an aggregate the gateway can't serve is `BadAggregateNotSupported` per node. - **MTConnect (#627/#628):** TIME_SERIES reads as a variable-length `double[]` (Float64 only); a count disagreeing with `sampleCount` is `BadDecodingError`. One MTConnect driver = one agent client = at most one `/raw` Device (`MTConnectMultipleDevices` deploy error; `RawTreeService.CreateDeviceAsync` refuses a second). - **MTConnect CONDITION alarms (#626):** a CONDITION raw tag is a native Part 9 condition only when its `TagConfig` carries the standard `alarm` object (opt-in — alarm tags leave the value subscription, so auto-promotion would break existing String tags). Normal→inactive, Warning→500, Fault→800; one condition per DataItem showing the most severe active nativeCode (ties: ordinal nativeCode, then type); ack is server-side only. ⚠️ UNAVAILABLE never clears: an active condition stays active+acked with a quality-only `BadNoCommunication` (same as #481 transport loss). The driver dedups against what the server was last TOLD (level+nativeCode+nativeSeverity+qualifier — not text), records only while a handler is attached, and clears that record when the subscribed alarm SET changes (a rebuild recreates the nodes inactive) but not on a reconnect of the same set — a replayed `/current` must never re-raise (un-ack). `AlarmEventArgs` gained optional `OpcUaSeverity` (exact Part 9 severity) and `QualityStatusCode`; a quality-only event goes out as `AlarmQualityUpdate` per node and never through the snapshot projection. - **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience tracker, *then* evicts its telemetry snapshots. ⚠️ Both are **tombstoned**, not just cleared: the removed child stops asynchronously and its in-flight subscribe/connect still completes through the invoker (and emits Health) on the thread pool *after* `StopChild`, which used to resurrect the state — so a removed driver's counters were republished forever. `Release` → tracker `Retire` and hub `Evict` ignore every later write for that id until it is spawned again (`Create` → `Reinstate`, `SpawnChild` → `Readmit`). A Watch/`Terminated` release would not work — the late writes land after the actor terminates. ## Library Preferences - **Logging**: Serilog with rolling daily file sink - **Unit tests**: xUnit + Shouldly for assertions - **Service hosting (Server, Admin)**: .NET generic host with `AddWindowsService` (decision #30 — replaced TopShelf in v2; see `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`) - **OPC UA**: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: `OPCFoundation.NetStandard.Opc.Ua.Server` ## OPC UA .NET Standard Documentation Use the DeepWiki MCP (`mcp__deepwiki`) to query documentation for the OPC UA .NET Standard stack: `https://deepwiki.com/OPCFoundation/UA-.NETStandard`. Tools: `read_wiki_structure`, `read_wiki_contents`, and `ask_question` with repo `OPCFoundation/UA-.NETStandard`. ## Testing Use the Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/` for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See `docs/Client.CLI.md` for full documentation. ```bash dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840 dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3 dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500 ``` **AdminUI interactive pages must wrap their body in ``** — `MainLayout` is static, so it cannot catch an interactive page's render failure. Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`. The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`. **v3 (Batch 2): device I/O is authored in the `/raw` project tree** (`GlobalRaw.razor` → `RawTree.razor`, `IRawTreeService`) — a cluster-rooted `Folder → Driver → Device → TagGroup → Tag` hierarchy with per-node context menus (the reusable `ContextMenu`), lazy loading, driver/device config modals (endpoint moved to `Device.DeviceConfig`; Test-connect probes the `DriverDeviceConfigMerger`-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed `/clusters/{id}/drivers` flow (`DriverTypePicker`, `DriverEditRouter`, the 8 `*DriverPage` shells, the Drivers list) is **retired** — its form bodies live on inside the `/raw` modals. The `DriverType` dispatch maps are keyed off the `DriverTypeNames` constants (fixing the `TwinCat`/`Focas` drift). A `Calculation` pseudo-driver (`DriverTypeNames.Calculation`, `IDependencyConsumer`, deploy-time `scriptId`-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See `docs/Raw.md`. **v3 (Batch 3): UNS Equipment is reference-only.** The equipment **Tags** tab no longer authors or binds tags — it holds `UnsTagReference` rows pointing at raw tags authored in `/raw`. "+ Add reference" opens the `RawTree` in picker mode, **cluster-scoped** (structurally + server-enforced `tag.cluster == equipment.cluster`); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. **Effective-name uniqueness** (references + VirtualTags + ScriptedAlarms share the `{EquipmentId}/{EffectiveName}` NodeId space) is enforced at authoring (`IEffectiveNameGuard`) and at the deploy gate (`DraftValidator.UnsEffectiveNameCollision` — catches rename-induced collisions). Scripts use **`ctx.GetTag("{{equip}}/")`** — resolved per-equipment through `UnsTagReference` effective names to the backing RawPath at both compose seams (`AddressSpaceComposer` + `DeploymentArtifact`, via the shared `EquipmentReferenceMap`, byte-parity); an unresolved `` is a deploy error (`EquipReferenceUnresolved`) AND a live Monaco diagnostic (`OTSCRIPT_EQUIPREF`) — editor accepts ⇔ publish accepts. `EquipmentScriptPaths.DeriveEquipmentBase` is deleted (the dot-joint `{{equip}}.X` became the slash-joint `{{equip}}/`). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (`RawTreeService` substring scan). `ImportEquipmentModal` dropped the `DriverInstanceId` column. See `docs/Uns.md` + `docs/ScriptEditor.md`. **v3 (Batch 4 = v3.0): dual-namespace address space.** The OPC UA server exposes **two namespaces** — `https://zb.com/otopcua/raw` (Raw device tree, `s=`) + `https://zb.com/otopcua/uns` (UNS tree, `s=///`) — replacing the single `.../ns` and the retired `EquipmentNodeIds` scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in `DriverHostActor` to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable `Organizes`-references its raw node. Writes route via **either** NodeId to the same driver ref under the same realm-qualified `WriteOperate` gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (`ConditionId = RawPath`) and fan via SDK `AddNotifier` to the raw device folder + every referencing equipment folder (one `ReportEvent`, one Server-object copy); both NodeIds register the SAME historian tagname (mux `HistorizedTagRef` stays single, keyed by RawPath). Identity authority: `V3NodeIds` + `AddressSpaceRealm` (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, `docs/Uns.md`, and `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`. The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`. ## Scripting / Script Editor C# virtual-tag scripts are authored in a **Roslyn-backed Monaco editor** on the ScriptEdit page (`/scripts/{id}`) and inline inside the virtual-tag modal on the `/uns` page. The editor provides completions, live diagnostics, hover, signature help, document formatting, and tag-path completions inside `ctx.GetTag("…")` / `ctx.SetVirtualTag("…")` literals — all backed by the same compiler context the runtime publish gate uses, so what the editor accepts/rejects matches publish exactly. The backend lives in `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/` (six minimal-API endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy (Administrator or Designer) via `RequireAuthorization` in `ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`). `Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted. See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`. ## Scripted Alarm Ack/Shelve Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`. ⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no** condition event — the old "SDK auto-fires its own event (E2)" comments were wrong. `WireMethodTransitionReporting` wraps every method's SDK handler (Ack/Confirm/AddComment/Enable/Disable/OneShotShelve/TimedShelve/Unshelve) and, on Good, reports once via `ReportConditionEvent` — **under `Lock`**, so the engine's/driver's re-projection (routed by a fire-and-forget Tell *before* the SDK applies state) waits and then hits the delta-gate with no delta. Things to keep: the SDK's shelve handlers never apply shelving, so the wrapper does; only a real before/after change is reported (a re-ack is silent and restores the Confirmed/Retain the SDK flips); `MaxTimeShelved` is set to `int.MaxValue` ms at materialise because `Create()` left it 0 and every TimedShelve failed. ⚠️ **Native alarms support Acknowledge and AddComment only** — Confirm, the shelve methods and Enable/Disable return `BadNotSupported` (no upstream path, no pair replication, and `NativeAlarmProjector`'s Confirmed/Unshelved constants own those fields); a native ack re-asserts `ConfirmedState = true` because the SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by `ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`). Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a **no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an `AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by `IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design). **Condition `Severity` is carried verbatim, clamped to 1..1000** — `ToPart9Severity`; the node manager used to re-bucket onto the five `EventSeverity` points (so an MTConnect Fault read 900, scripted Critical 1000 read 900, native Low 200 read 300). ScadaBridge only clamps, so exact values pass through. ⚠️ **Never select a two-state id with `AddSelectClause(type, "X/Id")`** — the string becomes ONE QualifiedName named `X/Id`, which no event carries; the Client.CLI/Client.UI filter did this since `a2883b82` and every alarm read inactive+unacked. Use `new SimpleAttributeOperand(type, [X, Id])` (ScadaBridge already does). **Condition children are instance-scoped (#629).** Every child of a materialised condition carries `{ConditionId}/{browse path}` in the condition's namespace (e.g. `…/temp_hi/ShelvingState/UnshelveTime`), assigned by the `OtOpcUaNodeManager.New` override — the SDK default kept each child on the TYPE's ns=0 id, so a direct Read hit the type (or `BadNodeIdUnknown`), translate failed, and children sat outside the ACL (the slash join now puts them under their condition's scope). ⚠️ The shelve methods resolve by the `AlarmConditionType_ShelvingState_*` ids only because `New` adopts those (a fixed set) as their `MethodDeclarationId`; calls by the `ShelvedStateMachineType_*` id are no longer supported (pinned by a test). Condition teardown must forget the whole `PredefinedNodes` subtree (`ForgetPredefinedSubtree`) or a removed condition's children keep answering. ## Historian / HistoryRead **Backend: HistorianGateway (sole historian backend).** As of the gateway-integration cutover, the historian read, alarm-write, and continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed **`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin `IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`, now a retired stub). A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default historian tagname, which is the tag's driver `FullName`. ### Read path (`ServerHistorian` section) The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the `GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default (historized nodes return `GoodNoData`, never an error). Supported variants: Raw, Processed (Average/Minimum/Maximum/Total/Count aggregates), and AtTime over historized variable nodes; Events over alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node); authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization. `ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`): | Key | Default | Notes | |---|---|---| | `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` | | `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) | | `ApiKey` | `""` | Peppered-HMAC key `histgw__` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** | | `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme | | `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) | | `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store | | `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read | | `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) | | `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential | | `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 | | `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded | ### Alarm-history path (`AlarmHistorian` section) Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink` default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The `AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`, `Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection (endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround). ### Continuous historization (`ContinuousHistorization` section) When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed `IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**) with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries only the recorder + outbox knobs: | Key | Default | Notes | |---|---|---| | `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder | | `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage | | `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands | | `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` | | `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup | | `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) | | `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" | | `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve | | `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance | | `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass | | `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff | ### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook) `AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's `EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues` land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the recorder likewise drops + meters non-numeric values. Continuous historization is **numeric-analog only** in v1 (`UInt16→UInt4` is a documented fallback). The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`), built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier. When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with `requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR #423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.) **DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists). `HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see the "Boolean historization maps to Int2" section in `docs/Historian.md`. ### Gateway-side prerequisites The target HistorianGateway OtOpcUa points at **must** run with: - `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization). - `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads). - An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**. ### Migration note (deployments upgrading from the Wonderware backend) The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env: | Old (Wonderware) key | New (gateway) key | |---|---| | `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) | | `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) | | `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) | The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`) were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from `ServerHistorian`. See `docs/Historian.md` for the full guide. ### Live-validation gate — PASSED (was "KNOWN LIMITATION 1") **No longer a limitation.** The cutover was live-validated against the real gateway on `wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot populate `Source_Object` — not a fixable bug). Evidence: `docs/plans/2026-06-27-otopcua-historian-followups.md`. Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to `wonder-sql-vd03`, gateway running the prerequisites above): ```bash export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip export HISTGW_GATEWAY_APIKEY=histgw__ # must carry historian:read + historian:write (+ tags:write) scopes export HISTGW_TEST_TAG= # read round-trip export HISTGW_WRITE_SANDBOX_TAG= # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write) export HISTGW_BOOL_SANDBOX_TAG= # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441) export HISTGW_ALARM_SOURCE= # alarm SendEvent → ReadEvents round-trip dotnet test --filter "Category=LiveIntegration" ``` The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the gate the operator runs on the VPN before trusting the cutover. ### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04) The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer + meters). It is spawned with an **empty *initial* ref set** (`Array.Empty()` in `WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the `AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe `Tell`; a faulting feed can never break a deploy). **The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a production-fatal defect** — see `docs/plans/2026-08-01-continuous-historization-live-gate.md`. Three things to know: - ⚠️ **`ConfigureAwait(false)` in an Akka handler destroys `Self`.** `OnValueChangedAsync` touched `Self` after awaiting `IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false)`, which discards Akka's `ActorTaskScheduler` synchronization context — so `ActorBase.Context` was gone and it threw `NotSupportedException` on the FIRST value. The throw reached supervision, the actor **restarted**, and because production seeds an **empty** ref set the restart discarded every historized ref: one value captured, then silence. Fixed by capturing `IActorRef self = Self` before the await (the idiom the `GetStatus` handler already used for `Sender`). - ⚠️ **A fake that completes synchronously hides this entire class of bug**, and 13 tests passed because of one. `ValueTask.CompletedTask` never yields, so the continuation ran inline with the context intact. Worse, the obvious fix to the fake — `await Task.Yield()` — is *also* vacuous: `Yield()` posts back to the captured context, so the caller still resumes on the actor thread. The fake must complete on a **pool** thread (`await Task.Delay(1).ConfigureAwait(false)`), as a real per-entry fsync does. Only reverting the production fix exposed that the first regression test proved nothing. - **Both pair nodes historized the same value** — the recorder had **no Primary gate**, unlike the alarm store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as **#537** and **fixed** — see the paragraph below. **Gate COMPLETE — #491 CLOSED 2026-08-04.** Outbox append, drain, `WriteLiveValues`, **restart-convergence**, `EnsureTags` provisioning (`ensured=2 failed=0`) and **read-back** all ✅. The read-back leg ran against the licensed `wonder-sql-vd03` with the VPN up: two registers carrying *distinct non-zero* constants (`8224`, `120`) each read back as its own value — a `0`-valued source would have proved nothing (the #536 trap). **Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison tag); evidence in the same plan doc. Two things to know before touching either.** - **#538 — the discriminator is "did another tag write this pass?", NOT a failure count.** `DrainOnceAsync` writes every per-tag group and acks **all-or-nothing** (deliberate: `RemoveAsync` truncates *through* an id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned the FIFO head while every *healthy* tag was re-submitted on every pass, forever — duplicate samples, an outbox that never truncated (observed 8.3 MB; signature = the gateway's `ok` **and** `error` counters climbing together). The drain now quarantines a tag after `MaxWriteAttempts` consecutive failures, but ⚠️ **only counts a failure on a pass where some other tag succeeded** — otherwise an ordinary gateway restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so the load-bearing test is the **falsification control** (`A_backend_that_fails_every_tag_abandons_nothing`). Abandonment is also deferred until *every* failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. - **#537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink.** The alarm queue is in LocalDb and *replicates*, so the Secondary holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of stale duplicates on promotion). ⚠️ **And the peer that matters is one recording the SAME refs, not one holding the same rows** — a node stands down only for a Primary in its own per-cluster mesh, and a node with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is usually running another cluster's configuration, and deferring to it ends this cluster's historization *silently* (the drain reports success on an empty outbox forever) — the same scope mismatch that once suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on `RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes (NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the #511 registration accident). Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType) provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those changed raw tags.