lmxopcua/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Goal

Build an OPC UA server (.NET 10) that exposes industrial data sources — including AVEVA System Platform (Wonderware) Galaxy — under a unified Equipment-based address space. Galaxy access flows through the in-process GalaxyDriver (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/) talking gRPC to a separately installed mxaccessgw gateway process. The gateway owns the MXAccess COM bitness constraint (its worker is x86 net48); everything in this repo is .NET 10. PR 7.2 retired the legacy in-process Galaxy.Host / Galaxy.Proxy / Galaxy.Shared projects + the OtOpcUaGalaxyHost Windows service.

See docs/v2/Galaxy.Performance.md for the runtime perf surface (tracing, metrics, soak harness).

Related Projects (scadaproj umbrella)

This repo is one of a family of related SCADA / OT / Wonderware / OPC UA sister projects that live as sibling directories under ~/Desktop/. They are separate repos and separate processes, coupled at runtime over wire protocols (gRPC + OPC UA) — not by project/compile references — and share the ZB.MOM.WW.* product namespace. The common subject is AVEVA System Platform (Wonderware) "Galaxy" data, and mxaccessgw is the linchpin the others connect through.

• ~/Desktop/scadaproj (../scadaproj/CLAUDE.md) — the umbrella/index workspace that aggregates the whole family (a high-level scan of each sister project: purpose, location, stack, primary commands). It also hosts the shared ZB.MOM.WW.* libraries — ZB.MOM.WW.Auth, ZB.MOM.WW.Theme, ZB.MOM.WW.Health, ZB.MOM.WW.Telemetry, ZB.MOM.WW.Configuration, and ZB.MOM.WW.GalaxyRepository. This repo is indexed there — see the OtOpcUa entry in ../scadaproj/CLAUDE.md.
• ~/Desktop/MxAccessGateway (mxaccessgw, ../MxAccessGateway/CLAUDE.md) — the gRPC gateway that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session x86 net48 worker). OtOpcUa's GalaxyDriver depends on this for live Galaxy read/write/subscribe and Galaxy Repository browse.
• ~/Desktop/HistorianGateway (../HistorianGateway/CLAUDE.md) — single-process gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse (via the shared ZB.MOM.WW.GalaxyRepository lib). Patterned on mxaccessgw but with no COM / no x86 worker. OtOpcUa's historian backend (Driver.Historian.Gateway) consumes it as the sole historian path.
• ~/Desktop/ScadaBridge (../ScadaBridge/CLAUDE.md) — the distributed SCADA platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's OPC UA address space.

Propagate cross-repo changes to the umbrella index. When a fact the index records about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow, shared-lib consumption, or per-project commands — update the OtOpcUa entry in ../scadaproj/CLAUDE.md in the same change so the index never drifts from this repo. Since 2026-09-03 that entry is a short summary row. The detailed OtOpcUa narrative — change history, live-gate records, redundancy/auth/ACL behaviour, and the per-issue consumer-impact record for ScadaBridge — lives in ../scadaproj/.claude/skills/sister-project-history/otopcua.md (the sister-project-history skill). Append detail there; keep the index row to a few lines and only touch it when the summary itself (purpose, wire contract, driver set, auth posture, commands) changes.

The index edit belongs on scadaproj's main, and must be pushed. scadaproj is a separate repo with its own checkout, so committing there inherits whatever branch it happens to be on — which is usually not main and is often an unrelated feature branch with no upstream. That silently drifts the index for as long as that branch stays unmerged, which is the exact failure the rule above exists to prevent. Do this explicitly:

cd ~/Desktop/scadaproj
git rev-parse --abbrev-ref HEAD            # note it, to restore afterwards
git stash list                             # never commit onto a dirty unrelated branch
git checkout main && git pull
#   …edit the OtOpcUa row in CLAUDE.md and/or .claude/skills/sister-project-history/otopcua.md…
git commit -am "docs(index): <what changed about OtOpcUa>"
git push origin main
git checkout -                             # restore the original branch

Observed 2026-07-27: the Sql poll driver's index entry had sat unpushed on an unrelated feature branch since 2026-07-24 because it was committed onto the current checkout, so the index disagreed with reality for three days despite this rule. If you find index commits stranded on a feature branch, cherry-pick them onto main rather than merging that branch.

⛔ Wonder box remote access (2026-09-11): no psbridge, no winrm. Both skills are retired/disabled — the WinRM remote-exec channel set off the security software on the wonder boxes (vd03/vq03/vq04). Do not run remote commands (service control, publish/swap, PowerShell) on those boxes and do not stand up a replacement channel until the user clears one. Still allowed: UNC/SMB file copies and logging directly into the SQL Servers. Deploy steps that need a command on the box are blocked; see the umbrella ../scadaproj/CLAUDE.md for the current hold.

Architecture Overview

Data Flow

1. Galaxy Repository DB (ZB) — SQL Server database holding the deployed object hierarchy and attribute definitions. The mxaccessgw's GalaxyRepositoryClient queries it via gRPC; the driver consumes the materialised hierarchy through IGalaxyHierarchySource.
2. MXAccess (via mxaccessgw) — Live read/write/subscribe over a gRPC session. The gateway owns the COM apartment + STA pump server-side; the driver speaks MxCommand / MxEvent protos exclusively.
3. OPC UA Server — Exposes the deployed tree under two OPC UA namespaces (see below). Galaxy tags are bound by TagConfig.FullName (tag_name.AttributeName); reads/writes/subscriptions are translated to that reference for MXAccess.

v3 OPC UA Address Space (Batch 4): dual namespace

The server exposes two OPC UA namespaces (replacing the single https://zb.com/otopcua/ns), built by AddressSpaceComposer / AddressSpaceApplier and served by OtOpcUaNodeManager (which registers both URIs; V3NodeIds + AddressSpaceRealm are the identity authority):

Namespace URI	Realm	Subtree	NodeId s= scheme
https://zb.com/otopcua/raw	AddressSpaceRealm.Raw	the /raw device tree (Folder → Driver → Device → TagGroup → Tag)	the node's RawPath (e.g. Plant/Modbus/dev1/Speed)
https://zb.com/otopcua/uns	AddressSpaceRealm.Uns	the UNS tree (Area → Line → Equipment → signal)	slash-joined Area/Line/Equipment/EffectiveName

• Single source, fanned to both. Every device value has exactly one source — the raw tag's node. A UnsTagReference projects that raw tag into an equipment as a UNS-namespace variable that carries an Organizes reference to its raw node and mirrors it. One driver publish for a RawPath fans (in DriverHostActor) to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp — the mux key stays single (RawPath).
• Writes via either NodeId route to the same driver ref under the same WriteOperate gating (the node-manager write gate is realm-qualified and fails closed); a failed device write reverts both NodeIds via the shared fan-out (write-outcome self-correction).
• Native alarms materialize once at the raw tag (ConditionId = RawPath) and fan via SDK notifiers to the raw device folder AND every referencing equipment folder — one ReportEvent, one Server-object copy; ack/confirm/shelve route on ConditionId = RawPath.
• Historian dual-registration. Both NodeIds register the same historian tagname; the mux HistorizedTagRef stays single (RawPath). HistoryRead works through either NodeId.

The retired EquipmentNodeIds ({equipmentId}/{folderPath}/{name}) scheme and the single-namespace EquipmentTags materialization path are gone. See docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md + docs/plans/2026-07-15-v3-batch4-address-space-plan.md, and docs/Uns.md.

How a deploy reaches that address space (surgical-deploys, 2026-08-15). The old blanket rule — ANY non-empty ChangedRawTags / ChangedUnsReferenceVariables routes to Rebuild — is gone, because it made a routine edit a fleet-wide outage: RawTagPlan.ReferencingEquipmentPaths is part of the record's equality, so adding or dropping one UnsTagReference flipped its backing raw tag to Changed and tore down every node on the server, killing every client subscription. AddressSpaceChangeClassifier is still default-closed; what changed is which deltas it whitelists.

• Still a full rebuild: an alarm-bearing raw-tag delta (Alarm != null on either side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an identity change on a paired raw-tag delta (TagId/DriverInstanceId/ParentNodeId/Realm — pairing is by RawPath, so this guard catches a tag deleted and re-created under the same path; a plain rename never pairs and is served surgically as remove-one-node + add-one-node with any referencing UNS mirror re-pointed, live-gated 2026-08-15), a raw container (Folder/Driver/Device/TagGroup) rename or removal, ChangedEquipment / ChangedAlarms, a non-surgical equipment-tag delta, and anything else the with-equality whitelist does not name (including any FUTURE field).
• Now surgical: a non-alarm raw-tag delta over the in-place-updatable attributes (Writable / IsHistorized / HistorianTagname / DataType / IsArray / ArrayLength) and/or ReferencingEquipmentPaths — applied via ISurgicalAddressSpaceSink.UpdateTagAttributes in the Raw realm — and every UnsReferenceDelta (add / remove / re-point / backing-edit).
• ⚠️ UNS mirrors repair through the RECORD, not the classifier. A raw-side historian or shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape (EnsureVariable early-returns on an existing node). UnsReferenceVariable therefore carries the backing tag's Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}, so the same edit makes the reference record unequal and the planner emits a coincident UnsReferenceDelta. Adding a newly-inherited field needs no classifier change — omitting one silently reopens the staleness. Repair is in place (UpdateTagAttributes, Uns realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an identity change (re-point, effective-name edit) is remove+recreate — and only for that one node.
• A non-rebuild apply materialises the DELTA, not the composition. MaterialiseHierarchy still runs full (parents must exist); the other passes take the plan's added/changed subsets. Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread.
• ⚠️ A degraded non-rebuild apply does NOT advance _lastApplied. The full passes used to re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the delta path removed that. Holding the composition back is the replacement — the next deploy re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct, bounded resync). A rebuild still advances even with failures.
• Any surgical false/throw still ratchets to SafeRebuild() — that one-way fallback is what makes every whitelist above safe.
• Announcements: AnnounceAddedNodes announces surgically-added Raw containers/tags (ComputeRawAddAnnouncements, #599 — the former "raw adds are undiscoverable until a re-browse" gap is closed) as well as added UNS reference variables.
• Array writability rides the same whitelist: ArrayWritable joins the surgical raw-tag attributes and UnsReferenceVariable carries BackingArrayWritable (see "Equipment-node writes" under Authentication). Removing an equipment that has references removes the references before the equipment subtree, and RemoveEquipmentSubtree severs the inverse Organizes edges.

See docs/plans/2026-08-15-surgical-deploys.md, docs/Uns.md and docs/Raw.md.

Key Concept: Tag Name and FullName

Galaxy objects have a tag_name — a globally unique system name used for MXAccess read/write — and attributes are referenced as tag_name.AttributeName (e.g. DelmiaReceiver_001.DownloadPath). This dot-separated reference is what TagConfig.FullName stores for a Galaxy equipment tag. The Galaxy address picker browses the live hierarchy using contained names for navigation, then resolves the selection to the tag_name.AttributeName form.

Galaxy is a standard Equipment-kind driver. Galaxy points are ordinary equipment Tags bound to GalaxyMxGateway via TagConfig.FullName (tag_name.AttributeName), authored through the standard Tag modal + Galaxy address picker on the equipment page's Tags tab — the same flow as Modbus or S7. There is no alias machinery, no SystemPlatform namespace kind, and no relay→alias converter. See docs/plans/2026-06-12-galaxy-standard-driver-design.md for the full design.

Data Type Mapping

Galaxy mx_data_type values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs.

Change Detection

DeployWatcher (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs) polls the gateway's deploy-event signal and raises IRediscoverable.OnRediscoveryNeeded when the Galaxy redeploys.

The signal is consumed as an operator prompt, not as an address-space rebuild (§8.2, 2026-07-27). DriverInstanceActor subscribes in PreStart and unsubscribes in PostStop — the detach is load-bearing, because the IDriver instance outlives the actor when the host respawns a child around the same driver object. A raise is recorded and rides the driver-health snapshot (DriverHealthChanged.RediscoveryNeededUtc / .RediscoveryReason) to the AdminUI /hosts page as a "re-browse" chip.

⚠️ It is advisory: a Galaxy redeploy still does NOT change the served address space. v3 authors raw tags explicitly through the /raw browse-commit flow, so an operator must re-browse the device and commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact records.

Two gotchas if you touch this:

• PublishHealthSnapshot dedups on a health fingerprint. The rediscovery timestamp must stay in that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the operator. Pinned by DriverInstanceActorRediscoverySignalTests.
• The shared test StubDriver.GetHealth() returns DateTime.UtcNow, so its fingerprint differs on every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests use a stub with stable health for exactly this reason.

#507 is closed as superseded, and the injection path is deleted. Its retained code read EquipmentNode.DriverInstanceId and EquipmentTags, both structurally empty in v3 (AddressSpaceComposer.cs, DeploymentArtifact.cs), so removing the guard would have changed a log line and injected nothing. Gone with it: HandleDiscoveredNodes, PartitionDiscoveredByDeviceHost, ApplyDiscoveredPlansForDriver, the redeploy re-inject tail, DiscoveredNodeMapper, AddressSpaceApplier.MaterialiseDiscoveredNodes, the connect-time discovery loop (StartDiscovery/RediscoverTick/DiscoveredNodesReady/TriggerRediscovery), and the 18 DiscoveryInjectionDormantV3 tests — v2 characterization tests asserting an equipment-rooted graft (EquipmentRootNodeId == "EQ-1") that v3 cannot produce. ITagDiscovery itself stays: the /raw browse picker drives it through Commons/Browsing/DiscoveryDriverBrowser, which never went through the actor.

IRediscoverable is the ONLY tag-change signal, and four drivers do not raise it. A periodic tag-set drift check briefly existed (shipped e77c8a35, removed by Gitea #523): every SupportsOnlineDiscovery driver was re-browsed on a 5-minute timer and the result diffed against its authored tags. It was taken back out because it generated recurring, unconfigurable (interval was a constructor parameter with no appsettings key), never-live-gated outbound traffic to real PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the design treated it as an implementation detail.

⚠️ Consequence to know before you rely on rediscovery: AbCip and FOCAS browse a live backend but implement no IRediscoverable, so they have no tag-change signal at all — a PLC program re-download or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT, MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an event-driven signal, or one that runs when something else already indicates change (a reconnect, a deploy) — not a wall-clock timer. Removed with the feature: TagSetDrift/TagSetDriftDetector, DriverInstanceActor's drift timer, and ITagDiscovery.AuthoredDiscoveryRefs / DiscoveryStreamIncludesAuthoredTags.

IHostConnectivityProbe is consumed (Gitea #521, closed). DriverInstanceActor subscribes OnHostStatusChanged, pulls GetHostStatuses() fresh, and the host set rides DriverHealthChanged to the AdminUI /hosts page — no SQL: the DriverHostStatus table is dropped, and so is DriverInstanceResilienceStatus (#524; the live data goes over the driver-resilience-status telemetry channel only). Multi-host dispatch is wired too: DriverInstanceActor resolves the breaker/host key per call via IPerCallHostResolver (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip).

Comms-loss detection: GetHealth() is load-bearing (Gitea #527, closed)

Until #527, DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed initial connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely — publishing Bad for every tag, never re-initialising, and not recovering when the device returned (live-confirmed: a Modbus driver against a stopped fixture needed a node restart). PublishHealthSnapshot read GetHealth() for reporting only.

The Connected health tick now runs a watchdog that self-sends DisconnectObserved when a driver reports a non-Healthy state and its LastSuccessfulRead has been frozen for 90 s. Things to know before touching it:

• ⚠️ The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal. Degraded means connected but some data is bad, and several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely. LastSuccessfulRead is the honest evidence — preserved across Degraded/Faulted writes, advanced by any success. Each conjunct has a test that goes red when the other is deleted.

• The mechanism rests on every driver reaching a non-Healthy state on sustained failure. Surveyed across all 12: nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip — correct, as it has no transport. Re-check this if a new driver lands.

• The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way. ModbusDriver (_transport) and MqttDriver (_connection) used to overwrite an IAsyncDisposable field inside InitializeAsync without disposing the old one, so re-initialising without ShutdownAsync leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose first: ModbusDriver.InitializeCoreAsync releases _transport and the _probeCts/_reprobeCts probe loops before rebuilding them (c3970a2e + the lifecycle gate in 409a58b6), deliberately not via the shared TeardownAsync() because that would also drop _poll and every active subscription with nothing to re-drive them; MqttDriver.InitializeCoreAsync calls its (idempotent) TeardownAsync() first, which reaches _connection and the primary-host connection (2b4b3fcb). The path is no longer latent — the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new IAsyncDisposable field on these two, must repeat the dispose-before-overwrite.

• The threshold is hardcoded (DefaultCommsLossTimeout), matching HealthPollInterval / DefaultReconnectInterval. Unlike the #523 drift detector this generates no traffic on a healthy system — it is a local GetHealth() read, and only reaches the network once a driver has already stopped reading.

• What comms loss now does downstream — one contract, six seams (archreview R3-02). "A value consumer never reports a quality better than its source's" is stated once in docs/DriverLifecycle.md §"The quality contract" and implemented by Core.Abstractions.InputQuality (WorstOf/IsUsable). Three things changed with it, all client-visible:

  • ⚠️ Value nodes of a comms-lost driver now go Bad. DriverHostActor.FanCommsLossToValueNodes stamps BadNoCommunication on the raw NodeId and every referencing UNS NodeId, value retained server-side, via the new status-only sink primitive IOpcUaAddressSpaceSink.WriteStatuses (the host holds no value cache, so only the node can keep the value). ⚠️ Retention is not client-observable: the SDK's NodeState.ReadAttribute nulls the value of any Bad-status read/sample, so clients see Value = null + the Bad status; the retained value is what the next Good re-publish and the (value, status) dedup compare against. #481 had wired comms loss to the mux and to Part 9 conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop the tag" (ScadaBridge) will see more Bads — that is the fix working.
  • Nothing Good is written on reconnect — recovery is the driver's own re-publish (the poll engine's forceRaise, MQTT's new initial push). Instead the host audits: refs still latched 30 s after a reconnect log a Warning and increment otopcua.driver.comms_latched_refs. Non-zero on that counter means a driver reconnects without re-publishing.
  • Every dedup on the recovery path is keyed on (value, status), not value. A device that comes back usually comes back holding the same reading, so a value-only dedup swallows the recovery and the node stays Bad forever — the v3-gate trap, now closed in the Calculation driver, the virtual-tag actor and (via a connection-loss stale) MQTT's plain-mode suppression. Any new consumer of driver values must repeat this; a regression test for it must replay the IDENTICAL value or it proves nothing.
  • Consequences per seam: MQTT/Sparkplug stale on connection loss and push cached snapshots on every (re)subscribe (docs/drivers/Mqtt.md); the Calculation driver and virtual tags hold while any input is Bad and inherit the worst input's sub-code, evaluate through Uncertain, and hand scripts the input's real StatusCode/SourceTimestampUtc (docs/drivers/Calculation.md, docs/ScriptEditor.md) — so a script that relied on receiving null for a dead input no longer runs at all while it is Bad.

See docs/DriverLifecycle.md §"The comms-loss watchdog" and §"The quality contract".

mxaccessgw

The gateway lives in a sibling repo at ~/Desktop/MxAccessGateway/ (mxaccessgw — see Related Projects). See docs/v2/Galaxy.ParityRig.md for the gw setup recipe (build, API key provisioning via apikey create-key, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its gateway.md) is the canonical MxAccess API doc; the standalone mxaccess_documentation.md previously kept in this repo retired in PR 7.3.

Build Commands

dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test ZB.MOM.WW.OtOpcUa.slnx                                # all tests
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests               # a single test project
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod"    # a single test

CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native SQLitePCLRaw.lib.e_sqlite3) is resolved — the temporary NuGetAuditSuppress was removed once the patched bundle shipped. Core.AlarmHistorian now carries a surgical direct pin of SQLitePCLRaw.bundle_e_sqlite3 to 2.1.12 (the first fixed native bundle, embedding the SQLite 3.50.2+ patch), overriding the 2.1.11 that Microsoft.Data.Sqlite pulls transitively. dotnet restore/build is clean with the audit active.

Test projects live under tests/<module>/ (Core, Server, Drivers, Drivers/Cli, Client, Tooling) — there is no single unit-test project. Unit suites are named *.Tests; integration suites are *.IntegrationTests and need their Docker fixture up (see Docker Workflow). DB-backed tests in *.Configuration.Tests, *.Admin.Tests, and *.Server.Tests require the central SQL Server.

Docker Workflow (driver fixtures + central SQL Server)

Infra index: infra/README.md is the canonical index of all testing + deployment infrastructure — the Docker host layout, driver fixtures, the env-gated live-gate recipes (S7 blackhole, HistorianGateway LiveIntegration), the integration-test harness, the docker-dev rig, and the deploy setup. Start there; the sections below + docs/v2/dev-environment.md carry the detail.

Migrated 2026-04-28: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (DOCKER, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in appsettings.json defaults, fixture-class default endpoints, and e2e-config.sample.json were rewritten to target 10.100.0.35. See docs/v2/dev-environment.md for the full rewrite (header dated 2026-04-28).

⚠️ The project: lmxopcua label is aspirational, not universal. This note used to claim every fixture compose file carries it. As of 2026-07-24 only the MQTT fixture actually does — grep -rn "labels:" tests --include=*.yml matches Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml and nothing else. So docker ps --filter label=project=lmxopcua returns the MQTT fixture alone, not the fleet. Add the label when you touch an older fixture; until then enumerate by container name.

Docker workloads run on a shared Linux host at 10.100.0.35 — not on this VM. Stacks live at /opt/otopcua-<driver>/ on the host and carry the project=lmxopcua label so they're discoverable via docker ps --filter label=project=lmxopcua.

docker -H ssh://... does NOT work from the Windows VM. Windows OpenSSH ↔ docker.exe stdio bridging hangs (docker system dial-stdio runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs docker compose server-side.

On macOS there is no lmxopcua-fix — drive the host over SSH directly. The helper is a Windows-VM script; ~/bin is empty on the Mac and the commands below will not resolve. Use passwordless SSH (and rsync in place of sync), which is what infra/README.md §1 documents as the Mac path:

ssh dohertj2@10.100.0.35 'docker ps --format "{{.Names}}\t{{.Status}}"'
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mqtt && docker compose up -d'
rsync -av tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/ \
    dohertj2@10.100.0.35:/opt/otopcua-mqtt/          # the `sync` step, by hand

A local container runtime does exist on the Mac (OrbStack), so the docker-dev/ rig itself runs locally there — it is only the shared fixtures that live on 10.100.0.35.

Use lmxopcua-fix.ps1 (in ~/bin) to control fixtures from the Windows VM:

lmxopcua-fix ls                            # list all lmxopcua-tagged containers on the host
lmxopcua-fix up   modbus      standard     # bring a profile up
lmxopcua-fix up   abcip       controllogix
lmxopcua-fix up   s7          s7_1500
lmxopcua-fix up   opcuaclient                # single-service stack, no profile arg
lmxopcua-fix down modbus                   # tear stack down
lmxopcua-fix logs modbus
lmxopcua-fix sync modbus                   # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/

sync is the deployment step. When you edit a fixture's compose file or Dockerfile under tests/.../Docker/, run lmxopcua-fix sync <driver> to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; /opt/otopcua-<driver>/ is a mirrored deployment.

Endpoints (defaults already point at the docker host):

• SQL Server (always-on): 10.100.0.35,14330 — used by appsettings.json for ConfigDb.
• Modbus: 10.100.0.35:5020 (MODBUS_SIM_ENDPOINT)
• AB CIP: 10.100.0.35:44818 (AB_SERVER_ENDPOINT)
• S7: 10.100.0.35:1102 (S7_SIM_ENDPOINT)
• OPC UA reference (opc-plc): opc.tcp://10.100.0.35:50000 (OPCUA_SIM_ENDPOINT)
• MQTT (Mosquitto, auth on both listeners — no anonymous fallback): 10.100.0.35:8883 TLS+auth (MQTT_FIXTURE_ENDPOINT) · 10.100.0.35:1883 plaintext-but-authenticated (MQTT_FIXTURE_PLAIN_ENDPOINT). Also needs MQTT_FIXTURE_USERNAME / MQTT_FIXTURE_PASSWORD, and MQTT_FIXTURE_CA_CERT to pin the fixture CA (scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt). A co-running publisher loops JSON under otopcua/fixture/….
• MQTT Sparkplug B: the same broker, plus the sparkplug compose profile's otopcua-sparkplug-sim (project-owned C# edge-node simulator) — group OtOpcUaSim, edge nodes EdgeA/EdgeB, EdgeA device Filler1, ~2 s cadence. It subscribes to spBv1.0/OtOpcUaSim/NCMD/{node} and re-births on request. Births are never retained, so docker restart otopcua-sparkplug-sim is how you force a fresh one while a browse window is open.

Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at opc.tcp://localhost:4840 — that's not on the docker host.

Local docker-dev rig — login is DISABLED, so do live /run verification yourself (don't wait for the user to sign in). The local docker-dev/docker-compose.yml stack (AdminUI at http://localhost:9200 via Traefik; OPC UA opc.tcp://localhost:4840 central-1 / :4841 central-2) runs the AdminUI with Security__Auth__DisableLogin: "true" — no sign-in form; it's auto-authenticated as a full-access admin. So AdminUI / Razor /run verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against http://localhost:9200) does not require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA data-plane auth is still real LDAP against the shared GLAuth on 10.100.0.35:3893 — that only gates Client.CLI read/write role operations, e.g. binding a multi-role / opc-writeop user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the ZB.MOM.WW.HistorianGateway sidecar — still need the user.)

See docs/v2/dev-environment.md for the full inventory and rationale.

Build & Runtime Constraints

• Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint is owned by the mxaccessgw worker (x86 net48), not by anything in this repo.
• The gateway's MXAccess worker requires a deployed ArchestrA Platform on the machine running the gateway. The OtOpcUa server itself does not.

Transport Security

The server supports configurable OPC UA transport security via the OpcUa:EnabledSecurityProfiles list in appsettings.json. Phase 1 profiles (the OpcUaSecurityProfile enum members): None (default), Basic256Sha256Sign, Basic256Sha256SignAndEncrypt. Security policies are built from the enabled profiles by BuildSecurityPolicies at startup. The server certificate is always created even for None-only deployments because UserName token encryption depends on it. See docs/security.md for the full guide.

Redundancy

The server supports non-transparent warm/hot redundancy via the Redundancy section in appsettings.json. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct MxAccess.ClientName values) but have unique ApplicationUri values. Each exposes RedundancySupport, ServerUriArray, and a dynamic ServiceLevel based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See docs/Redundancy.md for the full guide.

Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.

• Downing is auto-down, not keep-oldest. Cluster:SplitBrainResolverStrategy (default auto-down) selects the provider via ServiceCollectionExtensions.BuildDowningHocon; an unknown value fails host start. A two-node pair running the previous keep-oldest + down-if-alone could not survive a crash of the oldest node: Akka's KeepOldest.OldestDecision only lets down-if-alone rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs itself and exits. down-if-alone is a 3+-node feature. The accepted trade for auto-down is dual-active during a genuine network partition. Its live gate is CLOSED — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (c50ebcf7; docs/plans/2026-07-24-mesh-phase7-failover-drills.md).
• The Primary is the oldest Up driver member, not the role leader. RedundancyStateActor previously used ClusterState.RoleLeader("driver") (lowest address), while ClusterSingletonManager places singletons on the oldest. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary.

A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice. Even under auto-down, a node cold-starting while its peer was dead never came Up: Akka runs FirstSeedNodeProcess — the only bootstrap path that can form a new cluster when no peer answers InitJoin — exclusively when seed-nodes[0] is the node’s own address; every other node runs JoinSeedNodeProcess and retries InitJoin forever. Cluster:SeedNodes is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second (docker-dev central-2 was swapped; site nodes list only central-1 and are legitimately not seeds). AkkaClusterOptionsValidator (AddValidatedOptions/ValidateOnStart in AddOtOpcUaCluster) enforces it at boot — conditionally (self must be entry 0 only if self is in the list, or every site node would refuse to start), matching on PublicHostname+Port, never the 0.0.0.0 bind address. The earlier fix — a Cluster:SelfFormAfter timer calling Cluster.Join(SelfAddress) — is deleted: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by SelfFirstSeedBootstrapTests (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + AkkaClusterOptionsValidatorTests. See docs/Redundancy.md §"Bootstrap: self-first seed ordering".

Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, Cluster:BootstrapGuard:Enabled, default OFF). Self-first on BOTH nodes means when both cold-start at the same instant, each runs FirstSeedNodeProcess, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically lower host:port node is the preferred founder (self-first, forms immediately); the higher node TCP-probes the partner's Akka port up to PartnerProbeSeconds (25s) — reachable ⇒ peer-first (join the founder), unreachable ⇒ self-first (partner dead, form alone). The order is chosen BEFORE the single JoinSeedNodes, from an explicit reachability signal — it never re-forms mid-handshake (the SelfFormAfter failure mode). When on, Akka gets NO config seeds (BuildClusterOptions empties ClusterOptions.SeedNodes) and ClusterBootstrapCoordinator drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: site-a = guard demo (guard on, serialization removed); site-b = A/B control (guard off, depends_on: service_healthy serialization). The guard is the production-faithful fix (compose depends_on doesn't exist on real VMs). Pinned by ClusterBootstrapGuardTests + ClusterBootstrapCoordinatorTests (real 2-node clusters, incl. the higher-node-cold-start-alone case). See docs/Redundancy.md §"Bootstrap guard".

Assert the effective cluster config off a running ActorSystem (as SplitBrainResolverActivationTests does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.

RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster. The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.

Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one. The single fleet-wide Akka mesh was split into one mesh per application Cluster: the central pair (admin,driver,cluster-MAIN), site-a (driver,cluster-SITE-A), site-b (driver,cluster-SITE-B) — same ActorSystem name, separated purely by per-pair self-first seeds (each node's Cluster:SeedNodes lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:

• The redundancy singleton is cluster-scoped and spawned per driver node. RedundancyStateActor is now a cluster-{ClusterId}-scoped singleton (falls back to plain driver for a legacy node), so each pair elects and publishes its own Primary on its own mesh's DPS. Two or more Primaries exist fleet-wide, one per pair — by design, not a bug. ⚠️ Wiring landmine (fixed 3a4ed8dd, caught by the live gate): the singleton's cluster-role scope is computed at AddAkka-configurator time, so it must come from AkkaClusterOptions (config) — never by resolving IClusterRoleInfo from the SP there. ClusterRoleInfo depends on the ActorSystem, and that lambda runs while the ActorSystem is being built, so resolving it recurses into ActorSystem construction and stack-overflows every node at boot (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep WithOtOpcUaClusterRedundancySingleton/BuildClusterRedundancySingletonOptions taking AkkaClusterOptions.
• ClusterNodeAddressReconciler is own-cluster-scoped — an admin node can only reconcile ClusterNode rows in its own cluster; it has no gossip visibility into another mesh.
• CentralCommunicationActor dials each node directly over gRPC — one NodeCommandService.Execute call per enabled ClusterNode row carrying a GrpcPort — rather than fanning through a client/receptionist object. Superseded by the ClusterClient→gRPC migration: the Akka ClusterClient this bullet originally described (one per application Cluster, closing the TODO(Phase 6) Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
• SplitTopologyTransportValidator fails host start on a node carrying a cluster-{ClusterId} role unless it also runs the split-safe transports: MeshTransport:Mode=Grpc, Telemetry:Mode=Grpc (driver), TelemetryDial:Mode=Grpc (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
• The compiled transport-mode defaults were deliberately NOT flipped — MeshTransport:Mode/Telemetry:Mode/TelemetryDial:Mode all still default to Dps; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
• Secrets replication is pair-local — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).

See docs/Redundancy.md §"Per-cluster meshes (Phase 6)", docs/Configuration.md §"Per-cluster mesh split (Phase 6)", and docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.

Role authority (EffectiveRoles)

A node's roles are resolved in exactly one place (archreview R3-04, closing 01/S-12 = 03/C5 and 01/C-9): EffectiveRoles.Resolve (ZB.MOM.WW.OtOpcUa.Cluster), with one precedence — Cluster:Roles when non-empty, else OTOPCUA_ROLES, else empty — and two-layer normalization: a plain role (admin/driver/dev) is trimmed and lower-cased, but a cluster role's ClusterId suffix (cluster-SITE-A) is kept verbatim, because lower-casing it would rename the Akka role of a running deployment. An unknown role from either source fails host start. Program.cs resolves one EffectiveRoles instance at the top of the composition root and registers it as a singleton; that SAME instance drives both the conditional host wiring (hasAdmin/hasDriver) and AkkaClusterOptions.Roles (what Akka gossips) — there is no second derivation for the two to disagree about, which is why no runtime "do the views agree" assert exists. Before this, Program.cs read OTOPCUA_ROLES only, so a node configured solely through Cluster:Roles gossiped the right Akka roles while Program.cs's own wiring mounted none of them — the inverse of the pre-#532 field incident (docs/deployments/wonder-app-vd03-makino-z-34184.md).

• A role overlay may not itself set Cluster:Roles. Roles are resolved before the per-role appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing Cluster:Roles would leave the bound options disagreeing with the registered instance. AkkaClusterOptionsValidator fails host start if this ever happens, naming both views — a no-op today since none of the four shipped overlays carries a Cluster section.
• Both sources set to different normalized role sets logs a Warning at boot (Cluster:Roles still wins outright) — the boot banner also names which source won (source=Configured / Environment / None).
• RoleParser.GossipedRolesHave (case-insensitive) is defence in depth at the handful of production sites that read LIVE GOSSIP directly (Member.Roles) rather than through EffectiveRoles — a peer running an older, pre-R3-04 build can still gossip a miscased plain role during a rolling upgrade. ⚠️ Accepted residual (#620), not closed by R3-04: RedundancyStateActor's election delegates to ZB.MOM.WW.Health.Akka's ClusterActiveNode.OldestUpMember, which matches via Akka's OWN ordinal Member.HasRole. Deliberately not patched — neither in the library nor with a local case-insensitive pre-filter, which would split the Primary election from ClusterSingletonManager placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade window; finish the upgrade. Full rationale: docs/Redundancy.md ("Accepted residual (#620)").
• Program.cs wraps the composition root (#618): a start-up failure logs Fatal, writes FATAL: to stderr (the bootstrap logger is still silent before AddZbSerilog), flushes, and Environment.Exit(1)s — leftover Akka/Kestrel threads would otherwise keep the process alive. HostAbortedException (design-time tooling) passes through untouched.
• RoleAuthorityGuardTests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/) is the enforcement: a source scan failing on any production file outside EffectiveRoles.cs/RoleParser.cs that reads OTOPCUA_ROLES, Cluster:Roles as code, or declares a bare "admin"/"driver"/"dev" literal — with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI RBAC role).

See docs/Configuration.md §Roles, docs/ServiceHosting.md §"Role gating", and archreview/plans/R3-04-one-role-authority-plan.md.

Mesh command transport (MeshTransport)

The Akka ClusterClient transport described in this section's earlier history is DELETED (branch feat/grpc-phase3-deletion, ClusterClient→gRPC migration, Phase 3) — code, the akka.cluster.client HOCON block, the receptionist extension, MeshTransport:CentralContactPoints, and the ClusterClient mode value itself, which now fails validation loudly at boot (MeshTransportOptionsValidator names the key and the allowed values) rather than silently falling back to Dps. What remains is exactly two transports, selected by MeshTransport:Mode:

• Dps (default) — DistributedPubSub, the path this repo has always used. Pair-local only: it works only when central and the node share a gossip ring, which after Phase 6 means within one 2-node mesh.
• Grpc — unary gRPC calls on the mesh_transport.v1 contract. Required on any node carrying a cluster-{ClusterId} role; SplitTopologyTransportValidator refuses to boot such a node on anything else.

Central→node commands (deployments, driver-control, alarm-commands) ride NodeCommandService.Execute, served on each node's driver-role telemetry h2c listener (Telemetry:GrpcListenPort, the docker-dev rig's :4056 — it shares that listener with TelemetryStreamGrpcService rather than claiming a port of its own). CentralCommunicationActor dials it directly, one call per node — the target set is every enabled, non-maintenance ClusterNode row carrying a GrpcPort, dialled at Host+GrpcPort and refreshed every MeshTransport:ContactRefreshSeconds. There is no client/receptionist object and no rotation: each command goes to the specific node dialled, which is what makes per-node delivery provable — the old ClusterClient.SendToAll/Send distinction this section used to explain no longer applies to either direction.

Node→central deploy acks ride CentralCommandService.ReportApply, served on central's admin-role config-serve h2c listener (ConfigServe:GrpcListenPort, the rig's :4055 — again sharing rather than claiming a new port). GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and stops at the first success.

Auth is a per-cluster PSK bearer, both directions: MeshTransport:GrpcPsk (node-side, one key per application Cluster) and MeshTransport:ClusterPsks (central-side map, consulted before the Secrets store). MeshCommandAuthInterceptor (node) and CentralCommandAuthInterceptor (central) fail closed on every unresolvable case — constant-time compare, no key material logged. Production keys are ${secret:OT-MESH-PSK-<clusterId>}; the docker-dev rig supplies three literal dev keys via env (dev-mesh-psk-MAIN/-SITE-A/-SITE-B).

Things that carried over unchanged from the ClusterClient era, because they were never ClusterClient-specific to begin with:

• Fire-and-forget, no retry, on both transports. A failed gRPC dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery. The old buffer-size = 0 decision this section used to describe is gone with the HOCON block that set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
• Inbound commands re-emit on the node's EventStream, not back onto DPS. DPS is mesh-wide, so a DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (DriverHostActor, ScriptedAlarmHostActor) accept a command from either transport identically.
• Do not over-claim what fire-and-forget buys. The Phase 2 live gate observed a node applying a deployment 44 s after it was sealed TimedOut anyway, via DriverHostActor.Bootstrap() replaying the orphan Applying row on restart. That backstop is unaffected by the ClusterClient deletion — it was always about the deploy path, not the transport.

redundancy-state stays on DPS in both modes (bidirectional, and built from Cluster.State, which makes it genuinely mesh-bound). See docs/Configuration.md §MeshTransport, docs/Redundancy.md §"Command transport", and the migration's design doc / plan / tracking doc / live gate for the phase-by-phase history — those are historical records, not updated further by this sweep. docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md is the original Phase 2 design doc and is likewise left as a historical record of the transport this section now describes as deleted.

Phase 1 landed 2026-07-22 and changed a deploy-path behaviour. ConfigPublishCoordinator sources its expected-ack set from enabled ClusterNode rows, not from Akka.Cluster.State.Members filtered by the driver role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:

• A configured node that is switched off now FAILS the deployment at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the new ClusterNode.MaintenanceMode column, NOT Enabled = 0: the live gate showed Enabled = 0 is rejected outright by DraftValidator.ValidateClusterTopology (ClusterEnabledNodeCountMismatch — enabled count must equal ServerCluster.NodeCount), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. Enabled = part of the declared topology; MaintenanceMode = do not expect it now. The deadline log names the silent nodes and points at the right flag.
• Every ClusterNode row is now defined to be a driver node. The DB has no per-node role column and deliberately did not gain one (it would drift from Cluster:Roles). Giving an admin-only node a ClusterNode row makes every deploy time out.
• There is no such thing as a cluster-scoped deployment. Deployment has no ClusterId, ConfigComposer always snapshots the whole DB, and DeploymentArtifact.ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
• ClusterNode gained AkkaPort (NOT NULL, 4053) + GrpcPort (nullable) as central's dial targets for Phases 2/5. They duplicate the node's own Cluster:Port / Cluster:PublicHostname with no schema-level enforcement, so ClusterNodeAddressReconcilerActor (admin singleton) logs an Error on drift. Phase 2 must revisit it — once the meshes split, an admin node cannot see site members and every site row would report EnabledRowNotInCluster forever. Resolved, and not the way this bullet originally expected: the ClusterClient→gRPC migration deleted the Akka ClusterClient transport that would have dialled Host+AkkaPort; the Phase 2 command dial now uses Host+GrpcPort, the same column Phase 5 telemetry already dialled — AkkaPort is Akka remoting only again.

Config source (ConfigSource / ConfigServe)

Per-cluster mesh Phase 3 (config fetch-and-cache) added a second way a driver node obtains its deployed configuration, selected by ConfigSource:Mode: Direct (read Deployment.ArtifactBlob from central SQL — the default, byte-for-byte today's behaviour) or FetchAndCache (fetch the artifact bytes from central over a dedicated-h2c gRPC stream, verify SHA-256 == RevisionHash, cache in LocalDb, and read only the cache — no central-SQL config read). Central (admin role) serves via DeploymentArtifactService behind ConfigServeAuthInterceptor on ConfigServe:GrpcListenPort, gated by a shared node bearer key (ConfigServe:ApiKey == ConfigSource:ApiKey, FixedTimeEquals, fail-closed). Things worth knowing before touching it:

• The serve side is wired in both modes (mapped whenever ConfigServe:GrpcListenPort > 0 on an admin node), so flipping a node to FetchAndCache is a config change, not a redeploy — the same dark-switch discipline as MeshTransport.
• RevisionHash == SHA-256(artifact bytes) is load-bearing. ConfigComposer computes the revision hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the RevisionHash the dispatch already carries — no hash rides in the proto, and DispatchDeployment stays payload-free. A future change to how the revision hash is derived silently breaks every fetch. The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes — ConfigComposer.Pack gzips (CompressionLevel.Fastest) immediately after SerializeToUtf8Bytes and hashes the result, so the invariant holds unchanged and nothing between the composer and DeploymentArtifact.Open (SQL ArtifactBlob, the LocalDb cache chunks, the gRPC fetch stream) ever looks inside. Open sniffs the gzip magic 1F 8B — valid JSON can never start with 0x1F — so pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every decompression failure surfaces as JsonException, the same family the malformed-JSON paths already handle. Two operational consequences in docs/Configuration.md §ConfigSource: the first deploy after upgrade re-hashes every revision (a one-time full re-dispatch, not a no-op), and nodes must be upgraded BEFORE central starts emitting compressed blobs.
• A null fetch is "no answer," never an empty config (#485). All endpoints down / NotFound / SHA mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a FetchAndCache node restores served state from the LocalDb pointer with no central-SQL read; an empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale. Malformed non-empty bytes are "no answer" too — on cache boot, the FetchAndCache boot reconcile and the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does not adopt the revision (Direct ⇒ Stale; FetchAndCache ⇒ no revision, the re-dispatch fetches).
• Both pair nodes fetch independently — no Primary gating of the fetch (same shape as both reading SQL today; StoreAsync is idempotent and the cache replicates).
• The dedicated h2c listener coexists with the LocalDb-sync listener. A fused admin+driver node with both ports set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel Listen*-discards-ASPNETCORE_URLS trap the LocalDb block already warned about).
• Phase boundary: config reads only. A FetchAndCache node still writes its NodeDeploymentState ack row to central SQL; DbHealthProbe and EfAlarmConditionStateStore still touch SQL. Cutting those is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.

On the docker-dev rig the central pair carries ConfigServe (:4055, committed dev key) and stays hardcoded Direct via its admin role; the four site nodes carry ConfigSource (endpoints + matching key) hardcoded FetchAndCache — mandatory for driver-only nodes since Phase 4, not a flip. There is no OTOPCUA_CONFIG_MODE env var in the compose file. See docs/Configuration.md §ConfigSource / ConfigServe, docs/Redundancy.md §"Command transport", and docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md.

Live telemetry transport (Telemetry / TelemetryDial)

Per-cluster mesh Phase 5 (merged 35552a96; live gate PASSED f134935f) added a second transport for the four live-observability channels the AdminUI's /alerts, /script-log, and /hosts panels feed on, selected by Telemetry:Mode (node/serve side) and TelemetryDial:Mode (central/dial side): Dps — the four existing DPS SignalR bridges, the default, today's behaviour — or Grpc, one gRPC server-streaming contract carrying all four channels as oneof event kinds. The direction is inverted from ConfigSource/MeshTransport: the driver node hosts the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in — mirroring ScadaBridge's SiteStreamService shape, and required so telemetry survives once the meshes split (Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before touching it:

• Migrated (4 channels): alerts (AlarmTransitionEvent), script-logs (ScriptLogEntry), driver-health (DriverHealthChanged), driver-resilience-status (DriverResilienceStatusChanged) — feeding the exact same central sinks (IInProcessBroadcaster<...>, IDriverStatusSnapshotStore, IDriverResilienceStatusStore) in either mode.
• Deferred, with reasons (do not read "seven observability topics" from the program sketch as the delivered scope): redundancy-state stays on DPS in every mode (pair-local, built from Cluster.State, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an observability panel); deployment-acks already rides the Phase 2 gRPC command transport (MeshTransport:Mode=Grpc, CentralCommandService.ReportApply) as a command-plane reply, not telemetry. A third, fleet-status, was DELETED rather than deferred (2026-08-06, Gitea #542): FleetStatusBroadcaster → DPS → SignalR bridge → FleetStatusHub had zero clients end to end (no browser JS, no HubConnection — /fleet polls NodeDeploymentState in the Config DB), its DriverHostStatusHeartbeat input had no production sender, and Phase 6 left its "fleet" as just central's own pair. Fleet visibility is /fleet + the four per-node telemetry dials above.
• The node ALWAYS hosts and ALWAYS double-publishes, in both modes. Every publish seam emits into a node-local ITelemetryLocalHub and still publishes DPS, unconditionally; the gRPC server binds whenever Telemetry:GrpcListenPort > 0, independent of Telemetry:Mode. Only TelemetryDial:Mode on central actually switches anything: which of the two always-available sources (DPS bridges vs. the TelemetryDialSupervisor dialer) feeds the sinks. This is what makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as MeshTransport and ConfigSource.
• Authenticated from day one, fail-closed — a shared node bearer key (Telemetry:ApiKey == TelemetryDial:ApiKey), gated by TelemetryStreamAuthInterceptor (CryptographicOperations.FixedTimeEquals, PermissionDenied on mismatch, an unset key rejects every call). This supersedes design doc §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
• Reconnect story: central runs one reconnecting dialer per driver node (discovered from ClusterNode.Host/GrpcPort, refreshed every TelemetryDial:ContactRefreshSeconds), immediate first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event is ignored, and it never permanently gives up (observability, not data plane). TelemetryLocalHub.Evict drops a removed driver's snapshots (StopChild, via plan.ToStop; central-side stores are not evicted). The node-local hub replays last-value snapshots for driver-health/driver-resilience-status on every (re)subscribe so those stores re-prime immediately after a reconnect; alerts/script-logs are append logs and tolerate the gap. The /alerts//script-log pill reflects aggregate stream health (live when ≥1 node stream is up) — the same semantics the DPS pill has always had.

See docs/Telemetry.md for the full architecture, docs/Configuration.md §Telemetry/TelemetryDial for the config keys, and docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md for the implementation plan and live gate procedure.

LocalDb pair-local store (Phases 1 + 2)

Every driver-role node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database (the retired LiteDB LocalCache subsystem was deleted as superseded). It holds three replicated tables: deployment_artifacts + deployment_pointer (Phase 1) and alarm_sf_events (Phase 2).

Phase 1 caches the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention per cluster) so a driver node can boot from cache when central SQL Server is unreachable — DriverHostActor writes the cache after each successful apply (IDeploymentArtifactCache → LocalDbDeploymentArtifactCache, in Runtime/Deployment/) and reads it as a boot fallback, flagging running-from-cache. Wired by AddOtOpcUaLocalDb in the hasDriver branch only (Host/Configuration/LocalDbRegistration.cs + LocalDbSetup.OnReady, whose DDL→RegisterReplicated order is load-bearing); admin-only nodes never register it. The cache can optionally replicate to the node's redundant pair peer over the library's gRPC sync — default-OFF, fail-closed bearer auth (LocalDbSyncAuthInterceptor), gated on LocalDb:SyncListenPort (a dedicated Kestrel h2c listener) + LocalDb:Replication:*. On the docker-dev rig the site-a pair has replication on; central + site-b stay default-OFF (site-b is the pin). ApiKey via ${secret:}/env, never committed (the rig's dev key is the committed-dev-secret exception).

Phase 2 moved the alarm store-and-forward buffer off its standalone alarm-historian.db and into the same database as alarm_sf_events, so a node that dies holding undelivered alarm history no longer takes it to the grave. SqliteStoreAndForwardSink became LocalDbStoreAndForwardSink (Core.AlarmHistorian), and the breaking config key AlarmHistorian:DatabasePath was removed — it is still read, by AlarmSfLegacyMigrator, purely to find a pre-consolidation file to copy across on first boot (renamed .migrated after the copy commits). Because the buffer replicates, only the node holding the Primary role drains it: the sink takes a drainGate delegate, Runtime supplies it from the IRedundancyRoleView singleton that DriverHostActor publishes its PrimaryGatePolicy verdict to, and a gated-off node reports the new HistorianDrainState.NotPrimary. Delivery is therefore at-least-once across a failover, by design (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the same event accepted on both nodes — which happens in every boot window, since HistorianAdapterActor default-writes while its role is unknown — converges to one row. On the docker-dev rig the site-a pair and site-b both run AlarmHistorian__Enabled=true against a deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.

The library pin is 0.3.4 (Directory.Packages.props, both ZB.MOM.WW.LocalDb and .Replication), bumped from 0.2.1 on 2026-09-03 (was itself bumped from 0.1.3 on 2026-08-15). The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3 upgrades in place on open, so no rig or volume action was needed. RegisterReplicated(string) is unchanged and everything 0.2.x/0.3.x added is additive, except that RegisterReplicated (0.3.3) now throws LocalDbRegistrationException for a table that is the parent of a row-mutating ON DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire. 0.3.0 made the snapshot-resync flag per-table (see the ⚠️ below, now corrected); 0.3.2 added validated config Replication:KeepAlivePingDelay/KeepAlivePingTimeout (defaults 60s/20s, left unset here) and a public LocalDbOplogGapException, neither yet wired to a call site. Its two new APIs — DeregisterReplicated and RegisterReplicated(table, baselineExistingRows:) — are now both used (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two things are worth knowing:

• MaxBatchSize no longer means what the rig's 16 was chosen to mean. 0.2.0 made batching byte-budgeted — LocalDb:Replication:MaxBatchBytes (2 MB default, left unset here, sized under the 4 MB gRPC cap) bounds the wire message. MaxBatchSize is demoted to a secondary row cap and the DB read-page size in OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materialise a whole page in memory before the byte split runs. So the site-a pair keeps 16 — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" was corrected with the bump; do not restore that rationale.
• Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18). LocalDbSetup.OnReady reads one predicate, LocalDbReplicationPolicy.CapturesChanges (via LocalDbSetup.ReplicationIsConfigured): LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey is non-blank. The OR is not belt-and-braces — only the dialling half of a pair sets PeerAddress, so keying on it alone would strip capture from every passive node (site-a-2 carries ApiKey + LocalDb:SyncListenPort and nothing else) and the pair would converge in one direction only, silently. ⚠️ SyncListenPort deliberately does NOT join the OR: the passive endpoint is fail-closed on the key, so a port without an ApiKey rejects every inbound stream and has nothing to capture for.
  • ⚠️ The alarm DRAIN gate uses a different predicate — LocalDbReplicationPolicy.SharesRowsWithPeer: ApiKey and (PeerAddress or LocalDb:SyncListenPort > 0), i.e. "a peer actually holds these rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no peer must still drain, so the two predicates are deliberately not the same. The passive half (no PeerAddress) cannot name its peer, so it stands down for its mesh partner (the other member of its cluster-{ClusterId} role); a legacy passive half with no mesh role still drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the published Primary verdict (and the live data-plane gate, which must stay the same decision) allows — for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the Sparkplug primary-host claim with its Primary peer.
  • Configured ⇒ RegisterReplicated(table, baselineExistingRows: true), per table, in the same load-bearing DDL→register order. Baselining is what makes enabling replication on a node that has been running without it actually converge: capture is CDC and the snapshot streamer pages from __localdb_row_version, so pre-registration rows were in neither and a late opt-in used to replicate only writes made after the restart. It seeds the ledger at the LWW floor (HLC 0, this node's id) and flags a resync — a baselined row loses to any genuine remote write of the same key and wins only where the peer has no version at all. It is idempotent, so it is free on every boot after the first. As of 0.3.0 (this repo pins 0.3.4) the snapshot flag is per-table, not per-database — baselining one table no longer streams every registered table in both directions. (Prior to 0.3.0, on the 0.2.1 pin this repo carried until 2026-09-03, the flag was per-database and baselining anything streamed every registered table in full; the old advice to trim alarm_sf_events before enabling replication no longer applies at the current pin.)
  • Not configured ⇒ DeregisterReplicated(table) per table, an idempotent boot-time self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and prunes that table's oplog + row-version rows. Merely declining to register would leave every pre-2026-08-18 file capturing forever — two INSERTs plus a json_object of the whole row inside every write transaction — on precisely the node that has no peer to capture for. The call returns true when it removed something, so the Information line fires once, and only when it actually cleaned; a default-OFF node's ordinary boot stays silent. The logger is Serilog's static Log.Logger (this runs inside the AddZbLocalDb singleton factory, where resolving one from DI would nest a resolution inside a singleton construction; Program.cs assigns it before anything resolves ILocalDb), with an optional parameter so tests observe the line rather than infer it.
  • ⚠️ PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change. The sync handshake compares registered-table digests fail-closed, so a node that deregisters a table its peer still replicates stops syncing with a schema mismatch. Enabling replication on an established pair means setting the config and restarting both nodes — do it on both, or on neither.
  • Rollout note: the only replicating pair in the fleet is docker-dev site-a, and it was enabled from a fresh pair, so the impact of the old gap was low — central and site-b are default-OFF and simply shed their unused triggers on their next boot. LocalDbSetupTests pins both branches (registered-table set on a configured node, empty set + zero triggers + the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.

See docs/Configuration.md (LocalDb section), docs/Redundancy.md (pair-local config cache), docs/AlarmHistorian.md, and the runbook docs/operations/2026-07-20-localdb-pair-replication.md.

Phase 4 — cut the driver-side ConfigDb connection

Per-cluster mesh Phase 4 finished the job Phase 3 started: a driver-only node (Cluster:Roles has driver, not admin) now boots with no ConfigDb connection string at all. Program.cs gates AddOtOpcUaConfigDb (and its health check) on hasAdmin; a fused admin,driver node is unaffected (it keeps ConfigDb via its admin role). Consequences:

• ConfigSource:Mode must be FetchAndCache on a driver-only node — ConfigSourceOptionsValidator fails host start if such a node is left on Direct (there is nothing to read Direct from). A fused node may stay Direct.
• ServiceLevel semantics change — client-visible. With no ConfigDb, DbHealthProbeActor is not spawned (ServiceCollectionExtensions.WithOtOpcUaRuntimeActors skips it when the resolved db factory is null), and OpcUaPublishActor runs dbLess: DbReachable is fed constant true and Stale comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes 240 (or 250 as driver Primary) even with central SQL unreachable — the survive-alone posture — where a Direct/DB-backed node in the same state drops to 100/0. ServiceLevelCalculator itself is unchanged; only the inputs a DB-less node feeds it changed. See docs/Redundancy.md §"DB-less (driver-only) nodes".
• Scripted-alarm Part 9 condition state moved to LocalDb. LocalDbAlarmConditionStateStore (replicated alarm_condition_state table, pair-local like the Phase-2 alarm S&F buffer) replaces EfAlarmConditionStateStore on every driver-role node, fused central included. The DB-backed ScriptedAlarmState table was dropped (Task 9, 3a590a0c, migration 20260723183050_DropScriptedAlarmStateTable); EfAlarmConditionStateStore is deleted.
• Central persists deploy acks. DriverHostActor no longer writes NodeDeploymentState to SQL on a driver-only node — ConfigPublishCoordinator.PersistNodeAck (already fed by every ApplyAck) is the system of record.
• LDAP group→role DB grants don't reach driver-only nodes. They register the empty NullLdapGroupRoleMappingService (no ConfigDb to read LdapGroupRoleMapping from), so OtOpcUaGroupRoleMapper falls back to the Security:Ldap:GroupToRole appsettings baseline only — the same rows a driver node never received via config either, so this is not a regression.

Merged to master (26fad75c); live gate PASSED (1281aebf — driver nodes run ConfigDb-free) and Task 9 landed (3a590a0c). See docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md and docs/plans/2026-07-22-per-cluster-mesh-program.md §Phase 4.

Authentication: API keys on the data plane, LDAP on AdminUI (#520)

⚠️ Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all. It is machine-to-machine, so a client presents an OPC UA UserName token with the fixed username apikey and an otua_<keyId>_<secret> credential as the password. ApiKeyOpcUaUserAuthenticator (src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/) implements IOpcUaUserAuthenticator: one indexed lookup on keyId plus one peppered HMAC-SHA256, FixedTimeEquals, no network I/O. The key resolves to a principal; NodeAcl.Principal rows grant it permissions. LdapOpcUaUserAuthenticator is deleted, along with four Security:Ldap keys that existed only for it (AuthTimeoutMs, MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds) — they bounded an LDAP bind running inline in the SDK's synchronous, SessionManager-lock-holding impersonation callback, and nothing on the remaining LDAP path has that property.

LDAP remains, for AdminUI login only (Security:Ldap, OtOpcUaLdapAuthService → ILdapAuthService, group→role mapping via GroupToRole + LdapGroupRoleMapping). Do not "restore" the data-plane authenticator when you see LDAP wiring in Program.cs — it is there for the AdminUI.

Two things to know before relying on any of this:

• ⚠️ Node ACLs are enforced only behind Security:Acl:Enabled (default false). With the flag off, a key's grants collapse into the two server-wide role gates (WriteOperate on Value writes, AlarmAck on Part 9 methods), only Cluster-scoped grants count — ApiKeyInterimRoleMapper deliberately under-grants, because honouring a narrower grant's flags while unable to honour its scope would turn "WriteOperate on Equipment X" into server-wide write authority — and read, browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous. With the flag on (DataPlaneAuthorizer), scoped grants decide every data-plane operation: write and alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6).
  • The two role gates re-check the session's KEY on every call, flag off or on (Gitea #563). Roles are resolved once at activation, so an expired / deployment-disabled key used to keep WriteOperate / AlarmAck on its open session until the client disconnected. DataPlaneAuthorizer.AuthorizeKeyUsable runs the same FindUsableKey probe against the same snapshot + clock as the ACL path (one dictionary lookup, no I/O) and denies BadUserAccessDenied before the role check. Only those two gates — reads/subscriptions stay ungated with the flag off, by design. Anonymous carries no key and is unaffected. No flag: a dead key keeping write authority is a defect, not a rollout choice. ⚠️ On a fused node a key minted after the last deploy authenticates (ConfigDb) but cannot write/ack until a deployment carries it — the flag-on path always behaved this way.
  • ⚠️ The ACL decision REPLACES the role check — the two are never chained. Chained, a grant would have to confer the server-wide role as well as pass the scope check, and only Cluster-scoped grants confer a role, so every narrow grant would stay inert and the flag would appear to do nothing. The load-bearing test is therefore an allow — a session carrying no roles writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too.
  • The trie is built inside DataPlaneAuthorityState.Apply, not by a second wiring at the publish actor, and the state owns its PermissionTrieCache rather than being handed one. An uninstalled trie denies everything, which is indistinguishable from working enforcement until someone checks that an ALLOW is possible at all.
  • A deploy reaches sessions that are already open, in both directions: each decision stamps the current generation rather than the one the session activated under, and re-checks the authenticating key — so revoking a key or a grant stops an open session immediately instead of at the client's next reconnect. On a driver-only node the artifact is the only revocation channel.
  • UserAuthorizationState is built per decision walk, not per session — and since the hot-path change it is allocated only on a cache miss, so a cache hit never constructs one. Its LDAP-shaped freshness/staleness predicates are switched off EXPLICITLY (AuthCacheMaxStaleness = MaxValue). ⚠️ Relying on "the stamp is always fresh" was not safe: MembershipResolvedUtc comes from DataPlaneAuthorizer's TimeProvider while IsStale is evaluated against TriePermissionEvaluator's, so any divergence between the two clocks denies every request with a verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead.
  • Stage 6's hot seams (read/browse/subscribe) go through a per-(session, node, operation, realm) decision cache, cleared wholesale when the generation advances. It is sharded per API key (ApiKeyId ?? Principal — despite the type name, sessions sharing one key share one shard and one budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and not every other key's decisions. ⚠️ It caches the grant walk only — key revocation and expiry are re-checked before the cache lookup on every call, because expiry moves with the clock rather than with a deployment.
  • Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors. A grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward only, so PermissionTrieNode.BrowseTransitPrincipals marks the path. Planting a real Browse grant on each ancestor instead would be collected by the additive downward walk and leak Browse onto every sibling subtree beneath that ancestor. ⚠️ The topmost ancestor is the single shared root folder (s=OtOpcUa, raw namespace) that BOTH realm trees hang under — its identifier is a display name, not a path, so DataPlaneAuthorizer.BuildScope maps it onto the trie's zero-segment realm-root scope (V3NodeIds.SharedRootFolderId, realm-qualified: in the UNS namespace that identifier is an ordinary Area), and its Browse considers transit in either realm because it fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse gate denies the whole call when its source node is denied, so clients were stopped at the front door.
  • Revocation reaches live subscriptions (stage 6): every other operation re-asks per request, but a subscription is authorized once and then streams — and streaming is how machine consumers actually read this server. DataPlaneAuthorityState.Changed fires RevalidateMonitoredItems. ⚠️ A denied item is queued BadUserAccessDenied and then detached from its MonitoredNode2 fan-out — NOT set MonitoringMode.Disabled: disabling discards the item's queued notifications, so the client is silenced without ever being told why (caught only by the over-the-wire test). Detaching is also reversible, so restoring the grant resumes the feed with no client action. ⚠️ Changed is no longer the only trigger (arch-review 03/S19): AclRevalidationTimer, armed by OtOpcUaServerHostedService at Security:Acl:RevalidationIntervalSeconds (default 60 s, 0 disables, inert while Security:Acl:Enabled is false), runs the same pass on the clock — because ExpiresUtc passes with the clock, not with a deployment. The pass's verdict memo is pass-local and must stay that way; hoisting it across ticks would serve an expiry that landed between two ticks the earlier tick's allow. Event/alarm items ride both triggers too (Gitea #562) — a second map (_eventItems), fed by the SubscribeToEvents override, holds every direct notifier subscription; a denied one is detached via the SDK's own protected SubscribeToEvents(…, unsubscribe: true) (what a client delete does) and re-attached with unsubscribe: false on restore — again never Disabled. Unlike a data item it is not told (no value slot for a status; the feed goes quiet), and a client delete of a stood-down item is absorbed by the override (the SDK would otherwise remove an item that is not attached). Server-object (i=2253) event subscriptions are filtered per event at delivery (Gitea #586). The SubscribeToAllEvents override attaches them to nothing (just records them in _allEventsItems); each condition's own OnReportEvent (OnConditionReportEvent, fires once per event ⇒ still one Server-object copy) queues the event to each recorded item whose session holds Subscribe on the condition (RawPath) or any notifier folder it reports through — the per-folder path's question. The root-notifier forward to the core Server object (OnReportEvent override) is dropped: it was the ungated path, and it is the only one that served an item created before the alarm was deployed (the SDK's AddRootNotifier subscribes existing all-events items with unsubscribe: true). Revalidation is implicit — every event re-asks the authorizer, so a deploy or expiry lands on the next event with no pass. ConditionRefresh on an i=2253 item is filtered the same way. The dispatcher must keep the SDK's audit guard (AuditEventState only while Server.Auditing over SignAndEncrypt/https), because conditions report their Part 9 method audit events through themselves. ReconcileAllEventsMonitoring must keep AreEventsMonitored raised on every condition while any i=2253 item exists, because ConditionState gates its method audit reports on EventsMonitored() (method-driven state-change events no longer depend on it — #587, below). The protected SubscribeToEvents override that no-ops all-events items is what stops the SDK's AddRootNotifier (unsubscribe: true bug) taking that unit back. Both review findings (review of 75e559f6). ⚠️ The materialise path raises only the new condition's unit (RaiseAllEventsMonitoring); a full reconcile there made a deploy of N alarms O(N²) under Lock with ScadaBridge connected — keep the full reconcile to remove/rebuild/(un)subscribe. Model-change and audit events are gated too (Gitea #588). The manager's own server-wide events (GeneralModelChangeEvent, the rejected-write AuditWriteUpdateEvent) go through ReportToServerObject: flag off ⇒ Server.ReportEvent as before; flag on ⇒ delivered per item to _allEventsItems (same audience), a model change narrowed to the Affected nodes the session may Browse (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral one), an audit write only with Read on its source node plus the SDK's audit/encryption guard. Any other event type fails closed. Auditing is off by default (never configured in this repo).
• ns=0 standard nodes stay ungated by design, even with Security:Acl:Enabled (#617) — ServerStatus, NamespaceArray, ServiceLevel etc. are served by the SDK's core node manager outside our authorizer and clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node, event subscription and HistoryRead is still gated. docs/security.md.
• Equipment-node writes: EvaluateEquipmentWriteStructure is the ONLY type check. OnWriteValue bypasses the SDK's own type check on every writable equipment node, so OtOpcUaNodeManager.EvaluateEquipmentWriteStructure (plus EvaluateEquipmentWriteRange) is the whole gate. Scalars are deliberately lenient (drivers range-check; ScadaBridge sends Double to a Float node). Array writes (#610): a numeric array of another element type is converted, checked, element by element to the node's element type (ScadaBridge writes CLR int[]/double[] lists) — any out-of-range, fractional-to-integer or NaN/∞ element refuses the whole write BadTypeMismatch with nothing sent, so a UInt32 > Int32.MaxValue can never reach a Galaxy Int32 attribute as reinterpreted bits; non-numeric mismatches are refused (Byte[] ≈ ByteString); wrong length ⇒ BadTypeMismatch; an array to a scalar node is refused only on Modbus/S7 (EnsureVariable's refuseArrayOnScalar — pass-through drivers keep their browse-committed scalar-shaped array tags writable; a new sink wrapper must forward that argument, and the reflection guard checks only that the call arrives); IndexRange writes ⇒ BadWriteNotSupported, ValueRank > 1 out of scope. An array node is writable only when the driver can write the array (DriverArrayWriteSupport: Modbus bounded by maxRegistersPerWrite/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types with a declared arrayLength — an undeclared length stays read-only because a short whole-array write would resize the Galaxy attribute); arrayLength >= 1 is required, and the node's Writable && (!IsArray || ArrayWritable) is projected onto the raw node and every UNS mirror.
• Anonymous is a normal grantable principal, not a bypass. OpcUa:AllowAnonymous (default true, preserving today's behaviour) controls admission; when admitted the session authenticates as the reserved principal anonymous and gets roles from NodeAcl rows naming it, arriving at the gates as the same RoleCarryingUserIdentity a key produces. Set it false to make keys mandatory. A key may not use the anonymous principal, and a failure to read anonymous's grants denies rather than admitting with no roles.
• Keys reach a driver-only node inside the deployment artifact (stage 4). ConfigComposer snapshots the ApiKey rows — the peppered HMAC only, never a secret, and the pepper is node config that is deliberately not in the artifact — and DeploymentArtifact.ParseDataPlaneAuthority decodes them node-side alongside the cluster's NodeAcl rows. OpcUaPublishActor installs both on IDataPlaneAuthoritySource on every apply, before its plan.IsEmpty early-return, so a deployment that only revokes a key still lands. A node with a ConfigDb keeps ConfigDbApiKeyStore (per-activation read, wrapped — see below); a driver-only node gets ArtifactApiKeyStore. ⚠️ The asymmetry is ACTIVATION-scoped, not session-scoped — this line used to say "immediate on a ConfigDb node" and that was wrong. Clearing Enabled in the ConfigDb stops the next session activation on an admin/fused node (bounded by Security:ApiKeys:StoreCacheSeconds, default 5 s) and takes until the next deployment on a driver-only one — but an already-open session re-checks its key against the in-memory artifact snapshot (DataPlaneAuthorizer.Evaluate → snapshot.FindUsableKey), never against IApiKeyStore, so on both node kinds an Enabled revocation reaches an open session only at the next deployment. Expiry is the exception: FindUsableKey compares ExpiresUtc against the authorizer's clock on every call, so it is immediate everywhere — and since arch-review 03/S19 it reaches live subscriptions too, on the host's Security:Acl:RevalidationIntervalSeconds timer (default 60 s; the session is silenced with BadUserAccessDenied, not closed, and recovers on re-activation). Direct event/alarm subscriptions are revalidated on both triggers as well (#562); with the flag off, expiry and revocation reach the two role gates (#563) but not reads or subscriptions.
• ⚠️ The activation lookup is BOUNDED (arch-review 03/S18). The SDK raises impersonation synchronously under a SessionManager-wide lock, so a ConfigDb that is slow rather than refusing stalled every session activation on the server, Anonymous and X.509 included. BoundedApiKeyStore wraps the ConfigDb store — and only it, never ArtifactApiKeyStore, which is two in-memory dictionary probes — with Security:ApiKeys:StoreTimeoutMs (2000; exceeding it denies, via the authenticator's existing catch-all) plus a StoreCacheSeconds (5) read-through cache. Faults and timeouts are never cached, a cached hit whose ExpiresUtc has passed is re-read rather than served, and a timeout logs a distinct Warning + otopcua.opcua.apikey_store.bounded{outcome=timeout}. Both knobs at 0 restore the old behaviour exactly.
• ⚠️ The auth generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs with nothing logged. Pinned by DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases.
• Note OtOpcUaServerHostedService takes IOpcUaUserAuthenticator and IDataPlaneAuthoritySource as required ctor parameters — the built-in DI container does not fall back to a default value for an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a node with neither role) or it does not boot.

Grants are authored through a scope picker, not free text (stage 7). /clusters/{id}/acls opens AclScopePickerModal over the cluster's raw + UNS trees, and selecting a node supplies ScopeKind and ScopeId together — so RawDevice + a TagId, a cross-cluster scope, a Cluster grant carrying an id, and the retired FolderSegment kind are all unauthorable by construction rather than merely rejected later. ⚠️ A Cluster row must produce a NULL ScopeId — the deploy gate rejects one that carries an id (NodeAclScopeIdUnexpected), so a picker that helpfully filled in the cluster id would author rows that cannot be deployed. IAclScopeService.CheckScopeAsync mirrors DraftValidator.ValidateNodeAclScopes rule for rule and blocks the save when a scope resolves to nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches the picker); the list page marks such rows unresolved. UNS tags are deliberately not offered — a UNS signal is a UnsTagReference, and a Tag grant already covers the raw node and every UNS projection of it. The picker is AclScopeTree, a separate component from RawTree, which already carries two modes.

All eight stages are done (d4389f57 enforcement, stage 7 authoring after it), each live-gated on docker-dev. The rationale for every decision above is in docs/plans/2026-07-31-apikey-acl-enforcement-design.md. See docs/security.md for the full guide.

Dev/test LDAP is the shared GLAuth running on the Linux Docker host at 10.100.0.35:3893 (baseDN dc=zb,dc=local, plaintext/Transport=None). It is managed via scadaproj/infra/glauth/ (source of truth + deploy runbook). Single bind account cn=serviceaccount,dc=zb,dc=local / serviceaccount123; all test users password password. The docker-dev compose binds this shared instance directly — DevStubMode is no longer used. The per-VM NSSM GLAuth at C:\publish\glauth\ and the old base DNs dc=lmxopcua,dc=local / dc=otopcua,dc=local are obsolete. (The integration-test harness under tests/.../Host.IntegrationTests/ runs its own ephemeral GLAuth container on port 3894 — seeded by tests/.../Host.IntegrationTests/glauth/config.toml with dev users alice/bob + the cn=serviceaccount bind account — for the opt-in OTOPCUA_HARNESS_USE_LDAP=1 real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired bitnami/openldap:2.6 in PR #451, 2026-07-15.)

AB drivers: native transport (Transport)

AbCip and AbLegacy have a second, managed wire transport (Driver.Ab.Transport) beside libplctag, chosen by the driver-config key transport = Native | Libplctag (string; takes effect when the driver is constructed — every process start, or a deploy that recreates it, not ReinitializeAsync; AdminUI "Transport" dropdown; CLIs --transport). "Default" is encoded by omitting the key; the AdminUI must never re-emit a concrete default. The defaults differ per driver (final review, 2026-10-07):

• AbCip: an absent key resolves to the node default, Native unless pinned. Precedence (one place, AbCipTransportResolver, shared by driver, probe and CLI): explicit per-driver key → node-level appsetting Drivers:AbCip:DefaultTransport (Native | Libplctag, env Drivers__AbCip__DefaultTransport, AbCipDriverDefaultsOptions; a bad value fails host start) → compiled Native. Bound unconditionally because the driver factory AND the admin-node Test Connect probe both read it. ⚠️ So the upgrade restart alone moves no-key AbCip drivers to Native; to stay on libplctag, set the node key — NOT a per-driver key through the old AdminUI, whose typed form drops unknown keys (#576). Evidence: AbCip reads proven on a real CompactLogix (Z-35162, read-only); AbCip writes are ab_server-only.
• AbLegacy: an absent key resolves to Libplctag for every family; Native is opt-in (no node key). On a wire the native PCCC path is proven only for N-file reads/writes and one F8 read, all on ab_server; S:0 (the default probe), B/T/C/R/S/L/ST/A/O/I and the T/C/R sub-element offsets are encoded from the PCCC table and unverified natively. Explicit Native on PLC-5/LogixPccc logs a once-per-instance Warning. AbLegacy native refuses status-bit sub-element writes (T4:0.DN…, BadNotSupported) and its ST byte order is unverified.

Refusals are not comms faults: an Unsupported refusal (BadNotSupported) and an AbCip native type mismatch (BadTypeMismatch, where libplctag returned Good garbage for a same-width mis-authored type) both keep the tag handle, leave driver health alone (no Degraded, nothing for the comms-loss watchdog) and warn once per tag — only genuine transport failures evict and degrade. DeviceSessionPool.Shared logs through the host logger factory (wired in DriverFactoryBootstrap; the FOCAS and TwinCAT factories' Register(registry, ILoggerFactory?) take one too, #602 CONV-3). The AbCip browsePageDelayMs knob has an AdminUI field (AbCipDriverForm "Browse page delay (ms)"). libplctag is confined to Libplctag*.cs adapters by LibplctagConfinementGuardTests; do not reference it elsewhere. See docs/drivers/AbCip.md §Transport, docs/drivers/AbLegacy.md §Transport + §Wire evidence, docs/Configuration.md §Drivers:AbCip, and docs/plans/2026-10-07-native-ab-transport-gate-results.md.

Other driver notes

• S7: connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe tick); a write-side access-denied (PUT/GET disabled) latches a sticky Faulted that outranks later read passes; driver-side array writes cover non-Bool narrow/Int32/Float32 types only — Bool arrays are refused (a one-byte-per-element encode would clobber neighbour bits).
• MQTT: Plain writeTopic writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types BadNotSupported).
• FOCAS / AbLegacy: background loops are joined (≤ 5 s) on shutdown.
• Driver removal (#609): removing a driver releases its capability-invoker pipelines and resilience tracker, then evicts its telemetry snapshots. ⚠️ Both are tombstoned, not just cleared: the removed child stops asynchronously and its in-flight subscribe/connect still completes through the invoker (and emits Health) on the thread pool after StopChild, which used to resurrect the state — so a removed driver's counters were republished forever. Release → tracker Retire and hub Evict ignore every later write for that id until it is spawned again (Create → Reinstate, SpawnChild → Readmit). A Watch/Terminated release would not work — the late writes land after the actor terminates.

Library Preferences

• Logging: Serilog with rolling daily file sink
• Unit tests: xUnit + Shouldly for assertions
• Service hosting (Server, Admin): .NET generic host with AddWindowsService (decision #30 — replaced TopShelf in v2; see src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs)
• OPC UA: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: OPCFoundation.NetStandard.Opc.Ua.Server

OPC UA .NET Standard Documentation

Use the DeepWiki MCP (mcp__deepwiki) to query documentation for the OPC UA .NET Standard stack: https://deepwiki.com/OPCFoundation/UA-.NETStandard. Tools: read_wiki_structure, read_wiki_contents, and ask_question with repo OPCFoundation/UA-.NETStandard.

Testing

Use the Client CLI at src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/ for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See docs/Client.CLI.md for full documentation.

dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500

AdminUI interactive pages must wrap their body in <PageErrorBoundary> — MainLayout is static, so it cannot catch an interactive page's render failure.

Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see docs/plans/2026-05-28-driver-browsers-design.md.

The AdminUI's global UNS page (/uns) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See docs/Uns.md.

v3 (Batch 2): device I/O is authored in the /raw project tree (GlobalRaw.razor → RawTree.razor, IRawTreeService) — a cluster-rooted Folder → Driver → Device → TagGroup → Tag hierarchy with per-node context menus (the reusable ContextMenu), lazy loading, driver/device config modals (endpoint moved to Device.DeviceConfig; Test-connect probes the DriverDeviceConfigMerger-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed /clusters/{id}/drivers flow (DriverTypePicker, DriverEditRouter, the 8 *DriverPage shells, the Drivers list) is retired — its form bodies live on inside the /raw modals. The DriverType dispatch maps are keyed off the DriverTypeNames constants (fixing the TwinCat/Focas drift). A Calculation pseudo-driver (DriverTypeNames.Calculation, IDependencyConsumer, deploy-time scriptId-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See docs/Raw.md.

v3 (Batch 3): UNS Equipment is reference-only. The equipment Tags tab no longer authors or binds tags — it holds UnsTagReference rows pointing at raw tags authored in /raw. "+ Add reference" opens the RawTree in picker mode, cluster-scoped (structurally + server-enforced tag.cluster == equipment.cluster); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. Effective-name uniqueness (references + VirtualTags + ScriptedAlarms share the {EquipmentId}/{EffectiveName} NodeId space) is enforced at authoring (IEffectiveNameGuard) and at the deploy gate (DraftValidator.UnsEffectiveNameCollision — catches rename-induced collisions). Scripts use ctx.GetTag("{{equip}}/<RefName>") — resolved per-equipment through UnsTagReference effective names to the backing RawPath at both compose seams (AddressSpaceComposer + DeploymentArtifact, via the shared EquipmentReferenceMap, byte-parity); an unresolved <RefName> is a deploy error (EquipReferenceUnresolved) AND a live Monaco diagnostic (OTSCRIPT_EQUIPREF) — editor accepts ⇔ publish accepts. EquipmentScriptPaths.DeriveEquipmentBase is deleted (the dot-joint {{equip}}.X became the slash-joint {{equip}}/<RefName>). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (RawTreeService substring scan). ImportEquipmentModal dropped the DriverInstanceId column. See docs/Uns.md + docs/ScriptEditor.md.

v3 (Batch 4 = v3.0): dual-namespace address space. The OPC UA server exposes two namespaces — https://zb.com/otopcua/raw (Raw device tree, s=<RawPath>) + https://zb.com/otopcua/uns (UNS tree, s=<Area>/<Line>/<Equipment>/<EffectiveName>) — replacing the single .../ns and the retired EquipmentNodeIds scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in DriverHostActor to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable Organizes-references its raw node. Writes route via either NodeId to the same driver ref under the same realm-qualified WriteOperate gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (ConditionId = RawPath) and fan via SDK AddNotifier to the raw device folder + every referencing equipment folder (one ReportEvent, one Server-object copy); both NodeIds register the SAME historian tagname (mux HistorizedTagRef stays single, keyed by RawPath). Identity authority: V3NodeIds + AddressSpaceRealm (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, docs/Uns.md, and docs/plans/2026-07-15-v3-batch4-address-space-plan.md.

The /uns TagModal uses driver-typed tag-config editors: it dispatches by the bound driver's DriverType to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via TagConfigEditorMap, with client-side validation via TagConfigValidator; unmapped drivers (only Galaxy) fall back to the generic raw-TagConfig-JSON textarea. Each editor is a thin razor shell over a pure <Driver>TagConfigModel (FromJson/ToJson/Validate, preserves unknown keys). To add a driver's editor, copy the Modbus template under Components/Shared/Uns/TagEditors/ + Uns/TagEditors/, reusing the driver's enums + camelCase JSON property names, and register it in TagConfigEditorMap + TagConfigValidator. See docs/plans/2026-06-09-driver-typed-tag-editors-design.md.

Scripting / Script Editor

C# virtual-tag scripts are authored in a Roslyn-backed Monaco editor on the ScriptEdit page (/scripts/{id}) and inline inside the virtual-tag modal on the /uns page. The editor provides completions, live diagnostics, hover, signature help, document formatting, and tag-path completions inside ctx.GetTag("…") / ctx.SetVirtualTag("…") literals — all backed by the same compiler context the runtime publish gate uses, so what the editor accepts/rejects matches publish exactly. The backend lives in src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/ (six minimal-API endpoints under /api/script-analysis/*, gated by the ConfigEditor policy (Administrator or Designer) via RequireAuthorization in ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints). Core.VirtualTags is now just DependencyGraph, ITagUpstreamSource and IHistoryWriter — the dormant VirtualTagEngine/Source/Definition and TimerTriggerScheduler were deleted. See docs/ScriptEditor.md for the full guide. Scripts may use the {{equip}} token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in docs/ScriptEditor.md.

Scripted Alarm Ack/Shelve

Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the alarm-commands DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in OtOpcUaNodeManager, gated on the AlarmAck role — conferred by the session's API-key principal's NodeAcl grants (or the anonymous principal's), carried by RoleCarryingUserIdentity; LDAP is AdminUI-only; (2) AdminUI /alerts per-row buttons routed through the AdminOperationsActor singleton (gated by DriverOperator policy). ScriptedAlarmHostActor dispatches from the topic to the engine. Client.CLI supports ack, confirm, shelve commands. See docs/ScriptedAlarms.md §"Inbound operator ack/shelve" and docs/AlarmTracking.md.

⚠️ One condition event per accepted method call is OURS, not the SDK's (Gitea #587). The SDK's own report is gated on AutoReportStateChanges (left false), so before #587 a client Acknowledge emitted no condition event — the old "SDK auto-fires its own event (E2)" comments were wrong. WireMethodTransitionReporting wraps every method's SDK handler (Ack/Confirm/AddComment/Enable/Disable/OneShotShelve/TimedShelve/Unshelve) and, on Good, reports once via ReportConditionEvent — under Lock, so the engine's/driver's re-projection (routed by a fire-and-forget Tell before the SDK applies state) waits and then hits the delta-gate with no delta. Things to keep: the SDK's shelve handlers never apply shelving, so the wrapper does; only a real before/after change is reported (a re-ack is silent and restores the Confirmed/Retain the SDK flips); MaxTimeShelved is set to int.MaxValue ms at materialise because Create() left it 0 and every TimedShelve failed. ⚠️ Native alarms support Acknowledge and AddComment only — Confirm, the shelve methods and Enable/Disable return BadNotSupported (no upstream path, no pair replication, and NativeAlarmProjector's Confirmed/Unshelved constants own those fields); a native ack re-asserts ConfirmedState = true because the SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by ConditionMethodEventTests (exactly-once after a sentinel, folder + i=2253).

Behaviours that were wrong or missing and are now fixed — keep them: the SDK's OnTimedUnshelve is a no-op (the engine's AutoUnshelve owns timed-shelve expiry, and UnshelveTime reports the real remaining time from AlarmConditionSnapshot.UnshelveAtUtc); a commented re-ack is stored as an AddComment (durable, replicated) and reports one event, an uncommented re-ack stays silent; a native ack is refused BadInvalidState off-Primary via NativeAlarmAckPrimaryGate (fed by IDriverPrimaryRoleSource; DriverHostActor publishes its Primary verdict at host start and on member changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design).

Historian / HistoryRead

Backend: HistorianGateway (sole historian backend). As of the gateway-integration cutover, the historian read, alarm-write, and continuous-historization paths are all served by the ZB.MOM.WW.HistorianGateway sidecar, consumed as the Gitea-feed ZB.MOM.WW.HistorianGateway.Client gRPC package (historian_gateway.v1) behind a thin IHistorianGatewayClient seam in ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway. The bespoke Wonderware TCP/ArchestrA sidecar projects and the vestigial Historian.Wonderware driver type were retired — there is no Wonderware backend in the tree anymore (see docs/drivers/Historian.Wonderware.md, now a retired stub).

A tag is historized by adding "isHistorized": true to its TagConfig JSON blob (authored in the raw-JSON textarea on the /uns TagModal); an optional "historianTagname" field overrides the default historian tagname, which is the tag's driver FullName.

Read path (ServerHistorian section)

The server dispatches all OPC UA HistoryRead to the registered IHistorianDataSource — the GatewayHistorianDataSource read client when enabled, else the NullHistorianDataSource default (historized nodes return GoodNoData, never an error). Supported variants: Raw, Processed (Average/Minimum/Maximum/Total/Count aggregates), and AtTime over historized variable nodes; Events over alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node); authorization uses the standard AccessLevels.HistoryRead bit set at materialization.

ServerHistorian appsettings keys (ServerHistorianOptions; Enabled defaults to false):

Key	Default	Notes
Enabled	false	true registers the gateway read client; false keeps NullHistorianDataSource
Endpoint	""	Absolute gateway URI, e.g. https://host:5222. Scheme selects transport (https:// = TLS, http:// = h2c)
ApiKey	""	Peppered-HMAC key histgw_<id>_<secret> sent as Authorization: Bearer. Supply via env ServerHistorian__ApiKey — never commit
UseTls	true	Connect over TLS; must match the Endpoint scheme
AllowUntrustedServerCertificate	false	Accept a self-signed / untrusted server cert (dev / on-prem only)
CaCertificatePath	null	PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store
CallTimeout	00:00:30	Per-call deadline for each unary gateway read
MaxTieClusterOverfetch	65536	Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend)
HistoryReadBatchParallelism	4	Max nodes served concurrently within one HistoryRead batch (03/S3); ⌈N/P⌉ × CallTimeout worst case instead of sequential
MaxConcurrentHistoryReadBatches	16	Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to BadTooManyOperations. Total in-flight reads capped at 16 × 4 = 64
HistoryReadDeadline	00:01:00	Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces BadTimeout. Non-positive = unbounded

Alarm-history path (AlarmHistorian section)

Alarm events are written through GatewayAlarmHistorianWriter (the gateway SendEvent path) behind the durable LocalDbStoreAndForwardSink — AlarmHistorian:Enabled=true swaps the NullAlarmHistorianSink default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the replicated alarm_sf_events table (see the LocalDb section) and whose drain worker forwards batches to the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). Only the node holding the Primary role drains (a gated node reports HistorianDrainState.NotPrimary). The AlarmHistorian section carries only the Enabled gate + the queue knobs (DrainIntervalSeconds, Capacity, DeadLetterRetentionDays, BatchSize, MaxAttempts) — the downstream gateway connection (endpoint/key/TLS) is sourced from the ServerHistorian section. Alarm-history ReadEvents requires the target gateway deployed with RuntimeDb:EventReadsEnabled=true (the C2 SQL event-read workaround).

Continuous historization (ContinuousHistorization section)

When ContinuousHistorization:Enabled=true and ServerHistorian is configured, the Host builds a durable, crash-safe FasterLog outbox (FasterLogHistorizationOutbox) + a gateway-backed IHistorianValueWriter, and WithOtOpcUaRuntimeActors spawns the ContinuousHistorizationRecorder. The recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the crash boundary), and drains the outbox to the gateway's SQL live-value write path (WriteLiveValues) with exponential backoff. The gateway connection is sourced from ServerHistorian; this section carries only the recorder + outbox knobs:

Key	Default	Notes
Enabled	false	true (with ServerHistorian configured) wires + spawns the recorder
OutboxPath	"" (required when enabled)	Directory holding the FasterLog segment + commit files. In production set an absolute path on durable storage
CommitMode	PerEntry	PerEntry = fsync before the append returns (no loss window); Periodic = batched commits every CommitIntervalMs. ⚠️ PerEntry now means one fsync per ingest burst, not per value — the recorder appends a burst through AppendBatchAsync, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands
CommitIntervalMs	100	Periodic-mode commit cadence; required > 0 only under Periodic
DrainBatchSize	1024	Entries peeked + written per drain pass. Raised from 64: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs tags × latency — while 64 drew down ≤ 64 entries per DrainIntervalSeconds, letting the outbox fall permanently behind during any gateway hiccup
DrainIntervalSeconds	2	Steady drain cadence (and post-success reschedule)
DrainConcurrency	8	Historian tags one drain pass writes concurrently (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential tags × latency pass becomes ⌈tags / concurrency⌉ × latency with outstanding round-trips bounded. 1 = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass"
Capacity	500000	Max un-acked outbox entries before drop-oldest; 0 = unbounded. Non-zero by default since #538 — the disk backstop for a historian refusing every write, the one case the per-tag quarantine deliberately cannot resolve
MaxWriteAttempts	5	#538: consecutive failed drain passes one tag may accumulate — counted only across passes where other tags wrote successfully — before its queued entries are abandoned so the FIFO head can advance
MinBackoffSeconds	1	Initial retry backoff after a failed drain pass
MaxBackoffSeconds	30	Cap on the exponential retry backoff

Tag auto-provisioning (IHistorianProvisioning EnsureTags hook)

AddressSpaceApplier.Apply() fires a non-blocking, fire-and-forget IHistorianProvisioning.EnsureTagsAsync hook for added historized value tags — and, since #600.1, for changed raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed GatewayTagProvisioner calls the gateway's EnsureTags so a brand-new historized tag exists in the historian before the recorder's WriteLiveValues land. The hook is wrapped so a faulted/synchronously-throwing provisioner can never block or fail a deploy. Non-numeric (String/DateTime/Reference) data types are skipped (not provisioned); the recorder likewise drops + meters non-numeric values. Continuous historization is numeric-analog only in v1 (UInt16→UInt4 is a documented fallback).

The provisioner is registered by the Host's AddHistorianProvisioning (Runtime.ServiceCollectionExtensions), built via GatewayHistorian.CreateProvisioner, and gated on the same ServerHistorian:Enabled flag as the read path — WithOtOpcUaRuntimeActors resolves IHistorianProvisioning and passes it into the applier. When the section is disabled the applier binds the no-op NullHistorianProvisioning and provisioning does not run. Every dispatch logs a tally (dispatched/requested/ensured/skipped/failed); dispatched=N with requested=0 flags the dormant no-op. The API key must carry historian:tags:write for EnsureTags. (PR #423 shipped GatewayTagProvisioner but left this wiring out, so provisioning was dormant; restored 2026-06-27.) DataType mapping (issue #441): EnsureTags is an upsert (create-or-update, not skip-if-exists). HistorianTypeMapper maps Boolean → Int2 (Boolean historizes as a small 0/1 integer) because the historian's Int1 analog-tag creation path is server-degenerate — EnsureTags(Int1) stores an unusable stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see the "Boolean historization maps to Int2" section in docs/Historian.md.

Gateway-side prerequisites

The target HistorianGateway OtOpcUa points at must run with:

• RuntimeDb:Enabled=true — enables the WriteLiveValues SQL live path (continuous historization).
• RuntimeDb:EventReadsEnabled=true — enables ReadEvents from Runtime.dbo.Events (alarm-history reads).
• An API key carrying scopes historian:read, historian:write, historian:tags:write.

Migration note (deployments upgrading from the Wonderware backend)

The ServerHistorian section changed shape. Rename the old Wonderware keys and supply the key via env:

Old (Wonderware) key	New (gateway) key
ServerHistorian:Host + :Port	ServerHistorian:Endpoint (https://host:5222)
ServerHistorian:SharedSecret	ServerHistorian:ApiKey (env ServerHistorian__ApiKey)
ServerHistorian:ServerCertThumbprint	ServerHistorian:CaCertificatePath (+ UseTls / AllowUntrustedServerCertificate)

The AlarmHistorian section's old Wonderware connection keys (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from ServerHistorian. See docs/Historian.md for the full guide.

Live-validation gate — PASSED (was "KNOWN LIMITATION 1")

No longer a limitation. The cutover was live-validated against the real gateway on wonder-sql-vd03: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a confirmed protocol limitation (the captured CM_EVENT wire never carries SourceName, so the gateway cannot populate Source_Object — not a fixable bug). Evidence: docs/plans/2026-06-27-otopcua-historian-followups.md.

Kept here because it is the recipe to re-run the gate after any gateway or backend change (VPN to wonder-sql-vd03, gateway running the prerequisites above):

export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret>             # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag>                         # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag>          # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag>            # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name>                      # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"

The live suite skips cleanly when these env vars are absent (safe to run offline on macOS). It is the gate the operator runs on the VPN before trusting the cutover.

Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04)

The ContinuousHistorizationRecorder is fully wired (actor + FasterLog outbox + gateway value-writer + meters). It is spawned with an empty initial ref set (Array.Empty<string>() in WithOtOpcUaRuntimeActors) because the deployed address space — and thus the historized-ref set — is built later at deploy time, not at actor-spawn time. The ref-feed gap is now closed: on every deploy the AddressSpaceApplier.FeedHistorizedRefs hook computes the added/removed historized-ref delta and posts it through ActorHistorizedTagSubscriptionSink → the recorder's UpdateHistorizedRefs message, and the recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe Tell; a faulting feed can never break a deploy).

The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a production-fatal defect — see docs/plans/2026-08-01-continuous-historization-live-gate.md. Three things to know:

• ⚠️ ConfigureAwait(false) in an Akka handler destroys Self. OnValueChangedAsync touched Self after awaiting IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false), which discards Akka's ActorTaskScheduler synchronization context — so ActorBase.Context was gone and it threw NotSupportedException on the FIRST value. The throw reached supervision, the actor restarted, and because production seeds an empty ref set the restart discarded every historized ref: one value captured, then silence. Fixed by capturing IActorRef self = Self before the await (the idiom the GetStatus handler already used for Sender).
• ⚠️ A fake that completes synchronously hides this entire class of bug, and 13 tests passed because of one. ValueTask.CompletedTask never yields, so the continuation ran inline with the context intact. Worse, the obvious fix to the fake — await Task.Yield() — is also vacuous: Yield() posts back to the captured context, so the caller still resumes on the actor thread. The fake must complete on a pool thread (await Task.Delay(1).ConfigureAwait(false)), as a real per-entry fsync does. Only reverting the production fix exposed that the first regression test proved nothing.
• Both pair nodes historized the same value — the recorder had no Primary gate, unlike the alarm store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as #537 and fixed — see the paragraph below.

Gate COMPLETE — #491 CLOSED 2026-08-04. Outbox append, drain, WriteLiveValues, restart-convergence, EnsureTags provisioning (ensured=2 failed=0) and read-back all ✅. The read-back leg ran against the licensed wonder-sql-vd03 with the VPN up: two registers carrying distinct non-zero constants (8224, 120) each read back as its own value — a 0-valued source would have proved nothing (the #536 trap).

Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison tag); evidence in the same plan doc. Two things to know before touching either.

• #538 — the discriminator is "did another tag write this pass?", NOT a failure count. DrainOnceAsync writes every per-tag group and acks all-or-nothing (deliberate: RemoveAsync truncates through an id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned the FIFO head while every healthy tag was re-submitted on every pass, forever — duplicate samples, an outbox that never truncated (observed 8.3 MB; signature = the gateway's ok and error counters climbing together). The drain now quarantines a tag after MaxWriteAttempts consecutive failures, but ⚠️ only counts a failure on a pass where some other tag succeeded — otherwise an ordinary gateway restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so the load-bearing test is the falsification control (A_backend_that_fails_every_tag_abandons_nothing). Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway.
• #537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink. The alarm queue is in LocalDb and replicates, so the Secondary holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of stale duplicates on promotion). ⚠️ And the peer that matters is one recording the SAME refs, not one holding the same rows — a node stands down only for a Primary in its own per-cluster mesh, and a node with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is usually running another cluster's configuration, and deferring to it ends this cluster's historization silently (the drain reports success on an empty outbox forever) — the same scope mismatch that once suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on RedundancyRoleView behind IHistorizationRoleView, registered unconditionally on driver nodes (NOT alongside the alarm IRedundancyRoleView, which only exists when that section is enabled — the #511 registration accident).

Re-confirmed live in 2026-06 and since fixed (#600.1): EnsureTags provisioning used to fire only for newly-ADDED historized tags, so flipping an existing tag's historianTagname (or historization, or DataType) provisioned nothing (2026-06-27-historian-gateway-integration-issues.md Issue 5). It now also fires for those changed raw tags.