lmxopcua/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Goal

Build an OPC UA server (.NET 10) that exposes industrial data sources — including AVEVA System Platform (Wonderware) Galaxy — under a unified Equipment-based address space. Galaxy access flows through the in-process GalaxyDriver (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/) talking gRPC to a separately installed mxaccessgw gateway process. The gateway owns the MXAccess COM bitness constraint (its worker is x86 net48); everything in this repo is .NET 10. PR 7.2 retired the legacy in-process Galaxy.Host / Galaxy.Proxy / Galaxy.Shared projects + the OtOpcUaGalaxyHost Windows service.

See docs/v2/Galaxy.Performance.md for the runtime perf surface (tracing, metrics, soak harness).

Related Projects (scadaproj umbrella)

This repo is one of a family of related SCADA / OT / Wonderware / OPC UA sister projects that live as sibling directories under ~/Desktop/. They are separate repos and separate processes, coupled at runtime over wire protocols (gRPC + OPC UA) — not by project/compile references — and share the ZB.MOM.WW.* product namespace. The common subject is AVEVA System Platform (Wonderware) "Galaxy" data, and mxaccessgw is the linchpin the others connect through.

• ~/Desktop/scadaproj (../scadaproj/CLAUDE.md) — the umbrella/index workspace that aggregates the whole family (a high-level scan of each sister project: purpose, location, stack, primary commands). It also hosts the shared ZB.MOM.WW.* libraries — ZB.MOM.WW.Auth, ZB.MOM.WW.Theme, ZB.MOM.WW.Health, ZB.MOM.WW.Telemetry, ZB.MOM.WW.Configuration, and ZB.MOM.WW.GalaxyRepository. This repo is indexed there — see the OtOpcUa entry in ../scadaproj/CLAUDE.md.
• ~/Desktop/MxAccessGateway (mxaccessgw, ../MxAccessGateway/CLAUDE.md) — the gRPC gateway that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session x86 net48 worker). OtOpcUa's GalaxyDriver depends on this for live Galaxy read/write/subscribe and Galaxy Repository browse.
• ~/Desktop/HistorianGateway (../HistorianGateway/CLAUDE.md) — single-process gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse (via the shared ZB.MOM.WW.GalaxyRepository lib). Patterned on mxaccessgw but with no COM / no x86 worker. OtOpcUa's historian backend (Driver.Historian.Gateway) consumes it as the sole historian path.
• ~/Desktop/ScadaBridge (../ScadaBridge/CLAUDE.md) — the distributed SCADA platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's OPC UA address space.

Propagate cross-repo changes to the umbrella index. When a fact the index records about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow, shared-lib consumption, or per-project commands — update the OtOpcUa entry in ../scadaproj/CLAUDE.md in the same change so the index never drifts from this repo.

The index edit belongs on scadaproj's main, and must be pushed. scadaproj is a separate repo with its own checkout, so committing there inherits whatever branch it happens to be on — which is usually not main and is often an unrelated feature branch with no upstream. That silently drifts the index for as long as that branch stays unmerged, which is the exact failure the rule above exists to prevent. Do this explicitly:

cd ~/Desktop/scadaproj
git rev-parse --abbrev-ref HEAD            # note it, to restore afterwards
git stash list                             # never commit onto a dirty unrelated branch
git checkout main && git pull
#   …edit the OtOpcUa entry in CLAUDE.md…
git commit -am "docs(index): <what changed about OtOpcUa>"
git push origin main
git checkout -                             # restore the original branch

Observed 2026-07-27: the Sql poll driver's index entry had sat unpushed on an unrelated feature branch since 2026-07-24 because it was committed onto the current checkout, so the index disagreed with reality for three days despite this rule. If you find index commits stranded on a feature branch, cherry-pick them onto main rather than merging that branch.

Architecture Overview

Data Flow

1. Galaxy Repository DB (ZB) — SQL Server database holding the deployed object hierarchy and attribute definitions. The mxaccessgw's GalaxyRepositoryClient queries it via gRPC; the driver consumes the materialised hierarchy through IGalaxyHierarchySource.
2. MXAccess (via mxaccessgw) — Live read/write/subscribe over a gRPC session. The gateway owns the COM apartment + STA pump server-side; the driver speaks MxCommand / MxEvent protos exclusively.
3. OPC UA Server — Exposes the deployed tree under two OPC UA namespaces (see below). Galaxy tags are bound by TagConfig.FullName (tag_name.AttributeName); reads/writes/subscriptions are translated to that reference for MXAccess.

v3 OPC UA Address Space (Batch 4): dual namespace

The server exposes two OPC UA namespaces (replacing the single https://zb.com/otopcua/ns), built by AddressSpaceComposer / AddressSpaceApplier and served by OtOpcUaNodeManager (which registers both URIs; V3NodeIds + AddressSpaceRealm are the identity authority):

Namespace URI	Realm	Subtree	NodeId s= scheme
https://zb.com/otopcua/raw	AddressSpaceRealm.Raw	the /raw device tree (Folder → Driver → Device → TagGroup → Tag)	the node's RawPath (e.g. Plant/Modbus/dev1/Speed)
https://zb.com/otopcua/uns	AddressSpaceRealm.Uns	the UNS tree (Area → Line → Equipment → signal)	slash-joined Area/Line/Equipment/EffectiveName

• Single source, fanned to both. Every device value has exactly one source — the raw tag's node. A UnsTagReference projects that raw tag into an equipment as a UNS-namespace variable that carries an Organizes reference to its raw node and mirrors it. One driver publish for a RawPath fans (in DriverHostActor) to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp — the mux key stays single (RawPath).
• Writes via either NodeId route to the same driver ref under the same WriteOperate gating (the node-manager write gate is realm-qualified and fails closed); a failed device write reverts both NodeIds via the shared fan-out (write-outcome self-correction).
• Native alarms materialize once at the raw tag (ConditionId = RawPath) and fan via SDK notifiers to the raw device folder AND every referencing equipment folder — one ReportEvent, one Server-object copy; ack/confirm/shelve route on ConditionId = RawPath.
• Historian dual-registration. Both NodeIds register the same historian tagname; the mux HistorizedTagRef stays single (RawPath). HistoryRead works through either NodeId.

The retired EquipmentNodeIds ({equipmentId}/{folderPath}/{name}) scheme and the single-namespace EquipmentTags materialization path are gone. See docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md + docs/plans/2026-07-15-v3-batch4-address-space-plan.md, and docs/Uns.md.

How a deploy reaches that address space (surgical-deploys, 2026-08-15). The old blanket rule — ANY non-empty ChangedRawTags / ChangedUnsReferenceVariables routes to Rebuild — is gone, because it made a routine edit a fleet-wide outage: RawTagPlan.ReferencingEquipmentPaths is part of the record's equality, so adding or dropping one UnsTagReference flipped its backing raw tag to Changed and tore down every node on the server, killing every client subscription. AddressSpaceChangeClassifier is still default-closed; what changed is which deltas it whitelists.

• Still a full rebuild: an alarm-bearing raw-tag delta (Alarm != null on either side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an identity change on a paired raw-tag delta (TagId/DriverInstanceId/ParentNodeId/Realm — pairing is by RawPath, so this guard catches a tag deleted and re-created under the same path; a plain rename never pairs and is served surgically as remove-one-node + add-one-node with any referencing UNS mirror re-pointed, live-gated 2026-08-15), a raw container (Folder/Driver/Device/TagGroup) rename or removal, ChangedEquipment / ChangedAlarms, a non-surgical equipment-tag delta, and anything else the with-equality whitelist does not name (including any FUTURE field).
• Now surgical: a non-alarm raw-tag delta over the in-place-updatable attributes (Writable / IsHistorized / HistorianTagname / DataType / IsArray / ArrayLength) and/or ReferencingEquipmentPaths — applied via ISurgicalAddressSpaceSink.UpdateTagAttributes in the Raw realm — and every UnsReferenceDelta (add / remove / re-point / backing-edit).
• ⚠️ UNS mirrors repair through the RECORD, not the classifier. A raw-side historian or shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape (EnsureVariable early-returns on an existing node). UnsReferenceVariable therefore carries the backing tag's Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}, so the same edit makes the reference record unequal and the planner emits a coincident UnsReferenceDelta. Adding a newly-inherited field needs no classifier change — omitting one silently reopens the staleness. Repair is in place (UpdateTagAttributes, Uns realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an identity change (re-point, effective-name edit) is remove+recreate — and only for that one node.
• A non-rebuild apply materialises the DELTA, not the composition. MaterialiseHierarchy still runs full (parents must exist); the other passes take the plan's added/changed subsets. Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread.
• ⚠️ A degraded non-rebuild apply does NOT advance _lastApplied. The full passes used to re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the delta path removed that. Holding the composition back is the replacement — the next deploy re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct, bounded resync). A rebuild still advances even with failures.
• Any surgical false/throw still ratchets to SafeRebuild() — that one-way fallback is what makes every whitelist above safe.
• ⚠️ Known gap, no longer neutral: AnnounceAddedNodes is Uns-only, so surgically-added raw containers/tags are materialised but never announced — previously invisible (the rebuild forced a re-browse anyway), now genuinely undiscoverable until a client re-browses. Added UNS reference variables ARE announced.

See docs/plans/2026-08-15-surgical-deploys.md, docs/Uns.md and docs/Raw.md.

Key Concept: Tag Name and FullName

Galaxy objects have a tag_name — a globally unique system name used for MXAccess read/write — and attributes are referenced as tag_name.AttributeName (e.g. DelmiaReceiver_001.DownloadPath). This dot-separated reference is what TagConfig.FullName stores for a Galaxy equipment tag. The Galaxy address picker browses the live hierarchy using contained names for navigation, then resolves the selection to the tag_name.AttributeName form.

Galaxy is a standard Equipment-kind driver. Galaxy points are ordinary equipment Tags bound to GalaxyMxGateway via TagConfig.FullName (tag_name.AttributeName), authored through the standard Tag modal + Galaxy address picker on the equipment page's Tags tab — the same flow as Modbus or S7. There is no alias machinery, no SystemPlatform namespace kind, and no relay→alias converter. See docs/plans/2026-06-12-galaxy-standard-driver-design.md for the full design.

Data Type Mapping

Galaxy mx_data_type values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs.

Change Detection

DeployWatcher (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs) polls the gateway's deploy-event signal and raises IRediscoverable.OnRediscoveryNeeded when the Galaxy redeploys.

The signal is consumed as an operator prompt, not as an address-space rebuild (§8.2, 2026-07-27). DriverInstanceActor subscribes in PreStart and unsubscribes in PostStop — the detach is load-bearing, because the IDriver instance outlives the actor when the host respawns a child around the same driver object. A raise is recorded and rides the driver-health snapshot (DriverHealthChanged.RediscoveryNeededUtc / .RediscoveryReason) to the AdminUI /hosts page as a "re-browse" chip.

⚠️ It is advisory: a Galaxy redeploy still does NOT change the served address space. v3 authors raw tags explicitly through the /raw browse-commit flow, so an operator must re-browse the device and commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact records.

Two gotchas if you touch this:

• PublishHealthSnapshot dedups on a health fingerprint. The rediscovery timestamp must stay in that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the operator. Pinned by DriverInstanceActorRediscoverySignalTests.
• The shared test StubDriver.GetHealth() returns DateTime.UtcNow, so its fingerprint differs on every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests use a stub with stable health for exactly this reason.

#507 is closed as superseded, and the injection path is deleted. Its retained code read EquipmentNode.DriverInstanceId and EquipmentTags, both structurally empty in v3 (AddressSpaceComposer.cs, DeploymentArtifact.cs), so removing the guard would have changed a log line and injected nothing. Gone with it: HandleDiscoveredNodes, PartitionDiscoveredByDeviceHost, ApplyDiscoveredPlansForDriver, the redeploy re-inject tail, DiscoveredNodeMapper, AddressSpaceApplier.MaterialiseDiscoveredNodes, the connect-time discovery loop (StartDiscovery/RediscoverTick/DiscoveredNodesReady/TriggerRediscovery), and the 18 DiscoveryInjectionDormantV3 tests — v2 characterization tests asserting an equipment-rooted graft (EquipmentRootNodeId == "EQ-1") that v3 cannot produce. ITagDiscovery itself stays: the /raw browse picker drives it through Commons/Browsing/DiscoveryDriverBrowser, which never went through the actor.

IRediscoverable is the ONLY tag-change signal, and four drivers do not raise it. A periodic tag-set drift check briefly existed (shipped e77c8a35, removed by Gitea #523): every SupportsOnlineDiscovery driver was re-browsed on a 5-minute timer and the result diffed against its authored tags. It was taken back out because it generated recurring, unconfigurable (interval was a constructor parameter with no appsettings key), never-live-gated outbound traffic to real PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the design treated it as an implementation detail.

⚠️ Consequence to know before you rely on rediscovery: AbCip and FOCAS browse a live backend but implement no IRediscoverable, so they have no tag-change signal at all — a PLC program re-download or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT, MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an event-driven signal, or one that runs when something else already indicates change (a reconnect, a deploy) — not a wall-clock timer. Removed with the feature: TagSetDrift/TagSetDriftDetector, DriverInstanceActor's drift timer, and ITagDiscovery.AuthoredDiscoveryRefs / DiscoveryStreamIncludesAuthoredTags.

⚠️ IHostConnectivityProbe is still unconsumed. GetHostStatuses() has no production call site and nothing ever writes a DriverHostStatus row (the table was re-created in the v3 initial migration, so confirm intent before deleting). The separable OnHostStatusChanged event is the useful half. Tracked as Gitea #521.

Comms-loss detection: GetHealth() is load-bearing (Gitea #527, closed)

Until #527, DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed initial connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely — publishing Bad for every tag, never re-initialising, and not recovering when the device returned (live-confirmed: a Modbus driver against a stopped fixture needed a node restart). PublishHealthSnapshot read GetHealth() for reporting only.

The Connected health tick now runs a watchdog that self-sends DisconnectObserved when a driver reports a non-Healthy state and its LastSuccessfulRead has been frozen for 90 s. Things to know before touching it:

• ⚠️ The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal. Degraded means connected but some data is bad, and several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely. LastSuccessfulRead is the honest evidence — preserved across Degraded/Faulted writes, advanced by any success. Each conjunct has a test that goes red when the other is deleted.
• The mechanism rests on every driver reaching a non-Healthy state on sustained failure. Surveyed across all 12: nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip — correct, as it has no transport. Re-check this if a new driver lands.
• The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way. ModbusDriver (_transport) and MqttDriver (_connection) used to overwrite an IAsyncDisposable field inside InitializeAsync without disposing the old one, so re-initialising without ShutdownAsync leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose first: ModbusDriver.InitializeCoreAsync releases _transport and the _probeCts/_reprobeCts probe loops before rebuilding them (c3970a2e + the lifecycle gate in 409a58b6), deliberately not via the shared TeardownAsync() because that would also drop _poll and every active subscription with nothing to re-drive them; MqttDriver.InitializeCoreAsync calls its (idempotent) TeardownAsync() first, which reaches _connection and the primary-host connection (2b4b3fcb). The path is no longer latent — the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new IAsyncDisposable field on these two, must repeat the dispose-before-overwrite.
• The threshold is hardcoded (DefaultCommsLossTimeout), matching HealthPollInterval / DefaultReconnectInterval. Unlike the #523 drift detector this generates no traffic on a healthy system — it is a local GetHealth() read, and only reaches the network once a driver has already stopped reading.

See docs/DriverLifecycle.md §"The comms-loss watchdog".

mxaccessgw

The gateway lives in a sibling repo at ~/Desktop/MxAccessGateway/ (mxaccessgw — see Related Projects). See docs/v2/Galaxy.ParityRig.md for the gw setup recipe (build, API key provisioning via apikey create-key, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its gateway.md) is the canonical MxAccess API doc; the standalone mxaccess_documentation.md previously kept in this repo retired in PR 7.3.

Build Commands

dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test ZB.MOM.WW.OtOpcUa.slnx                                # all tests
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests               # a single test project
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod"    # a single test

CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native SQLitePCLRaw.lib.e_sqlite3) is resolved — the temporary NuGetAuditSuppress was removed once the patched bundle shipped. Core.AlarmHistorian now carries a surgical direct pin of SQLitePCLRaw.bundle_e_sqlite3 to 2.1.12 (the first fixed native bundle, embedding the SQLite 3.50.2+ patch), overriding the 2.1.11 that Microsoft.Data.Sqlite pulls transitively. dotnet restore/build is clean with the audit active.

Test projects live under tests/<module>/ (Core, Server, Drivers, Drivers/Cli, Client, Tooling) — there is no single unit-test project. Unit suites are named *.Tests; integration suites are *.IntegrationTests and need their Docker fixture up (see Docker Workflow). DB-backed tests in *.Configuration.Tests, *.Admin.Tests, and *.Server.Tests require the central SQL Server.

Docker Workflow (driver fixtures + central SQL Server)

Infra index: infra/README.md is the canonical index of all testing + deployment infrastructure — the Docker host layout, driver fixtures, the env-gated live-gate recipes (S7 blackhole, HistorianGateway LiveIntegration), the integration-test harness, the docker-dev rig, and the deploy setup. Start there; the sections below + docs/v2/dev-environment.md carry the detail.

Migrated 2026-04-28: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (DOCKER, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in appsettings.json defaults, fixture-class default endpoints, and e2e-config.sample.json were rewritten to target 10.100.0.35. See docs/v2/dev-environment.md for the full rewrite (header dated 2026-04-28).

⚠️ The project: lmxopcua label is aspirational, not universal. This note used to claim every fixture compose file carries it. As of 2026-07-24 only the MQTT fixture actually does — grep -rn "labels:" tests --include=*.yml matches Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml and nothing else. So docker ps --filter label=project=lmxopcua returns the MQTT fixture alone, not the fleet. Add the label when you touch an older fixture; until then enumerate by container name.

Docker workloads run on a shared Linux host at 10.100.0.35 — not on this VM. Stacks live at /opt/otopcua-<driver>/ on the host and carry the project=lmxopcua label so they're discoverable via docker ps --filter label=project=lmxopcua.

docker -H ssh://... does NOT work from the Windows VM. Windows OpenSSH ↔ docker.exe stdio bridging hangs (docker system dial-stdio runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs docker compose server-side.

On macOS there is no lmxopcua-fix — drive the host over SSH directly. The helper is a Windows-VM script; ~/bin is empty on the Mac and the commands below will not resolve. Use passwordless SSH (and rsync in place of sync), which is what infra/README.md §1 documents as the Mac path:

ssh dohertj2@10.100.0.35 'docker ps --format "{{.Names}}\t{{.Status}}"'
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mqtt && docker compose up -d'
rsync -av tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/ \
    dohertj2@10.100.0.35:/opt/otopcua-mqtt/          # the `sync` step, by hand

A local container runtime does exist on the Mac (OrbStack), so the docker-dev/ rig itself runs locally there — it is only the shared fixtures that live on 10.100.0.35.

Use lmxopcua-fix.ps1 (in ~/bin) to control fixtures from the Windows VM:

lmxopcua-fix ls                            # list all lmxopcua-tagged containers on the host
lmxopcua-fix up   modbus      standard     # bring a profile up
lmxopcua-fix up   abcip       controllogix
lmxopcua-fix up   s7          s7_1500
lmxopcua-fix up   opcuaclient                # single-service stack, no profile arg
lmxopcua-fix down modbus                   # tear stack down
lmxopcua-fix logs modbus
lmxopcua-fix sync modbus                   # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/

sync is the deployment step. When you edit a fixture's compose file or Dockerfile under tests/.../Docker/, run lmxopcua-fix sync <driver> to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; /opt/otopcua-<driver>/ is a mirrored deployment.

Endpoints (defaults already point at the docker host):

• SQL Server (always-on): 10.100.0.35,14330 — used by appsettings.json for ConfigDb.
• Modbus: 10.100.0.35:5020 (MODBUS_SIM_ENDPOINT)
• AB CIP: 10.100.0.35:44818 (AB_SERVER_ENDPOINT)
• S7: 10.100.0.35:1102 (S7_SIM_ENDPOINT)
• OPC UA reference (opc-plc): opc.tcp://10.100.0.35:50000 (OPCUA_SIM_ENDPOINT)
• MQTT (Mosquitto, auth on both listeners — no anonymous fallback): 10.100.0.35:8883 TLS+auth (MQTT_FIXTURE_ENDPOINT) · 10.100.0.35:1883 plaintext-but-authenticated (MQTT_FIXTURE_PLAIN_ENDPOINT). Also needs MQTT_FIXTURE_USERNAME / MQTT_FIXTURE_PASSWORD, and MQTT_FIXTURE_CA_CERT to pin the fixture CA (scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt). A co-running publisher loops JSON under otopcua/fixture/….
• MQTT Sparkplug B: the same broker, plus the sparkplug compose profile's otopcua-sparkplug-sim (project-owned C# edge-node simulator) — group OtOpcUaSim, edge nodes EdgeA/EdgeB, EdgeA device Filler1, ~2 s cadence. It subscribes to spBv1.0/OtOpcUaSim/NCMD/{node} and re-births on request. Births are never retained, so docker restart otopcua-sparkplug-sim is how you force a fresh one while a browse window is open.

Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at opc.tcp://localhost:4840 — that's not on the docker host.

Local docker-dev rig — login is DISABLED, so do live /run verification yourself (don't wait for the user to sign in). The local docker-dev/docker-compose.yml stack (AdminUI at http://localhost:9200 via Traefik; OPC UA opc.tcp://localhost:4840 central-1 / :4841 central-2) runs the AdminUI with Security__Auth__DisableLogin: "true" — no sign-in form; it's auto-authenticated as a full-access admin. So AdminUI / Razor /run verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against http://localhost:9200) does not require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA data-plane auth is still real LDAP against the shared GLAuth on 10.100.0.35:3893 — that only gates Client.CLI read/write role operations, e.g. binding a multi-role / opc-writeop user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the ZB.MOM.WW.HistorianGateway sidecar — still need the user.)

See docs/v2/dev-environment.md for the full inventory and rationale.

Build & Runtime Constraints

• Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint is owned by the mxaccessgw worker (x86 net48), not by anything in this repo.
• The gateway's MXAccess worker requires a deployed ArchestrA Platform on the machine running the gateway. The OtOpcUa server itself does not.

Transport Security

The server supports configurable OPC UA transport security via the OpcUa:EnabledSecurityProfiles list in appsettings.json. Phase 1 profiles (the OpcUaSecurityProfile enum members): None (default), Basic256Sha256Sign, Basic256Sha256SignAndEncrypt. Security policies are built from the enabled profiles by BuildSecurityPolicies at startup. The server certificate is always created even for None-only deployments because UserName token encryption depends on it. See docs/security.md for the full guide.

Redundancy

The server supports non-transparent warm/hot redundancy via the Redundancy section in appsettings.json. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct MxAccess.ClientName values) but have unique ApplicationUri values. Each exposes RedundancySupport, ServerUriArray, and a dynamic ServiceLevel based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See docs/Redundancy.md for the full guide.

Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.

• Downing is auto-down, not keep-oldest. Cluster:SplitBrainResolverStrategy (default auto-down) selects the provider via ServiceCollectionExtensions.BuildDowningHocon; an unknown value fails host start. A two-node pair running the previous keep-oldest + down-if-alone could not survive a crash of the oldest node: Akka's KeepOldest.OldestDecision only lets down-if-alone rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs itself and exits. down-if-alone is a 3+-node feature. The accepted trade for auto-down is dual-active during a genuine network partition. Its live gate is CLOSED — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (c50ebcf7; docs/plans/2026-07-24-mesh-phase7-failover-drills.md).
• The Primary is the oldest Up driver member, not the role leader. RedundancyStateActor previously used ClusterState.RoleLeader("driver") (lowest address), while ClusterSingletonManager places singletons on the oldest. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary.

A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice. Even under auto-down, a node cold-starting while its peer was dead never came Up: Akka runs FirstSeedNodeProcess — the only bootstrap path that can form a new cluster when no peer answers InitJoin — exclusively when seed-nodes[0] is the node’s own address; every other node runs JoinSeedNodeProcess and retries InitJoin forever. Cluster:SeedNodes is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second (docker-dev central-2 was swapped; site nodes list only central-1 and are legitimately not seeds). AkkaClusterOptionsValidator (AddValidatedOptions/ValidateOnStart in AddOtOpcUaCluster) enforces it at boot — conditionally (self must be entry 0 only if self is in the list, or every site node would refuse to start), matching on PublicHostname+Port, never the 0.0.0.0 bind address. The earlier fix — a Cluster:SelfFormAfter timer calling Cluster.Join(SelfAddress) — is deleted: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by SelfFirstSeedBootstrapTests (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + AkkaClusterOptionsValidatorTests. See docs/Redundancy.md §"Bootstrap: self-first seed ordering".

Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, Cluster:BootstrapGuard:Enabled, default OFF). Self-first on BOTH nodes means when both cold-start at the same instant, each runs FirstSeedNodeProcess, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically lower host:port node is the preferred founder (self-first, forms immediately); the higher node TCP-probes the partner's Akka port up to PartnerProbeSeconds (25s) — reachable ⇒ peer-first (join the founder), unreachable ⇒ self-first (partner dead, form alone). The order is chosen BEFORE the single JoinSeedNodes, from an explicit reachability signal — it never re-forms mid-handshake (the SelfFormAfter failure mode). When on, Akka gets NO config seeds (BuildClusterOptions empties ClusterOptions.SeedNodes) and ClusterBootstrapCoordinator drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: site-a = guard demo (guard on, serialization removed); site-b = A/B control (guard off, depends_on: service_healthy serialization). The guard is the production-faithful fix (compose depends_on doesn't exist on real VMs). Pinned by ClusterBootstrapGuardTests + ClusterBootstrapCoordinatorTests (real 2-node clusters, incl. the higher-node-cold-start-alone case). See docs/Redundancy.md §"Bootstrap guard".

Assert the effective cluster config off a running ActorSystem (as SplitBrainResolverActivationTests does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.

RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster. The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.

Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one. The single fleet-wide Akka mesh was split into one mesh per application Cluster: the central pair (admin,driver,cluster-MAIN), site-a (driver,cluster-SITE-A), site-b (driver,cluster-SITE-B) — same ActorSystem name, separated purely by per-pair self-first seeds (each node's Cluster:SeedNodes lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:

• The redundancy singleton is cluster-scoped and spawned per driver node. RedundancyStateActor is now a cluster-{ClusterId}-scoped singleton (falls back to plain driver for a legacy node), so each pair elects and publishes its own Primary on its own mesh's DPS. Two or more Primaries exist fleet-wide, one per pair — by design, not a bug. ⚠️ Wiring landmine (fixed 3a4ed8dd, caught by the live gate): the singleton's cluster-role scope is computed at AddAkka-configurator time, so it must come from AkkaClusterOptions (config) — never by resolving IClusterRoleInfo from the SP there. ClusterRoleInfo depends on the ActorSystem, and that lambda runs while the ActorSystem is being built, so resolving it recurses into ActorSystem construction and stack-overflows every node at boot (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep WithOtOpcUaClusterRedundancySingleton/BuildClusterRedundancySingletonOptions taking AkkaClusterOptions.
• ClusterNodeAddressReconciler is own-cluster-scoped — an admin node can only reconcile ClusterNode rows in its own cluster; it has no gossip visibility into another mesh.
• CentralCommunicationActor dials each node directly over gRPC — one NodeCommandService.Execute call per enabled ClusterNode row carrying a GrpcPort — rather than fanning through a client/receptionist object. Superseded by the ClusterClient→gRPC migration: the Akka ClusterClient this bullet originally described (one per application Cluster, closing the TODO(Phase 6) Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
• SplitTopologyTransportValidator fails host start on a node carrying a cluster-{ClusterId} role unless it also runs the split-safe transports: MeshTransport:Mode=Grpc, Telemetry:Mode=Grpc (driver), TelemetryDial:Mode=Grpc (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
• The compiled transport-mode defaults were deliberately NOT flipped — MeshTransport:Mode/Telemetry:Mode/TelemetryDial:Mode all still default to Dps; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
• Secrets replication is pair-local — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).

See docs/Redundancy.md §"Per-cluster meshes (Phase 6)", docs/Configuration.md §"Per-cluster mesh split (Phase 6)", and docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.

Mesh command transport (MeshTransport)

The Akka ClusterClient transport described in this section's earlier history is DELETED (branch feat/grpc-phase3-deletion, ClusterClient→gRPC migration, Phase 3) — code, the akka.cluster.client HOCON block, the receptionist extension, MeshTransport:CentralContactPoints, and the ClusterClient mode value itself, which now fails validation loudly at boot (MeshTransportOptionsValidator names the key and the allowed values) rather than silently falling back to Dps. What remains is exactly two transports, selected by MeshTransport:Mode:

• Dps (default) — DistributedPubSub, the path this repo has always used. Pair-local only: it works only when central and the node share a gossip ring, which after Phase 6 means within one 2-node mesh.
• Grpc — unary gRPC calls on the mesh_transport.v1 contract. Required on any node carrying a cluster-{ClusterId} role; SplitTopologyTransportValidator refuses to boot such a node on anything else.

Central→node commands (deployments, driver-control, alarm-commands) ride NodeCommandService.Execute, served on each node's driver-role telemetry h2c listener (Telemetry:GrpcListenPort, the docker-dev rig's :4056 — it shares that listener with TelemetryStreamGrpcService rather than claiming a port of its own). CentralCommunicationActor dials it directly, one call per node — the target set is every enabled, non-maintenance ClusterNode row carrying a GrpcPort, dialled at Host+GrpcPort and refreshed every MeshTransport:ContactRefreshSeconds. There is no client/receptionist object and no rotation: each command goes to the specific node dialled, which is what makes per-node delivery provable — the old ClusterClient.SendToAll/Send distinction this section used to explain no longer applies to either direction.

Node→central deploy acks ride CentralCommandService.ReportApply, served on central's admin-role config-serve h2c listener (ConfigServe:GrpcListenPort, the rig's :4055 — again sharing rather than claiming a new port). GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and stops at the first success.

Auth is a per-cluster PSK bearer, both directions: MeshTransport:GrpcPsk (node-side, one key per application Cluster) and MeshTransport:ClusterPsks (central-side map, consulted before the Secrets store). MeshCommandAuthInterceptor (node) and CentralCommandAuthInterceptor (central) fail closed on every unresolvable case — constant-time compare, no key material logged. Production keys are ${secret:OT-MESH-PSK-<clusterId>}; the docker-dev rig supplies three literal dev keys via env (dev-mesh-psk-MAIN/-SITE-A/-SITE-B).

Things that carried over unchanged from the ClusterClient era, because they were never ClusterClient-specific to begin with:

• Fire-and-forget, no retry, on both transports. A failed gRPC dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery. The old buffer-size = 0 decision this section used to describe is gone with the HOCON block that set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
• Inbound commands re-emit on the node's EventStream, not back onto DPS. DPS is mesh-wide, so a DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (DriverHostActor, ScriptedAlarmHostActor) accept a command from either transport identically.
• Do not over-claim what fire-and-forget buys. The Phase 2 live gate observed a node applying a deployment 44 s after it was sealed TimedOut anyway, via DriverHostActor.Bootstrap() replaying the orphan Applying row on restart. That backstop is unaffected by the ClusterClient deletion — it was always about the deploy path, not the transport.

redundancy-state stays on DPS in both modes (bidirectional, and built from Cluster.State, which makes it genuinely mesh-bound). See docs/Configuration.md §MeshTransport, docs/Redundancy.md §"Command transport", and the migration's design doc / plan / tracking doc / live gate for the phase-by-phase history — those are historical records, not updated further by this sweep. docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md is the original Phase 2 design doc and is likewise left as a historical record of the transport this section now describes as deleted.

Phase 1 landed 2026-07-22 and changed a deploy-path behaviour. ConfigPublishCoordinator sources its expected-ack set from enabled ClusterNode rows, not from Akka.Cluster.State.Members filtered by the driver role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:

• A configured node that is switched off now FAILS the deployment at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the new ClusterNode.MaintenanceMode column, NOT Enabled = 0: the live gate showed Enabled = 0 is rejected outright by DraftValidator.ValidateClusterTopology (ClusterEnabledNodeCountMismatch — enabled count must equal ServerCluster.NodeCount), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. Enabled = part of the declared topology; MaintenanceMode = do not expect it now. The deadline log names the silent nodes and points at the right flag.
• Every ClusterNode row is now defined to be a driver node. The DB has no per-node role column and deliberately did not gain one (it would drift from Cluster:Roles). Giving an admin-only node a ClusterNode row makes every deploy time out.
• There is no such thing as a cluster-scoped deployment. Deployment has no ClusterId, ConfigComposer always snapshots the whole DB, and DeploymentArtifact.ResolveClusterScope is node-side self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
• ClusterNode gained AkkaPort (NOT NULL, 4053) + GrpcPort (nullable) as central's dial targets for Phases 2/5. They duplicate the node's own Cluster:Port / Cluster:PublicHostname with no schema-level enforcement, so ClusterNodeAddressReconcilerActor (admin singleton) logs an Error on drift. Phase 2 must revisit it — once the meshes split, an admin node cannot see site members and every site row would report EnabledRowNotInCluster forever. Resolved, and not the way this bullet originally expected: the ClusterClient→gRPC migration deleted the Akka ClusterClient transport that would have dialled Host+AkkaPort; the Phase 2 command dial now uses Host+GrpcPort, the same column Phase 5 telemetry already dialled — AkkaPort is Akka remoting only again.

Config source (ConfigSource / ConfigServe)

Per-cluster mesh Phase 3 (config fetch-and-cache) added a second way a driver node obtains its deployed configuration, selected by ConfigSource:Mode: Direct (read Deployment.ArtifactBlob from central SQL — the default, byte-for-byte today's behaviour) or FetchAndCache (fetch the artifact bytes from central over a dedicated-h2c gRPC stream, verify SHA-256 == RevisionHash, cache in LocalDb, and read only the cache — no central-SQL config read). Central (admin role) serves via DeploymentArtifactService behind ConfigServeAuthInterceptor on ConfigServe:GrpcListenPort, gated by a shared node bearer key (ConfigServe:ApiKey == ConfigSource:ApiKey, FixedTimeEquals, fail-closed). Things worth knowing before touching it:

• The serve side is wired in both modes (mapped whenever ConfigServe:GrpcListenPort > 0 on an admin node), so flipping a node to FetchAndCache is a config change, not a redeploy — the same dark-switch discipline as MeshTransport.
• RevisionHash == SHA-256(artifact bytes) is load-bearing. ConfigComposer computes the revision hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the RevisionHash the dispatch already carries — no hash rides in the proto, and DispatchDeployment stays payload-free. A future change to how the revision hash is derived silently breaks every fetch. The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes — ConfigComposer.Pack gzips (CompressionLevel.Fastest) immediately after SerializeToUtf8Bytes and hashes the result, so the invariant holds unchanged and nothing between the composer and DeploymentArtifact.Open (SQL ArtifactBlob, the LocalDb cache chunks, the gRPC fetch stream) ever looks inside. Open sniffs the gzip magic 1F 8B — valid JSON can never start with 0x1F — so pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every decompression failure surfaces as JsonException, the same family the malformed-JSON paths already handle. Two operational consequences in docs/Configuration.md §ConfigSource: the first deploy after upgrade re-hashes every revision (a one-time full re-dispatch, not a no-op), and nodes must be upgraded BEFORE central starts emitting compressed blobs.
• A null fetch is "no answer," never an empty config (#485). All endpoints down / NotFound / SHA mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance, and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a FetchAndCache node restores served state from the LocalDb pointer with no central-SQL read; an empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
• Both pair nodes fetch independently — no Primary gating of the fetch (same shape as both reading SQL today; StoreAsync is idempotent and the cache replicates).
• The dedicated h2c listener coexists with the LocalDb-sync listener. A fused admin+driver node with both ports set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel Listen*-discards-ASPNETCORE_URLS trap the LocalDb block already warned about).
• Phase boundary: config reads only. A FetchAndCache node still writes its NodeDeploymentState ack row to central SQL; DbHealthProbe and EfAlarmConditionStateStore still touch SQL. Cutting those is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.

On the docker-dev rig the central pair carries ConfigServe (:4055, committed dev key) and stays hardcoded Direct via its admin role; the four site nodes carry ConfigSource (endpoints + matching key) hardcoded FetchAndCache — mandatory for driver-only nodes since Phase 4, not a flip. There is no OTOPCUA_CONFIG_MODE env var in the compose file. See docs/Configuration.md §ConfigSource / ConfigServe, docs/Redundancy.md §"Command transport", and docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md.

Live telemetry transport (Telemetry / TelemetryDial)

Per-cluster mesh Phase 5 (merged 35552a96; live gate PASSED f134935f) added a second transport for the four live-observability channels the AdminUI's /alerts, /script-log, and /hosts panels feed on, selected by Telemetry:Mode (node/serve side) and TelemetryDial:Mode (central/dial side): Dps — the four existing DPS SignalR bridges, the default, today's behaviour — or Grpc, one gRPC server-streaming contract carrying all four channels as oneof event kinds. The direction is inverted from ConfigSource/MeshTransport: the driver node hosts the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in — mirroring ScadaBridge's SiteStreamService shape, and required so telemetry survives once the meshes split (Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before touching it:

• Migrated (4 channels): alerts (AlarmTransitionEvent), script-logs (ScriptLogEntry), driver-health (DriverHealthChanged), driver-resilience-status (DriverResilienceStatusChanged) — feeding the exact same central sinks (IInProcessBroadcaster<...>, IDriverStatusSnapshotStore, IDriverResilienceStatusStore) in either mode.
• Deferred, with reasons (do not read "seven observability topics" from the program sketch as the delivered scope): redundancy-state stays on DPS in every mode (pair-local, built from Cluster.State, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an observability panel); deployment-acks already rides the Phase 2 gRPC command transport (MeshTransport:Mode=Grpc, CentralCommandService.ReportApply) as a command-plane reply, not telemetry. A third, fleet-status, was DELETED rather than deferred (2026-08-06, Gitea #542): FleetStatusBroadcaster → DPS → SignalR bridge → FleetStatusHub had zero clients end to end (no browser JS, no HubConnection — /fleet polls NodeDeploymentState in the Config DB), its DriverHostStatusHeartbeat input had no production sender, and Phase 6 left its "fleet" as just central's own pair. Fleet visibility is /fleet + the four per-node telemetry dials above.
• The node ALWAYS hosts and ALWAYS double-publishes, in both modes. Every publish seam emits into a node-local ITelemetryLocalHub and still publishes DPS, unconditionally; the gRPC server binds whenever Telemetry:GrpcListenPort > 0, independent of Telemetry:Mode. Only TelemetryDial:Mode on central actually switches anything: which of the two always-available sources (DPS bridges vs. the TelemetryDialSupervisor dialer) feeds the sinks. This is what makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as MeshTransport and ConfigSource.
• Authenticated from day one, fail-closed — a shared node bearer key (Telemetry:ApiKey == TelemetryDial:ApiKey), gated by TelemetryStreamAuthInterceptor (CryptographicOperations.FixedTimeEquals, PermissionDenied on mismatch, an unset key rejects every call). This supersedes design doc §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
• Reconnect story: central runs one reconnecting dialer per driver node (discovered from ClusterNode.Host/GrpcPort, refreshed every TelemetryDial:ContactRefreshSeconds), immediate first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event is ignored, and it never permanently gives up (observability, not data plane). The node-local hub replays last-value snapshots for driver-health/driver-resilience-status on every (re)subscribe so those stores re-prime immediately after a reconnect; alerts/script-logs are append logs and tolerate the gap. The /alerts//script-log pill reflects aggregate stream health (live when ≥1 node stream is up) — the same semantics the DPS pill has always had.

See docs/Telemetry.md for the full architecture, docs/Configuration.md §Telemetry/TelemetryDial for the config keys, and docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md for the implementation plan and live gate procedure.

LocalDb pair-local store (Phases 1 + 2)

Every driver-role node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database (the retired LiteDB LocalCache subsystem was deleted as superseded). It holds three replicated tables: deployment_artifacts + deployment_pointer (Phase 1) and alarm_sf_events (Phase 2).

Phase 1 caches the deployed-configuration artifact (chunked, SHA-256-verified, newest-2 retention per cluster) so a driver node can boot from cache when central SQL Server is unreachable — DriverHostActor writes the cache after each successful apply (IDeploymentArtifactCache → LocalDbDeploymentArtifactCache, in Runtime/Deployment/) and reads it as a boot fallback, flagging running-from-cache. Wired by AddOtOpcUaLocalDb in the hasDriver branch only (Host/Configuration/LocalDbRegistration.cs + LocalDbSetup.OnReady, whose DDL→RegisterReplicated order is load-bearing); admin-only nodes never register it. The cache can optionally replicate to the node's redundant pair peer over the library's gRPC sync — default-OFF, fail-closed bearer auth (LocalDbSyncAuthInterceptor), gated on LocalDb:SyncListenPort (a dedicated Kestrel h2c listener) + LocalDb:Replication:*. On the docker-dev rig the site-a pair has replication on; central + site-b stay default-OFF (site-b is the pin). ApiKey via ${secret:}/env, never committed (the rig's dev key is the committed-dev-secret exception).

Phase 2 moved the alarm store-and-forward buffer off its standalone alarm-historian.db and into the same database as alarm_sf_events, so a node that dies holding undelivered alarm history no longer takes it to the grave. SqliteStoreAndForwardSink became LocalDbStoreAndForwardSink (Core.AlarmHistorian), and the breaking config key AlarmHistorian:DatabasePath was removed — it is still read, by AlarmSfLegacyMigrator, purely to find a pre-consolidation file to copy across on first boot (renamed .migrated after the copy commits). Because the buffer replicates, only the node holding the Primary role drains it: the sink takes a drainGate delegate, Runtime supplies it from the IRedundancyRoleView singleton that DriverHostActor publishes its PrimaryGatePolicy verdict to, and a gated-off node reports the new HistorianDrainState.NotPrimary. Delivery is therefore at-least-once across a failover, by design (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the same event accepted on both nodes — which happens in every boot window, since HistorianAdapterActor default-writes while its role is unknown — converges to one row. On the docker-dev rig the site-a pair and site-b both run AlarmHistorian__Enabled=true against a deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.

The library pin is 0.2.1 (Directory.Packages.props, both ZB.MOM.WW.LocalDb and .Replication), bumped from 0.1.3 on 2026-08-15. It was a pure pin bump — no code change: RegisterReplicated(string) is unchanged, everything 0.2.x added is additive, the wire protocol is untouched (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2 upgrades in place on open, so no rig or volume action was needed. Its two new APIs — DeregisterReplicated and RegisterReplicated(table, baselineExistingRows:) — are now both used (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two things are worth knowing:

• MaxBatchSize no longer means what the rig's 16 was chosen to mean. 0.2.0 made batching byte-budgeted — LocalDb:Replication:MaxBatchBytes (2 MB default, left unset here, sized under the 4 MB gRPC cap) bounds the wire message. MaxBatchSize is demoted to a secondary row cap and the DB read-page size in OplogStore.ReadBatchAboveAsync/SnapshotStreamer, which materialise a whole page in memory before the byte split runs. So the site-a pair keeps 16 — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap" was corrected with the bump; do not restore that rationale.
• Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18). LocalDbSetup.OnReady reads one predicate, LocalDbSetup.ReplicationIsConfigured: LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey is non-blank. The OR is not belt-and-braces — only the dialling half of a pair sets PeerAddress, so keying on it alone would strip capture from every passive node (site-a-2 carries ApiKey + LocalDb:SyncListenPort and nothing else) and the pair would converge in one direction only, silently. ⚠️ SyncListenPort deliberately does NOT join the OR: the passive endpoint is fail-closed on the key, so a port without an ApiKey rejects every inbound stream and has nothing to capture for.
  • Configured ⇒ RegisterReplicated(table, baselineExistingRows: true), per table, in the same load-bearing DDL→register order. Baselining is what makes enabling replication on a node that has been running without it actually converge: capture is CDC and the snapshot streamer pages from __localdb_row_version, so pre-registration rows were in neither and a late opt-in used to replicate only writes made after the restart. It seeds the ledger at the LWW floor (HLC 0, this node's id) and flags a resync — a baselined row loses to any genuine remote write of the same key and wins only where the peer has no version at all. It is idempotent, so it is free on every boot after the first. ⚠️ On the pinned 0.2.1 the snapshot flag is per-database, not per-table, so baselining anything streams every registered table in full, in both directions at once — trim alarm_sf_events before enabling replication on a node that has been buffering alarm history for months, and expect one slow first sync.
  • Not configured ⇒ DeregisterReplicated(table) per table, an idempotent boot-time self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and prunes that table's oplog + row-version rows. Merely declining to register would leave every pre-2026-08-18 file capturing forever — two INSERTs plus a json_object of the whole row inside every write transaction — on precisely the node that has no peer to capture for. The call returns true when it removed something, so the Information line fires once, and only when it actually cleaned; a default-OFF node's ordinary boot stays silent. The logger is Serilog's static Log.Logger (this runs inside the AddZbLocalDb singleton factory, where resolving one from DI would nest a resolution inside a singleton construction; Program.cs assigns it before anything resolves ILocalDb), with an optional parameter so tests observe the line rather than infer it.
  • ⚠️ PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change. The sync handshake compares registered-table digests fail-closed, so a node that deregisters a table its peer still replicates stops syncing with a schema mismatch. Enabling replication on an established pair means setting the config and restarting both nodes — do it on both, or on neither.
  • Rollout note: the only replicating pair in the fleet is docker-dev site-a, and it was enabled from a fresh pair, so the impact of the old gap was low — central and site-b are default-OFF and simply shed their unused triggers on their next boot. LocalDbSetupTests pins both branches (registered-table set on a configured node, empty set + zero triggers + the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.

See docs/Configuration.md (LocalDb section), docs/Redundancy.md (pair-local config cache), docs/AlarmHistorian.md, and the runbook docs/operations/2026-07-20-localdb-pair-replication.md.

Phase 4 — cut the driver-side ConfigDb connection

Per-cluster mesh Phase 4 finished the job Phase 3 started: a driver-only node (Cluster:Roles has driver, not admin) now boots with no ConfigDb connection string at all. Program.cs gates AddOtOpcUaConfigDb (and its health check) on hasAdmin; a fused admin,driver node is unaffected (it keeps ConfigDb via its admin role). Consequences:

• ConfigSource:Mode must be FetchAndCache on a driver-only node — ConfigSourceOptionsValidator fails host start if such a node is left on Direct (there is nothing to read Direct from). A fused node may stay Direct.
• ServiceLevel semantics change — client-visible. With no ConfigDb, DbHealthProbeActor is not spawned (ServiceCollectionExtensions.WithOtOpcUaRuntimeActors skips it when the resolved db factory is null), and OpcUaPublishActor runs dbLess: DbReachable is fed constant true and Stale comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes 240 (or 250 as driver Primary) even with central SQL unreachable — the survive-alone posture — where a Direct/DB-backed node in the same state drops to 100/0. ServiceLevelCalculator itself is unchanged; only the inputs a DB-less node feeds it changed. See docs/Redundancy.md §"DB-less (driver-only) nodes".
• Scripted-alarm Part 9 condition state moved to LocalDb. LocalDbAlarmConditionStateStore (replicated alarm_condition_state table, pair-local like the Phase-2 alarm S&F buffer) replaces EfAlarmConditionStateStore on every driver-role node, fused central included. The DB-backed ScriptedAlarmState table was dropped (Task 9, 3a590a0c, migration 20260723183050_DropScriptedAlarmStateTable); EfAlarmConditionStateStore is deleted.
• Central persists deploy acks. DriverHostActor no longer writes NodeDeploymentState to SQL on a driver-only node — ConfigPublishCoordinator.PersistNodeAck (already fed by every ApplyAck) is the system of record.
• LDAP group→role DB grants don't reach driver-only nodes. They register the empty NullLdapGroupRoleMappingService (no ConfigDb to read LdapGroupRoleMapping from), so OtOpcUaGroupRoleMapper falls back to the Security:Ldap:GroupToRole appsettings baseline only — the same rows a driver node never received via config either, so this is not a regression.

Merged to master (26fad75c); live gate PASSED (1281aebf — driver nodes run ConfigDb-free) and Task 9 landed (3a590a0c). See docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md and docs/plans/2026-07-22-per-cluster-mesh-program.md §Phase 4.

Authentication: API keys on the data plane, LDAP on AdminUI (#520)

⚠️ Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all. It is machine-to-machine, so a client presents an OPC UA UserName token with the fixed username apikey and an otua_<keyId>_<secret> credential as the password. ApiKeyOpcUaUserAuthenticator (src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/) implements IOpcUaUserAuthenticator: one indexed lookup on keyId plus one peppered HMAC-SHA256, FixedTimeEquals, no network I/O. The key resolves to a principal; NodeAcl.Principal rows grant it permissions. LdapOpcUaUserAuthenticator is deleted, along with four Security:Ldap keys that existed only for it (AuthTimeoutMs, MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds) — they bounded an LDAP bind running inline in the SDK's synchronous, SessionManager-lock-holding impersonation callback, and nothing on the remaining LDAP path has that property.

LDAP remains, for AdminUI login only (Security:Ldap, OtOpcUaLdapAuthService → ILdapAuthService, group→role mapping via GroupToRole + LdapGroupRoleMapping). Do not "restore" the data-plane authenticator when you see LDAP wiring in Program.cs — it is there for the AdminUI.

Two things to know before relying on any of this:

• ⚠️ Node ACLs are enforced only behind Security:Acl:Enabled (default false). With the flag off, a key's grants collapse into the two server-wide role gates (WriteOperate on Value writes, AlarmAck on Part 9 methods), only Cluster-scoped grants count — ApiKeyInterimRoleMapper deliberately under-grants, because honouring a narrower grant's flags while unable to honour its scope would turn "WriteOperate on Equipment X" into server-wide write authority — and read, browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous. With the flag on (DataPlaneAuthorizer), scoped grants decide every data-plane operation: write and alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6).
  • ⚠️ The ACL decision REPLACES the role check — the two are never chained. Chained, a grant would have to confer the server-wide role as well as pass the scope check, and only Cluster-scoped grants confer a role, so every narrow grant would stay inert and the flag would appear to do nothing. The load-bearing test is therefore an allow — a session carrying no roles writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too.
  • The trie is built inside DataPlaneAuthorityState.Apply, not by a second wiring at the publish actor, and the state owns its PermissionTrieCache rather than being handed one. An uninstalled trie denies everything, which is indistinguishable from working enforcement until someone checks that an ALLOW is possible at all.
  • A deploy reaches sessions that are already open, in both directions: each decision stamps the current generation rather than the one the session activated under, and re-checks the authenticating key — so revoking a key or a grant stops an open session immediately instead of at the client's next reconnect. On a driver-only node the artifact is the only revocation channel.
  • UserAuthorizationState is built per decision walk, not per session — and since the hot-path change it is allocated only on a cache miss, so a cache hit never constructs one. Its LDAP-shaped freshness/staleness predicates are switched off EXPLICITLY (AuthCacheMaxStaleness = MaxValue). ⚠️ Relying on "the stamp is always fresh" was not safe: MembershipResolvedUtc comes from DataPlaneAuthorizer's TimeProvider while IsStale is evaluated against TriePermissionEvaluator's, so any divergence between the two clocks denies every request with a verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead.
  • Stage 6's hot seams (read/browse/subscribe) go through a per-(session, node, operation, realm) decision cache, cleared wholesale when the generation advances. It is sharded per API key (ApiKeyId ?? Principal — despite the type name, sessions sharing one key share one shard and one budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and not every other key's decisions. ⚠️ It caches the grant walk only — key revocation and expiry are re-checked before the cache lookup on every call, because expiry moves with the clock rather than with a deployment.
  • Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors. A grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward only, so PermissionTrieNode.BrowseTransitPrincipals marks the path. Planting a real Browse grant on each ancestor instead would be collected by the additive downward walk and leak Browse onto every sibling subtree beneath that ancestor. ⚠️ The topmost ancestor is the single shared root folder (s=OtOpcUa, raw namespace) that BOTH realm trees hang under — its identifier is a display name, not a path, so DataPlaneAuthorizer.BuildScope maps it onto the trie's zero-segment realm-root scope (V3NodeIds.SharedRootFolderId, realm-qualified: in the UNS namespace that identifier is an ordinary Area), and its Browse considers transit in either realm because it fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse gate denies the whole call when its source node is denied, so clients were stopped at the front door.
  • Revocation reaches live subscriptions (stage 6): every other operation re-asks per request, but a subscription is authorized once and then streams — and streaming is how machine consumers actually read this server. DataPlaneAuthorityState.Changed fires RevalidateMonitoredItems. ⚠️ A denied item is queued BadUserAccessDenied and then detached from its MonitoredNode2 fan-out — NOT set MonitoringMode.Disabled: disabling discards the item's queued notifications, so the client is silenced without ever being told why (caught only by the over-the-wire test). Detaching is also reversible, so restoring the grant resumes the feed with no client action.
• Anonymous is a normal grantable principal, not a bypass. OpcUa:AllowAnonymous (default true, preserving today's behaviour) controls admission; when admitted the session authenticates as the reserved principal anonymous and gets roles from NodeAcl rows naming it, arriving at the gates as the same RoleCarryingUserIdentity a key produces. Set it false to make keys mandatory. A key may not use the anonymous principal, and a failure to read anonymous's grants denies rather than admitting with no roles.
• Keys reach a driver-only node inside the deployment artifact (stage 4). ConfigComposer snapshots the ApiKey rows — the peppered HMAC only, never a secret, and the pepper is node config that is deliberately not in the artifact — and DeploymentArtifact.ParseDataPlaneAuthority decodes them node-side alongside the cluster's NodeAcl rows. OpcUaPublishActor installs both on IDataPlaneAuthoritySource on every apply, before its plan.IsEmpty early-return, so a deployment that only revokes a key still lands. A node with a ConfigDb keeps ConfigDbApiKeyStore (per-request read); a driver-only node gets ArtifactApiKeyStore. ⚠️ The consequence is asymmetric revocation: immediate on a ConfigDb node, next-deployment on a driver-only one.
• ⚠️ The auth generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs with nothing logged. Pinned by DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases.
• Note OtOpcUaServerHostedService takes IOpcUaUserAuthenticator and IDataPlaneAuthoritySource as required ctor parameters — the built-in DI container does not fall back to a default value for an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a node with neither role) or it does not boot.

Grants are authored through a scope picker, not free text (stage 7). /clusters/{id}/acls opens AclScopePickerModal over the cluster's raw + UNS trees, and selecting a node supplies ScopeKind and ScopeId together — so RawDevice + a TagId, a cross-cluster scope, a Cluster grant carrying an id, and the retired FolderSegment kind are all unauthorable by construction rather than merely rejected later. ⚠️ A Cluster row must produce a NULL ScopeId — the deploy gate rejects one that carries an id (NodeAclScopeIdUnexpected), so a picker that helpfully filled in the cluster id would author rows that cannot be deployed. IAclScopeService.CheckScopeAsync mirrors DraftValidator.ValidateNodeAclScopes rule for rule and blocks the save when a scope resolves to nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches the picker); the list page marks such rows unresolved. UNS tags are deliberately not offered — a UNS signal is a UnsTagReference, and a Tag grant already covers the raw node and every UNS projection of it. The picker is AclScopeTree, a separate component from RawTree, which already carries two modes.

All eight stages are done (d4389f57 enforcement, stage 7 authoring after it), each live-gated on docker-dev. The rationale for every decision above is in docs/plans/2026-07-31-apikey-acl-enforcement-design.md. See docs/security.md for the full guide.

Dev/test LDAP is the shared GLAuth running on the Linux Docker host at 10.100.0.35:3893 (baseDN dc=zb,dc=local, plaintext/Transport=None). It is managed via scadaproj/infra/glauth/ (source of truth + deploy runbook). Single bind account cn=serviceaccount,dc=zb,dc=local / serviceaccount123; all test users password password. The docker-dev compose binds this shared instance directly — DevStubMode is no longer used. The per-VM NSSM GLAuth at C:\publish\glauth\ and the old base DNs dc=lmxopcua,dc=local / dc=otopcua,dc=local are obsolete. (The integration-test harness under tests/.../Host.IntegrationTests/ runs its own ephemeral GLAuth container on port 3894 — seeded by tests/.../Host.IntegrationTests/glauth/config.toml with dev users alice/bob + the cn=serviceaccount bind account — for the opt-in OTOPCUA_HARNESS_USE_LDAP=1 real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired bitnami/openldap:2.6 in PR #451, 2026-07-15.)

Library Preferences

• Logging: Serilog with rolling daily file sink
• Unit tests: xUnit + Shouldly for assertions
• Service hosting (Server, Admin): .NET generic host with AddWindowsService (decision #30 — replaced TopShelf in v2; see src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs)
• OPC UA: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: OPCFoundation.NetStandard.Opc.Ua.Server

OPC UA .NET Standard Documentation

Use the DeepWiki MCP (mcp__deepwiki) to query documentation for the OPC UA .NET Standard stack: https://deepwiki.com/OPCFoundation/UA-.NETStandard. Tools: read_wiki_structure, read_wiki_contents, and ask_question with repo OPCFoundation/UA-.NETStandard.

Testing

Use the Client CLI at src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/ for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See docs/Client.CLI.md for full documentation.

dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500

Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see docs/plans/2026-05-28-driver-browsers-design.md.

The AdminUI's global UNS page (/uns) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See docs/Uns.md.

v3 (Batch 2): device I/O is authored in the /raw project tree (GlobalRaw.razor → RawTree.razor, IRawTreeService) — a cluster-rooted Folder → Driver → Device → TagGroup → Tag hierarchy with per-node context menus (the reusable ContextMenu), lazy loading, driver/device config modals (endpoint moved to Device.DeviceConfig; Test-connect probes the DriverDeviceConfigMerger-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed /clusters/{id}/drivers flow (DriverTypePicker, DriverEditRouter, the 8 *DriverPage shells, the Drivers list) is retired — its form bodies live on inside the /raw modals. The DriverType dispatch maps are keyed off the DriverTypeNames constants (fixing the TwinCat/Focas drift). A Calculation pseudo-driver (DriverTypeNames.Calculation, IDependencyConsumer, deploy-time scriptId-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See docs/Raw.md.

v3 (Batch 3): UNS Equipment is reference-only. The equipment Tags tab no longer authors or binds tags — it holds UnsTagReference rows pointing at raw tags authored in /raw. "+ Add reference" opens the RawTree in picker mode, cluster-scoped (structurally + server-enforced tag.cluster == equipment.cluster); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. Effective-name uniqueness (references + VirtualTags + ScriptedAlarms share the {EquipmentId}/{EffectiveName} NodeId space) is enforced at authoring (IEffectiveNameGuard) and at the deploy gate (DraftValidator.UnsEffectiveNameCollision — catches rename-induced collisions). Scripts use ctx.GetTag("{{equip}}/<RefName>") — resolved per-equipment through UnsTagReference effective names to the backing RawPath at both compose seams (AddressSpaceComposer + DeploymentArtifact, via the shared EquipmentReferenceMap, byte-parity); an unresolved <RefName> is a deploy error (EquipReferenceUnresolved) AND a live Monaco diagnostic (OTSCRIPT_EQUIPREF) — editor accepts ⇔ publish accepts. EquipmentScriptPaths.DeriveEquipmentBase is deleted (the dot-joint {{equip}}.X became the slash-joint {{equip}}/<RefName>). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (RawTreeService substring scan). ImportEquipmentModal dropped the DriverInstanceId column. See docs/Uns.md + docs/ScriptEditor.md.

v3 (Batch 4 = v3.0): dual-namespace address space. The OPC UA server exposes two namespaces — https://zb.com/otopcua/raw (Raw device tree, s=<RawPath>) + https://zb.com/otopcua/uns (UNS tree, s=<Area>/<Line>/<Equipment>/<EffectiveName>) — replacing the single .../ns and the retired EquipmentNodeIds scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in DriverHostActor to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable Organizes-references its raw node. Writes route via either NodeId to the same driver ref under the same realm-qualified WriteOperate gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (ConditionId = RawPath) and fan via SDK AddNotifier to the raw device folder + every referencing equipment folder (one ReportEvent, one Server-object copy); both NodeIds register the SAME historian tagname (mux HistorizedTagRef stays single, keyed by RawPath). Identity authority: V3NodeIds + AddressSpaceRealm (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, docs/Uns.md, and docs/plans/2026-07-15-v3-batch4-address-space-plan.md.

The /uns TagModal uses driver-typed tag-config editors: it dispatches by the bound driver's DriverType to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via TagConfigEditorMap, with client-side validation via TagConfigValidator; unmapped drivers (only Galaxy) fall back to the generic raw-TagConfig-JSON textarea. Each editor is a thin razor shell over a pure <Driver>TagConfigModel (FromJson/ToJson/Validate, preserves unknown keys). To add a driver's editor, copy the Modbus template under Components/Shared/Uns/TagEditors/ + Uns/TagEditors/, reusing the driver's enums + camelCase JSON property names, and register it in TagConfigEditorMap + TagConfigValidator. See docs/plans/2026-06-09-driver-typed-tag-editors-design.md.

Scripting / Script Editor

C# virtual-tag scripts are authored in a Roslyn-backed Monaco editor on the ScriptEdit page (/scripts/{id}) and inline inside the virtual-tag modal on the /uns page. The editor provides completions, live diagnostics, hover, signature help, document formatting, and tag-path completions inside ctx.GetTag("…") / ctx.SetVirtualTag("…") literals — all backed by the same compiler context the runtime publish gate uses, so what the editor accepts/rejects matches publish exactly. The backend lives in src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/ (six minimal-API endpoints under /api/script-analysis/*, gated by the ConfigEditor policy (Administrator or Designer) via RequireAuthorization in ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints). See docs/ScriptEditor.md for the full guide. Scripts may use the {{equip}} token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in docs/ScriptEditor.md.

Scripted Alarm Ack/Shelve

Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the alarm-commands DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in OtOpcUaNodeManager, gated on the AlarmAck LDAP role via RoleCarryingUserIdentity; (2) AdminUI /alerts per-row buttons routed through the AdminOperationsActor singleton (gated by DriverOperator policy). ScriptedAlarmHostActor dispatches from the topic to the engine. Client.CLI supports ack, confirm, shelve commands. See docs/ScriptedAlarms.md §"Inbound operator ack/shelve" and docs/AlarmTracking.md.

Historian / HistoryRead

Backend: HistorianGateway (sole historian backend). As of the gateway-integration cutover, the historian read, alarm-write, and continuous-historization paths are all served by the ZB.MOM.WW.HistorianGateway sidecar, consumed as the Gitea-feed ZB.MOM.WW.HistorianGateway.Client gRPC package (historian_gateway.v1) behind a thin IHistorianGatewayClient seam in ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway. The bespoke Wonderware TCP/ArchestrA sidecar projects and the vestigial Historian.Wonderware driver type were retired — there is no Wonderware backend in the tree anymore (see docs/drivers/Historian.Wonderware.md, now a retired stub).

A tag is historized by adding "isHistorized": true to its TagConfig JSON blob (authored in the raw-JSON textarea on the /uns TagModal); an optional "historianTagname" field overrides the default historian tagname, which is the tag's driver FullName.

Read path (ServerHistorian section)

The server dispatches all OPC UA HistoryRead to the registered IHistorianDataSource — the GatewayHistorianDataSource read client when enabled, else the NullHistorianDataSource default (historized nodes return GoodNoData, never an error). Supported variants: Raw, Processed (Average/Minimum/Maximum/Total/Count aggregates), and AtTime over historized variable nodes; Events over alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node); authorization uses the standard AccessLevels.HistoryRead bit set at materialization.

ServerHistorian appsettings keys (ServerHistorianOptions; Enabled defaults to false):

Key	Default	Notes
Enabled	false	true registers the gateway read client; false keeps NullHistorianDataSource
Endpoint	""	Absolute gateway URI, e.g. https://host:5222. Scheme selects transport (https:// = TLS, http:// = h2c)
ApiKey	""	Peppered-HMAC key histgw_<id>_<secret> sent as Authorization: Bearer. Supply via env ServerHistorian__ApiKey — never commit
UseTls	true	Connect over TLS; must match the Endpoint scheme
AllowUntrustedServerCertificate	false	Accept a self-signed / untrusted server cert (dev / on-prem only)
CaCertificatePath	null	PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store
CallTimeout	00:00:30	Per-call deadline for each unary gateway read
MaxTieClusterOverfetch	65536	Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend)
HistoryReadBatchParallelism	4	Max nodes served concurrently within one HistoryRead batch (03/S3); ⌈N/P⌉ × CallTimeout worst case instead of sequential
MaxConcurrentHistoryReadBatches	16	Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to BadTooManyOperations. Total in-flight reads capped at 16 × 4 = 64
HistoryReadDeadline	00:01:00	Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces BadTimeout. Non-positive = unbounded

Alarm-history path (AlarmHistorian section)

Alarm events are written through GatewayAlarmHistorianWriter (the gateway SendEvent path) behind the durable LocalDbStoreAndForwardSink — AlarmHistorian:Enabled=true swaps the NullAlarmHistorianSink default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the replicated alarm_sf_events table (see the LocalDb section) and whose drain worker forwards batches to the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). Only the node holding the Primary role drains (a gated node reports HistorianDrainState.NotPrimary). The AlarmHistorian section carries only the Enabled gate + the queue knobs (DrainIntervalSeconds, Capacity, DeadLetterRetentionDays, BatchSize, MaxAttempts) — the downstream gateway connection (endpoint/key/TLS) is sourced from the ServerHistorian section. Alarm-history ReadEvents requires the target gateway deployed with RuntimeDb:EventReadsEnabled=true (the C2 SQL event-read workaround).

Continuous historization (ContinuousHistorization section)

When ContinuousHistorization:Enabled=true and ServerHistorian is configured, the Host builds a durable, crash-safe FasterLog outbox (FasterLogHistorizationOutbox) + a gateway-backed IHistorianValueWriter, and WithOtOpcUaRuntimeActors spawns the ContinuousHistorizationRecorder. The recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the crash boundary), and drains the outbox to the gateway's SQL live-value write path (WriteLiveValues) with exponential backoff. The gateway connection is sourced from ServerHistorian; this section carries only the recorder + outbox knobs:

Key	Default	Notes
Enabled	false	true (with ServerHistorian configured) wires + spawns the recorder
OutboxPath	"" (required when enabled)	Directory holding the FasterLog segment + commit files. In production set an absolute path on durable storage
CommitMode	PerEntry	PerEntry = fsync before the append returns (no loss window); Periodic = batched commits every CommitIntervalMs. ⚠️ PerEntry now means one fsync per ingest burst, not per value — the recorder appends a burst through AppendBatchAsync, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands
CommitIntervalMs	100	Periodic-mode commit cadence; required > 0 only under Periodic
DrainBatchSize	1024	Entries peeked + written per drain pass. Raised from 64: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs tags × latency — while 64 drew down ≤ 64 entries per DrainIntervalSeconds, letting the outbox fall permanently behind during any gateway hiccup
DrainIntervalSeconds	2	Steady drain cadence (and post-success reschedule)
DrainConcurrency	8	Historian tags one drain pass writes concurrently (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential tags × latency pass becomes ⌈tags / concurrency⌉ × latency with outstanding round-trips bounded. 1 = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass"
Capacity	500000	Max un-acked outbox entries before drop-oldest; 0 = unbounded. Non-zero by default since #538 — the disk backstop for a historian refusing every write, the one case the per-tag quarantine deliberately cannot resolve
MaxWriteAttempts	5	#538: consecutive failed drain passes one tag may accumulate — counted only across passes where other tags wrote successfully — before its queued entries are abandoned so the FIFO head can advance
MinBackoffSeconds	1	Initial retry backoff after a failed drain pass
MaxBackoffSeconds	30	Cap on the exponential retry backoff

Tag auto-provisioning (IHistorianProvisioning EnsureTags hook)

AddressSpaceApplier.Apply() fires a non-blocking, fire-and-forget IHistorianProvisioning.EnsureTagsAsync hook for added historized value tags — the gateway-backed GatewayTagProvisioner calls the gateway's EnsureTags so a brand-new historized tag exists in the historian before the recorder's WriteLiveValues land. The hook is wrapped so a faulted/synchronously-throwing provisioner can never block or fail a deploy. Non-numeric (String/DateTime/Reference) data types are skipped (not provisioned); the recorder likewise drops + meters non-numeric values. Continuous historization is numeric-analog only in v1 (UInt16→UInt4 is a documented fallback).

The provisioner is registered by the Host's AddHistorianProvisioning (Runtime.ServiceCollectionExtensions), built via GatewayHistorian.CreateProvisioner, and gated on the same ServerHistorian:Enabled flag as the read path — WithOtOpcUaRuntimeActors resolves IHistorianProvisioning and passes it into the applier. When the section is disabled the applier binds the no-op NullHistorianProvisioning and provisioning does not run. Every dispatch logs a tally (dispatched/requested/ensured/skipped/failed); dispatched=N with requested=0 flags the dormant no-op. The API key must carry historian:tags:write for EnsureTags. (PR #423 shipped GatewayTagProvisioner but left this wiring out, so provisioning was dormant; restored 2026-06-27.) DataType mapping (issue #441): EnsureTags is an upsert (create-or-update, not skip-if-exists). HistorianTypeMapper maps Boolean → Int2 (Boolean historizes as a small 0/1 integer) because the historian's Int1 analog-tag creation path is server-degenerate — EnsureTags(Int1) stores an unusable stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see the "Boolean historization maps to Int2" section in docs/Historian.md.

Gateway-side prerequisites

The target HistorianGateway OtOpcUa points at must run with:

• RuntimeDb:Enabled=true — enables the WriteLiveValues SQL live path (continuous historization).
• RuntimeDb:EventReadsEnabled=true — enables ReadEvents from Runtime.dbo.Events (alarm-history reads).
• An API key carrying scopes historian:read, historian:write, historian:tags:write.

Migration note (deployments upgrading from the Wonderware backend)

The ServerHistorian section changed shape. Rename the old Wonderware keys and supply the key via env:

Old (Wonderware) key	New (gateway) key
ServerHistorian:Host + :Port	ServerHistorian:Endpoint (https://host:5222)
ServerHistorian:SharedSecret	ServerHistorian:ApiKey (env ServerHistorian__ApiKey)
ServerHistorian:ServerCertThumbprint	ServerHistorian:CaCertificatePath (+ UseTls / AllowUntrustedServerCertificate)

The AlarmHistorian section's old Wonderware connection keys (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from ServerHistorian. See docs/Historian.md for the full guide.

Live-validation gate — PASSED (was "KNOWN LIMITATION 1")

No longer a limitation. The cutover was live-validated against the real gateway on wonder-sql-vd03: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a confirmed protocol limitation (the captured CM_EVENT wire never carries SourceName, so the gateway cannot populate Source_Object — not a fixable bug). Evidence: docs/plans/2026-06-27-otopcua-historian-followups.md.

Kept here because it is the recipe to re-run the gate after any gateway or backend change (VPN to wonder-sql-vd03, gateway running the prerequisites above):

export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret>             # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag>                         # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag>          # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag>            # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name>                      # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"

The live suite skips cleanly when these env vars are absent (safe to run offline on macOS). It is the gate the operator runs on the VPN before trusting the cutover.

KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)

The ContinuousHistorizationRecorder is fully wired (actor + FasterLog outbox + gateway value-writer + meters). It is spawned with an empty initial ref set (Array.Empty<string>() in WithOtOpcUaRuntimeActors) because the deployed address space — and thus the historized-ref set — is built later at deploy time, not at actor-spawn time. The ref-feed gap is now closed: on every deploy the AddressSpaceApplier.FeedHistorizedRefs hook computes the added/removed historized-ref delta and posts it through ActorHistorizedTagSubscriptionSink → the recorder's UpdateHistorizedRefs message, and the recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe Tell; a faulting feed can never break a deploy).

The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a production-fatal defect — see docs/plans/2026-08-01-continuous-historization-live-gate.md. Three things to know:

• ⚠️ ConfigureAwait(false) in an Akka handler destroys Self. OnValueChangedAsync touched Self after awaiting IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false), which discards Akka's ActorTaskScheduler synchronization context — so ActorBase.Context was gone and it threw NotSupportedException on the FIRST value. The throw reached supervision, the actor restarted, and because production seeds an empty ref set the restart discarded every historized ref: one value captured, then silence. Fixed by capturing IActorRef self = Self before the await (the idiom the GetStatus handler already used for Sender).
• ⚠️ A fake that completes synchronously hides this entire class of bug, and 13 tests passed because of one. ValueTask.CompletedTask never yields, so the continuation ran inline with the context intact. Worse, the obvious fix to the fake — await Task.Yield() — is also vacuous: Yield() posts back to the captured context, so the caller still resumes on the actor thread. The fake must complete on a pool thread (await Task.Delay(1).ConfigureAwait(false)), as a real per-entry fsync does. Only reverting the production fix exposed that the first regression test proved nothing.
• Both pair nodes historized the same value — the recorder had no Primary gate, unlike the alarm store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as #537 and fixed — see the paragraph below.

Gate COMPLETE — #491 CLOSED 2026-08-04. Outbox append, drain, WriteLiveValues, restart-convergence, EnsureTags provisioning (ensured=2 failed=0) and read-back all ✅. The read-back leg ran against the licensed wonder-sql-vd03 with the VPN up: two registers carrying distinct non-zero constants (8224, 120) each read back as its own value — a 0-valued source would have proved nothing (the #536 trap).

Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison tag); evidence in the same plan doc. Two things to know before touching either.

• #538 — the discriminator is "did another tag write this pass?", NOT a failure count. DrainOnceAsync writes every per-tag group and acks all-or-nothing (deliberate: RemoveAsync truncates through an id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned the FIFO head while every healthy tag was re-submitted on every pass, forever — duplicate samples, an outbox that never truncated (observed 8.3 MB; signature = the gateway's ok and error counters climbing together). The drain now quarantines a tag after MaxWriteAttempts consecutive failures, but ⚠️ only counts a failure on a pass where some other tag succeeded — otherwise an ordinary gateway restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so the load-bearing test is the falsification control (A_backend_that_fails_every_tag_abandons_nothing). Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway.
• #537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink. The alarm queue is in LocalDb and replicates, so the Secondary holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of stale duplicates on promotion). ⚠️ And the peer that matters is one recording the SAME refs, not one holding the same rows — a node stands down only for a Primary in its own per-cluster mesh, and a node with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is usually running another cluster's configuration, and deferring to it ends this cluster's historization silently (the drain reports success on an empty outbox forever) — the same scope mismatch that once suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on RedundancyRoleView behind IHistorizationRoleView, registered unconditionally on driver nodes (NOT alongside the alarm IRedundancyRoleView, which only exists when that section is enabled — the #511 registration accident).

Also re-confirmed live: EnsureTags provisioning fires only for newly-ADDED historized tags — flipping an existing tag's historianTagname produces no provisioning at all (2026-06-27-historian-gateway-integration-issues.md Issue 5).