dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
docs/s7-counter-evidence
lmxopcua/CLAUDE.md
T

1590 lines
164 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Goal
Build an OPC UA server (.NET 10) that exposes industrial data sources —
including AVEVA System Platform (Wonderware) Galaxy — under a unified
Equipment-based address space. Galaxy access flows through the in-process
`GalaxyDriver` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) talking gRPC
to a separately installed **mxaccessgw** gateway process. The gateway owns the
MXAccess COM bitness constraint (its worker is x86 net48); everything in this
repo is .NET 10. PR 7.2 retired the legacy in-process `Galaxy.Host` /
`Galaxy.Proxy` / `Galaxy.Shared` projects + the `OtOpcUaGalaxyHost` Windows
service.
See `docs/v2/Galaxy.Performance.md` for the runtime perf surface
(tracing, metrics, soak harness).
## Related Projects (scadaproj umbrella)
This repo is one of a family of related SCADA / OT / Wonderware / OPC UA
**sister projects** that live as sibling directories under `~/Desktop/`. They are
**separate repos and separate processes**, coupled at runtime over wire protocols
(gRPC + OPC UA) — not by project/compile references — and share the
`ZB.MOM.WW.*` product namespace. The common subject is AVEVA System Platform
(Wonderware) "Galaxy" data, and `mxaccessgw` is the linchpin the others connect
through.
- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) —
the umbrella/index workspace that aggregates the whole family (a high-level scan
of each sister project: purpose, location, stack, primary commands). It also hosts
the shared `ZB.MOM.WW.*` libraries — `ZB.MOM.WW.Auth`, `ZB.MOM.WW.Theme`,
`ZB.MOM.WW.Health`, `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Configuration`, and
`ZB.MOM.WW.GalaxyRepository`. **This repo is indexed there** — see the OtOpcUa
entry in `../scadaproj/CLAUDE.md`.
- **`~/Desktop/MxAccessGateway`** (`mxaccessgw`,
[`../MxAccessGateway/CLAUDE.md`](../MxAccessGateway/CLAUDE.md)) — the gRPC gateway
that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session
x86 net48 worker). **OtOpcUa's `GalaxyDriver` depends on this** for live Galaxy
read/write/subscribe and Galaxy Repository browse.
- **`~/Desktop/HistorianGateway`**
([`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process
gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse
(via the shared `ZB.MOM.WW.GalaxyRepository` lib). Patterned on `mxaccessgw` but with
no COM / no x86 worker. **OtOpcUa's default historian backend** (`Driver.Historian.Gateway`)
consumes it; `ServerHistorian:Backend=SqlServer` selects a plain SQL Server historian instead (see Historian section).
- **`~/Desktop/ScadaBridge`**
([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) — the distributed SCADA
platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's
OPC UA address space.
**Propagate cross-repo changes to the umbrella index.** When a fact the index records
about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow,
shared-lib consumption, or per-project commands — update the **OtOpcUa entry in
`../scadaproj/CLAUDE.md`** in the same change so the index never drifts from this repo.
**Since 2026-09-03 that entry is a short summary row.** The detailed OtOpcUa narrative —
change history, live-gate records, redundancy/auth/ACL behaviour, and the per-issue
consumer-impact record for ScadaBridge — lives in
`../scadaproj/.claude/skills/sister-project-history/otopcua.md` (the `sister-project-history`
skill). Append detail **there**; keep the index row to a few lines and only touch it when
the summary itself (purpose, wire contract, driver set, auth posture, commands) changes.
**The index edit belongs on `scadaproj`'s `main`, and must be pushed.** `scadaproj` is a
separate repo with its own checkout, so committing there inherits whatever branch it
happens to be on — which is usually *not* `main` and is often an unrelated feature branch
with no upstream. That silently drifts the index for as long as that branch stays
unmerged, which is the exact failure the rule above exists to prevent. Do this explicitly:
```bash
cd ~/Desktop/scadaproj
git rev-parse --abbrev-ref HEAD # note it, to restore afterwards
git stash list # never commit onto a dirty unrelated branch
git checkout main && git pull
# …edit the OtOpcUa row in CLAUDE.md and/or .claude/skills/sister-project-history/otopcua.md…
git commit -am "docs(index): <what changed about OtOpcUa>"
git push origin main
git checkout - # restore the original branch
```
Observed 2026-07-27: the `Sql` poll driver's index entry had sat unpushed on an unrelated
feature branch since 2026-07-24 because it was committed onto the current checkout, so the
index disagreed with reality for three days despite this rule. If you find index commits
stranded on a feature branch, cherry-pick them onto `main` rather than merging that branch.
> ⛔ **Wonder box remote access (2026-09-11): no `psbridge`, no `winrm`.** Both skills are retired/disabled —
> the WinRM remote-exec channel set off the security software on the wonder boxes (vd03/vq03/vq04). Do not
> run remote commands (service control, publish/swap, PowerShell) on those boxes and do not stand up a
> replacement channel until the user clears one. Still allowed: UNC/SMB file copies and logging directly
> into the SQL Servers. Deploy steps that need a command on the box are blocked; see the umbrella
> [`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md) for the current hold.
## Architecture Overview
### Data Flow
1. **Galaxy Repository DB (ZB)** — SQL Server database holding the
deployed object hierarchy and attribute definitions. The
mxaccessgw's `GalaxyRepositoryClient` queries it via gRPC; the
driver consumes the materialised hierarchy through
`IGalaxyHierarchySource`.
2. **MXAccess (via mxaccessgw)** — Live read/write/subscribe over a
gRPC session. The gateway owns the COM apartment + STA pump
server-side; the driver speaks `MxCommand` / `MxEvent` protos
exclusively.
3. **OPC UA Server** — Exposes the deployed tree under **two OPC UA namespaces**
(see below). Galaxy tags are bound by `TagConfig.FullName`
(`tag_name.AttributeName`); reads/writes/subscriptions are translated to that
reference for MXAccess.
### v3 OPC UA Address Space (Batch 4): dual namespace
The server exposes **two OPC UA namespaces** (replacing the single
`https://zb.com/otopcua/ns`), built by `AddressSpaceComposer` /
`AddressSpaceApplier` and served by `OtOpcUaNodeManager` (which registers both
URIs; `V3NodeIds` + `AddressSpaceRealm` are the identity authority):
| Namespace URI | Realm | Subtree | NodeId `s=` scheme |
|---|---|---|---|
| `https://zb.com/otopcua/raw` | `AddressSpaceRealm.Raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
| `https://zb.com/otopcua/uns` | `AddressSpaceRealm.Uns` | the UNS tree (Area → Line → Equipment → signal) | slash-joined **`Area/Line/Equipment/EffectiveName`** |
- **Single source, fanned to both.** Every device value has exactly one source —
the raw tag's node. A `UnsTagReference` projects that raw tag into an equipment
as a UNS-namespace variable that carries an **`Organizes` reference to its raw
node** and mirrors it. One driver publish for a RawPath fans (in `DriverHostActor`)
to the raw NodeId AND every referencing UNS NodeId with identical
value/quality/timestamp — the mux key stays single (RawPath).
- **Writes via either NodeId** route to the same driver ref under the same
`WriteOperate` gating (the node-manager write gate is realm-qualified and fails
closed); a failed device write reverts both NodeIds via the shared fan-out
(write-outcome self-correction).
- **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and
fan via SDK notifiers to the raw device folder AND every referencing equipment
folder — one `ReportEvent`, one Server-object copy; ack/confirm/shelve route on
`ConditionId = RawPath`.
- **Historian dual-registration.** Both NodeIds register the **same** historian
tagname; the mux `HistorizedTagRef` stays single (RawPath). HistoryRead works
through either NodeId.
The retired `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme and
the single-namespace `EquipmentTags` materialization path are gone. See
`docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` +
`docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`.
**How a deploy reaches that address space (surgical-deploys, 2026-08-15).** The old
blanket rule — ANY non-empty `ChangedRawTags` / `ChangedUnsReferenceVariables` routes to
`Rebuild` — is gone, because it made a routine edit a fleet-wide outage:
`RawTagPlan.ReferencingEquipmentPaths` is part of the record's equality, so adding or
dropping **one** `UnsTagReference` flipped its backing raw tag to Changed and tore down
every node on the server, killing every client subscription. `AddressSpaceChangeClassifier`
is still default-closed; what changed is which deltas it whitelists.
- **Still a full rebuild:** an **alarm-bearing** raw-tag delta (`Alarm != null` on either
side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an
**identity** change on a *paired* raw-tag delta (`TagId`/`DriverInstanceId`/`ParentNodeId`/`Realm`
— pairing is by RawPath, so this guard catches a tag deleted and re-created under the same path;
a plain **rename never pairs** and is served surgically as remove-one-node + add-one-node with any
referencing UNS mirror re-pointed, live-gated 2026-08-15),
a raw **container** (Folder/Driver/Device/TagGroup) rename or removal, `ChangedEquipment` /
`ChangedAlarms`, a non-surgical equipment-tag delta, and anything else the
`with`-equality whitelist does not name (including any FUTURE field).
- **Now surgical:** a **non-alarm** raw-tag delta over the in-place-updatable attributes
(`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` / `ArrayLength`)
and/or `ReferencingEquipmentPaths` — applied via `ISurgicalAddressSpaceSink.UpdateTagAttributes`
in the **Raw** realm — and **every** `UnsReferenceDelta` (add / remove / re-point /
backing-edit).
- ⚠️ **UNS mirrors repair through the RECORD, not the classifier.** A raw-side historian or
shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape
(`EnsureVariable` early-returns on an existing node). `UnsReferenceVariable` therefore
carries the backing tag's `Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}`, so the
same edit makes the reference record unequal and the planner emits a **coincident**
`UnsReferenceDelta`. Adding a newly-inherited field needs no classifier change — **omitting
one silently reopens the staleness.** Repair is **in place** (`UpdateTagAttributes`, Uns
realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an
identity change (re-point, effective-name edit) is remove+recreate — and only for that
one node.
- **A non-rebuild apply materialises the DELTA, not the composition.** `MaterialiseHierarchy`
still runs full (parents must exist); the other passes take the plan's added/changed subsets.
Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread.
- ⚠️ **A degraded non-rebuild apply does NOT advance `_lastApplied`.** The full passes used to
re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the
delta path removed that. Holding the composition back is the replacement — the next deploy
re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that
apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct,
bounded resync). A **rebuild** still advances even with failures.
- **…and the deploy ack now SAYS so (#601).** The Applied ack waits for `OpcUaPublishActor`'s
`RebuildCompleted` (timeout = shared `DeployTimings.ApplyDeadline` − 15 s = 105 s ⇒ degraded "outcome not reported in time", never clean) and carries an additive `Degraded`/`DegradedReason`
on both ack transports; central seals but records `NodeDeploymentState.FailureReason = "Applied degraded: …"`
(`/fleet` "Applied (degraded)"), and a re-dispatch of a degraded revision re-applies instead of short-circuiting.
See `docs/Redundancy.md` §"What an `Applied` ack means".
- **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is
what makes every whitelist above safe.
- **Announcements:** `AnnounceAddedNodes` announces surgically-added **Raw** containers/tags
(`ComputeRawAddAnnouncements`, #599 — the former "raw adds are undiscoverable until a re-browse" gap is
closed) as well as added UNS reference variables.
- **Array writability rides the same whitelist:** `ArrayWritable` joins the surgical raw-tag attributes and
`UnsReferenceVariable` carries `BackingArrayWritable` (see "Equipment-node writes" under Authentication). Removing
an equipment that has references removes the references before the equipment subtree, and
`RemoveEquipmentSubtree` severs the inverse `Organizes` edges.
See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`.
### Key Concept: Tag Name and FullName
Galaxy objects have a **tag_name** — a globally unique system name used for
MXAccess read/write — and attributes are referenced as `tag_name.AttributeName`
(e.g. `DelmiaReceiver_001.DownloadPath`). This dot-separated reference is what
`TagConfig.FullName` stores for a Galaxy equipment tag. The Galaxy address
picker browses the live hierarchy using contained names for navigation, then
resolves the selection to the `tag_name.AttributeName` form.
**Galaxy is a standard Equipment-kind driver.** Galaxy points are ordinary
equipment `Tag`s bound to `GalaxyMxGateway` via `TagConfig.FullName`
(`tag_name.AttributeName`), authored through the standard Tag modal + Galaxy
address picker on the equipment page's Tags tab — the same flow as Modbus or
S7. There is no alias machinery, no `SystemPlatform` namespace kind, and no
relay→alias converter. See
`docs/plans/2026-06-12-galaxy-standard-driver-design.md` for the full design.
### Data Type Mapping
Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`.
### Change Detection
`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys.
**The signal is consumed as an operator prompt, not as an address-space rebuild** (§8.2, 2026-07-27).
`DriverInstanceActor` subscribes in `PreStart` and unsubscribes in `PostStop` — the detach is
load-bearing, because the `IDriver` instance outlives the actor when the host respawns a child around
the same driver object. A raise is recorded and rides the driver-health snapshot
(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`) to the AdminUI `/hosts` page as a
**"re-browse" chip**.
⚠️ **It is advisory: a Galaxy redeploy still does NOT change the served address space.** v3 authors raw
tags explicitly through the `/raw` browse-commit flow, so an operator must re-browse the device and
commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact
records.
Two gotchas if you touch this:
- `PublishHealthSnapshot` dedups on a health fingerprint. The rediscovery timestamp **must** stay in
that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the
operator. Pinned by `DriverInstanceActorRediscoverySignalTests`.
- The shared test `StubDriver.GetHealth()` returns `DateTime.UtcNow`, so its fingerprint differs on
every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests
use a stub with stable health for exactly this reason.
**#507 is closed as superseded, and the injection path is deleted.** Its retained code read
`EquipmentNode.DriverInstanceId` and `EquipmentTags`, both *structurally empty* in v3
(`AddressSpaceComposer.cs`, `DeploymentArtifact.cs`), so removing the guard would have changed a log
line and injected nothing. Gone with it: `HandleDiscoveredNodes`, `PartitionDiscoveredByDeviceHost`,
`ApplyDiscoveredPlansForDriver`, the redeploy re-inject tail, `DiscoveredNodeMapper`,
`AddressSpaceApplier.MaterialiseDiscoveredNodes`, the connect-time discovery loop
(`StartDiscovery`/`RediscoverTick`/`DiscoveredNodesReady`/`TriggerRediscovery`), and the 18
`DiscoveryInjectionDormantV3` tests — v2 characterization tests asserting an equipment-rooted graft
(`EquipmentRootNodeId == "EQ-1"`) that v3 cannot produce. `ITagDiscovery` itself stays: the `/raw`
browse picker drives it through `Commons/Browsing/DiscoveryDriverBrowser`, which never went through
the actor.
**`IRediscoverable` is the ONLY tag-change signal, and four drivers do not raise it.** A periodic
tag-set drift check briefly existed (shipped `e77c8a35`, **removed** by Gitea **#523**): every
`SupportsOnlineDiscovery` driver was re-browsed on a 5-minute timer and the result diffed against its
authored tags. It was taken back out because it generated recurring, **unconfigurable** (interval was a
constructor parameter with no appsettings key), **never-live-gated** outbound traffic to real
PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the
design treated it as an implementation detail.
⚠️ **Consequence to know before you rely on rediscovery:** **AbCip and FOCAS browse a live backend but
implement no `IRediscoverable`, so they have no tag-change signal at all** — a PLC program re-download
or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT,
MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an
event-driven signal, or one that runs when something *else* already indicates change (a reconnect, a
deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`TagSetDriftDetector`,
`DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` /
`DiscoveryStreamIncludesAuthoredTags`.
**`IHostConnectivityProbe` is consumed (Gitea #521, closed).** `DriverInstanceActor` subscribes
`OnHostStatusChanged`, pulls `GetHostStatuses()` fresh, and the host set rides `DriverHealthChanged` to the
AdminUI `/hosts` page — no SQL: the `DriverHostStatus` table is dropped, and so is
`DriverInstanceResilienceStatus` (#524; the live data goes over the `driver-resilience-status` telemetry
channel only). **Multi-host dispatch is wired too:** `DriverInstanceActor` resolves the breaker/host key
per call via `IPerCallHostResolver` (Modbus, FOCAS, AbLegacy, TwinCAT, AbCip).
### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed)
Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into
`Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected
and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never
re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a
stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*.
The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a
non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it:
- ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.**
`Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the
read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a
perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely.
`LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any
success. Each conjunct has a test that goes red when the *other* is deleted.
- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across
all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect
supervisor, OpcUaClient via a bad session keep-alive (#670 — it used to stay `Healthy` forever; its SDK
`SessionReconnectHandler` is deleted, a lost session publishes `BadNoCommunication` once per outage and the
watchdog's re-init is the recovery; good keep-alives + notifications advance `LastSuccessfulRead`). `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has
no transport. Re-check this if a new driver lands.
- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.**
`ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable`
field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync`
leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose
first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts`
probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not*
via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with
nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first,
which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent —
the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new
`IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite.
- The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` /
`DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy
system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped
reading.
- **Fast path for DEFINITIVE losses (#676):** a driver implementing the opt-in `IDriverConnectionLossSource` raises
`ConnectionLost` (once per connection) when its protocol proves the connection can never recover in place — OpcUaClient:
keep-alive `BadSessionIdInvalid`/`BadSessionClosed`/`BadSessionNotActivated`/`BadServerHalted`, a SUSTAINED closed
connection (2nd `BadSecureChannelClosed`/`BadConnectionClosed` in one outage — what a real process stop produces; the
SDK never reopens it) or a subscription the SERVER reported dropped (its publishing-stall heuristic does not raise) — and `DriverInstanceActor` (attach PreStart, detach PostStop) turns it into the same disconnect only
while Connected; ambiguous failures (timeouts, `BadNoCommunication`, a stalled socket) must never raise and stay with
the watchdog's conjunction. `docs/DriverLifecycle.md`.
- **What comms loss now does downstream — one contract, six seams (archreview R3-02).** "A value consumer
never reports a quality better than its source's" is stated once in `docs/DriverLifecycle.md` §"The
quality contract" and implemented by `Core.Abstractions.InputQuality` (`WorstOf`/`IsUsable`). Three
things changed with it, all client-visible:
- ⚠️ **Value nodes of a comms-lost driver now go Bad.** `DriverHostActor.FanCommsLossToValueNodes`
stamps `BadNoCommunication` on the raw NodeId **and every referencing UNS NodeId**, **value retained
server-side**, via the new status-only sink primitive `IOpcUaAddressSpaceSink.WriteStatuses` (the host
holds no value cache, so only the node can keep the value). ⚠️ Retention is **not client-observable**:
the SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read/sample, so clients see
`Value = null` + the Bad status; the retained value is what the next Good re-publish and the
`(value, status)` dedup compare against. #481 had wired comms loss to the mux and to Part 9
conditions and stopped one seam short of the external data plane. A consumer that treats Bad as "drop
the tag" (ScadaBridge) will see more Bads — that is the fix working.
- **Nothing Good is written on reconnect** — recovery is the driver's own re-publish (the poll engine's
`forceRaise`, MQTT's new initial push). Instead the host **audits**: refs still latched 30 s after a
reconnect log a Warning and increment `otopcua.driver.comms_latched_refs`. Non-zero on that counter
means a driver reconnects without re-publishing.
- **Every dedup on the recovery path is keyed on (value, status), not value.** A device that comes back
usually comes back holding the same reading, so a value-only dedup swallows the recovery and the node
stays Bad forever — the v3-gate trap, now closed in the Calculation driver, the virtual-tag actor and
(via a connection-loss stale) MQTT's plain-mode suppression. **Any new consumer of driver values must
repeat this**; a regression test for it must replay the IDENTICAL value or it proves nothing.
- Consequences per seam: MQTT/Sparkplug stale on connection loss and push cached snapshots on every
(re)subscribe (`docs/drivers/Mqtt.md`); the Calculation driver and virtual tags **hold** while any
input is Bad and inherit the worst input's sub-code, evaluate through Uncertain, and hand scripts the
input's real `StatusCode`/`SourceTimestampUtc` (`docs/drivers/Calculation.md`, `docs/ScriptEditor.md`)
— so a script that relied on receiving `null` for a dead input no longer runs at all while it is Bad.
See `docs/DriverLifecycle.md` §"The comms-loss watchdog" and §"The quality contract".
## mxaccessgw
The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.
## Build Commands
```bash
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test ZB.MOM.WW.OtOpcUa.slnx # all tests
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single test project
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test
```
`Category=SlowUnit` (#606) tags the 16 wall-clock-bound Driver.Mqtt unit tests (blackhole connect-timeout /
reconnect-backoff windows, ~1 min of the project's 73 s). CI runs them (its filter excludes only E2E +
LiveIntegration); for a fast local loop add `--filter "Category!=SlowUnit"`.
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`)
is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle
shipped. `Core.AlarmHistorian` now carries a surgical direct pin of
`SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the
SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls
transitively. `dotnet restore`/`build` is clean with the audit active.
Test projects live under `tests/<module>/` (Core, Server, Drivers,
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
Unit suites are named `*.Tests`; integration suites are `*.IntegrationTests`
and need their Docker fixture up (see Docker Workflow). DB-backed tests in
`*.Configuration.Tests`, `*.Admin.Tests`, and `*.Server.Tests` require the
central SQL Server.
## Docker Workflow (driver fixtures + central SQL Server)
> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing +
> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate**
> recipes (S7 blackhole, HistorianGateway LiveIntegration), the integration-test
> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail.
> **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28).
> ⚠️ **The `project: lmxopcua` label is aspirational, not universal.** This note used to claim every fixture compose file carries it. As of 2026-07-24 only the **MQTT** fixture actually does — `grep -rn "labels:" tests --include=*.yml` matches `Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml` and nothing else. So `docker ps --filter label=project=lmxopcua` returns the MQTT fixture alone, not the fleet. Add the label when you touch an older fixture; until then enumerate by container name.
Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-<driver>/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`.
**`docker -H ssh://...` does NOT work from the Windows VM.** Windows OpenSSH ↔ docker.exe stdio bridging hangs (`docker system dial-stdio` runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs `docker compose` server-side.
> **On macOS there is no `lmxopcua-fix` — drive the host over SSH directly.** The helper is a Windows-VM
> script; `~/bin` is empty on the Mac and the commands below will not resolve. Use passwordless SSH
> (and `rsync` in place of `sync`), which is what `infra/README.md` §1 documents as the Mac path:
>
> ```bash
> ssh dohertj2@10.100.0.35 'docker ps --format "{{.Names}}\t{{.Status}}"'
> ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mqtt && docker compose up -d'
> rsync -av tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/ \
> dohertj2@10.100.0.35:/opt/otopcua-mqtt/ # the `sync` step, by hand
> ```
>
> A local container runtime *does* exist on the Mac (OrbStack), so the `docker-dev/` rig itself runs
> locally there — it is only the shared **fixtures** that live on `10.100.0.35`.
**Use `lmxopcua-fix.ps1` (in `~/bin`) to control fixtures from the Windows VM:**
```powershell
lmxopcua-fix ls # list all lmxopcua-tagged containers on the host
lmxopcua-fix up modbus standard # bring a profile up
lmxopcua-fix up abcip controllogix
lmxopcua-fix up s7 s7_1500
lmxopcua-fix up opcuaclient # single-service stack, no profile arg
lmxopcua-fix down modbus # tear stack down
lmxopcua-fix logs modbus
lmxopcua-fix sync modbus # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/
```
**`sync` is the deployment step.** When you edit a fixture's compose file or Dockerfile under `tests/.../Docker/`, run `lmxopcua-fix sync <driver>` to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; `/opt/otopcua-<driver>/` is a mirrored deployment.
**Endpoints (defaults already point at the docker host):**
- SQL Server (always-on): `10.100.0.35,14330` — used by `appsettings.json` for `ConfigDb`.
- Modbus: `10.100.0.35:5020` (`MODBUS_SIM_ENDPOINT`)
- AB CIP: `10.100.0.35:44818` (`AB_SERVER_ENDPOINT`)
- S7: `10.100.0.35:1102` (`S7_SIM_ENDPOINT`)
- OPC UA reference (opc-plc): `opc.tcp://10.100.0.35:50000` (`OPCUA_SIM_ENDPOINT`)
- MQTT (Mosquitto, **auth on both listeners — no anonymous fallback**): `10.100.0.35:8883` TLS+auth
(`MQTT_FIXTURE_ENDPOINT`) · `10.100.0.35:1883` plaintext-but-authenticated (`MQTT_FIXTURE_PLAIN_ENDPOINT`).
Also needs `MQTT_FIXTURE_USERNAME` / `MQTT_FIXTURE_PASSWORD`, and `MQTT_FIXTURE_CA_CERT` to pin the
fixture CA (`scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt`). A co-running publisher loops
JSON under `otopcua/fixture/…`.
- MQTT **Sparkplug B**: the same broker, plus the `sparkplug` compose profile's `otopcua-sparkplug-sim`
(project-owned C# edge-node simulator) — group **`OtOpcUaSim`**, edge nodes **`EdgeA`**/**`EdgeB`**,
`EdgeA` device **`Filler1`**, ~2 s cadence. It subscribes to `spBv1.0/OtOpcUaSim/NCMD/{node}` and
re-births on request. **Births are never retained**, so `docker restart otopcua-sparkplug-sim` is how
you force a fresh one while a browse window is open.
Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at `opc.tcp://localhost:4840` — **that's not on the docker host**.
**Local docker-dev rig — login is DISABLED, so do live `/run` verification yourself (don't wait for the user to sign in).** The local `docker-dev/docker-compose.yml` stack (AdminUI at `http://localhost:9200` via Traefik; OPC UA `opc.tcp://localhost:4840` central-1 / `:4841` central-2) runs the AdminUI with `Security__Auth__DisableLogin: "true"` — **no sign-in form; it's auto-authenticated as a full-access admin.** So AdminUI / Razor `/run` verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against `http://localhost:9200`) does **not** require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA *data-plane* auth is still real LDAP against the shared GLAuth on `10.100.0.35:3893` — that only gates Client.CLI read/write **role** operations, e.g. binding a `multi-role` / `opc-writeop` user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the `ZB.MOM.WW.HistorianGateway` sidecar — still need the user.)
See `docs/v2/dev-environment.md` for the full inventory and rationale.
## Build & Runtime Constraints
- Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint
is owned by the mxaccessgw worker (x86 net48), not by anything in
this repo.
- The gateway's MXAccess worker requires a deployed ArchestrA Platform
on the machine running the gateway. The OtOpcUa server itself does
not.
## Transport Security
The server supports configurable OPC UA transport security via the `OpcUa:EnabledSecurityProfiles` list in `appsettings.json`. Phase 1 profiles (the `OpcUaSecurityProfile` enum members): `None` (default), `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`. Security policies are built from the enabled profiles by `BuildSecurityPolicies` at startup. The server certificate is always created even for `None`-only deployments because `UserName` token encryption depends on it. See `docs/security.md` for the full guide.
## Redundancy
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard".
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
## Role authority (`EffectiveRoles`)
**A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and
01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence —
`Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a
plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix
(`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running
deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one
`EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that
SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and
`AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to
disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs`
read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right
Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field
incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`).
- **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role
appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing
`Cluster:Roles` would leave the bound options disagreeing with the registered instance.
`AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op
today since none of the four shipped overlays carries a `Cluster` section.
- **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles`
still wins outright) — the boot banner also names which source won (`source=Configured` /
`Environment` / `None`).
- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of
production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through
`EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role
during a rolling upgrade. ⚠️ **Accepted residual (#620), not closed by R3-04:**
`RedundancyStateActor`'s election delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`,
which matches via Akka's OWN ordinal `Member.HasRole`. Deliberately **not** patched — neither in the
library nor with a local case-insensitive pre-filter, which would split the Primary election from
`ClusterSingletonManager` placement (also ordinal). It only bites a mixed fixed/pre-fix rolling-upgrade
window; finish the upgrade. Full rationale: `docs/Redundancy.md` ("Accepted residual (#620)").
- **`Program.cs` wraps the composition root** (#618): a start-up failure logs Fatal, writes `FATAL:` to
stderr (the bootstrap logger is still silent before `AddZbSerilog`), flushes, and `Environment.Exit(1)`s
— leftover Akka/Kestrel threads would otherwise keep the process alive. `HostAbortedException`
(design-time tooling) passes through untouched.
- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement:
a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads
`OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal —
with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same
English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI
RBAC role).
See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and
`archreview/plans/R3-04-one-role-authority-plan.md`.
## Mesh command transport (`MeshTransport`)
**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
`MeshTransport:Mode`:
- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
it works only when central and the node share a gossip ring, which after Phase 6 means within one
2-node mesh.
- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
node on anything else.
Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
applies to either direction.
Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
stops at the first success.
Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
on every unresolvable case — constant-time compare, no key material logged. Production keys are
`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
Things that carried over unchanged from the `ClusterClient` era, because they were never
`ClusterClient`-specific to begin with:
- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
`ScriptedAlarmHostActor`) accept a command from either transport identically.
- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
deletion — it was always about the deploy path, not the transport.
`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
is the original Phase 2 design doc and is likewise left as a historical record of the transport this
section now describes as deleted.
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
## Config source (`ConfigSource` / `ConfigServe`)
Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
fail-closed). Things worth knowing before touching it:
- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
`RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
**The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** —
`ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and
hashes the result, so the invariant holds unchanged and nothing between the composer and
`DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever
looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so
pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every
decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already
handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after
upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be
upgraded BEFORE central starts emitting compressed blobs.**
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
`FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
**Malformed non-empty bytes are "no answer" too** — on cache boot, the `FetchAndCache` boot reconcile and
the subscription push alike (the lenient swallow leg is closed). A cache boot from malformed bytes does
**not** adopt the revision (`Direct` ⇒ Stale; `FetchAndCache` ⇒ no revision, the re-dispatch fetches).
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
SQL today; `StoreAsync` is idempotent and the cache replicates).
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
`Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching
key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is
no `OTOPCUA_CONFIG_MODE` env var in the compose file. See
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
touching it:
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
either mode.
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport
(`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542):
`FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end**
(no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its
`DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just
central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
is ignored, and it never permanently gives up (observability, not data plane). `TelemetryLocalHub.Evict` drops a removed driver's snapshots (`StopChild`, via `plan.ToStop`; central-side
stores are not evicted). The node-local hub
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
node stream is up) — the same semantics the DPS pill has always had.
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
§`Telemetry`/`TelemetryDial` for the config keys, and
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
gate procedure.
## LocalDb pair-local store (Phases 1 + 2)
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
**Phase 1** caches the
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
exception).
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
same event accepted on both nodes — which happens in every boot window, since
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
**The library pin is `0.3.4`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and
`.Replication`), bumped from `0.2.1` on 2026-09-03 (was itself bumped from `0.1.3` on 2026-08-15).
The 0.2.1→0.3.4 bump is wire-compatible with 0.2.x, and the on-disk bookkeeping schema v2 → v3
upgrades in place on open, so no rig or volume action was needed. `RegisterReplicated(string)` is
unchanged and everything 0.2.x/0.3.x added is additive, **except** that `RegisterReplicated` (0.3.3)
now throws `LocalDbRegistrationException` for a table that is the parent of a row-mutating ON
DELETE FK — none of this repo's four replicated tables have such an FK, so this did not fire.
0.3.0 made the snapshot-resync flag **per-table** (see the ⚠️ below, now corrected); 0.3.2 added
validated config `Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, left
unset here) and a public `LocalDbOplogGapException`, neither yet wired to a call site. **Its two
new APIs — `DeregisterReplicated` and `RegisterReplicated(table, baselineExistingRows:)` — are now
both used** (2026-08-18); the bullets below are the shape that landed, not the bump itself. Two
things are worth knowing:
- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching
**byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized
under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row
cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which
materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps
`16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the
message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap"
was corrected with the bump; do not restore that rationale.
- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).**
`LocalDbSetup.OnReady` reads one predicate, `LocalDbReplicationPolicy.CapturesChanges`
(via `LocalDbSetup.ReplicationIsConfigured`):
**`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is
not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it
alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` +
`LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only,
silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is
fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has
nothing to capture for.
- ⚠️ **The alarm DRAIN gate uses a different predicate** — `LocalDbReplicationPolicy.SharesRowsWithPeer`:
`ApiKey` **and** (`PeerAddress` **or** `LocalDb:SyncListenPort > 0`), i.e. "a peer actually holds these
rows". Capture must be wider than the drain gate (a dialling-only node still captures); a node with no
peer must still drain, so the two predicates are deliberately not the same.
The **passive** half (no `PeerAddress`) cannot name its peer, so it stands down for its **mesh partner**
(the other member of its `cluster-{ClusterId}` role); a legacy passive half with no mesh role still
drains alongside its partner (duplicate historian rows). ⚠️ Before a node has joined its mesh the
published Primary verdict (and the live data-plane gate, which must stay the same decision) **allows** —
for seconds, up to ~25 s under the bootstrap guard — so a cold-starting node can briefly co-hold the
Sparkplug primary-host claim with its Primary peer.
- **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the
same load-bearing DDL→register order. Baselining is what makes enabling replication on a node
that has been running *without* it actually converge: capture is CDC and the snapshot streamer
pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in
used to replicate only writes made after the restart. It seeds the ledger at the LWW floor
(HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote
write of the same key and wins only where the peer has no version at all. It is idempotent, so
it is free on every boot after the first. As of `0.3.0` (this repo pins `0.3.4`) the snapshot
flag is **per-table**, not per-database — baselining one table no longer streams every
registered table in both directions. (Prior to `0.3.0`, on the `0.2.1` pin this repo carried
until 2026-09-03, the flag was per-database and baselining anything streamed *every*
registered table in full; the old advice to trim `alarm_sf_events` before enabling replication
no longer applies at the current pin.)
- **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time
self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and
prunes that table's oplog + row-version rows. Merely declining to register would leave every
pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row
inside every write transaction — on precisely the node that has no peer to capture for. The
call returns `true` when it removed something, so the Information line fires **once, and only
when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is
Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where
resolving one from DI would nest a resolution inside a singleton construction; `Program.cs`
assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe
the line rather than infer it.
- ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake
compares registered-table digests **fail-closed**, so a node that deregisters a table its peer
still replicates stops syncing with a schema mismatch. Enabling replication on an established
pair means setting the config and **restarting both nodes** — do it on both, or on neither.
- **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was
enabled from a fresh pair, so the impact of the old gap was low — central and site-b are
default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests`
pins both branches (registered-table set on a configured node, empty set + zero triggers +
the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
cache), `docs/AlarmHistorian.md`, and the runbook
`docs/operations/2026-07-20-localdb-pair-replication.md`.
## Phase 4 — cut the driver-side ConfigDb connection
Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles`
has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs`
gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is
unaffected (it keeps ConfigDb via its admin role). Consequences:
- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator`
fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused
node may stay `Direct`.
- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not
spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db
factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and
`Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes
**240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture
— where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself
is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less
(driver-only) nodes".
- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore`
(replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces
`EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed
`ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration
`20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted.
- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on
a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is
the system of record.
- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty
`NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so
`OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only —
the same rows a driver node never received via config either, so this is not a regression.
Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and
Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4.
## Authentication: API keys on the data plane, LDAP on AdminUI (#520)
⚠️ **Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine,
so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an
`otua_<keyId>_<secret>` credential as the password. `ApiKeyOpcUaUserAuthenticator`
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/`) implements `IOpcUaUserAuthenticator`: one indexed lookup
on `keyId` plus one peppered HMAC-SHA256, `FixedTimeEquals`, **no network I/O**. The key resolves to a
**principal**; `NodeAcl.Principal` rows grant it permissions. `LdapOpcUaUserAuthenticator` is
**deleted**, along with four `Security:Ldap` keys that existed only for it (`AuthTimeoutMs`,
`MaxConcurrentAuthentications`, `OutageFailureThreshold`, `OutageCooldownSeconds`) — they bounded an
LDAP bind running inline in the SDK's synchronous, `SessionManager`-lock-holding impersonation
callback, and nothing on the remaining LDAP path has that property.
**LDAP remains, for AdminUI login only** (`Security:Ldap`, `OtOpcUaLdapAuthService` →
`ILdapAuthService`, group→role mapping via `GroupToRole` + `LdapGroupRoleMapping`). Do not "restore"
the data-plane authenticator when you see LDAP wiring in `Program.cs` — it is there for the AdminUI.
Two things to know before relying on any of this:
- ⚠️ **Node ACLs are enforced only behind `Security:Acl:Enabled` (default `false`).** With the flag
**off**, a key's grants collapse into the two server-wide role gates (`WriteOperate` on Value writes,
`AlarmAck` on Part 9 methods), **only `Cluster`-scoped grants count** — `ApiKeyInterimRoleMapper`
deliberately under-grants, because honouring a narrower grant's flags while unable to honour its
*scope* would turn "WriteOperate on Equipment X" into server-wide write authority — and **read,
browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous**. With the
flag **on** (`DataPlaneAuthorizer`), scoped grants decide **every** data-plane operation: write and
alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6).
- **The two role gates re-check the session's KEY on every call, flag off or on (Gitea #563).** Roles
are resolved once at activation, so an expired / deployment-disabled key used to keep `WriteOperate` /
`AlarmAck` on its open session until the client disconnected. `DataPlaneAuthorizer.AuthorizeKeyUsable`
runs the same `FindUsableKey` probe against the same snapshot + clock as the ACL path (one dictionary
lookup, no I/O) and denies `BadUserAccessDenied` before the role check. **Only those two gates** —
reads/subscriptions stay ungated with the flag off, by design. Anonymous carries no key and is
unaffected. No flag: a dead key keeping write authority is a defect, not a rollout choice. ⚠️ On a
fused node a key minted after the last deploy authenticates (ConfigDb) but cannot write/ack until a
deployment carries it — the flag-on path always behaved this way.
- ⚠️ **The ACL decision REPLACES the role check — the two are never chained.** Chained, a grant would
have to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped
grants confer a role, so every narrow grant would stay inert and the flag would appear to do
nothing. The load-bearing test is therefore an **allow** — a session carrying *no roles* writing
successfully on the strength of its grant. A deny-only test passes against a build where the ACL is
never consulted, because the role gate denies too.
- **The trie is built inside `DataPlaneAuthorityState.Apply`**, not by a second wiring at the publish
actor, and the state owns its `PermissionTrieCache` rather than being handed one. An uninstalled
trie denies everything, which is indistinguishable from working enforcement until someone checks
that an ALLOW is possible at all.
- **A deploy reaches sessions that are already open**, in both directions: each decision stamps the
*current* generation rather than the one the session activated under, and re-checks the
authenticating key — so revoking a key or a grant stops an open session immediately instead of at
the client's next reconnect. On a driver-only node the artifact is the only revocation channel.
- `UserAuthorizationState` is built **per decision walk, not per session** — and since the hot-path
change it is allocated only on a cache **miss**, so a cache hit never constructs one. Its LDAP-shaped
freshness/staleness predicates are switched off EXPLICITLY (`AuthCacheMaxStaleness = MaxValue`).
⚠️ Relying on "the stamp is always fresh" was not safe: `MembershipResolvedUtc` comes from
`DataPlaneAuthorizer`'s `TimeProvider` while `IsStale` is evaluated against
`TriePermissionEvaluator`'s, so **any divergence between the two clocks denies every request** with a
verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead.
- **Stage 6's hot seams (read/browse/subscribe) go through a per-`(session, node, operation, realm)`
decision cache**, cleared wholesale when the generation advances. It is **sharded per API key**
(`ApiKeyId ?? Principal` — despite the type name, sessions sharing one key share one shard and one
budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and
not every other key's decisions. ⚠️ It caches the **grant walk only** — key revocation and expiry are
re-checked **before** the cache lookup on every call, because expiry moves with the clock rather than
with a deployment.
- **Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors.** A
grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward
only, so `PermissionTrieNode.BrowseTransitPrincipals` marks the path. Planting a real `Browse` grant
on each ancestor instead would be collected by the additive downward walk and leak `Browse` onto
every sibling subtree beneath that ancestor. ⚠️ **The topmost ancestor is the single shared root
folder** (`s=OtOpcUa`, raw namespace) that BOTH realm trees hang under — its identifier is a display
name, not a path, so `DataPlaneAuthorizer.BuildScope` maps it onto the trie's **zero-segment
realm-root scope** (`V3NodeIds.SharedRootFolderId`, realm-qualified: in the UNS namespace that
identifier is an ordinary Area), and its Browse considers transit in **either** realm because it
fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse
gate denies the whole call when its source node is denied, so clients were stopped at the front door.
- **Revocation reaches live subscriptions** (stage 6): every other operation re-asks per request, but a
subscription is authorized once and then streams — and streaming is how machine consumers actually
read this server. `DataPlaneAuthorityState.Changed` fires `RevalidateMonitoredItems`. ⚠️ A denied
item is queued `BadUserAccessDenied` and then **detached from its `MonitoredNode2` fan-out — NOT set
`MonitoringMode.Disabled`**: disabling discards the item's queued notifications, so the client is
silenced without ever being told why (caught only by the over-the-wire test). Detaching is also
reversible, so restoring the grant resumes the feed with no client action. ⚠️ **`Changed` is no longer
the only trigger (arch-review 03/S19):** `AclRevalidationTimer`, armed by `OtOpcUaServerHostedService`
at `Security:Acl:RevalidationIntervalSeconds` (default 60 s, `0` disables, inert while
`Security:Acl:Enabled` is false), runs the same pass on the clock — because `ExpiresUtc` passes with
the clock, not with a deployment. The pass's verdict memo is **pass-local and must stay that way**;
hoisting it across ticks would serve an expiry that landed between two ticks the earlier tick's allow.
**Event/alarm items ride both triggers too (Gitea #562)** — a second map (`_eventItems`), fed by the
`SubscribeToEvents` override, holds every *direct* notifier subscription; a denied one is detached via
the SDK's own protected `SubscribeToEvents(…, unsubscribe: true)` (what a client delete does) and
re-attached with `unsubscribe: false` on restore — again never `Disabled`. Unlike a data item it is
**not told** (no value slot for a status; the feed goes quiet), and a client delete of a stood-down
item is absorbed by the override (the SDK would otherwise remove an item that is not attached).
**Server-object (`i=2253`) event subscriptions are filtered per event at delivery (Gitea #586).** The
`SubscribeToAllEvents` override attaches them to **nothing** (just records them in `_allEventsItems`);
each condition's own `OnReportEvent` (`OnConditionReportEvent`, fires once per event ⇒ still one
Server-object copy) queues the event to each recorded item whose session holds `Subscribe` on the
condition (RawPath) **or** any notifier folder it reports through — the per-folder path's question. The
root-notifier forward to the core Server object (`OnReportEvent` override) is **dropped**: it was the
ungated path, and it is the only one that served an item created before the alarm was deployed (the
SDK's `AddRootNotifier` subscribes existing all-events items with `unsubscribe: true`). Revalidation is
**implicit** — every event re-asks the authorizer, so a deploy or expiry lands on the next event with no
pass. `ConditionRefresh` on an `i=2253` item is filtered the same way. The dispatcher **must** keep
the SDK's audit guard (`AuditEventState` only while `Server.Auditing` over SignAndEncrypt/https),
because conditions report their Part 9 method audit events through themselves.
`ReconcileAllEventsMonitoring` must keep `AreEventsMonitored` raised on every condition while any
`i=2253` item exists, because `ConditionState` gates its method **audit** reports on
`EventsMonitored()` (method-driven state-change events no longer depend on it — #587, below). The
protected `SubscribeToEvents` override that no-ops all-events items is what
stops the SDK's `AddRootNotifier` (`unsubscribe: true` bug) taking that unit back. Both review
findings (review of `75e559f6`). ⚠️ The materialise path raises only the **new** condition's unit
(`RaiseAllEventsMonitoring`); a full reconcile there made a deploy of N alarms O(N²) under `Lock` with
ScadaBridge connected — keep the full reconcile to remove/rebuild/(un)subscribe.
**Model-change and audit events are gated too (Gitea #588).** The manager's own server-wide events
(`GeneralModelChangeEvent`, the rejected-write `AuditWriteUpdateEvent`) go through
`ReportToServerObject`: flag **off** ⇒ `Server.ReportEvent` as before; flag **on** ⇒ delivered per item to
`_allEventsItems` (same audience), a model change narrowed to the `Affected` nodes the session may
**Browse** (dropped when none — the builders' Messages name the node, so a narrowed copy gets a neutral
one), an audit write only with `Read` on its source node plus the SDK's audit/encryption guard. Any other
event type fails closed. Auditing is off by default (never configured in this repo).
- **ns=0 standard nodes stay ungated by design, even with `Security:Acl:Enabled`** (#617) — `ServerStatus`,
`NamespaceArray`, `ServiceLevel` etc. are served by the SDK's core node manager outside our authorizer and
clients read them to connect and to judge redundancy health; they carry no plant data. Every raw/UNS node,
event subscription and HistoryRead is still gated. `docs/security.md`.
- **Equipment-node writes: `EvaluateEquipmentWriteStructure` is the ONLY type check.** `OnWriteValue` bypasses
the SDK's own type check on every writable equipment node, so `OtOpcUaNodeManager.EvaluateEquipmentWriteStructure`
(plus `EvaluateEquipmentWriteRange`) is the whole gate. Scalars are deliberately lenient (drivers
range-check; ScadaBridge sends Double to a Float node). **Array writes** (#610): a numeric array of another
element type is **converted, checked, element by element** to the node's element type (ScadaBridge writes
CLR `int[]`/`double[]` lists) — any out-of-range, fractional-to-integer or NaN/∞ element refuses the whole
write `BadTypeMismatch` with nothing sent, so a `UInt32` > `Int32.MaxValue` can never reach a Galaxy Int32
attribute as reinterpreted bits; non-numeric mismatches are refused (`Byte[]` ≈ ByteString); wrong length ⇒
`BadTypeMismatch`; an array to a **scalar** node is refused only on Modbus/S7 (`EnsureVariable`'s
`refuseArrayOnScalar` — pass-through drivers keep their browse-committed scalar-shaped array tags writable;
a new sink wrapper must forward that argument, and the reflection guard checks only that the call arrives);
`IndexRange` writes ⇒ `BadWriteNotSupported`, `ValueRank > 1` out of scope. An
array node is writable only when the driver can write the array (`DriverArrayWriteSupport`: Modbus bounded
by `maxRegistersPerWrite`/coil cap, S7 non-Bool/non-wide types, OpcUaClient, Galaxy signed types **with a
declared `arrayLength`** — an undeclared length stays read-only because a short whole-array write would
resize the Galaxy attribute); `arrayLength >= 1` is required, and the node's `Writable && (!IsArray ||
ArrayWritable)` is projected onto the raw node and every UNS mirror.
- **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`,
preserving today's behaviour) controls *admission*; when admitted the session authenticates as the
reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as
the same `RoleCarryingUserIdentity` a key produces. Set it `false` to make keys mandatory. A key may
**not** use the `anonymous` principal, and a failure to read anonymous's grants **denies** rather
than admitting with no roles.
- **Keys reach a driver-only node inside the deployment artifact** (stage 4). `ConfigComposer` snapshots
the `ApiKey` rows — the peppered HMAC only, never a secret, and the pepper is node config that is
deliberately *not* in the artifact — and `DeploymentArtifact.ParseDataPlaneAuthority` decodes them
node-side alongside the cluster's `NodeAcl` rows. `OpcUaPublishActor` installs both on
`IDataPlaneAuthoritySource` on every apply, **before** its `plan.IsEmpty` early-return, so a deployment
that only revokes a key still lands. A node with a ConfigDb keeps `ConfigDbApiKeyStore` (per-activation
read, wrapped — see below); a driver-only node gets `ArtifactApiKeyStore`. ⚠️ **The asymmetry is
ACTIVATION-scoped, not session-scoped — this line used to say "immediate on a ConfigDb node" and that was
wrong.** Clearing `Enabled` in the ConfigDb stops the **next session activation** on an admin/fused node
(bounded by `Security:ApiKeys:StoreCacheSeconds`, default 5 s) and takes until the next deployment on a
driver-only one — but an **already-open** session re-checks its key against the in-memory *artifact
snapshot* (`DataPlaneAuthorizer.Evaluate` → `snapshot.FindUsableKey`), never against `IApiKeyStore`, so on
**both** node kinds an `Enabled` revocation reaches an open session only at the next deployment.
**Expiry is the exception**: `FindUsableKey` compares `ExpiresUtc` against the authorizer's clock on every
call, so it is immediate everywhere — and since arch-review 03/S19 it reaches live subscriptions too, on
the host's `Security:Acl:RevalidationIntervalSeconds` timer (default 60 s; the session is **silenced with
`BadUserAccessDenied`, not closed**, and recovers on re-activation). Direct event/alarm subscriptions
are revalidated on both triggers as well (#562); with the flag **off**, expiry and revocation reach the
two role gates (#563) but not reads or subscriptions.
- ⚠️ **The activation lookup is BOUNDED (arch-review 03/S18).** The SDK raises impersonation synchronously
under a `SessionManager`-wide lock, so a ConfigDb that is *slow* rather than *refusing* stalled **every**
session activation on the server, Anonymous and X.509 included. `BoundedApiKeyStore` wraps the ConfigDb
store — and only it, never `ArtifactApiKeyStore`, which is two in-memory dictionary probes — with
`Security:ApiKeys:StoreTimeoutMs` (2000; exceeding it **denies**, via the authenticator's existing
catch-all) plus a `StoreCacheSeconds` (5) read-through cache. Faults and timeouts are never cached, a
cached hit whose `ExpiresUtc` has passed is re-read rather than served, and a timeout logs a distinct
Warning + `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. Both knobs at `0` restore the old
behaviour exactly.
- ⚠️ **The auth generation is a node-local COUNTER driven by the `RevisionHash`, not derived from its
bits.** `PermissionTrieCache` treats the highest `GenerationId` as current, so a generation taken from
hash bytes would install a *lower* number about half the time and the cache would go on serving the
previous deployment's ACLs with nothing logged. Pinned by
`DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases`.
- Note `OtOpcUaServerHostedService` takes `IOpcUaUserAuthenticator` **and** `IDataPlaneAuthoritySource`
as **required** ctor parameters — the built-in DI container does not fall back to a default value for
an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a
node with neither role) or it does not boot.
**Grants are authored through a scope picker, not free text (stage 7).** `/clusters/{id}/acls` opens
`AclScopePickerModal` over the cluster's raw + UNS trees, and selecting a node supplies `ScopeKind`
**and** `ScopeId` together — so `RawDevice` + a TagId, a cross-cluster scope, a `Cluster` grant carrying
an id, and the retired `FolderSegment` kind are all unauthorable by construction rather than merely
rejected later. ⚠️ **A `Cluster` row must produce a NULL `ScopeId`** — the deploy gate rejects one that
carries an id (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would
author rows that cannot be deployed. `IAclScopeService.CheckScopeAsync` mirrors
`DraftValidator.ValidateNodeAclScopes` rule for rule and **blocks the save** when a scope resolves to
nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches
the picker); the list page marks such rows `unresolved`. **UNS tags are deliberately not offered** — a
UNS signal is a `UnsTagReference`, and a `Tag` grant already covers the raw node *and* every UNS
projection of it. The picker is `AclScopeTree`, a separate component from `RawTree`, which already
carries two modes.
**All eight stages are done** (`d4389f57` enforcement, stage 7 authoring after it), each live-gated on
docker-dev. The rationale for every decision above is in
`docs/plans/2026-07-31-apikey-acl-enforcement-design.md`. See `docs/security.md` for the full guide.
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)
## AB drivers: native transport (`Transport`)
AbCip and AbLegacy have a second, managed wire transport (`Driver.Ab.Transport`) beside libplctag, chosen by
the driver-config key `transport` = `Native` | `Libplctag` (string; takes effect when the driver is
**constructed** — every process start, or a deploy that recreates it, not `ReinitializeAsync`; AdminUI
"Transport" dropdown; CLIs `--transport`). "Default" is encoded by omitting the key; the AdminUI must never
re-emit a concrete default. The defaults differ per driver (final review, 2026-10-07):
- **AbCip: an absent key resolves to the node default, `Native` unless pinned.** Precedence (one place,
`AbCipTransportResolver`, shared by driver, probe and CLI): explicit per-driver key → node-level appsetting
**`Drivers:AbCip:DefaultTransport`** (`Native` | `Libplctag`, env `Drivers__AbCip__DefaultTransport`,
`AbCipDriverDefaultsOptions`; a bad value **fails host start**) → compiled `Native`. Bound unconditionally
because the driver factory AND the admin-node Test Connect probe both read it. ⚠️ So **the upgrade restart
alone moves no-key AbCip drivers to Native**; to stay on libplctag, set the node key — NOT a per-driver key
through the old AdminUI, whose typed form drops unknown keys (#576). Evidence: AbCip reads proven on a real
CompactLogix (Z-35162); scalar native writes (DINT/BOOL/REAL) proven on it once (2026-10-09, user-approved one-off, restored) — and native UDT member writes (DINT/BIT/REAL, #611 gate, same day) and a native STRING write (hw5, same day, restored byte-exact); and a native array-element write (`EM01_PowerUnit._SEQ_EP[31]`, user-approved, an element no alias targets, restored; neighbours unchanged); libplctag member and fragmented writes are `ab_server`-only. #566: a real Logix drops TCP on a connection idled past its timeout (no 0x0107/0x0203); the faulted session reopens on the next request.
- **AbLegacy: an absent key resolves to `Libplctag` for every family; `Native` is opt-in** (no node key). On
a real MicroLogix 1400 (Z-33323, idle, 2026-10-09) native read S/N/F/B/T/C/R/ST/O/I incl. `S:0`, T/C/R
sub-elements and status bits, and ST byte order, all correct, and wrote N/F8/B-bit (restored). libplctag (the default)
used to fail `S:`/`O:`/`I:` and `R` status bits there (incl. the default probe `S:0`) — its `pccc.c` needs explicit
file numbers and has no R status-bit mnemonics; since #659 `LibplctagLegacyName` translates (`S2:`/`O0:`/`I1:`,
`R6:0.CON/13`) and both transports read identically on that press. The authored grammar is unchanged. Read-side
only: a libplctag WRITE to any address the translation changes (`S:`/`O:`/`I:` incl. bit writes, R `EN…FD`, `C…UA`)
is refused `BadNotSupported` before any runtime is created (none could reach the PLC before — no write-surface
widening on the default transport); libplctag T/C status-bit writes it parses itself are unchanged.
L/A files, function files, SLC, PLC-5 and `logixpccc` remain unverified natively. Explicit Native on PLC-5/`LogixPccc` logs a once-per-instance Warning.
AbLegacy native refuses status-bit sub-element writes (`T4:0.DN`…, `BadNotSupported`).
Refusals are not comms faults: an Unsupported refusal (`BadNotSupported`) and an AbCip native **type
mismatch** (`BadTypeMismatch`, where libplctag returned Good garbage for a same-width mis-authored type) both
keep the tag handle, leave driver health alone (no `Degraded`, nothing for the comms-loss watchdog) and warn
once per tag — only genuine transport failures evict and degrade (a runtime `NotSupportedException` too, #611 —
it used to latch `Degraded`). AbCip width-checks every scalar **write** against the controller's bytes before
sending, on both transports (#611; libplctag used to patch part of its buffer and send it all). Controller-discovered
UDT members are writable once browse-committed and authored `ReadWrite` (BIT members included — the controller
applies the bit) — native member writes incl. a BIT member are live-gated on Z35162 (2026-10-09); sibling-bit preservation and the libplctag path are ab_server-only; whole-UDT writes are refused `BadNotSupported` by decision (`docs/drivers/AbCip.md` §Member writes). `DeviceSessionPool.Shared` logs through the
host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT factories' `Register(registry, ILoggerFactory?)` take one too, #602 CONV-3). The AbCip `browsePageDelayMs` knob has an AdminUI field (`AbCipDriverForm` "Browse page delay (ms)"). libplctag is confined to `Libplctag*.cs` adapters by
`LibplctagConfinementGuardTests`; do not reference it elsewhere. See `docs/drivers/AbCip.md` §Transport,
`docs/drivers/AbLegacy.md` §Transport + §Wire evidence, `docs/Configuration.md` §`Drivers:AbCip`, and
`docs/plans/2026-10-07-native-ab-transport-gate-results.md`.
## Other driver notes
- **S7:** connect attempts back off 1 s doubling to 30 s (one attempt per window, not one per call/poll/probe
tick); a write-side access-denied (PUT/GET disabled) latches a **sticky Faulted** that outranks later read
passes; driver-side array writes cover non-Bool narrow/`Int32`/`Float32` types only — Bool arrays are
refused (a one-byte-per-element encode would clobber neighbour bits).
- **MQTT:** Plain `writeTopic` writes publish at QoS 1 (Sparkplug NCMD/DCMD stay QoS 0); the Sparkplug
primary-host claim backs off 2 s → 30 s; writes encode the birth's own datatype (unsupported birth types
`BadNotSupported`).
- **No driver-side history (#655):** `IHistoryProvider`, `DriverCapability.HistoryRead`, `HistoryReadStatusException` and
the OpcUaClient history passthrough are deleted — nothing ever dispatched HistoryRead to a driver; it is served only by the
server-wide `IHistorianDataSource`. `isHistorized` on an OpcUaClient tag means OtOpcUa records it, not that upstream
history is proxied. A stored `ResilienceConfig` still carrying a `HistoryRead` key parses (diagnostic only).
- **FOCAS / AbLegacy:** background loops are joined (≤ 5 s) on shutdown.
- **FOCAS (#257/#266):** `cnc_statinfo` flags are a `Status/` fixed-tree subtree (`FIXED:Status/Run`…, capability-probed);
adjacent PMC tags in one poll batch coalesce into one `pmc_rdpmcrng` (`PmcCoalesce` config, default on) — a refused range
falls back to per-tag reads, a transport failure fails all members without retry. Parts count stays `PARAM:6711` (docs only).
- **FOCAS alarms (#647):** an `ALARM:` address (`*`, `<Type>`, `<Type>/<n>`, `<Type>/<lo>-<hi>`, `T<code>`, `OTHER`) +
the `alarm` object = ONE condition at the RawPath (either alone is skipped + reported). Overlapping selectors all
fire; `OTHER` = what no type selector claims. ⚠️ A failed `cnc_rdalmmsg2` THROWS (it used to return `[]` ⇒ mass
clear + un-acked re-raise) — and annotates its conditions `BadNoCommunication` once per outage (quality-only) and
degrades driver health with `LastSuccessfulRead` frozen, so an alarm-only driver trips the #527 watchdog. ⚠️ A CNC
REFUSAL (non-OK rc, transport fine) is `BadNotSupported` + one Warning and does NOT touch health — else the
watchdog reconnect-loops a production control. Health = worst of the value and alarm paths, computed together. The told
record lives on the driver INSTANCE (reconnect / in-place re-init re-raise nothing); a deploy RESPAWNS the driver,
and its Raise for a still-active alarm is a host-side re-statement (below). Type names are the 16i-era table only
(30i codes unconfirmed, not invented — use `T<code>` there). Read-only; ack server-side. `AlarmProjection.Enabled`
is a no-op. E2E: `Runtime.Tests/FocasNativeAlarmEndToEndTests` (incl. a respawn). `docs/drivers/FOCAS.md`.
- **A Raise on an already-active condition with the SAME message is a RE-STATEMENT (#647 review):** `NativeAlarmProjector` keeps its ack and
`ForwardNativeAlarm` writes no second "Activated" `/alerts` row — every native driver's told-record dies with its
instance, and a deploy respawns the driver. Only `Retrigger` (or inactive → active) un-acks, so a source whose
UPSTREAM is authoritative for the ack sends Retrigger for an unacked re-statement: OpcUaClient on first sight of a
condition (no prior told), Galaxy for an unacked active snapshot entry. **Those two also flag
`AlarmEventArgs.IsRestatement` (#656)** — history-only: a flagged Retrigger on an Active+unacked condition with the
same message AND severity, or a flagged Clear on an inactive one, writes no `/alerts` row (every Galaxy reconnect used
to duplicate them); a flagged `Unspecified` on an already-active condition (AbCip ALMA governing-limit drop-out)
moves the node's text/severity, keeps active/acked and writes no row. Active+acked is the atomic **`AlarmTransitionKind.ActiveAcknowledged`** (Galaxy snapshot
`ActiveAcked`, OpcUaClient inactive→acked-active) — never Acknowledge (kept a cold host's `Active=false`: the alarm was
invisible) nor Raise+Acknowledge (an unacked intermediate). Its mirror **`InactiveUnacknowledged`** (#657, OpcUaClient)
replaces Raise/Retrigger+Clear (an Active flicker; rows only for what changed). OpcUaClient's first sight of an inactive
condition is a flagged InactiveUnacknowledged, or flagged Clear+Acknowledge for inactive+acked (a flagged Acknowledge on an
acked condition writes no row) — never `Unspecified`, which wrote "Activated" for an inactive alarm and left a
retained-Active condition Active. `NativeAlarmProjector` returns the rows to write; unflagged
sources (FOCAS/MTConnect, live Galaxy) are unchanged. E2E: `Runtime.Tests/NativeAlarmRestatementEndToEndTests`.
- **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync`
first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip alarm projection;
FOCAS's is PAUSED, its told record kept, #647) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes
after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw
"tag name collision"); S7 swaps `Plc` under `_gate`.
- **Data and alarm subscriptions are independent in `DriverInstanceActor` (#662):** a data re-subscribe (every
deploy's bulk push to a kept driver) detaches DATA only (`DetachDataSubscription`) — it used to unhook
`OnAlarmEvent` too, silently killing the native-alarm feed until a respawn. The health-poll reconcile
re-attaches/re-subscribes alarms as well as data, so a failed alarm subscribe is retried every tick.
- **A config change is a respawn — there is NO in-place path (#665).** The `ApplyDelta`/`ApplyResult`/`DeltaApplied`
seam (host `ApplyChildDelta`, `DriverSpawnPlan.ToApplyDelta`, the actor's Connected reinit and mid-connect
`AdoptConfigDuringInit`) was unreachable since #516 and is deleted; #663/#664 were latent bugs in it. `IDriver.ReinitializeAsync`
stays as a per-driver contract the runtime never calls. Every driver's `InitializeAsync` still serves the config it is
handed (Calculation adopts, Galaxy refuses a non-equivalent one — #664), and still releases prior state (#630).
`docs/DriverLifecycle.md`.
- **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME
surface as Int32 ms); an undecodable buffer or unrepresentable write is `BadOutOfRange` with no health impact.
Optional `localTsap`/`remoteTsap`/`connectionType` override the slot TSAP (`remoteTsap` ⊕ `connectionType`). Test
Connect reads 2 bytes of `MB0` so a PUT/GET-disabled CPU fails the probe. The negotiated PDU size is logged, on
`S7Driver.GetDiagnostics()`, and (#640) on `/hosts` via the optional `IDriverDiagnostics` bag (string map on
`DriverHealthChanged.Diagnostics` / proto `has_diagnostics`+`diagnostics`; its digest is in the publish fingerprint).
- **TwinCAT (#637):** TIME/TOD publish Float64 ms, DATE/DT a UTC DateTime; a tag authored `UDInt` keeps the raw counter
(and keeps historizing — DateTime is not historized).
- **TwinCAT (#645, #310/#314):** opt-in ADS sum-read (`useSumRead`, `sumReadMaxItems` default 200 / ceiling 500, 32 KiB payload
budget). It reads raw bytes and decodes as the declared type, so only a scalar passing the strict width+kind check
(`CheckNotification`) rides a batch; value-converted pairs (DInt over INT, Real over LREAL/DINT) read singly. Only overall
ADS 1793/1794 latch the device to single reads; any other non-comms overall code (1792/1795/1797/1798...) reads that chunk singly for the poll without latching, with a once-per-device Warning; a comms failure neither.
Opt-in `diagnosticsEnabled` (default off) puts ADS state/app/online-change/cycle/exec/`HeadroomMs` on the `/hosts`
`IDriverDiagnostics` bag (≤ 4 extra reads per device per probe tick once settled; TC2 falls back to `.SYSTEMINFO` /
`.SYSTEMTASKINFOARR[1]`, costing 8 on the first tick and after each reconnect). Live-gated read-only 2026-10-08 on
Z35474 (TC3) + Z34834 (TC2, which answers sum commands) via a harness driving `TwinCATDriver` — `/hosts` end-to-end
not run (a stock host has no AMS router); the gate found ⚠️ Beckhoff's `SumRead` ctor throws for a `List<>` of entities (only an array survives
its internal cast) — `AdsTwinCATClient.CreateSumRead` owns it. Flat browse stays (VirtualTree is a strict subset
on TC3). Sum-write not ported.
- **AbLegacy (#636):** `AbLegacyAddress` (in `AbLegacy.Contracts`) is family-aware: PLC-5 `I:`/`O:` indices are OCTAL
and are emitted to libplctag as decimal (libplctag's `pccc.c` parses decimal only). MicroLogix function files parse
but read `BadNotSupported`. Indirect `N7:[N7:0]` (#646) is resolved by the driver at read/write time (index re-read
every call, never cached; bad index = `BadOutOfRange`; indirect writes are not atomic). One check (`AbLegacyAddress.Validate`) backs the
editor (family-independent) and the deploy gate (family from the tag's `DeviceConfig.PlcFamily`); an unparseable
address reads `BadNodeIdUnknown` for that tag only.
- **OpcUaClient (#635):** OperationLimits are honoured by the SDK session (`SessionClientBatched`) — do not re-add
driver-side chunking. Subscription tuning keys are range-checked at `ParseOptions`. `discoveryUrl` endpoints go
AFTER configured ones; more than one discovered server without `discoveryServerUri` fails the connect. Session
events are the OTel counter `otopcua.driver.opcuaclient.events`. `HistoryAggregateType` carries the full Part 13
set; an aggregate the gateway can't serve is `BadAggregateNotSupported` per node.
- **MTConnect (#627/#628):** TIME_SERIES reads as a variable-length `double[]` (Float64 only); a count disagreeing
with `sampleCount` is `BadDecodingError`. One MTConnect driver = one agent client = at most one `/raw` Device
(`MTConnectMultipleDevices` deploy error; `RawTreeService.CreateDeviceAsync` refuses a second).
- **MTConnect CONDITION alarms (#626):** a CONDITION raw tag is a native Part 9 condition only when its `TagConfig`
carries the standard `alarm` object (opt-in — alarm tags leave the value subscription, so auto-promotion would break
existing String tags). Normal→inactive, Warning→500, Fault→800; one condition per DataItem showing the most severe
active nativeCode (ties: ordinal nativeCode, then type); ack is server-side only. ⚠️ UNAVAILABLE never clears: an
active condition stays active+acked with a quality-only `BadNoCommunication` (same as #481 transport loss). The
driver dedups against what the server was last TOLD (level+nativeCode+nativeSeverity+qualifier — not text), records
only while a handler is attached, and drops the record of only the conditions that LEFT the set when the subscribed
alarm SET changes (#641) — never on a reconnect of the same set; a replayed `/current` must never re-raise (un-ack).
`AlarmEventArgs` gained optional `OpcUaSeverity` (exact Part 9 severity) and `QualityStatusCode`; a quality-only
event goes out as `AlarmQualityUpdate` per node and never through the snapshot projection.
- **Native condition state survives a full rebuild (#641):** `RebuildAddressSpace` captures each NATIVE condition's
live node state (client acks included), the same-apply re-materialise restores it, and `CompleteAddressSpaceApply`
(called by `OpcUaPublishActor` at the end of every apply) reports ONE event per restored off-normal condition and
discards unclaimed captures. `NativeAlarmProjector` is pruned per apply, not cleared. Native routing is an exact
`ConditionId == RawPath` match — AbCip's `#active` suffix was fixed; FOCAS routes since #647 (below).
- **Driver removal (#609):** removing a driver releases its capability-invoker pipelines and resilience
tracker, *then* evicts its telemetry snapshots. ⚠️ Both are **tombstoned**, not just cleared: the removed
child stops asynchronously and its in-flight subscribe/connect still completes through the invoker (and
emits Health) on the thread pool *after* `StopChild`, which used to resurrect the state — so a removed
driver's counters were republished forever. `Release` → tracker `Retire` and hub `Evict` ignore every later
write for that id until it is spawned again (`Create` → `Reinstate`, `SpawnChild` → `Readmit`). A
Watch/`Terminated` release would not work — the late writes land after the actor terminates.
## Library Preferences
- **Logging**: Serilog with rolling daily file sink
- **Unit tests**: xUnit + Shouldly for assertions
- **Service hosting (Server, Admin)**: .NET generic host with `AddWindowsService` (decision #30 — replaced TopShelf in v2; see `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`)
- **OPC UA**: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: `OPCFoundation.NetStandard.Opc.Ua.Server`
## OPC UA .NET Standard Documentation
Use the DeepWiki MCP (`mcp__deepwiki`) to query documentation for the OPC UA .NET Standard stack: `https://deepwiki.com/OPCFoundation/UA-.NETStandard`. Tools: `read_wiki_structure`, `read_wiki_contents`, and `ask_question` with repo `OPCFoundation/UA-.NETStandard`.
## Testing
Use the Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/` for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See `docs/Client.CLI.md` for full documentation.
```bash
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500
```
**AdminUI interactive pages must wrap their body in `<PageErrorBoundary>`** — `MainLayout` is static, so it cannot catch an interactive page's render failure.
Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`.
The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`.
**v3 (Batch 2): device I/O is authored in the `/raw` project tree** (`GlobalRaw.razor` → `RawTree.razor`, `IRawTreeService`) — a cluster-rooted `Folder → Driver → Device → TagGroup → Tag` hierarchy with per-node context menus (the reusable `ContextMenu`), lazy loading, driver/device config modals (endpoint moved to `Device.DeviceConfig`; Test-connect probes the `DriverDeviceConfigMerger`-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed `/clusters/{id}/drivers` flow (`DriverTypePicker`, `DriverEditRouter`, the 8 `*DriverPage` shells, the Drivers list) is **retired** — its form bodies live on inside the `/raw` modals. The `DriverType` dispatch maps are keyed off the `DriverTypeNames` constants (fixing the `TwinCat`/`Focas` drift). A `Calculation` pseudo-driver (`DriverTypeNames.Calculation`, `IDependencyConsumer`, deploy-time `scriptId`-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See `docs/Raw.md`.
**v3 (Batch 3): UNS Equipment is reference-only.** The equipment **Tags** tab no longer authors or binds tags — it holds `UnsTagReference` rows pointing at raw tags authored in `/raw`. "+ Add reference" opens the `RawTree` in picker mode, **cluster-scoped** (structurally + server-enforced `tag.cluster == equipment.cluster`); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. **Effective-name uniqueness** (references + VirtualTags + ScriptedAlarms share the `{EquipmentId}/{EffectiveName}` NodeId space) is enforced at authoring (`IEffectiveNameGuard`) and at the deploy gate (`DraftValidator.UnsEffectiveNameCollision` — catches rename-induced collisions). Scripts use **`ctx.GetTag("{{equip}}/<RefName>")`** — resolved per-equipment through `UnsTagReference` effective names to the backing RawPath at both compose seams (`AddressSpaceComposer` + `DeploymentArtifact`, via the shared `EquipmentReferenceMap`, byte-parity); an unresolved `<RefName>` is a deploy error (`EquipReferenceUnresolved`) AND a live Monaco diagnostic (`OTSCRIPT_EQUIPREF`) — editor accepts ⇔ publish accepts. `EquipmentScriptPaths.DeriveEquipmentBase` is deleted (the dot-joint `{{equip}}.X` became the slash-joint `{{equip}}/<RefName>`). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (`RawTreeService` substring scan). `ImportEquipmentModal` dropped the `DriverInstanceId` column. See `docs/Uns.md` + `docs/ScriptEditor.md`.
**v3 (Batch 4 = v3.0): dual-namespace address space.** The OPC UA server exposes **two namespaces** — `https://zb.com/otopcua/raw` (Raw device tree, `s=<RawPath>`) + `https://zb.com/otopcua/uns` (UNS tree, `s=<Area>/<Line>/<Equipment>/<EffectiveName>`) — replacing the single `.../ns` and the retired `EquipmentNodeIds` scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in `DriverHostActor` to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable `Organizes`-references its raw node. Writes route via **either** NodeId to the same driver ref under the same realm-qualified `WriteOperate` gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (`ConditionId = RawPath`) and fan via SDK `AddNotifier` to the raw device folder + every referencing equipment folder (one `ReportEvent`, one Server-object copy); both NodeIds register the SAME historian tagname (mux `HistorizedTagRef` stays single, keyed by RawPath). Identity authority: `V3NodeIds` + `AddressSpaceRealm` (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, `docs/Uns.md`, and `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`.
The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.
## Scripting / Script Editor
C# virtual-tag scripts are authored in a **Roslyn-backed Monaco editor** on the
ScriptEdit page (`/scripts/{id}`) and inline inside the virtual-tag modal on the
`/uns` page. The editor provides completions, live diagnostics, hover, signature
help, document formatting, and tag-path completions inside `ctx.GetTag("…")` /
`ctx.SetVirtualTag("…")` literals — all backed by the same compiler context the
runtime publish gate uses, so what the editor accepts/rejects matches publish
exactly. The backend lives in
`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/` (six minimal-API
endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
(Administrator or Designer) via `RequireAuthorization` in
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
`Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted.
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.
## Scripted Alarm Ack/Shelve
Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` role — conferred by the session's API-key principal's `NodeAcl` grants (or the `anonymous` principal's), carried by `RoleCarryingUserIdentity`; LDAP is AdminUI-only; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.
⚠️ **One condition event per accepted method call is OURS, not the SDK's (Gitea #587).** The SDK's own
report is gated on `AutoReportStateChanges` (left `false`), so before #587 a client Acknowledge emitted **no**
condition event — the old "SDK auto-fires its own event (E2)" comments were wrong. `WireMethodTransitionReporting`
wraps every method's SDK handler (Ack/Confirm/AddComment/Enable/Disable/OneShotShelve/TimedShelve/Unshelve) and,
on Good, reports once via `ReportConditionEvent` — **under `Lock`**, so the engine's/driver's re-projection
(routed by a fire-and-forget Tell *before* the SDK applies state) waits and then hits the delta-gate with no
delta. Things to keep: the SDK's shelve handlers never apply shelving, so the wrapper does; only a real
before/after change is reported (a re-ack is silent and restores the Confirmed/Retain the SDK flips);
`MaxTimeShelved` is set to `int.MaxValue` ms at materialise because `Create()` left it 0 and every TimedShelve
failed. ⚠️ **Native alarms support Acknowledge and AddComment only** — Confirm, the shelve methods and
Enable/Disable return `BadNotSupported` (no upstream path, no pair replication, and `NativeAlarmProjector`'s
Confirmed/Unshelved constants own those fields); a native ack re-asserts `ConfirmedState = true` because the
SDK's ack unconfirms any confirm-capable condition and ScadaBridge maps that field. Pinned over the wire by
`ConditionMethodEventTests` (exactly-once after a sentinel, folder + `i=2253`).
Behaviours that were wrong or missing and are now fixed — keep them: the SDK's `OnTimedUnshelve` is a
**no-op** (the engine's `AutoUnshelve` owns timed-shelve expiry, and `UnshelveTime` reports the real
remaining time from `AlarmConditionSnapshot.UnshelveAtUtc`); a **commented** re-ack is stored as an
`AddComment` (durable, replicated) and reports one event, an uncommented re-ack stays silent; a **native
ack is refused `BadInvalidState` off-Primary** via `NativeAlarmAckPrimaryGate` (fed by
`IDriverPrimaryRoleSource`; `DriverHostActor` publishes its Primary verdict at host start and on member
changes, so the boot window before the first redundancy snapshot is closed — the same fix covers the MQTT
Sparkplug primary-host claim). A native re-ack is still re-sent to the driver (the retry path, by design).
**Condition `Severity` is carried verbatim, clamped to 1..1000** — `ToPart9Severity`; the node manager used to re-bucket onto
the five `EventSeverity` points (so an MTConnect Fault read 900, scripted Critical 1000 read 900, native Low 200 read 300).
ScadaBridge only clamps, so exact values pass through. ⚠️ **Never select a two-state id with `AddSelectClause(type, "X/Id")`** —
the string becomes ONE QualifiedName named `X/Id`, which no event carries; the Client.CLI/Client.UI filter did this since
`a2883b82` and every alarm read inactive+unacked. Use `new SimpleAttributeOperand(type, [X, Id])` (ScadaBridge already does).
**Condition children are instance-scoped (#629).** Every child of a materialised condition carries
`{ConditionId}/{browse path}` in the condition's namespace (e.g. `…/temp_hi/ShelvingState/UnshelveTime`), assigned by
the `OtOpcUaNodeManager.New` override — the SDK default kept each child on the TYPE's ns=0 id, so a direct Read hit
the type (or `BadNodeIdUnknown`), translate failed, and children sat outside the ACL (the slash join now puts them
under their condition's scope). ⚠️ The shelve methods resolve by the `AlarmConditionType_ShelvingState_*` ids only
because `New` adopts those (a fixed set) as their `MethodDeclarationId`; calls by the `ShelvedStateMachineType_*` id
are no longer supported (pinned by a test). Condition teardown must forget the whole `PredefinedNodes` subtree
(`ForgetPredefinedSubtree`) or a removed condition's children keep answering.
## Historian / HistoryRead
**Backend: pluggable — `ServerHistorian:Backend` = `Gateway` (default), `SqlServer` or `QuestDb`** (see the subsection below).
As of the gateway-integration cutover, the **Gateway** backend's historian read, alarm-write, and
continuous-historization paths are all served by the **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed
**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin
`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke
Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were
retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`,
now a retired stub).
A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the
raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default
historian tagname, which is the tag's driver `FullName`.
### Pluggable backends, SQL Server and HistoryUpdate (2026-10)
- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads".
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ **Breaking for SDK-default clients:** the .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true` — a client that cleared only `IsReadModified` now gets bounds, and one that set neither is a modified read with bounds ⇒ `Bad_InvalidArgument` (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ `GoodNoData`, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch `ServerHistorian:HonorReturnBounds` (default `true`; `false` = pre-#691). `docs/Historian.md` §Bounding values.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history.
### Read path (`ServerHistorian` section)
The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed
(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes; Events over
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.
`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`):
| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` |
| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) |
| `ApiKey` | `""` | Peppered-HMAC key `histgw_<id>_<secret>` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** |
| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme |
| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) |
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
| `HonorReturnBounds` | `true` | #691 kill switch: `false` ignores HistoryRead-Raw `ReturnBounds` as before (no bounding values, no modified-read `Bad_InvalidArgument`) |
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway |
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |
### Alarm-history path (`AlarmHistorian` section)
Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
### Continuous historization (`ContinuousHistorization` section)
When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a
durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed
`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The
recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the
crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**)
with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries
only the recorder + outbox knobs:
| Key | Default | Notes |
|---|---|---|
| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder |
| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage |
| `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands |
| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` |
| `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup |
| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) |
| `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" |
| `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve |
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
| `CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). `false` = pre-#672 |
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync`
hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
deploy. **Gateway backend:** non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not
provisioned), the recorder likewise drops + meters non-numeric values, and continuous historization is
**numeric-analog only** (`UInt16→UInt4` is a documented fallback). The **SqlServer** backend provisions and
records every type (see "Pluggable backends" above).
The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`),
built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the
read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier.
When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
the "Boolean historization maps to Int2" section in `docs/Historian.md`.
### Gateway-side prerequisites
The target HistorianGateway OtOpcUa points at **must** run with:
- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read
(`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would
lose the newest events of a segment it cut.
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.
### Migration note (deployments upgrading from the Wonderware backend)
The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env:
| Old (Wonderware) key | New (gateway) key |
|---|---|
| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) |
| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) |
| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) |
The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`)
were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
`ServerHistorian`. See `docs/Historian.md` for the full guide.
### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
**No longer a limitation.** The cutover was live-validated against the real gateway on
`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
populate `Source_Object` — not a fixable bug). Evidence:
`docs/plans/2026-06-27-otopcua-historian-followups.md`.
Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
`wonder-sql-vd03`, gateway running the prerequisites above):
```bash
export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret> # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag> # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag> # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name> # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"
```
The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
gate the operator runs on the VPN before trusting the cutover.
### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04)
The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built
later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the
`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it
through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the
recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the
deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe
`Tell`; a faulting feed can never break a deploy).
**The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a
production-fatal defect** — see `docs/plans/2026-08-01-continuous-historization-live-gate.md`. Three things
to know:
- ⚠️ **`ConfigureAwait(false)` in an Akka handler destroys `Self`.** `OnValueChangedAsync` touched `Self`
after awaiting `IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false)`, which discards Akka's
`ActorTaskScheduler` synchronization context — so `ActorBase.Context` was gone and it threw
`NotSupportedException` on the FIRST value. The throw reached supervision, the actor **restarted**, and
because production seeds an **empty** ref set the restart discarded every historized ref: one value
captured, then silence. Fixed by capturing `IActorRef self = Self` before the await (the idiom the
`GetStatus` handler already used for `Sender`).
- ⚠️ **A fake that completes synchronously hides this entire class of bug**, and 13 tests passed because
of one. `ValueTask.CompletedTask` never yields, so the continuation ran inline with the context intact.
Worse, the obvious fix to the fake — `await Task.Yield()` — is *also* vacuous: `Yield()` posts back to
the captured context, so the caller still resumes on the actor thread. The fake must complete on a
**pool** thread (`await Task.Delay(1).ConfigureAwait(false)`), as a real per-entry fsync does. Only
reverting the production fix exposed that the first regression test proved nothing.
- **Both pair nodes historized the same value** — the recorder had **no Primary gate**, unlike the alarm
store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as
**#537** and **fixed** — see the paragraph below.
**Gate COMPLETE — #491 CLOSED 2026-08-04.** Outbox append, drain, `WriteLiveValues`,
**restart-convergence**, `EnsureTags` provisioning (`ensured=2 failed=0`) and **read-back** all ✅. The
read-back leg ran against the licensed `wonder-sql-vd03` with the VPN up: two registers carrying
*distinct non-zero* constants (`8224`, `120`) each read back as its own value — a `0`-valued source
would have proved nothing (the #536 trap).
**Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison
tag); evidence in the same plan doc. Two things to know before touching either.**
- **#538 — the discriminator is "did another tag write this pass?", NOT a failure count.** `DrainOnceAsync`
writes every per-tag group and acks **all-or-nothing** (deliberate: `RemoveAsync` truncates *through* an
id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned
the FIFO head while every *healthy* tag was re-submitted on every pass, forever — duplicate samples, an
outbox that never truncated (observed 8.3 MB; signature = the gateway's `ok` **and** `error` counters
climbing together). The drain now quarantines a tag after `MaxWriteAttempts` consecutive failures, but
⚠️ **only counts a failure on a pass where some other tag succeeded** — otherwise an ordinary gateway
restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug
into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so
the load-bearing test is the **falsification control** (`A_backend_that_fails_every_tag_abandons_nothing`).
Abandonment is also deferred until *every* failing tag is over budget, so nothing is discarded on a pass
that could not have advanced the head anyway.
- **#537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink.** The alarm
queue is in LocalDb and *replicates*, so the Secondary holds the Primary's rows and gating the drain
loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its
drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of
stale duplicates on promotion). ⚠️ **And the peer that matters is one recording the SAME refs, not one
holding the same rows** — a node stands down only for a Primary in its own per-cluster mesh, and a node
with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is
usually running another cluster's configuration, and deferring to it ends this cluster's historization
*silently* (the drain reports success on an empty outbox forever) — the same scope mismatch that once
suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
#511 registration accident).
- **#672 — an UNREACHABLE recording Primary does not count (crash-failover gap).** A SIGKILLed Primary
stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the
redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and
*nobody* captured for ~30 s. `DriverHostActor` now holds its **own** `ReachabilityEvent` (+`MemberUp`)
subscription — not inside `StartPrimaryVerdictPublishing`, which early-returns with no primary-role
source — handled in **all three** behaviours, caches the last snapshot, and on a reachability change
re-derives and republishes **only** the historization verdict with unreachable peers excluded. Gap ⇒ the
failure-detector window (~10–12 s). Gated by `ContinuousHistorization:CaptureWhilePrimaryUnreachable`
(default `true`). ⚠️ **The alarm drain and `ShouldServiceAsPrimary`/`PublishPrimaryVerdict` must NOT
adopt this** (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay
put. ⚠️ A **removed** member deliberately stays marked unreachable (only `MemberUp` at that address
clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the
gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real
partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each
node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.
Re-confirmed live in 2026-06 and **since fixed (#600.1)**: `EnsureTags` provisioning used to fire only for
newly-ADDED historized tags, so flipping an existing tag's `historianTagname` (or historization, or DataType)
provisioned nothing (`2026-06-27-historian-gateway-integration-issues.md` Issue 5). It now also fires for those
changed raw tags.
Reference in New Issue View Git Blame Copy Permalink