Joseph Doherty
22ae18365b
CLAUDE.md's "Browse needs ancestor visibility" bullet and docs/security.md's ancestor-visibility bullet now name the topmost ancestor: the single shared root folder (s=OtOpcUa, raw namespace) that both realm trees hang under, its mapping onto the trie's zero-segment realm-root scope, the either-realm transit (it fronts both trees), and the realm qualification. Also corrects the adjacent security.md bullet that claimed "the two namespace root folders remain visible" — there is one anchor, not one per namespace, and what an ungranted session sees is the ns=0 Objects listing of it rather than a browse into it. Flagged rather than silently left, since it contradicts the mechanism the new bullet documents.
1133 lines
102 KiB
Markdown
1133 lines
102 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project Goal
|
||
|
||
Build an OPC UA server (.NET 10) that exposes industrial data sources —
|
||
including AVEVA System Platform (Wonderware) Galaxy — under a unified
|
||
Equipment-based address space. Galaxy access flows through the in-process
|
||
`GalaxyDriver` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/`) talking gRPC
|
||
to a separately installed **mxaccessgw** gateway process. The gateway owns the
|
||
MXAccess COM bitness constraint (its worker is x86 net48); everything in this
|
||
repo is .NET 10. PR 7.2 retired the legacy in-process `Galaxy.Host` /
|
||
`Galaxy.Proxy` / `Galaxy.Shared` projects + the `OtOpcUaGalaxyHost` Windows
|
||
service.
|
||
|
||
See `docs/v2/Galaxy.Performance.md` for the runtime perf surface
|
||
(tracing, metrics, soak harness).
|
||
|
||
## Related Projects (scadaproj umbrella)
|
||
|
||
This repo is one of a family of related SCADA / OT / Wonderware / OPC UA
|
||
**sister projects** that live as sibling directories under `~/Desktop/`. They are
|
||
**separate repos and separate processes**, coupled at runtime over wire protocols
|
||
(gRPC + OPC UA) — not by project/compile references — and share the
|
||
`ZB.MOM.WW.*` product namespace. The common subject is AVEVA System Platform
|
||
(Wonderware) "Galaxy" data, and `mxaccessgw` is the linchpin the others connect
|
||
through.
|
||
|
||
- **`~/Desktop/scadaproj`** ([`../scadaproj/CLAUDE.md`](../scadaproj/CLAUDE.md)) —
|
||
the umbrella/index workspace that aggregates the whole family (a high-level scan
|
||
of each sister project: purpose, location, stack, primary commands). It also hosts
|
||
the shared `ZB.MOM.WW.*` libraries — `ZB.MOM.WW.Auth`, `ZB.MOM.WW.Theme`,
|
||
`ZB.MOM.WW.Health`, `ZB.MOM.WW.Telemetry`, `ZB.MOM.WW.Configuration`, and
|
||
`ZB.MOM.WW.GalaxyRepository`. **This repo is indexed there** — see the OtOpcUa
|
||
entry in `../scadaproj/CLAUDE.md`.
|
||
- **`~/Desktop/MxAccessGateway`** (`mxaccessgw`,
|
||
[`../MxAccessGateway/CLAUDE.md`](../MxAccessGateway/CLAUDE.md)) — the gRPC gateway
|
||
that owns the MXAccess 32-bit COM bitness + STA pump (gateway x64 .NET 10 + per-session
|
||
x86 net48 worker). **OtOpcUa's `GalaxyDriver` depends on this** for live Galaxy
|
||
read/write/subscribe and Galaxy Repository browse.
|
||
- **`~/Desktop/HistorianGateway`**
|
||
([`../HistorianGateway/CLAUDE.md`](../HistorianGateway/CLAUDE.md)) — single-process
|
||
gRPC sidecar exposing the AVEVA Historian read/write API + read-only Galaxy browse
|
||
(via the shared `ZB.MOM.WW.GalaxyRepository` lib). Patterned on `mxaccessgw` but with
|
||
no COM / no x86 worker. **OtOpcUa's historian backend** (`Driver.Historian.Gateway`)
|
||
consumes it as the sole historian path.
|
||
- **`~/Desktop/ScadaBridge`**
|
||
([`../ScadaBridge/CLAUDE.md`](../ScadaBridge/CLAUDE.md)) — the distributed SCADA
|
||
platform (Akka.NET hub-and-spoke) whose Data Connection Layer consumes this server's
|
||
OPC UA address space.
|
||
|
||
**Propagate cross-repo changes to the umbrella index.** When a fact the index records
|
||
about OtOpcUa changes here — remote/push status, the driver set, the Galaxy data flow,
|
||
shared-lib consumption, or per-project commands — update the **OtOpcUa entry in
|
||
`../scadaproj/CLAUDE.md`** in the same change so the index never drifts from this repo.
|
||
|
||
**The index edit belongs on `scadaproj`'s `main`, and must be pushed.** `scadaproj` is a
|
||
separate repo with its own checkout, so committing there inherits whatever branch it
|
||
happens to be on — which is usually *not* `main` and is often an unrelated feature branch
|
||
with no upstream. That silently drifts the index for as long as that branch stays
|
||
unmerged, which is the exact failure the rule above exists to prevent. Do this explicitly:
|
||
|
||
```bash
|
||
cd ~/Desktop/scadaproj
|
||
git rev-parse --abbrev-ref HEAD # note it, to restore afterwards
|
||
git stash list # never commit onto a dirty unrelated branch
|
||
git checkout main && git pull
|
||
# …edit the OtOpcUa entry in CLAUDE.md…
|
||
git commit -am "docs(index): <what changed about OtOpcUa>"
|
||
git push origin main
|
||
git checkout - # restore the original branch
|
||
```
|
||
|
||
Observed 2026-07-27: the `Sql` poll driver's index entry had sat unpushed on an unrelated
|
||
feature branch since 2026-07-24 because it was committed onto the current checkout, so the
|
||
index disagreed with reality for three days despite this rule. If you find index commits
|
||
stranded on a feature branch, cherry-pick them onto `main` rather than merging that branch.
|
||
|
||
## Architecture Overview
|
||
|
||
### Data Flow
|
||
|
||
1. **Galaxy Repository DB (ZB)** — SQL Server database holding the
|
||
deployed object hierarchy and attribute definitions. The
|
||
mxaccessgw's `GalaxyRepositoryClient` queries it via gRPC; the
|
||
driver consumes the materialised hierarchy through
|
||
`IGalaxyHierarchySource`.
|
||
2. **MXAccess (via mxaccessgw)** — Live read/write/subscribe over a
|
||
gRPC session. The gateway owns the COM apartment + STA pump
|
||
server-side; the driver speaks `MxCommand` / `MxEvent` protos
|
||
exclusively.
|
||
3. **OPC UA Server** — Exposes the deployed tree under **two OPC UA namespaces**
|
||
(see below). Galaxy tags are bound by `TagConfig.FullName`
|
||
(`tag_name.AttributeName`); reads/writes/subscriptions are translated to that
|
||
reference for MXAccess.
|
||
|
||
### v3 OPC UA Address Space (Batch 4): dual namespace
|
||
|
||
The server exposes **two OPC UA namespaces** (replacing the single
|
||
`https://zb.com/otopcua/ns`), built by `AddressSpaceComposer` /
|
||
`AddressSpaceApplier` and served by `OtOpcUaNodeManager` (which registers both
|
||
URIs; `V3NodeIds` + `AddressSpaceRealm` are the identity authority):
|
||
|
||
| Namespace URI | Realm | Subtree | NodeId `s=` scheme |
|
||
|---|---|---|---|
|
||
| `https://zb.com/otopcua/raw` | `AddressSpaceRealm.Raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
|
||
| `https://zb.com/otopcua/uns` | `AddressSpaceRealm.Uns` | the UNS tree (Area → Line → Equipment → signal) | slash-joined **`Area/Line/Equipment/EffectiveName`** |
|
||
|
||
- **Single source, fanned to both.** Every device value has exactly one source —
|
||
the raw tag's node. A `UnsTagReference` projects that raw tag into an equipment
|
||
as a UNS-namespace variable that carries an **`Organizes` reference to its raw
|
||
node** and mirrors it. One driver publish for a RawPath fans (in `DriverHostActor`)
|
||
to the raw NodeId AND every referencing UNS NodeId with identical
|
||
value/quality/timestamp — the mux key stays single (RawPath).
|
||
- **Writes via either NodeId** route to the same driver ref under the same
|
||
`WriteOperate` gating (the node-manager write gate is realm-qualified and fails
|
||
closed); a failed device write reverts both NodeIds via the shared fan-out
|
||
(write-outcome self-correction).
|
||
- **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and
|
||
fan via SDK notifiers to the raw device folder AND every referencing equipment
|
||
folder — one `ReportEvent`, one Server-object copy; ack/confirm/shelve route on
|
||
`ConditionId = RawPath`.
|
||
- **Historian dual-registration.** Both NodeIds register the **same** historian
|
||
tagname; the mux `HistorizedTagRef` stays single (RawPath). HistoryRead works
|
||
through either NodeId.
|
||
|
||
The retired `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme and
|
||
the single-namespace `EquipmentTags` materialization path are gone. See
|
||
`docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` +
|
||
`docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`.
|
||
|
||
**How a deploy reaches that address space (surgical-deploys, 2026-08-15).** The old
|
||
blanket rule — ANY non-empty `ChangedRawTags` / `ChangedUnsReferenceVariables` routes to
|
||
`Rebuild` — is gone, because it made a routine edit a fleet-wide outage:
|
||
`RawTagPlan.ReferencingEquipmentPaths` is part of the record's equality, so adding or
|
||
dropping **one** `UnsTagReference` flipped its backing raw tag to Changed and tore down
|
||
every node on the server, killing every client subscription. `AddressSpaceChangeClassifier`
|
||
is still default-closed; what changed is which deltas it whitelists.
|
||
|
||
- **Still a full rebuild:** an **alarm-bearing** raw-tag delta (`Alarm != null` on either
|
||
side — the WP4 guard: only a rebuild clears and re-wires native-alarm notifiers), an
|
||
**identity** change on a *paired* raw-tag delta (`TagId`/`DriverInstanceId`/`ParentNodeId`/`Realm`
|
||
— pairing is by RawPath, so this guard catches a tag deleted and re-created under the same path;
|
||
a plain **rename never pairs** and is served surgically as remove-one-node + add-one-node with any
|
||
referencing UNS mirror re-pointed, live-gated 2026-08-15),
|
||
a raw **container** (Folder/Driver/Device/TagGroup) rename or removal, `ChangedEquipment` /
|
||
`ChangedAlarms`, a non-surgical equipment-tag delta, and anything else the
|
||
`with`-equality whitelist does not name (including any FUTURE field).
|
||
- **Now surgical:** a **non-alarm** raw-tag delta over the in-place-updatable attributes
|
||
(`Writable` / `IsHistorized` / `HistorianTagname` / `DataType` / `IsArray` / `ArrayLength`)
|
||
and/or `ReferencingEquipmentPaths` — applied via `ISurgicalAddressSpaceSink.UpdateTagAttributes`
|
||
in the **Raw** realm — and **every** `UnsReferenceDelta` (add / remove / re-point /
|
||
backing-edit).
|
||
- ⚠️ **UNS mirrors repair through the RECORD, not the classifier.** A raw-side historian or
|
||
shape edit would otherwise leave every referencing UNS mirror on the old tagname/shape
|
||
(`EnsureVariable` early-returns on an existing node). `UnsReferenceVariable` therefore
|
||
carries the backing tag's `Backing{IsHistorized,HistorianTagname,IsArray,ArrayLength}`, so the
|
||
same edit makes the reference record unequal and the planner emits a **coincident**
|
||
`UnsReferenceDelta`. Adding a newly-inherited field needs no classifier change — **omitting
|
||
one silently reopens the staleness.** Repair is **in place** (`UpdateTagAttributes`, Uns
|
||
realm) when the mirror's identity is unchanged, so its MonitoredItems survive; only an
|
||
identity change (re-point, effective-name edit) is remove+recreate — and only for that
|
||
one node.
|
||
- **A non-rebuild apply materialises the DELTA, not the composition.** `MaterialiseHierarchy`
|
||
still runs full (parents must exist); the other passes take the plan's added/changed subsets.
|
||
Adding one tag to a 50k-node server was ~50k sink calls on the pinned publish thread.
|
||
- ⚠️ **A degraded non-rebuild apply does NOT advance `_lastApplied`.** The full passes used to
|
||
re-ensure every node on every apply, accidentally self-healing any swallowed sink fault; the
|
||
delta path removed that. Holding the composition back is the replacement — the next deploy
|
||
re-diffs the same composition and re-ensures what failed, at the cost of re-emitting that
|
||
apply's removes (a re-remove returns false and ratchets to a rebuild, which is the correct,
|
||
bounded resync). A **rebuild** still advances even with failures.
|
||
- **Any surgical `false`/throw still ratchets to `SafeRebuild()`** — that one-way fallback is
|
||
what makes every whitelist above safe.
|
||
- ⚠️ **Known gap, no longer neutral:** `AnnounceAddedNodes` is Uns-only, so surgically-added
|
||
**raw** containers/tags are materialised but never announced — previously invisible (the
|
||
rebuild forced a re-browse anyway), now genuinely undiscoverable until a client re-browses.
|
||
Added UNS reference variables ARE announced.
|
||
|
||
See `docs/plans/2026-08-15-surgical-deploys.md`, `docs/Uns.md` and `docs/Raw.md`.
|
||
|
||
### Key Concept: Tag Name and FullName
|
||
|
||
Galaxy objects have a **tag_name** — a globally unique system name used for
|
||
MXAccess read/write — and attributes are referenced as `tag_name.AttributeName`
|
||
(e.g. `DelmiaReceiver_001.DownloadPath`). This dot-separated reference is what
|
||
`TagConfig.FullName` stores for a Galaxy equipment tag. The Galaxy address
|
||
picker browses the live hierarchy using contained names for navigation, then
|
||
resolves the selection to the `tag_name.AttributeName` form.
|
||
|
||
**Galaxy is a standard Equipment-kind driver.** Galaxy points are ordinary
|
||
equipment `Tag`s bound to `GalaxyMxGateway` via `TagConfig.FullName`
|
||
(`tag_name.AttributeName`), authored through the standard Tag modal + Galaxy
|
||
address picker on the equipment page's Tags tab — the same flow as Modbus or
|
||
S7. There is no alias machinery, no `SystemPlatform` namespace kind, and no
|
||
relay→alias converter. See
|
||
`docs/plans/2026-06-12-galaxy-standard-driver-design.md` for the full design.
|
||
|
||
### Data Type Mapping
|
||
|
||
Galaxy `mx_data_type` values map to OPC UA types (Boolean, Int32, Float, Double, String, DateTime, etc.). Array attributes use ValueRank=1 with ArrayDimensions from the Galaxy attribute definition. The driver-side mapping lives in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`.
|
||
|
||
### Change Detection
|
||
|
||
`DeployWatcher` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DeployWatcher.cs`) polls the gateway's deploy-event signal and raises `IRediscoverable.OnRediscoveryNeeded` when the Galaxy redeploys.
|
||
|
||
**The signal is consumed as an operator prompt, not as an address-space rebuild** (§8.2, 2026-07-27).
|
||
`DriverInstanceActor` subscribes in `PreStart` and unsubscribes in `PostStop` — the detach is
|
||
load-bearing, because the `IDriver` instance outlives the actor when the host respawns a child around
|
||
the same driver object. A raise is recorded and rides the driver-health snapshot
|
||
(`DriverHealthChanged.RediscoveryNeededUtc` / `.RediscoveryReason`) to the AdminUI `/hosts` page as a
|
||
**"re-browse" chip**.
|
||
|
||
⚠️ **It is advisory: a Galaxy redeploy still does NOT change the served address space.** v3 authors raw
|
||
tags explicitly through the `/raw` browse-commit flow, so an operator must re-browse the device and
|
||
commit. Injecting nodes at runtime would materialise nodes nobody approved and no deployment artifact
|
||
records.
|
||
|
||
Two gotchas if you touch this:
|
||
|
||
- `PublishHealthSnapshot` dedups on a health fingerprint. The rediscovery timestamp **must** stay in
|
||
that tuple, or a raise on an otherwise-unchanged Healthy driver is swallowed and never reaches the
|
||
operator. Pinned by `DriverInstanceActorRediscoverySignalTests`.
|
||
- The shared test `StubDriver.GetHealth()` returns `DateTime.UtcNow`, so its fingerprint differs on
|
||
every call and the dedup never engages — a test built on it passes vacuously. The rediscovery tests
|
||
use a stub with stable health for exactly this reason.
|
||
|
||
**#507 is closed as superseded, and the injection path is deleted.** Its retained code read
|
||
`EquipmentNode.DriverInstanceId` and `EquipmentTags`, both *structurally empty* in v3
|
||
(`AddressSpaceComposer.cs`, `DeploymentArtifact.cs`), so removing the guard would have changed a log
|
||
line and injected nothing. Gone with it: `HandleDiscoveredNodes`, `PartitionDiscoveredByDeviceHost`,
|
||
`ApplyDiscoveredPlansForDriver`, the redeploy re-inject tail, `DiscoveredNodeMapper`,
|
||
`AddressSpaceApplier.MaterialiseDiscoveredNodes`, the connect-time discovery loop
|
||
(`StartDiscovery`/`RediscoverTick`/`DiscoveredNodesReady`/`TriggerRediscovery`), and the 18
|
||
`DiscoveryInjectionDormantV3` tests — v2 characterization tests asserting an equipment-rooted graft
|
||
(`EquipmentRootNodeId == "EQ-1"`) that v3 cannot produce. `ITagDiscovery` itself stays: the `/raw`
|
||
browse picker drives it through `Commons/Browsing/DiscoveryDriverBrowser`, which never went through
|
||
the actor.
|
||
|
||
**`IRediscoverable` is the ONLY tag-change signal, and four drivers do not raise it.** A periodic
|
||
tag-set drift check briefly existed (shipped `e77c8a35`, **removed** by Gitea **#523**): every
|
||
`SupportsOnlineDiscovery` driver was re-browsed on a 5-minute timer and the result diffed against its
|
||
authored tags. It was taken back out because it generated recurring, **unconfigurable** (interval was a
|
||
constructor parameter with no appsettings key), **never-live-gated** outbound traffic to real
|
||
PLCs/CNCs/Agents — how often to browse someone's plant equipment is an operational decision, and the
|
||
design treated it as an implementation detail.
|
||
|
||
⚠️ **Consequence to know before you rely on rediscovery:** **AbCip and FOCAS browse a live backend but
|
||
implement no `IRediscoverable`, so they have no tag-change signal at all** — a PLC program re-download
|
||
or a CNC option install is invisible until someone re-browses the device by hand. Galaxy, TwinCAT,
|
||
MTConnect and MQTT raise natively and are unaffected. If that coverage matters later, reach for an
|
||
event-driven signal, or one that runs when something *else* already indicates change (a reconnect, a
|
||
deploy) — not a wall-clock timer. Removed with the feature: `TagSetDrift`/`TagSetDriftDetector`,
|
||
`DriverInstanceActor`'s drift timer, and `ITagDiscovery.AuthoredDiscoveryRefs` /
|
||
`DiscoveryStreamIncludesAuthoredTags`.
|
||
|
||
⚠️ **`IHostConnectivityProbe` is still unconsumed.** `GetHostStatuses()` has no production call site and
|
||
nothing ever writes a `DriverHostStatus` row (the table was re-created in the v3 initial migration, so
|
||
confirm intent before deleting). The separable `OnHostStatusChanged` event is the useful half. Tracked
|
||
as Gitea **#521**.
|
||
|
||
### Comms-loss detection: `GetHealth()` is load-bearing (Gitea #527, closed)
|
||
|
||
Until #527, **`DriverInstanceActor.DisconnectObserved` had no production sender.** The only paths into
|
||
`Reconnecting` were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected
|
||
and then lost its device stayed `Connected` indefinitely — publishing Bad for every tag, never
|
||
re-initialising, and **not recovering when the device returned** (live-confirmed: a Modbus driver against a
|
||
stopped fixture needed a node restart). `PublishHealthSnapshot` read `GetHealth()` for *reporting only*.
|
||
|
||
The `Connected` health tick now runs a watchdog that self-sends `DisconnectObserved` when a driver reports a
|
||
non-`Healthy` state **and** its `LastSuccessfulRead` has been frozen for 90 s. Things to know before touching it:
|
||
|
||
- ⚠️ **The conjunction is the design — "non-Healthy for N polls" is wrong and was the issue's own proposal.**
|
||
`Degraded` means *connected but some data is bad*, and several drivers write health **per tag inside the
|
||
read loop** (`ModbusDriver.ReadAsync`), so one unreadable address latches `Degraded` forever against a
|
||
perfectly reachable device. State-only tripping would reconnect that driver every 90 s indefinitely.
|
||
`LastSuccessfulRead` is the honest evidence — preserved across `Degraded`/`Faulted` writes, advanced by any
|
||
success. Each conjunct has a test that goes red when the *other* is deleted.
|
||
- **The mechanism rests on every driver reaching a non-Healthy state on sustained failure.** Surveyed across
|
||
all 12: nine poll drivers via `Degraded`, MQTT via live connection state, Galaxy via its reconnect
|
||
supervisor. `CalculationDriver` reports `Healthy` while running and so can never trip — correct, as it has
|
||
no transport. Re-check this if a new driver lands.
|
||
- **The reconnect retry MUST tear down before re-initialising — FIXED in both drivers, keep it that way.**
|
||
`ModbusDriver` (`_transport`) and `MqttDriver` (`_connection`) used to overwrite an `IAsyncDisposable`
|
||
field inside `InitializeAsync` without disposing the old one, so re-initialising without `ShutdownAsync`
|
||
leaked a socket per attempt — ~360 for an hour-long outage at the 10 s retry interval. Both now dispose
|
||
first: `ModbusDriver.InitializeCoreAsync` releases `_transport` **and** the `_probeCts`/`_reprobeCts`
|
||
probe loops before rebuilding them (`c3970a2e` + the lifecycle gate in `409a58b6`), deliberately *not*
|
||
via the shared `TeardownAsync()` because that would also drop `_poll` and every active subscription with
|
||
nothing to re-drive them; `MqttDriver.InitializeCoreAsync` calls its (idempotent) `TeardownAsync()` first,
|
||
which reaches `_connection` and the primary-host connection (`2b4b3fcb`). The path is no longer latent —
|
||
the comms-loss watchdog reaches it, not just a manual Force Reconnect — so a new driver, or a new
|
||
`IAsyncDisposable` field on these two, must repeat the dispose-before-overwrite.
|
||
- The threshold is hardcoded (`DefaultCommsLossTimeout`), matching `HealthPollInterval` /
|
||
`DefaultReconnectInterval`. Unlike the #523 drift detector this generates **no** traffic on a healthy
|
||
system — it is a local `GetHealth()` read, and only reaches the network once a driver has already stopped
|
||
reading.
|
||
|
||
See `docs/DriverLifecycle.md` §"The comms-loss watchdog".
|
||
|
||
## mxaccessgw
|
||
|
||
The gateway lives in a sibling repo at `~/Desktop/MxAccessGateway/` (`mxaccessgw` — see [Related Projects](#related-projects-scadaproj-umbrella)). See `docs/v2/Galaxy.ParityRig.md` for the gw setup recipe (build, API key provisioning via `apikey create-key`, env-var overrides for HTTP/2 cleartext + worker path). The gw's MXAccess Toolkit reference (its `gateway.md`) is the canonical MxAccess API doc; the standalone `mxaccess_documentation.md` previously kept in this repo retired in PR 7.3.
|
||
|
||
## Build Commands
|
||
|
||
```bash
|
||
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
|
||
dotnet build ZB.MOM.WW.OtOpcUa.slnx
|
||
dotnet test ZB.MOM.WW.OtOpcUa.slnx # all tests
|
||
dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests # a single test project
|
||
dotnet test --filter "FullyQualifiedName~MyTestClass.MyMethod" # a single test
|
||
```
|
||
|
||
CVE-2025-6965 / GHSA-2m69-gcr7-jv3q (transitive native `SQLitePCLRaw.lib.e_sqlite3`)
|
||
is **resolved** — the temporary `NuGetAuditSuppress` was removed once the patched bundle
|
||
shipped. `Core.AlarmHistorian` now carries a surgical direct pin of
|
||
`SQLitePCLRaw.bundle_e_sqlite3` to `2.1.12` (the first fixed native bundle, embedding the
|
||
SQLite 3.50.2+ patch), overriding the `2.1.11` that `Microsoft.Data.Sqlite` pulls
|
||
transitively. `dotnet restore`/`build` is clean with the audit active.
|
||
|
||
Test projects live under `tests/<module>/` (Core, Server, Drivers,
|
||
Drivers/Cli, Client, Tooling) — there is no single unit-test project.
|
||
Unit suites are named `*.Tests`; integration suites are `*.IntegrationTests`
|
||
and need their Docker fixture up (see Docker Workflow). DB-backed tests in
|
||
`*.Configuration.Tests`, `*.Admin.Tests`, and `*.Server.Tests` require the
|
||
central SQL Server.
|
||
|
||
## Docker Workflow (driver fixtures + central SQL Server)
|
||
|
||
> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing +
|
||
> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate**
|
||
> recipes (S7 blackhole, HistorianGateway LiveIntegration), the integration-test
|
||
> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail.
|
||
|
||
> **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28).
|
||
|
||
> ⚠️ **The `project: lmxopcua` label is aspirational, not universal.** This note used to claim every fixture compose file carries it. As of 2026-07-24 only the **MQTT** fixture actually does — `grep -rn "labels:" tests --include=*.yml` matches `Driver.Mqtt.IntegrationTests/Docker/docker-compose.yml` and nothing else. So `docker ps --filter label=project=lmxopcua` returns the MQTT fixture alone, not the fleet. Add the label when you touch an older fixture; until then enumerate by container name.
|
||
|
||
Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-<driver>/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`.
|
||
|
||
**`docker -H ssh://...` does NOT work from the Windows VM.** Windows OpenSSH ↔ docker.exe stdio bridging hangs (`docker system dial-stdio` runs server-side but no API data flows). Use the helper below — it SSHes into the docker host and runs `docker compose` server-side.
|
||
|
||
> **On macOS there is no `lmxopcua-fix` — drive the host over SSH directly.** The helper is a Windows-VM
|
||
> script; `~/bin` is empty on the Mac and the commands below will not resolve. Use passwordless SSH
|
||
> (and `rsync` in place of `sync`), which is what `infra/README.md` §1 documents as the Mac path:
|
||
>
|
||
> ```bash
|
||
> ssh dohertj2@10.100.0.35 'docker ps --format "{{.Names}}\t{{.Status}}"'
|
||
> ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mqtt && docker compose up -d'
|
||
> rsync -av tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Mqtt.IntegrationTests/Docker/ \
|
||
> dohertj2@10.100.0.35:/opt/otopcua-mqtt/ # the `sync` step, by hand
|
||
> ```
|
||
>
|
||
> A local container runtime *does* exist on the Mac (OrbStack), so the `docker-dev/` rig itself runs
|
||
> locally there — it is only the shared **fixtures** that live on `10.100.0.35`.
|
||
|
||
**Use `lmxopcua-fix.ps1` (in `~/bin`) to control fixtures from the Windows VM:**
|
||
|
||
```powershell
|
||
lmxopcua-fix ls # list all lmxopcua-tagged containers on the host
|
||
lmxopcua-fix up modbus standard # bring a profile up
|
||
lmxopcua-fix up abcip controllogix
|
||
lmxopcua-fix up s7 s7_1500
|
||
lmxopcua-fix up opcuaclient # single-service stack, no profile arg
|
||
lmxopcua-fix down modbus # tear stack down
|
||
lmxopcua-fix logs modbus
|
||
lmxopcua-fix sync modbus # rsync this repo's tests/.../Docker/ → /opt/otopcua-modbus/
|
||
```
|
||
|
||
**`sync` is the deployment step.** When you edit a fixture's compose file or Dockerfile under `tests/.../Docker/`, run `lmxopcua-fix sync <driver>` to push the changes to the docker host before bringing the stack up. The repo files are the source of truth; `/opt/otopcua-<driver>/` is a mirrored deployment.
|
||
|
||
**Endpoints (defaults already point at the docker host):**
|
||
- SQL Server (always-on): `10.100.0.35,14330` — used by `appsettings.json` for `ConfigDb`.
|
||
- Modbus: `10.100.0.35:5020` (`MODBUS_SIM_ENDPOINT`)
|
||
- AB CIP: `10.100.0.35:44818` (`AB_SERVER_ENDPOINT`)
|
||
- S7: `10.100.0.35:1102` (`S7_SIM_ENDPOINT`)
|
||
- OPC UA reference (opc-plc): `opc.tcp://10.100.0.35:50000` (`OPCUA_SIM_ENDPOINT`)
|
||
- MQTT (Mosquitto, **auth on both listeners — no anonymous fallback**): `10.100.0.35:8883` TLS+auth
|
||
(`MQTT_FIXTURE_ENDPOINT`) · `10.100.0.35:1883` plaintext-but-authenticated (`MQTT_FIXTURE_PLAIN_ENDPOINT`).
|
||
Also needs `MQTT_FIXTURE_USERNAME` / `MQTT_FIXTURE_PASSWORD`, and `MQTT_FIXTURE_CA_CERT` to pin the
|
||
fixture CA (`scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt`). A co-running publisher loops
|
||
JSON under `otopcua/fixture/…`.
|
||
- MQTT **Sparkplug B**: the same broker, plus the `sparkplug` compose profile's `otopcua-sparkplug-sim`
|
||
(project-owned C# edge-node simulator) — group **`OtOpcUaSim`**, edge nodes **`EdgeA`**/**`EdgeB`**,
|
||
`EdgeA` device **`Filler1`**, ~2 s cadence. It subscribes to `spBv1.0/OtOpcUaSim/NCMD/{node}` and
|
||
re-births on request. **Births are never retained**, so `docker restart otopcua-sparkplug-sim` is how
|
||
you force a fresh one while a browse window is open.
|
||
|
||
Override any endpoint via the env var to point at a real PLC. The local OtOpcUa server runs on this VM at `opc.tcp://localhost:4840` — **that's not on the docker host**.
|
||
|
||
**Local docker-dev rig — login is DISABLED, so do live `/run` verification yourself (don't wait for the user to sign in).** The local `docker-dev/docker-compose.yml` stack (AdminUI at `http://localhost:9200` via Traefik; OPC UA `opc.tcp://localhost:4840` central-1 / `:4841` central-2) runs the AdminUI with `Security__Auth__DisableLogin: "true"` — **no sign-in form; it's auto-authenticated as a full-access admin.** So AdminUI / Razor `/run` verification (deploy a config, drive a page, confirm behavior — e.g. via the Chrome browser-automation tools against `http://localhost:9200`) does **not** require the user to log in. Run it yourself; do not defer it as "user-driven sign-in required." (Caveat: OPC UA *data-plane* auth is still real LDAP against the shared GLAuth on `10.100.0.35:3893` — that only gates Client.CLI read/write **role** operations, e.g. binding a `multi-role` / `opc-writeop` user, and is independent of the AdminUI login. Things genuinely outside the local rig — real PLCs, or the AVEVA Historian reached via the `ZB.MOM.WW.HistorianGateway` sidecar — still need the user.)
|
||
|
||
See `docs/v2/dev-environment.md` for the full inventory and rationale.
|
||
|
||
## Build & Runtime Constraints
|
||
|
||
- Language: C#, .NET 10, AnyCPU. The MXAccess COM bitness constraint
|
||
is owned by the mxaccessgw worker (x86 net48), not by anything in
|
||
this repo.
|
||
- The gateway's MXAccess worker requires a deployed ArchestrA Platform
|
||
on the machine running the gateway. The OtOpcUa server itself does
|
||
not.
|
||
|
||
## Transport Security
|
||
|
||
The server supports configurable OPC UA transport security via the `OpcUa:EnabledSecurityProfiles` list in `appsettings.json`. Phase 1 profiles (the `OpcUaSecurityProfile` enum members): `None` (default), `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`. Security policies are built from the enabled profiles by `BuildSecurityPolicies` at startup. The server certificate is always created even for `None`-only deployments because `UserName` token encryption depends on it. See `docs/security.md` for the full guide.
|
||
|
||
## Redundancy
|
||
|
||
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
|
||
|
||
**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
|
||
|
||
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is CLOSED** — Phase 7 drill D4 (2026-07-24) ran the 1-vs-1 crash-the-oldest survival drill on the three-mesh docker-dev rig and every survivor stayed Up, never self-downing (`c50ebcf7`; `docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
|
||
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
|
||
|
||
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
|
||
|
||
**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard".
|
||
|
||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||
|
||
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
|
||
|
||
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
|
||
|
||
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
|
||
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
|
||
- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
|
||
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
|
||
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
|
||
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
|
||
|
||
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||
|
||
## Mesh command transport (`MeshTransport`)
|
||
|
||
**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
|
||
(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
|
||
Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
|
||
`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
|
||
validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
|
||
rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
|
||
`MeshTransport:Mode`:
|
||
|
||
- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
|
||
it works only when central and the node share a gossip ring, which after Phase 6 means within one
|
||
2-node mesh.
|
||
- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
|
||
carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
|
||
node on anything else.
|
||
|
||
Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
|
||
served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
|
||
rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
|
||
of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
|
||
every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
|
||
and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
|
||
no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
|
||
provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
|
||
applies to either direction.
|
||
|
||
Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
|
||
config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
|
||
claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
|
||
stops at the first success.
|
||
|
||
Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
|
||
application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
|
||
store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
|
||
on every unresolvable case — constant-time compare, no key material logged. Production keys are
|
||
`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
|
||
(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
|
||
|
||
Things that carried over unchanged from the `ClusterClient` era, because they were never
|
||
`ClusterClient`-specific to begin with:
|
||
|
||
- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
|
||
and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
|
||
nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
|
||
The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
|
||
set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
|
||
- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
|
||
DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
|
||
`ScriptedAlarmHostActor`) accept a command from either transport identically.
|
||
- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
|
||
deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
|
||
replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
|
||
deletion — it was always about the deploy path, not the transport.
|
||
|
||
`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
|
||
makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
|
||
§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
|
||
[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
|
||
[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
|
||
those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
|
||
is the original Phase 2 design doc and is likewise left as a historical record of the transport this
|
||
section now describes as deleted.
|
||
|
||
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
|
||
|
||
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
|
||
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
|
||
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
|
||
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
|
||
|
||
## Config source (`ConfigSource` / `ConfigServe`)
|
||
|
||
Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
|
||
deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
|
||
central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
|
||
bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
|
||
LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
|
||
`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
|
||
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
|
||
fail-closed). Things worth knowing before touching it:
|
||
|
||
- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
|
||
admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
|
||
dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
|
||
- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
|
||
hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
|
||
`RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
|
||
stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
|
||
**The blob is gzip-compressed since 2026-08-15, and the hash covers the COMPRESSED bytes** —
|
||
`ConfigComposer.Pack` gzips (`CompressionLevel.Fastest`) immediately after `SerializeToUtf8Bytes` and
|
||
hashes the result, so the invariant holds unchanged and nothing between the composer and
|
||
`DeploymentArtifact.Open` (SQL `ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream) ever
|
||
looks inside. `Open` sniffs the gzip magic `1F 8B` — valid JSON can never start with `0x1F` — so
|
||
pre-upgrade uncompressed rows keep parsing; inflation runs under a 512 MiB guard and every
|
||
decompression failure surfaces as `JsonException`, the same family the malformed-JSON paths already
|
||
handle. Two operational consequences in `docs/Configuration.md` §`ConfigSource`: the first deploy after
|
||
upgrade **re-hashes every revision** (a one-time full re-dispatch, not a no-op), and **nodes must be
|
||
upgraded BEFORE central starts emitting compressed blobs.**
|
||
- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
|
||
mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
|
||
and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
|
||
`FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
|
||
empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
|
||
- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
|
||
SQL today; `StoreAsync` is idempotent and the cache replicates).
|
||
- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
|
||
both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
|
||
ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
|
||
`Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
|
||
- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
|
||
ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
|
||
is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
|
||
|
||
On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
|
||
hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching
|
||
key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is
|
||
no `OTOPCUA_CONFIG_MODE` env var in the compose file. See
|
||
`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
|
||
`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
|
||
|
||
## Live telemetry transport (`Telemetry` / `TelemetryDial`)
|
||
|
||
Per-cluster mesh **Phase 5** (merged `35552a96`; live gate PASSED `f134935f`) added a second
|
||
transport for the four live-observability channels the AdminUI's `/alerts`, `/script-log`, and
|
||
`/hosts` panels feed on, selected by `Telemetry:Mode` (node/serve side) and `TelemetryDial:Mode`
|
||
(central/dial side): `Dps` — the four existing DPS SignalR bridges, **the default**, today's
|
||
behaviour — or `Grpc`, one gRPC server-streaming contract carrying all four channels as `oneof`
|
||
event kinds. **The direction is inverted from `ConfigSource`/`MeshTransport`: the driver node hosts
|
||
the gRPC server (dedicated Kestrel h2c port), and central is the client that dials in** — mirroring
|
||
ScadaBridge's `SiteStreamService` shape, and required so telemetry survives once the meshes split
|
||
(Phase 6) and central no longer shares a gossip ring with a site node. Things worth knowing before
|
||
touching it:
|
||
|
||
- **Migrated (4 channels):** `alerts` (`AlarmTransitionEvent`), `script-logs` (`ScriptLogEntry`),
|
||
`driver-health` (`DriverHealthChanged`), `driver-resilience-status`
|
||
(`DriverResilienceStatusChanged`) — feeding the exact same central sinks
|
||
(`IInProcessBroadcaster<...>`, `IDriverStatusSnapshotStore`, `IDriverResilienceStatusStore`) in
|
||
either mode.
|
||
- **Deferred, with reasons (do not read "seven observability topics" from the program sketch as the
|
||
delivered scope):** `redundancy-state` stays on DPS in every mode (pair-local, built from
|
||
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
|
||
observability panel); `deployment-acks` already rides the Phase 2 gRPC command transport
|
||
(`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
|
||
telemetry. A third, `fleet-status`, was **DELETED** rather than deferred (2026-08-06, Gitea #542):
|
||
`FleetStatusBroadcaster` → DPS → SignalR bridge → `FleetStatusHub` had **zero clients end to end**
|
||
(no browser JS, no `HubConnection` — `/fleet` polls `NodeDeploymentState` in the Config DB), its
|
||
`DriverHostStatusHeartbeat` input had no production sender, and Phase 6 left its "fleet" as just
|
||
central's own pair. Fleet visibility is `/fleet` + the four per-node telemetry dials above.
|
||
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
|
||
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
|
||
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
|
||
`TelemetryDial:Mode` on central actually switches anything: which of the two always-available
|
||
sources (DPS bridges vs. the `TelemetryDialSupervisor` dialer) feeds the sinks. This is what
|
||
makes the flip a config change, not a coordinated redeploy — same dark-switch discipline as
|
||
[MeshTransport](#mesh-command-transport-meshtransport) and [ConfigSource](#config-source-configsource--configserve).
|
||
- **Authenticated from day one, fail-closed** — a shared node bearer key (`Telemetry:ApiKey` ==
|
||
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
|
||
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, an unset key rejects
|
||
every call). This **supersedes** design doc §6.3's "match ScadaBridge's unauthenticated posture
|
||
for now" — ScadaBridge itself has since closed that gap with this identical interceptor pattern.
|
||
- **Reconnect story:** central runs one reconnecting dialer per driver node (discovered from
|
||
`ClusterNode.Host`/`GrpcPort`, refreshed every `TelemetryDial:ContactRefreshSeconds`), immediate
|
||
first retry then ~5s fixed backoff, generation-stamped so a superseded stream's late error/event
|
||
is ignored, and it never permanently gives up (observability, not data plane). The node-local hub
|
||
replays last-value snapshots for `driver-health`/`driver-resilience-status` on every (re)subscribe
|
||
so those stores re-prime immediately after a reconnect; `alerts`/`script-logs` are append logs and
|
||
tolerate the gap. The `/alerts`/`/script-log` pill reflects aggregate stream health (live when ≥1
|
||
node stream is up) — the same semantics the DPS pill has always had.
|
||
|
||
See `docs/Telemetry.md` for the full architecture, `docs/Configuration.md`
|
||
§`Telemetry`/`TelemetryDial` for the config keys, and
|
||
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan and live
|
||
gate procedure.
|
||
|
||
## LocalDb pair-local store (Phases 1 + 2)
|
||
|
||
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
|
||
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
|
||
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
|
||
|
||
**Phase 1** caches the
|
||
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
|
||
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
|
||
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
|
||
in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
|
||
`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
|
||
`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
|
||
never register it.** The cache can optionally replicate to the node's redundant pair peer over the
|
||
library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
|
||
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
|
||
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
|
||
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
|
||
exception).
|
||
|
||
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
|
||
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
|
||
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
|
||
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
|
||
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
|
||
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
|
||
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
|
||
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
|
||
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
|
||
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
|
||
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
|
||
same event accepted on both nodes — which happens in every boot window, since
|
||
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
|
||
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
|
||
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
|
||
|
||
**The library pin is `0.2.1`** (`Directory.Packages.props`, both `ZB.MOM.WW.LocalDb` and
|
||
`.Replication`), bumped from `0.1.3` on 2026-08-15. It was a **pure pin bump — no code change**:
|
||
`RegisterReplicated(string)` is unchanged, everything 0.2.x added is additive, the wire protocol is
|
||
untouched (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2 upgrades in place on
|
||
open, so no rig or volume action was needed. **Its two new APIs — `DeregisterReplicated` and
|
||
`RegisterReplicated(table, baselineExistingRows:)` — are now both used** (2026-08-18); the bullets
|
||
below are the shape that landed, not the bump itself. Two things are worth knowing:
|
||
|
||
- **`MaxBatchSize` no longer means what the rig's `16` was chosen to mean.** 0.2.0 made batching
|
||
**byte-budgeted** — `LocalDb:Replication:MaxBatchBytes` (2 MB default, left unset here, sized
|
||
under the 4 MB gRPC cap) bounds the wire message. `MaxBatchSize` is demoted to a secondary row
|
||
cap **and** the DB read-page size in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which
|
||
materialise a whole page in memory *before* the byte split runs. So the site-a pair keeps
|
||
`16` — it now bounds that transient allocation (≈2.7 MB at ≈171 KB artifact chunk rows), not the
|
||
message size. Every comment and doc row that called it "row-count-only against gRPC's 4 MB cap"
|
||
was corrected with the bump; do not restore that rationale.
|
||
- **Registration FOLLOWS the replication config — it is no longer unconditional (2026-08-18).**
|
||
`LocalDbSetup.OnReady` reads one predicate, `LocalDbSetup.ReplicationIsConfigured`:
|
||
**`LocalDb:Replication:PeerAddress` OR `LocalDb:Replication:ApiKey` is non-blank.** The **OR is
|
||
not belt-and-braces** — only the dialling half of a pair sets `PeerAddress`, so keying on it
|
||
alone would strip capture from every *passive* node (site-a-2 carries `ApiKey` +
|
||
`LocalDb:SyncListenPort` and nothing else) and the pair would converge in one direction only,
|
||
silently. ⚠️ **`SyncListenPort` deliberately does NOT join the OR:** the passive endpoint is
|
||
fail-closed on the key, so a port without an `ApiKey` rejects every inbound stream and has
|
||
nothing to capture for.
|
||
- **Configured ⇒ `RegisterReplicated(table, baselineExistingRows: true)`**, per table, in the
|
||
same load-bearing DDL→register order. Baselining is what makes enabling replication on a node
|
||
that has been running *without* it actually converge: capture is CDC and the snapshot streamer
|
||
pages from `__localdb_row_version`, so pre-registration rows were in neither and a late opt-in
|
||
used to replicate only writes made after the restart. It seeds the ledger at the LWW floor
|
||
(HLC `0`, this node's id) and flags a resync — a baselined row loses to any genuine remote
|
||
write of the same key and wins only where the peer has no version at all. It is idempotent, so
|
||
it is free on every boot after the first. ⚠️ On the pinned **0.2.1** the snapshot flag is
|
||
per-**database**, not per-table, so baselining anything streams *every* registered table in
|
||
full, in both directions at once — trim `alarm_sf_events` before enabling replication on a node
|
||
that has been buffering alarm history for months, and expect one slow first sync.
|
||
- **Not configured ⇒ `DeregisterReplicated(table)` per table**, an idempotent boot-time
|
||
self-heal: it drops the capture triggers an earlier (unconditional) build left in the file and
|
||
prunes that table's oplog + row-version rows. Merely declining to register would leave every
|
||
pre-2026-08-18 file capturing forever — two INSERTs plus a `json_object` of the whole row
|
||
inside every write transaction — on precisely the node that has no peer to capture for. The
|
||
call returns `true` when it removed something, so the Information line fires **once, and only
|
||
when it actually cleaned**; a default-OFF node's ordinary boot stays silent. The logger is
|
||
Serilog's static `Log.Logger` (this runs inside the `AddZbLocalDb` singleton factory, where
|
||
resolving one from DI would nest a resolution inside a singleton construction; `Program.cs`
|
||
assigns it before anything resolves `ILocalDb`), with an optional parameter so tests observe
|
||
the line rather than infer it.
|
||
- ⚠️ **PAIR SYMMETRY: flipping registration is a BOTH-NODES-TOGETHER change.** The sync handshake
|
||
compares registered-table digests **fail-closed**, so a node that deregisters a table its peer
|
||
still replicates stops syncing with a schema mismatch. Enabling replication on an established
|
||
pair means setting the config and **restarting both nodes** — do it on both, or on neither.
|
||
- **Rollout note:** the only replicating pair in the fleet is docker-dev **site-a**, and it was
|
||
enabled from a fresh pair, so the impact of the old gap was low — central and site-b are
|
||
default-OFF and simply shed their unused triggers on their next boot. `LocalDbSetupTests`
|
||
pins both branches (registered-table set on a configured node, empty set + zero triggers +
|
||
the HLC-0 baseline ledger entry on the other) and the DDL→register ordering, unchanged.
|
||
|
||
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
|
||
cache), `docs/AlarmHistorian.md`, and the runbook
|
||
`docs/operations/2026-07-20-localdb-pair-replication.md`.
|
||
|
||
## Phase 4 — cut the driver-side ConfigDb connection
|
||
|
||
Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles`
|
||
has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs`
|
||
gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is
|
||
unaffected (it keeps ConfigDb via its admin role). Consequences:
|
||
|
||
- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator`
|
||
fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused
|
||
node may stay `Direct`.
|
||
- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not
|
||
spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db
|
||
factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and
|
||
`Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes
|
||
**240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture
|
||
— where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself
|
||
is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less
|
||
(driver-only) nodes".
|
||
- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore`
|
||
(replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces
|
||
`EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed
|
||
`ScriptedAlarmState` table **was dropped** (Task 9, `3a590a0c`, migration
|
||
`20260723183050_DropScriptedAlarmStateTable`); `EfAlarmConditionStateStore` is deleted.
|
||
- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on
|
||
a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is
|
||
the system of record.
|
||
- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty
|
||
`NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so
|
||
`OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only —
|
||
the same rows a driver node never received via config either, so this is not a regression.
|
||
|
||
Merged to master (`26fad75c`); **live gate PASSED** (`1281aebf` — driver nodes run ConfigDb-free) and
|
||
Task 9 landed (`3a590a0c`). See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and
|
||
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4.
|
||
|
||
## Authentication: API keys on the data plane, LDAP on AdminUI (#520)
|
||
|
||
⚠️ **Changed 2026-07-31. The OPC UA data plane no longer uses LDAP at all.** It is machine-to-machine,
|
||
so a client presents an OPC UA **UserName** token with the fixed username **`apikey`** and an
|
||
`otua_<keyId>_<secret>` credential as the password. `ApiKeyOpcUaUserAuthenticator`
|
||
(`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/`) implements `IOpcUaUserAuthenticator`: one indexed lookup
|
||
on `keyId` plus one peppered HMAC-SHA256, `FixedTimeEquals`, **no network I/O**. The key resolves to a
|
||
**principal**; `NodeAcl.Principal` rows grant it permissions. `LdapOpcUaUserAuthenticator` is
|
||
**deleted**, along with four `Security:Ldap` keys that existed only for it (`AuthTimeoutMs`,
|
||
`MaxConcurrentAuthentications`, `OutageFailureThreshold`, `OutageCooldownSeconds`) — they bounded an
|
||
LDAP bind running inline in the SDK's synchronous, `SessionManager`-lock-holding impersonation
|
||
callback, and nothing on the remaining LDAP path has that property.
|
||
|
||
**LDAP remains, for AdminUI login only** (`Security:Ldap`, `OtOpcUaLdapAuthService` →
|
||
`ILdapAuthService`, group→role mapping via `GroupToRole` + `LdapGroupRoleMapping`). Do not "restore"
|
||
the data-plane authenticator when you see LDAP wiring in `Program.cs` — it is there for the AdminUI.
|
||
|
||
Two things to know before relying on any of this:
|
||
|
||
- ⚠️ **Node ACLs are enforced only behind `Security:Acl:Enabled` (default `false`).** With the flag
|
||
**off**, a key's grants collapse into the two server-wide role gates (`WriteOperate` on Value writes,
|
||
`AlarmAck` on Part 9 methods), **only `Cluster`-scoped grants count** — `ApiKeyInterimRoleMapper`
|
||
deliberately under-grants, because honouring a narrower grant's flags while unable to honour its
|
||
*scope* would turn "WriteOperate on Equipment X" into server-wide write authority — and **read,
|
||
browse, subscribe and HistoryRead are ungated for any admitted session including Anonymous**. With the
|
||
flag **on** (`DataPlaneAuthorizer`), scoped grants decide **every** data-plane operation: write and
|
||
alarm commands and HistoryRead (stage 5), read and browse and subscription creation (stage 6).
|
||
- ⚠️ **The ACL decision REPLACES the role check — the two are never chained.** Chained, a grant would
|
||
have to confer the server-wide role *as well as* pass the scope check, and only `Cluster`-scoped
|
||
grants confer a role, so every narrow grant would stay inert and the flag would appear to do
|
||
nothing. The load-bearing test is therefore an **allow** — a session carrying *no roles* writing
|
||
successfully on the strength of its grant. A deny-only test passes against a build where the ACL is
|
||
never consulted, because the role gate denies too.
|
||
- **The trie is built inside `DataPlaneAuthorityState.Apply`**, not by a second wiring at the publish
|
||
actor, and the state owns its `PermissionTrieCache` rather than being handed one. An uninstalled
|
||
trie denies everything, which is indistinguishable from working enforcement until someone checks
|
||
that an ALLOW is possible at all.
|
||
- **A deploy reaches sessions that are already open**, in both directions: each decision stamps the
|
||
*current* generation rather than the one the session activated under, and re-checks the
|
||
authenticating key — so revoking a key or a grant stops an open session immediately instead of at
|
||
the client's next reconnect. On a driver-only node the artifact is the only revocation channel.
|
||
- `UserAuthorizationState` is built **per decision walk, not per session** — and since the hot-path
|
||
change it is allocated only on a cache **miss**, so a cache hit never constructs one. Its LDAP-shaped
|
||
freshness/staleness predicates are switched off EXPLICITLY (`AuthCacheMaxStaleness = MaxValue`).
|
||
⚠️ Relying on "the stamp is always fresh" was not safe: `MembershipResolvedUtc` comes from
|
||
`DataPlaneAuthorizer`'s `TimeProvider` while `IsStale` is evaluated against
|
||
`TriePermissionEvaluator`'s, so **any divergence between the two clocks denies every request** with a
|
||
verdict indistinguishable from "no grant matched". Found by a test clock two hours ahead.
|
||
- **Stage 6's hot seams (read/browse/subscribe) go through a per-`(session, node, operation, realm)`
|
||
decision cache**, cleared wholesale when the generation advances. It is **sharded per API key**
|
||
(`ApiKeyId ?? Principal` — despite the type name, sessions sharing one key share one shard and one
|
||
budget), so a client crawling a large tree past the per-key ceiling evicts only that key's shard and
|
||
not every other key's decisions. ⚠️ It caches the **grant walk only** — key revocation and expiry are
|
||
re-checked **before** the cache lookup on every call, because expiry moves with the clock rather than
|
||
with a deployment.
|
||
- **Browse needs ancestor visibility, and it must NOT be a synthesized grant on the ancestors.** A
|
||
grant deep in the tree is reached by browsing the nodes above it, and the trie inherits downward
|
||
only, so `PermissionTrieNode.BrowseTransitPrincipals` marks the path. Planting a real `Browse` grant
|
||
on each ancestor instead would be collected by the additive downward walk and leak `Browse` onto
|
||
every sibling subtree beneath that ancestor. ⚠️ **The topmost ancestor is the single shared root
|
||
folder** (`s=OtOpcUa`, raw namespace) that BOTH realm trees hang under — its identifier is a display
|
||
name, not a path, so `DataPlaneAuthorizer.BuildScope` maps it onto the trie's **zero-segment
|
||
realm-root scope** (`V3NodeIds.SharedRootFolderId`, realm-qualified: in the UNS namespace that
|
||
identifier is an ordinary Area), and its Browse considers transit in **either** realm because it
|
||
fronts both trees. Before this every narrow Browse grant was unreachable by navigation — the Browse
|
||
gate denies the whole call when its source node is denied, so clients were stopped at the front door.
|
||
- **Revocation reaches live subscriptions** (stage 6): every other operation re-asks per request, but a
|
||
subscription is authorized once and then streams — and streaming is how machine consumers actually
|
||
read this server. `DataPlaneAuthorityState.Changed` fires `RevalidateMonitoredItems`. ⚠️ A denied
|
||
item is queued `BadUserAccessDenied` and then **detached from its `MonitoredNode2` fan-out — NOT set
|
||
`MonitoringMode.Disabled`**: disabling discards the item's queued notifications, so the client is
|
||
silenced without ever being told why (caught only by the over-the-wire test). Detaching is also
|
||
reversible, so restoring the grant resumes the feed with no client action.
|
||
- **Anonymous is a normal grantable principal**, not a bypass. `OpcUa:AllowAnonymous` (default `true`,
|
||
preserving today's behaviour) controls *admission*; when admitted the session authenticates as the
|
||
reserved principal `anonymous` and gets roles from `NodeAcl` rows naming it, arriving at the gates as
|
||
the same `RoleCarryingUserIdentity` a key produces. Set it `false` to make keys mandatory. A key may
|
||
**not** use the `anonymous` principal, and a failure to read anonymous's grants **denies** rather
|
||
than admitting with no roles.
|
||
- **Keys reach a driver-only node inside the deployment artifact** (stage 4). `ConfigComposer` snapshots
|
||
the `ApiKey` rows — the peppered HMAC only, never a secret, and the pepper is node config that is
|
||
deliberately *not* in the artifact — and `DeploymentArtifact.ParseDataPlaneAuthority` decodes them
|
||
node-side alongside the cluster's `NodeAcl` rows. `OpcUaPublishActor` installs both on
|
||
`IDataPlaneAuthoritySource` on every apply, **before** its `plan.IsEmpty` early-return, so a deployment
|
||
that only revokes a key still lands. A node with a ConfigDb keeps `ConfigDbApiKeyStore` (per-request
|
||
read); a driver-only node gets `ArtifactApiKeyStore`. ⚠️ **The consequence is asymmetric revocation:
|
||
immediate on a ConfigDb node, next-deployment on a driver-only one.**
|
||
- ⚠️ **The auth generation is a node-local COUNTER driven by the `RevisionHash`, not derived from its
|
||
bits.** `PermissionTrieCache` treats the highest `GenerationId` as current, so a generation taken from
|
||
hash bytes would install a *lower* number about half the time and the cache would go on serving the
|
||
previous deployment's ACLs with nothing logged. Pinned by
|
||
`DataPlaneAuthorityStateTests.The_generation_only_ever_increases_even_when_the_hash_decreases`.
|
||
- Note `OtOpcUaServerHostedService` takes `IOpcUaUserAuthenticator` **and** `IDataPlaneAuthoritySource`
|
||
as **required** ctor parameters — the built-in DI container does not fall back to a default value for
|
||
an unregistered service, so both must be registered explicitly on every node (deny-all / empty on a
|
||
node with neither role) or it does not boot.
|
||
|
||
**Grants are authored through a scope picker, not free text (stage 7).** `/clusters/{id}/acls` opens
|
||
`AclScopePickerModal` over the cluster's raw + UNS trees, and selecting a node supplies `ScopeKind`
|
||
**and** `ScopeId` together — so `RawDevice` + a TagId, a cross-cluster scope, a `Cluster` grant carrying
|
||
an id, and the retired `FolderSegment` kind are all unauthorable by construction rather than merely
|
||
rejected later. ⚠️ **A `Cluster` row must produce a NULL `ScopeId`** — the deploy gate rejects one that
|
||
carries an id (`NodeAclScopeIdUnexpected`), so a picker that helpfully filled in the cluster id would
|
||
author rows that cannot be deployed. `IAclScopeService.CheckScopeAsync` mirrors
|
||
`DraftValidator.ValidateNodeAclScopes` rule for rule and **blocks the save** when a scope resolves to
|
||
nothing (re-checked on submit, since a row loaded unchanged is exactly where the operator never touches
|
||
the picker); the list page marks such rows `unresolved`. **UNS tags are deliberately not offered** — a
|
||
UNS signal is a `UnsTagReference`, and a `Tag` grant already covers the raw node *and* every UNS
|
||
projection of it. The picker is `AclScopeTree`, a separate component from `RawTree`, which already
|
||
carries two modes.
|
||
|
||
**All eight stages are done** (`d4389f57` enforcement, stage 7 authoring after it), each live-gated on
|
||
docker-dev. The rationale for every decision above is in
|
||
`docs/plans/2026-07-31-apikey-acl-enforcement-design.md`. See `docs/security.md` for the full guide.
|
||
|
||
Dev/test LDAP is the **shared GLAuth** running on the Linux Docker host at `10.100.0.35:3893` (baseDN `dc=zb,dc=local`, plaintext/`Transport=None`). It is managed via `scadaproj/infra/glauth/` (source of truth + deploy runbook). Single bind account `cn=serviceaccount,dc=zb,dc=local` / `serviceaccount123`; all test users password `password`. The docker-dev compose binds this shared instance directly — `DevStubMode` is no longer used. The per-VM NSSM GLAuth at `C:\publish\glauth\` and the old base DNs `dc=lmxopcua,dc=local` / `dc=otopcua,dc=local` are obsolete. (The integration-test harness under `tests/.../Host.IntegrationTests/` runs its own ephemeral **GLAuth** container on port 3894 — seeded by `tests/.../Host.IntegrationTests/glauth/config.toml` with dev users alice/bob + the `cn=serviceaccount` bind account — for the opt-in `OTOPCUA_HARNESS_USE_LDAP=1` real-bind mode; distinct from the shared dev GLAuth on 3893. It replaced the retired `bitnami/openldap:2.6` in PR #451, 2026-07-15.)
|
||
|
||
## Library Preferences
|
||
|
||
- **Logging**: Serilog with rolling daily file sink
|
||
- **Unit tests**: xUnit + Shouldly for assertions
|
||
- **Service hosting (Server, Admin)**: .NET generic host with `AddWindowsService` (decision #30 — replaced TopShelf in v2; see `src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs`)
|
||
- **OPC UA**: OPC Foundation UA .NET Standard stack (https://github.com/opcfoundation/ua-.netstandard) — NuGet: `OPCFoundation.NetStandard.Opc.Ua.Server`
|
||
|
||
## OPC UA .NET Standard Documentation
|
||
|
||
Use the DeepWiki MCP (`mcp__deepwiki`) to query documentation for the OPC UA .NET Standard stack: `https://deepwiki.com/OPCFoundation/UA-.NETStandard`. Tools: `read_wiki_structure`, `read_wiki_contents`, and `ask_question` with repo `OPCFoundation/UA-.NETStandard`.
|
||
|
||
## Testing
|
||
|
||
Use the Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/` for manual testing against the running OPC UA server. Supports connect, read, write, browse, subscribe, historyread, alarms, and redundancy commands. See `docs/Client.CLI.md` for full documentation.
|
||
|
||
```bash
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
|
||
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500
|
||
```
|
||
|
||
Address pickers in AdminUI support live browse for OpcUaClient and Galaxy drivers — see `docs/plans/2026-05-28-driver-browsers-design.md`.
|
||
|
||
The AdminUI's global **UNS** page (`/uns`) is the single surface for managing the unified namespace fleet-wide (Area → Line → Equipment → Tag/VirtualTag), replacing the old per-cluster UNS/Equipment/Tags tabs. See `docs/Uns.md`.
|
||
|
||
**v3 (Batch 2): device I/O is authored in the `/raw` project tree** (`GlobalRaw.razor` → `RawTree.razor`, `IRawTreeService`) — a cluster-rooted `Folder → Driver → Device → TagGroup → Tag` hierarchy with per-node context menus (the reusable `ContextMenu`), lazy loading, driver/device config modals (endpoint moved to `Device.DeviceConfig`; Test-connect probes the `DriverDeviceConfigMerger`-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed `/clusters/{id}/drivers` flow (`DriverTypePicker`, `DriverEditRouter`, the 8 `*DriverPage` shells, the Drivers list) is **retired** — its form bodies live on inside the `/raw` modals. The `DriverType` dispatch maps are keyed off the `DriverTypeNames` constants (fixing the `TwinCat`/`Focas` drift). A `Calculation` pseudo-driver (`DriverTypeNames.Calculation`, `IDependencyConsumer`, deploy-time `scriptId`-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See `docs/Raw.md`.
|
||
|
||
**v3 (Batch 3): UNS Equipment is reference-only.** The equipment **Tags** tab no longer authors or binds tags — it holds `UnsTagReference` rows pointing at raw tags authored in `/raw`. "+ Add reference" opens the `RawTree` in picker mode, **cluster-scoped** (structurally + server-enforced `tag.cluster == equipment.cluster`); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. **Effective-name uniqueness** (references + VirtualTags + ScriptedAlarms share the `{EquipmentId}/{EffectiveName}` NodeId space) is enforced at authoring (`IEffectiveNameGuard`) and at the deploy gate (`DraftValidator.UnsEffectiveNameCollision` — catches rename-induced collisions). Scripts use **`ctx.GetTag("{{equip}}/<RefName>")`** — resolved per-equipment through `UnsTagReference` effective names to the backing RawPath at both compose seams (`AddressSpaceComposer` + `DeploymentArtifact`, via the shared `EquipmentReferenceMap`, byte-parity); an unresolved `<RefName>` is a deploy error (`EquipReferenceUnresolved`) AND a live Monaco diagnostic (`OTSCRIPT_EQUIPREF`) — editor accepts ⇔ publish accepts. `EquipmentScriptPaths.DeriveEquipmentBase` is deleted (the dot-joint `{{equip}}.X` became the slash-joint `{{equip}}/<RefName>`). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (`RawTreeService` substring scan). `ImportEquipmentModal` dropped the `DriverInstanceId` column. See `docs/Uns.md` + `docs/ScriptEditor.md`.
|
||
|
||
**v3 (Batch 4 = v3.0): dual-namespace address space.** The OPC UA server exposes **two namespaces** — `https://zb.com/otopcua/raw` (Raw device tree, `s=<RawPath>`) + `https://zb.com/otopcua/uns` (UNS tree, `s=<Area>/<Line>/<Equipment>/<EffectiveName>`) — replacing the single `.../ns` and the retired `EquipmentNodeIds` scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in `DriverHostActor` to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable `Organizes`-references its raw node. Writes route via **either** NodeId to the same driver ref under the same realm-qualified `WriteOperate` gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (`ConditionId = RawPath`) and fan via SDK `AddNotifier` to the raw device folder + every referencing equipment folder (one `ReportEvent`, one Server-object copy); both NodeIds register the SAME historian tagname (mux `HistorizedTagRef` stays single, keyed by RawPath). Identity authority: `V3NodeIds` + `AddressSpaceRealm` (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, `docs/Uns.md`, and `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`.
|
||
|
||
The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.
|
||
|
||
## Scripting / Script Editor
|
||
|
||
C# virtual-tag scripts are authored in a **Roslyn-backed Monaco editor** on the
|
||
ScriptEdit page (`/scripts/{id}`) and inline inside the virtual-tag modal on the
|
||
`/uns` page. The editor provides completions, live diagnostics, hover, signature
|
||
help, document formatting, and tag-path completions inside `ctx.GetTag("…")` /
|
||
`ctx.SetVirtualTag("…")` literals — all backed by the same compiler context the
|
||
runtime publish gate uses, so what the editor accepts/rejects matches publish
|
||
exactly. The backend lives in
|
||
`src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/` (six minimal-API
|
||
endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
|
||
(Administrator or Designer) via `RequireAuthorization` in
|
||
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
|
||
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.
|
||
|
||
## Scripted Alarm Ack/Shelve
|
||
|
||
Inbound operator acknowledge/shelve for scripted alarms is fully implemented. Two surfaces converge on the `alarm-commands` DPS topic: (1) OPC UA Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) wired in `OtOpcUaNodeManager`, gated on the `AlarmAck` LDAP role via `RoleCarryingUserIdentity`; (2) AdminUI `/alerts` per-row buttons routed through the `AdminOperationsActor` singleton (gated by `DriverOperator` policy). `ScriptedAlarmHostActor` dispatches from the topic to the engine. Client.CLI supports `ack`, `confirm`, `shelve` commands. See `docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" and `docs/AlarmTracking.md`.
|
||
|
||
## Historian / HistoryRead
|
||
|
||
**Backend: HistorianGateway (sole historian backend).** As of the gateway-integration cutover, the
|
||
historian read, alarm-write, and continuous-historization paths are all served by the
|
||
**`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the Gitea-feed
|
||
**`ZB.MOM.WW.HistorianGateway.Client`** gRPC package (`historian_gateway.v1`) behind a thin
|
||
`IHistorianGatewayClient` seam in `ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`. **The bespoke
|
||
Wonderware TCP/ArchestrA sidecar projects and the vestigial `Historian.Wonderware` driver type were
|
||
retired** — there is no Wonderware backend in the tree anymore (see `docs/drivers/Historian.Wonderware.md`,
|
||
now a retired stub).
|
||
|
||
A tag is historized by adding `"isHistorized": true` to its `TagConfig` JSON blob (authored in the
|
||
raw-JSON textarea on the `/uns` TagModal); an optional `"historianTagname"` field overrides the default
|
||
historian tagname, which is the tag's driver `FullName`.
|
||
|
||
### Read path (`ServerHistorian` section)
|
||
|
||
The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
|
||
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
|
||
(historized nodes return `GoodNoData`, never an error). Supported variants: Raw, Processed
|
||
(Average/Minimum/Maximum/Total/Count aggregates), and AtTime over historized variable nodes; Events over
|
||
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
|
||
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.
|
||
|
||
`ServerHistorian` appsettings keys (`ServerHistorianOptions`; `Enabled` defaults to `false`):
|
||
|
||
| Key | Default | Notes |
|
||
|---|---|---|
|
||
| `Enabled` | `false` | `true` registers the gateway read client; `false` keeps `NullHistorianDataSource` |
|
||
| `Endpoint` | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c) |
|
||
| `ApiKey` | `""` | Peppered-HMAC key `histgw_<id>_<secret>` sent as `Authorization: Bearer`. **Supply via env `ServerHistorian__ApiKey` — never commit** |
|
||
| `UseTls` | `true` | Connect over TLS; must match the `Endpoint` scheme |
|
||
| `AllowUntrustedServerCertificate` | `false` | Accept a self-signed / untrusted server cert (dev / on-prem only) |
|
||
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
|
||
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
|
||
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
|
||
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
|
||
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
|
||
| `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |
|
||
|
||
### Alarm-history path (`AlarmHistorian` section)
|
||
|
||
Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
|
||
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
|
||
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
|
||
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
|
||
the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
|
||
holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
|
||
`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
|
||
`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
|
||
(endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
|
||
target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
|
||
|
||
### Continuous historization (`ContinuousHistorization` section)
|
||
|
||
When `ContinuousHistorization:Enabled=true` **and** `ServerHistorian` is configured, the Host builds a
|
||
durable, crash-safe **FasterLog** outbox (`FasterLogHistorizationOutbox`) + a gateway-backed
|
||
`IHistorianValueWriter`, and `WithOtOpcUaRuntimeActors` spawns the `ContinuousHistorizationRecorder`. The
|
||
recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the
|
||
crash boundary), and drains the outbox to the gateway's SQL live-value write path (**`WriteLiveValues`**)
|
||
with exponential backoff. The gateway connection is sourced from `ServerHistorian`; this section carries
|
||
only the recorder + outbox knobs:
|
||
|
||
| Key | Default | Notes |
|
||
|---|---|---|
|
||
| `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder |
|
||
| `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage |
|
||
| `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands |
|
||
| `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` |
|
||
| `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup |
|
||
| `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) |
|
||
| `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" |
|
||
| `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve |
|
||
| `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
|
||
| `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |
|
||
| `MaxBackoffSeconds` | `30` | Cap on the exponential retry backoff |
|
||
|
||
### Tag auto-provisioning (`IHistorianProvisioning` EnsureTags hook)
|
||
|
||
`AddressSpaceApplier.Apply()` fires a **non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync`
|
||
hook for added historized value tags — the gateway-backed `GatewayTagProvisioner` calls the gateway's
|
||
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
|
||
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
|
||
deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the
|
||
recorder likewise drops + meters non-numeric values. Continuous historization is **numeric-analog only** in
|
||
v1 (`UInt16→UInt4` is a documented fallback).
|
||
|
||
The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`),
|
||
built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the
|
||
read path** — `WithOtOpcUaRuntimeActors` resolves `IHistorianProvisioning` and passes it into the applier.
|
||
When the section is disabled the applier binds the no-op `NullHistorianProvisioning` and provisioning does not
|
||
run. Every dispatch logs a tally (`dispatched/requested/ensured/skipped/failed`); `dispatched=N` with
|
||
`requested=0` flags the dormant no-op. The API key must carry `historian:tags:write` for `EnsureTags`. (PR
|
||
#423 shipped `GatewayTagProvisioner` but left this wiring out, so provisioning was dormant; restored 2026-06-27.)
|
||
**DataType mapping (issue #441):** `EnsureTags` is an **upsert** (create-or-update, not skip-if-exists).
|
||
`HistorianTypeMapper` maps **`Boolean → Int2`** (Boolean historizes as a small 0/1 integer) because the
|
||
historian's `Int1` analog-tag creation path is **server-degenerate** — `EnsureTags(Int1)` stores an unusable
|
||
stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are
|
||
supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see
|
||
the "Boolean historization maps to Int2" section in `docs/Historian.md`.
|
||
|
||
### Gateway-side prerequisites
|
||
|
||
The target HistorianGateway OtOpcUa points at **must** run with:
|
||
|
||
- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
|
||
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
|
||
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.
|
||
|
||
### Migration note (deployments upgrading from the Wonderware backend)
|
||
|
||
The `ServerHistorian` section changed shape. Rename the old Wonderware keys and supply the key via env:
|
||
|
||
| Old (Wonderware) key | New (gateway) key |
|
||
|---|---|
|
||
| `ServerHistorian:Host` + `:Port` | `ServerHistorian:Endpoint` (`https://host:5222`) |
|
||
| `ServerHistorian:SharedSecret` | `ServerHistorian:ApiKey` (**env `ServerHistorian__ApiKey`**) |
|
||
| `ServerHistorian:ServerCertThumbprint` | `ServerHistorian:CaCertificatePath` (+ `UseTls` / `AllowUntrustedServerCertificate`) |
|
||
|
||
The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`UseTls`/`ServerCertThumbprint`/`SharedSecret`)
|
||
were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
|
||
`ServerHistorian`. See `docs/Historian.md` for the full guide.
|
||
|
||
### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
|
||
|
||
**No longer a limitation.** The cutover was live-validated against the real gateway on
|
||
`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
|
||
protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
|
||
populate `Source_Object` — not a fixable bug). Evidence:
|
||
`docs/plans/2026-06-27-otopcua-historian-followups.md`.
|
||
|
||
Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
|
||
`wonder-sql-vd03`, gateway running the prerequisites above):
|
||
|
||
```bash
|
||
export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222 # absolute gateway URI; absent ⇒ all live tests skip
|
||
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret> # must carry historian:read + historian:write (+ tags:write) scopes
|
||
export HISTGW_TEST_TAG=<existing-tag> # read round-trip
|
||
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag> # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
|
||
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag> # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
|
||
export HISTGW_ALARM_SOURCE=<source-name> # alarm SendEvent → ReadEvents round-trip
|
||
dotnet test --filter "Category=LiveIntegration"
|
||
```
|
||
|
||
The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
|
||
gate the operator runs on the VPN before trusting the cutover.
|
||
|
||
### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
|
||
|
||
The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
|
||
meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
|
||
`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built
|
||
later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the
|
||
`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it
|
||
through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the
|
||
recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the
|
||
deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe
|
||
`Tell`; a faulting feed can never break a deploy).
|
||
|
||
**The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a
|
||
production-fatal defect** — see `docs/plans/2026-08-01-continuous-historization-live-gate.md`. Three things
|
||
to know:
|
||
|
||
- ⚠️ **`ConfigureAwait(false)` in an Akka handler destroys `Self`.** `OnValueChangedAsync` touched `Self`
|
||
after awaiting `IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false)`, which discards Akka's
|
||
`ActorTaskScheduler` synchronization context — so `ActorBase.Context` was gone and it threw
|
||
`NotSupportedException` on the FIRST value. The throw reached supervision, the actor **restarted**, and
|
||
because production seeds an **empty** ref set the restart discarded every historized ref: one value
|
||
captured, then silence. Fixed by capturing `IActorRef self = Self` before the await (the idiom the
|
||
`GetStatus` handler already used for `Sender`).
|
||
- ⚠️ **A fake that completes synchronously hides this entire class of bug**, and 13 tests passed because
|
||
of one. `ValueTask.CompletedTask` never yields, so the continuation ran inline with the context intact.
|
||
Worse, the obvious fix to the fake — `await Task.Yield()` — is *also* vacuous: `Yield()` posts back to
|
||
the captured context, so the caller still resumes on the actor thread. The fake must complete on a
|
||
**pool** thread (`await Task.Delay(1).ConfigureAwait(false)`), as a real per-entry fsync does. Only
|
||
reverting the production fix exposed that the first regression test proved nothing.
|
||
- **Both pair nodes historized the same value** — the recorder had **no Primary gate**, unlike the alarm
|
||
store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as
|
||
**#537** and **fixed** — see the paragraph below.
|
||
|
||
**Gate COMPLETE — #491 CLOSED 2026-08-04.** Outbox append, drain, `WriteLiveValues`,
|
||
**restart-convergence**, `EnsureTags` provisioning (`ensured=2 failed=0`) and **read-back** all ✅. The
|
||
read-back leg ran against the licensed `wonder-sql-vd03` with the VPN up: two registers carrying
|
||
*distinct non-zero* constants (`8224`, `120`) each read back as its own value — a `0`-valued source
|
||
would have proved nothing (the #536 trap).
|
||
|
||
**Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison
|
||
tag); evidence in the same plan doc. Two things to know before touching either.**
|
||
|
||
- **#538 — the discriminator is "did another tag write this pass?", NOT a failure count.** `DrainOnceAsync`
|
||
writes every per-tag group and acks **all-or-nothing** (deliberate: `RemoveAsync` truncates *through* an
|
||
id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned
|
||
the FIFO head while every *healthy* tag was re-submitted on every pass, forever — duplicate samples, an
|
||
outbox that never truncated (observed 8.3 MB; signature = the gateway's `ok` **and** `error` counters
|
||
climbing together). The drain now quarantines a tag after `MaxWriteAttempts` consecutive failures, but
|
||
⚠️ **only counts a failure on a pass where some other tag succeeded** — otherwise an ordinary gateway
|
||
restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug
|
||
into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so
|
||
the load-bearing test is the **falsification control** (`A_backend_that_fails_every_tag_abandons_nothing`).
|
||
Abandonment is also deferred until *every* failing tag is over budget, so nothing is discarded on a pass
|
||
that could not have advanced the head anyway.
|
||
- **#537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink.** The alarm
|
||
queue is in LocalDb and *replicates*, so the Secondary holds the Primary's rows and gating the drain
|
||
loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its
|
||
drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of
|
||
stale duplicates on promotion). ⚠️ **And the peer that matters is one recording the SAME refs, not one
|
||
holding the same rows** — a node stands down only for a Primary in its own per-cluster mesh, and a node
|
||
with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is
|
||
usually running another cluster's configuration, and deferring to it ends this cluster's historization
|
||
*silently* (the drain reports success on an empty outbox forever) — the same scope mismatch that once
|
||
suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on
|
||
`RedundancyRoleView` behind **`IHistorizationRoleView`**, registered unconditionally on driver nodes
|
||
(NOT alongside the alarm `IRedundancyRoleView`, which only exists when that section is enabled — the
|
||
#511 registration accident).
|
||
|
||
Also re-confirmed live: **`EnsureTags` provisioning fires only for newly-ADDED historized tags** —
|
||
flipping an existing tag's `historianTagname` produces no provisioning at all
|
||
(`2026-06-27-historian-gateway-integration-issues.md` Issue 5).
|