Joseph Doherty
154171f48c
The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.
Two hypotheses were tested and DISPROVED by measurement before anything was changed:
- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
— 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
Not the cause.
FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):
1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
captures the sender of the Register message as it arrives. Fully deterministic; no
timing involved. (Swept the other four LastSender sites: all drain the Unregister
first, so their ordering is fixed. Only this one was unsafe.)
2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
still in Connecting, which deliberately fast-fails writes ("driver not connected"),
and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
This does not weaken the assertion — every rejection branch replies WITHOUT reaching
the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.
3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
(ExpectNoMsg). Different quantities that happened to share a number, so the presence
budget could not be raised without slowing every absence check. Split into
AssertTimeout (5 s) and Settle (500 ms).
4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
could win the race against the recorder and return having observed the INITIAL state,
after which the CallCount>=2 check outside the block failed against a recorder that
had not run. Both conditions are now polled together with the retry count as the
discriminator. (The preceding test in the same file documents this exact trap.)
THE PRESENCE-BUDGET CLASS:
After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.
RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.
Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.
ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.
Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.
NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
OtOpcUa
OPC UA server (.NET 10 AnyCPU) that exposes a fleet of industrial drivers as a single OPC UA address space. Drivers ship in-process for AVEVA System Platform Galaxy (via the sibling mxaccessgw repo), Modbus TCP, Siemens S7, Allen-Bradley CIP (ControlLogix / CompactLogix), Allen-Bradley Legacy (SLC 500 / MicroLogix), Beckhoff TwinCAT (ADS), FANUC FOCAS, and OPC UA Client (gateway).
A cross-platform client stack (.NET 10) — shared library, CLI, and Avalonia desktop app — connects to any OPC UA server.
Architecture
OPC UA Clients (CLI, Desktop UI, 3rd-party)
|
v
+-------------------------------------+
| OtOpcUa.Server (.NET 10 AnyCPU) |
| address space + capability fan-out|
+-------------------------------------+
| | | | | | | |
Galaxy Modbus S7 AbCip AbLeg TwinCAT FOCAS OpcUaClient
|
v
mxaccessgw (sibling repo, gRPC)
|
v
MXAccess COM (x86 worker, on AVEVA box)
Galaxy is the only driver with an external runtime: it speaks gRPC to a separately installed mxaccessgw server (sibling repo at c:\Users\dohertj2\Desktop\mxaccessgw\) which owns the MXAccess COM apartment and the x86/STA bitness constraint server-side. Everything in this repo is platform-agnostic .NET 10.
Prerequisites
- .NET 10 SDK (server, drivers, clients all target .NET 10)
- SQL Server reachable for the central config DB
- For Galaxy specifically: a running
mxaccessgwdeployment — see docs/v2/Galaxy.ParityRig.md - For historian read-back / alarm history / continuous historization: a running
ZB.MOM.WW.HistorianGatewaydeployment (the sole historian backend; consumed as theZB.MOM.WW.HistorianGateway.ClientgRPC package). It must runRuntimeDb:Enabled=true+RuntimeDb:EventReadsEnabled=true, and the API key must carryhistorian:read+historian:write+historian:tags:writescopes.
Quick Start
dotnet restore ZB.MOM.WW.OtOpcUa.slnx
dotnet build ZB.MOM.WW.OtOpcUa.slnx
dotnet test ZB.MOM.WW.OtOpcUa.slnx
# Run the server in dev (foreground)
dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host
The server starts on opc.tcp://localhost:4840 with the None security profile. Configure Security.Profiles in src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json to enable Basic256Sha256-Sign or Basic256Sha256-SignAndEncrypt. See docs/security.md.
Install as Windows Services
Production deployment is driven by scripts/install/Install-Services.ps1, which registers the OtOpcUa server service under a chosen service account. Historian support requires a separately deployed ZB.MOM.WW.HistorianGateway and Galaxy support a separately installed mxaccessgw — neither this repo nor the install script provisions them.
.\scripts\install\Install-Services.ps1 `
-InstallRoot 'C:\Program Files\OtOpcUa' `
-ServiceAccount 'DOMAIN\svc-otopcua'
The historian backend is the external ZB.MOM.WW.HistorianGateway (not installed by this script). See the script header and docs/ServiceHosting.md for full options.
Client CLI
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- connect -u opc.tcp://localhost:4840
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- browse -u opc.tcp://localhost:4840 -r -d 3
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode"
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- write -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -v 42
dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- subscribe -u opc.tcp://localhost:4840 -n "ns=2;s=SomeNode" -i 500
See docs/Client.CLI.md and docs/Client.UI.md.
Documentation
Architecture deep-dives
| Topic | Doc |
|---|---|
| OPC UA server composition, namespace fan-out, Polly invoker | docs/OpcUaServer.md |
| Address space layout | docs/AddressSpace.md |
| Read / Write dispatch (driver vs virtual vs scripted-alarm) | docs/ReadWriteOperations.md |
| Incremental sync (driver-backend rediscovery + config publishes) | docs/IncrementalSync.md |
| Service hosting (Server + Admin; external HistorianGateway backend) | docs/ServiceHosting.md |
| Security (transport, LDAP, certificates) | docs/security.md |
| Redundancy | docs/Redundancy.md |
| Status dashboard | docs/StatusDashboard.md |
Drivers
| Topic | Doc |
|---|---|
| Driver specs (per-driver capability surface, config, addressing) | docs/v2/driver-specs.md |
| Galaxy driver | docs/drivers/Galaxy.md |
| Modbus / S7 / AbCip / AbLegacy / TwinCAT / FOCAS / OpcUaClient | docs/drivers/ |
| Galaxy parity rig (mxaccessgw setup) | docs/v2/Galaxy.ParityRig.md |
| Galaxy performance + tracing | docs/v2/Galaxy.Performance.md |
Clients
| Topic | Doc |
|---|---|
| Client CLI | docs/Client.CLI.md |
| Client UI (Avalonia desktop) | docs/Client.UI.md |
v1 archive
The original v1 in-process MXAccess docs (Galaxy.Host topology, Configuration env vars, AlarmTracking, DataTypeMapping, HistoricalDataAccess, Subscriptions, etc.) are preserved under docs/v1/ — historical reference only. PR 7.2 retired the v1 architecture on 2026-04-30; current state is documented in the sections above.
License
Internal use only.