The reported symptom was "Runtime.Tests occasionally reports Failed: 1" with no test
name. Instrumenting 30 full-assembly runs reproduced it at 13% (4 runs) and showed it
was never ONE flaky test: five failures across three distinct tests in that batch, and
five more distinct tests over the verification rounds that followed.
Two hypotheses were tested and DISPROVED by measurement before anything was changed:
- Cluster-formation timeout. Every test in this assembly forms a real single-node Akka
cluster over TCP in RuntimeActorTestBase's constructor and waits 5 s for MemberStatus.Up
— 219 formations per run. Instrumented: idle max 960 ms, under-load max 1470 ms, zero
timeouts. A 3.4x margin; not the cause.
- Ephemeral-port exhaustion from that bind churn. TIME_WAIT measured at ~0 throughout.
Not the cause.
FOUR GENUINE ORDERING/LOGIC DEFECTS (none is a timeout):
1. VirtualTagHostActorTests.ApplyVirtualTags_respawns_child_when_plan_changes_in_place
The test loops to accept RegisterInterest/UnregisterInterest in EITHER order — and
then asserts on mux.LastSender, which DEPENDS on that order. Under the [Register,
Unregister] interleaving LastSender is the DYING child and the assertion fails. Now
captures the sender of the Register message as it arrives. Fully deterministic; no
timing involved. (Swept the other four LastSender sites: all drain the Unregister
first, so their ordering is fixed. Only this one was unsafe.)
2. DriverHostActorWriteRoutingTests.Primary_routes_write_via_uns_nodeid_to_driver_by_rawpath
ApplyAck marks the end of the APPLY, not the point a write can succeed: the child is
still in Connecting, which deliberately fast-fails writes ("driver not connected"),
and the NodeId->driver reverse map has not been pushed. Now retries until accepted.
This does not weaken the assertion — every rejection branch replies WITHOUT reaching
the driver, so Writes.Count.ShouldBe(1) still means exactly what it did.
3. HistorianAdapterActorTests.Redundancy_snapshot_without_local_node_...
One 500 ms constant served both presence budgets (AwaitAssert) and absence windows
(ExpectNoMsg). Different quantities that happened to share a number, so the presence
budget could not be raised without slowing every absence check. Split into
AssertTimeout (5 s) and Settle (500 ms).
4. ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks
Waited on `outbox == 0`, which is ALSO true before the first append — so the poll
could win the race against the recorder and return having observed the INITIAL state,
after which the CallCount>=2 check outside the block failed against a recorder that
had not run. Both conditions are now polled together with the retry count as the
discriminator. (The preceding test in the same file documents this exact trap.)
THE PRESENCE-BUDGET CLASS:
After those four, three consecutive 30-run rounds each still failed once — on a
DIFFERENT test, in a DIFFERENT file, every time (OpcUaPublishActorApplyFailureTests 2 s,
OpcUaPublishActorRebuildTests 2 s, OpcUaPublishActorTests 500 ms). That is a
distribution, not a defect, and fixing it one test at a time was the wrong method.
RuntimeActorTestBase now carries a documented PresenceBudget (15 s) and 57 sub-3 s
AwaitAssert/AwaitCondition durations across four files route through it.
Why this is safe rather than sloppy: a presence budget is an upper bound before giving
up, not a wait — these helpers poll and return the instant the condition holds. Raising
one costs nothing on the happy path and CANNOT make a genuinely failing assertion pass;
it only changes how quickly a real breakage reports. Absence windows are the opposite
(the elapsed time IS the assertion) and were deliberately left untouched with their own
short, individually-calibrated literals.
ScriptedAlarmHostActorTests is a special case, diagnosed rather than merely raised: its
8 s budget was sized for message passing but actually waits on a REAL Roslyn compile
(ApplyScriptedAlarms -> ScriptedAlarmEngine.LoadAsync compiles every predicate). Cold
script compiles are the slowest and most variable thing in the assembly, so that class
gets 30 s with the reason recorded.
Verification: 30 consecutive full-assembly runs, 30 passed / 0 failed, against a 13%
baseline. Full solution builds; all 41 unit-test projects green.
NOT fixed here, filed as #501: DriverHostActorNativeAlarmAckRoutingTests:90 and :116 use
AwaitAssert for an ABSENCE assertion, so they return instantly and prove nothing. They
are excluded from the budget change on purpose — a bigger number does not fix them, and
rewriting them to settle-then-assert may legitimately turn them red.
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good
so every native + scripted condition reported Good unconditionally — a
comms-lost device still showed a healthy, inactive, Good condition (a
wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs
bucketing on IsGood) could not tell "genuinely inactive" from "lost contact".
Layer 1 — make Quality a real, plumbed field:
- AlarmConditionSnapshot gains OpcUaQuality Quality (default Good).
- MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good).
- WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality
member so a quality-bucket change fires a Part 9 event.
Layer 2 — drive native quality from driver connectivity (a comms-lost driver
emits no alarm transitions, and an alarm-bearing raw tag has no value variable,
so quality can't come from either existing channel):
- DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting.
- DriverHostActor fans it to every native condition the driver owns as
OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
- New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and
fires only on a bucket change — never touches Active/Acked/Retain (an active
alarm that loses comms stays active). Not a full-snapshot re-projection, so it
can't clobber severity/message and works for a never-fired condition.
Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the
reflection forwarding guard). Ungated by redundancy role; no /alerts row.
Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3).
Tests: node-level (materialise/project/no-clobber/unknown-node no-op),
NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor
fan-out, OpcUaPublishActor routing, and the wire-level guard
(Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified
against a simulated pre-fix always-Good server. Existing DriverInstanceActor
parent probes ignore the new ConnectivityChanged.
Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)";
design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md.
Materialize each native alarm ONCE at the raw tag (ConditionId = RawPath, Raw
realm); wire the single condition as an SDK event notifier of each referencing
equipment's UNS folder so one ReportEvent fans to every root without
re-reporting per root (which would break Server-object dedup + Part 9 ack
correlation).
- New sink method WireAlarmNotifiers(alarmNodeId, alarmRealm,
notifierFolderNodeIds, notifierFolderRealm) on IOpcUaAddressSpaceSink,
forwarded through DeferredAddressSpaceSink + SdkAddressSpaceSink +
NullOpcUaAddressSpaceSink (the forwarding-trap guard); auto-covered by the
DeferredSinkForwardingReflectionTests realm + forwarding guards + a
hand-written forward test.
- OtOpcUaNodeManager: the normative AddNotifier(isInverse) pair + idempotent
EnsureFolderIsEventNotifier per equipment folder; tracked per condition in
_alarmNotifierWiring. Teardown symmetry: RemoveNotifier(bidirectional:true) on
RebuildAddressSpace, RemoveAlarmConditionNode, and RemoveEquipmentSubtree so no
inverse-notifier entry leaks across redeploys.
- AddressSpaceApplier.MaterialiseRawSubtree wires notifiers for each native alarm
tag, resolving its ReferencingEquipmentPaths (Area/Line/Equipment) to the
EquipmentId folder NodeIds via BuildEquipmentIdByFolderPath.
- AlarmTransitionEvent gains ReferencingEquipmentPaths (empty default); /alerts
renders the referencing-equipment list as display metadata.
- Un-skipped + rewrote the native-alarm dark tests
(DriverHostActorNativeAlarmTests x6, DriverHostActorNativeAlarmAckRoutingTests
x1) for the v3 raw-condition model; new NodeManagerMultiNotifierAlarmTests
proves multi-notifier wiring + teardown symmetry (no leaked duplicates after a
re-trip) + applier wiring test.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only.
A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare
strings; the bare-only key let a colliding raw+UNS pair route to the WRONG
driver ref (last-writer-wins). The realm the node manager resolves (RealmOf)
is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite ->
_driverRefByNodeId keyed by (realm, bareId). New regression test:
Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm.
M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes
hard-short-circuits (single enforcement point; _discoveredByDriver never
populates so the re-inject tail is inert too), with a clear log pointing at the
/raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3;
the 16+2 v2 injection scenarios re-pointed to an accurate skip reason
(DiscoveryInjectionDormantV3).
M2 (MEDIUM): realm-qualified dual-node self-correction tests —
Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched +
Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped).
L1: removed the = AddressSpaceRealm.Uns defaults from the consequential
node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/
EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/
RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the
AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit
realm; read-only accessors + internal builders retain their defaults.
L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the
retired EquipmentNodeIds.Variable).
Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity
round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Wave B of Batch 4 — the runtime binding seam for the dual namespace.
Applier (both realms, explicit realm at every sink call site):
- MaterialiseRawSubtree: Raw containers as folders + Raw tags as variables
keyed by RawPath (native-alarm tag → single Part 9 condition at the RawPath),
all in AddressSpaceRealm.Raw; historian tagname = override else RawPath.
- MaterialiseUnsReferences: each UNS reference Variable under its equipment
folder (Uns realm) + an Organizes UNS->Raw edge; inherits writable/array/
historian tagname from the backing raw tag (both NodeIds -> one tagname).
- FeedHistorizedRefs / ProvisionHistorizedTags now source RAW tags (mux ref
stays single, keyed by RawPath); ApplyPureRemove tears down raw tags + UNS
refs in place (raw-container removal falls back to rebuild).
DriverHostActor (dual-NodeId, single-source fan-out):
- _nodeIdByDriverRef value gains a realm (NodeRealmRef); rebuilt from
RawTags UNION UnsReferenceVariables so one (DriverInstanceId, RawPath) fans
to the raw NodeId AND every referencing UNS NodeId with identical
value/quality/timestamp. Write inverse map keyed by the bare id; the
ns-qualified NodeId the write hook passes is normalised (BareNodeId) so a
write to either NodeId resolves the same driver ref (-> RawPath write).
- Native raw alarm condition routing is realm-tagged (Raw); AttributeValueUpdate
+ AlarmStateUpdate carry the realm through to the sink.
Retire EquipmentNodeIds -> V3NodeIds.Uns (applier/VirtualTagHostActor) and
RawPaths.Combine (DiscoveredNodeMapper, discovered nodes are Raw now).
DeploymentArtifact.ParseComposition emits the Raw + UNS subtrees byte-parity
with the composer (reconstruct entities -> AddressSpaceComposer.Compose).
Sink surface: removed the transitional `= AddressSpaceRealm.Uns` defaults from
IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink / SdkAddressSpaceSink /
DeferredAddressSpaceSink / NullOpcUaAddressSpaceSink — every call site is now
explicit (realm reordered before the trailing optionals on EnsureVariable +
MaterialiseAlarmCondition). Node-manager convenience methods keep their
defaults (they are not the interface impl; Sdk delegates explicitly).
Tests: rewrote DriverHostActorLiveValueTests (fan-out drift, 1:N) +
DriverHostActorWriteRoutingTests (dual-NodeId raw/uns write routing) to the v3
raw+uns model; new AddressSpaceApplierRawUnsTests; migrated the EquipmentTags
provisioning/feed tests to RawTags; swept EquipmentNodeIds test callers.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Close the Wave-A M1 gap: the sink surface had no way to wire the mandated
cross-tree Organizes reference from each UNS reference Variable to its backing
Raw node, forcing WP3 to reopen the frozen surface. Add a dedicated
realm-qualified AddReference method instead.
- IOpcUaAddressSpaceSink.AddReference(sourceNodeId, sourceRealm, targetNodeId,
targetRealm, referenceType="Organizes"): realm-qualified both ends; idempotent;
a missing endpoint is a logged no-op (never throws) so a mid-rebuild race can't
fault a deploy. Base-interface capability (no surgical sniff, no transitional
default — WP3 wires it explicitly per UNS reference variable).
- SdkAddressSpaceSink forwards to the node manager; DeferredAddressSpaceSink
forwards unconditionally (like EnsureFolder); NullOpcUaAddressSpaceSink no-ops.
- OtOpcUaNodeManager.AddReference: resolve both nodes by full ns-qualified key
(variable/folder/condition via ResolveNodeState), guard both-exist, then wire
the edge bidirectionally (forward Organizes on source, inverse on target) with a
ReferenceExists idempotency guard + ClearChangeMasks on mutated sides. Reference
type resolved by ResolveReferenceType (Organizes default). Added internal
GetNodeReferences test/diagnostic accessor.
- Reflection guard: the exhaustive-forwarding test + the realm-discriminator guard
auto-cover AddReference (it has two AddressSpaceRealm params) — no edit needed.
- New SdkAddressSpaceSinkTests: Organizes UNS→Raw edge created bidirectionally +
idempotent; missing endpoint is a safe no-op.
- All 15 IOpcUaAddressSpaceSink test doubles gain the AddReference no-op so the
solution still builds.
Build: dotnet build ZB.MOM.WW.OtOpcUa.slnx = 0 errors.
Tests: Commons.Tests 310/310; OpcUaServer.Tests 337 passed / 4 pre-existing skips.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Register both v3 namespaces (Raw first, UNS second) on OtOpcUaNodeManager and
thread an AddressSpaceRealm discriminator through every node-naming sink method
so a bare node id is resolved to the correct namespace by realm — never parsed
out of the id string.
- IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink: every node-naming method
gains `AddressSpaceRealm realm = AddressSpaceRealm.Uns` (transitional default so
un-migrated WP3 call sites still compile; WP3/Wave B makes them explicit and
removes the default). Null + SdkAddressSpaceSink impls updated; DeferredAddress-
SpaceSink forwards realm through every method (the forwarding trap).
- DeferredSinkForwardingReflectionTests: existing exhaustive-forwarding guard
auto-covers the new signatures; added an explicit guard that every node-naming
sink method carries an AddressSpaceRealm parameter (RebuildAddressSpace exempt).
- OtOpcUaNodeManager: register RawNamespaceUri + UnsNamespaceUri; realm->namespace
index via NamespaceIndexForRealm; all node maps (_variables/_folders/
_alarmConditions/_nativeAlarmNodeIds/_historizedTagnames/_eventNotifierSources)
re-keyed by the full ns-qualified NodeId string so Raw and UNS nodes sharing a
bare id stay distinct; _notifierFolders already NodeId-keyed. A historized UNS
reference node registers the SAME historian tagname as its backing raw node
(both NodeIds -> one tagname). Inbound-write hook routes the full ns-qualified
NodeId to the write gateway (realm-aware) and reverts by bare id + realm.
HistoryRead seams resolve via NodeId directly. DefaultNamespaceUri kept as a
transitional alias to UnsNamespaceUri for the SubscriptionSurvivalTests.
- Test doubles across Commons.Tests / OpcUaServer.Tests / Runtime.Tests updated to
the new interface signatures; SdkAddressSpaceSinkTests asserts the UNS namespace
index for default-realm nodes.
Build: dotnet build ZB.MOM.WW.OtOpcUa.slnx = 0 errors.
Tests: Commons.Tests 310/310; OpcUaServer.Tests 335 passed / 4 pre-existing skips.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Resolves the 12 reported build errors (7 CS0535 sink fakes + 5 CLI CS1587).
Runtime.Tests green (74). NOTE: OpcUaServer.Tests still has pre-existing CS7036
errors from the in-progress Galaxy-tag workstream (Phase7Plan/Phase7CompositionResult
new required params) — separate, test-only, not addressed here.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
Phase7Composer now carries UnsAreaProjection + UnsLineProjection lists so
the applier can materialise the full UNS topology in the OPC UA address
space. New IOpcUaAddressSpaceSink.EnsureFolder(folderNodeId, parentNodeId,
displayName) seam (no-op default, recorded in tests, forwarded by
DeferredAddressSpaceSink, implemented by SdkAddressSpaceSink). The SDK-
side OtOpcUaNodeManager gains an EnsureFolder API that creates
FolderState nodes with proper parent linkage; RebuildAddressSpace now
clears folders too so re-applies don't accumulate stale topology.
Phase7Applier.MaterialiseHierarchy walks composition.UnsAreas →
composition.UnsLines → composition.EquipmentNodes, calling EnsureFolder
with the correct parent at each level. Idempotent — calling twice with
the same composition is a no-op. OpcUaPublishActor.HandleRebuild invokes
it after Phase7Applier.Apply so OPC UA clients browsing the server now
see Area/Line/Equipment as proper folders rather than flat tag ids.
DeploymentArtifact.ParseComposition reads UnsAreas + UnsLines from the
JSON snapshot the ControlPlane emits, populating the new fields when
present.
Phase7Composer.Compose now accepts UnsAreas + UnsLines; a 3-arg overload
preserves the old signature for legacy callers + existing tests. The
Phase7CompositionResult convenience ctor likewise keeps the planner
tests working without UNS data.
3 new hierarchy tests (pure unit + boot-verify against a real
OtOpcUaSdkServer); OpcUaServer suite is 48/48 green (was 45, +3),
Runtime 74/74 unchanged.
Closes#85.
OpcUaPublishActor now routes through pluggable seams instead of just
incrementing a counter:
- IOpcUaAddressSpaceSink (Commons.OpcUa) — WriteValue / WriteAlarmState
/ RebuildAddressSpace. OpcUaQuality enum moved here from the actor's
nested type so producers don't have to reference the actor itself.
- IServiceLevelPublisher — Publish(byte). NullServiceLevelPublisher
retains the last level for inspection.
- The actor subscribes to the redundancy-state DPS topic in PreStart
and maps the local node's NodeRedundancyState to a coarse
ServiceLevel (Primary+leader=240, Primary=200, Secondary=100,
Detached=0). This keeps the local SDK's ServiceLevel node honest
without round-tripping back through the admin-singleton calculator.
- ServiceLevelChanged dedupes identical levels so the SDK doesn't see
redundant writes.
- Sink + publisher exceptions are caught and logged; the actor never
crashes its own dispatcher.
- PropsForTests gets optional sink/publisher/localNode params and
skips the DPS subscribe so unit tests stay on a vanilla TestKit
cluster.
Production binding to a real SDK NodeManager + Variable nodes is the
remaining residual — split as F10b. Task 60 still blocked on F10b.
Tests: Runtime 40 -> 46 (+6):
- AttributeValueUpdate routes to sink
- AlarmStateUpdate routes to sink
- RebuildAddressSpace calls sink.Rebuild
- ServiceLevelChanged dedupes
- RedundancyStateChanged for primary-leader publishes 240
- RedundancyStateChanged for secondary publishes 100
All 6 v2 test suites green: 132 tests passing.
Joseph Doherty