RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good
so every native + scripted condition reported Good unconditionally — a
comms-lost device still showed a healthy, inactive, Good condition (a
wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs
bucketing on IsGood) could not tell "genuinely inactive" from "lost contact".
Layer 1 — make Quality a real, plumbed field:
- AlarmConditionSnapshot gains OpcUaQuality Quality (default Good).
- MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good).
- WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality
member so a quality-bucket change fires a Part 9 event.
Layer 2 — drive native quality from driver connectivity (a comms-lost driver
emits no alarm transitions, and an alarm-bearing raw tag has no value variable,
so quality can't come from either existing channel):
- DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting.
- DriverHostActor fans it to every native condition the driver owns as
OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
- New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and
fires only on a bucket change — never touches Active/Acked/Retain (an active
alarm that loses comms stays active). Not a full-snapshot re-projection, so it
can't clobber severity/message and works for a never-fired condition.
Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the
reflection forwarding guard). Ungated by redundancy role; no /alerts row.
Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3).
Tests: node-level (materialise/project/no-clobber/unknown-node no-op),
NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor
fan-out, OpcUaPublishActor routing, and the wire-level guard
(Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified
against a simulated pre-fix always-Good server. Existing DriverInstanceActor
parent probes ignore the new ConnectivityChanged.
Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)";
design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md.
Materialize each native alarm ONCE at the raw tag (ConditionId = RawPath, Raw
realm); wire the single condition as an SDK event notifier of each referencing
equipment's UNS folder so one ReportEvent fans to every root without
re-reporting per root (which would break Server-object dedup + Part 9 ack
correlation).
- New sink method WireAlarmNotifiers(alarmNodeId, alarmRealm,
notifierFolderNodeIds, notifierFolderRealm) on IOpcUaAddressSpaceSink,
forwarded through DeferredAddressSpaceSink + SdkAddressSpaceSink +
NullOpcUaAddressSpaceSink (the forwarding-trap guard); auto-covered by the
DeferredSinkForwardingReflectionTests realm + forwarding guards + a
hand-written forward test.
- OtOpcUaNodeManager: the normative AddNotifier(isInverse) pair + idempotent
EnsureFolderIsEventNotifier per equipment folder; tracked per condition in
_alarmNotifierWiring. Teardown symmetry: RemoveNotifier(bidirectional:true) on
RebuildAddressSpace, RemoveAlarmConditionNode, and RemoveEquipmentSubtree so no
inverse-notifier entry leaks across redeploys.
- AddressSpaceApplier.MaterialiseRawSubtree wires notifiers for each native alarm
tag, resolving its ReferencingEquipmentPaths (Area/Line/Equipment) to the
EquipmentId folder NodeIds via BuildEquipmentIdByFolderPath.
- AlarmTransitionEvent gains ReferencingEquipmentPaths (empty default); /alerts
renders the referencing-equipment list as display metadata.
- Un-skipped + rewrote the native-alarm dark tests
(DriverHostActorNativeAlarmTests x6, DriverHostActorNativeAlarmAckRoutingTests
x1) for the v3 raw-condition model; new NodeManagerMultiNotifierAlarmTests
proves multi-notifier wiring + teardown symmetry (no leaked duplicates after a
re-trip) + applier wiring test.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only.
A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare
strings; the bare-only key let a colliding raw+UNS pair route to the WRONG
driver ref (last-writer-wins). The realm the node manager resolves (RealmOf)
is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite ->
_driverRefByNodeId keyed by (realm, bareId). New regression test:
Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm.
M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes
hard-short-circuits (single enforcement point; _discoveredByDriver never
populates so the re-inject tail is inert too), with a clear log pointing at the
/raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3;
the 16+2 v2 injection scenarios re-pointed to an accurate skip reason
(DiscoveryInjectionDormantV3).
M2 (MEDIUM): realm-qualified dual-node self-correction tests —
Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched +
Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped).
L1: removed the = AddressSpaceRealm.Uns defaults from the consequential
node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/
EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/
RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the
AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit
realm; read-only accessors + internal builders retain their defaults.
L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the
retired EquipmentNodeIds.Variable).
Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity
round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Wave B of Batch 4 — the runtime binding seam for the dual namespace.
Applier (both realms, explicit realm at every sink call site):
- MaterialiseRawSubtree: Raw containers as folders + Raw tags as variables
keyed by RawPath (native-alarm tag → single Part 9 condition at the RawPath),
all in AddressSpaceRealm.Raw; historian tagname = override else RawPath.
- MaterialiseUnsReferences: each UNS reference Variable under its equipment
folder (Uns realm) + an Organizes UNS->Raw edge; inherits writable/array/
historian tagname from the backing raw tag (both NodeIds -> one tagname).
- FeedHistorizedRefs / ProvisionHistorizedTags now source RAW tags (mux ref
stays single, keyed by RawPath); ApplyPureRemove tears down raw tags + UNS
refs in place (raw-container removal falls back to rebuild).
DriverHostActor (dual-NodeId, single-source fan-out):
- _nodeIdByDriverRef value gains a realm (NodeRealmRef); rebuilt from
RawTags UNION UnsReferenceVariables so one (DriverInstanceId, RawPath) fans
to the raw NodeId AND every referencing UNS NodeId with identical
value/quality/timestamp. Write inverse map keyed by the bare id; the
ns-qualified NodeId the write hook passes is normalised (BareNodeId) so a
write to either NodeId resolves the same driver ref (-> RawPath write).
- Native raw alarm condition routing is realm-tagged (Raw); AttributeValueUpdate
+ AlarmStateUpdate carry the realm through to the sink.
Retire EquipmentNodeIds -> V3NodeIds.Uns (applier/VirtualTagHostActor) and
RawPaths.Combine (DiscoveredNodeMapper, discovered nodes are Raw now).
DeploymentArtifact.ParseComposition emits the Raw + UNS subtrees byte-parity
with the composer (reconstruct entities -> AddressSpaceComposer.Compose).
Sink surface: removed the transitional `= AddressSpaceRealm.Uns` defaults from
IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink / SdkAddressSpaceSink /
DeferredAddressSpaceSink / NullOpcUaAddressSpaceSink — every call site is now
explicit (realm reordered before the trailing optionals on EnsureVariable +
MaterialiseAlarmCondition). Node-manager convenience methods keep their
defaults (they are not the interface impl; Sdk delegates explicitly).
Tests: rewrote DriverHostActorLiveValueTests (fan-out drift, 1:N) +
DriverHostActorWriteRoutingTests (dual-NodeId raw/uns write routing) to the v3
raw+uns model; new AddressSpaceApplierRawUnsTests; migrated the EquipmentTags
provisioning/feed tests to RawTags; swept EquipmentNodeIds test callers.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Close the Wave-A M1 gap: the sink surface had no way to wire the mandated
cross-tree Organizes reference from each UNS reference Variable to its backing
Raw node, forcing WP3 to reopen the frozen surface. Add a dedicated
realm-qualified AddReference method instead.
- IOpcUaAddressSpaceSink.AddReference(sourceNodeId, sourceRealm, targetNodeId,
targetRealm, referenceType="Organizes"): realm-qualified both ends; idempotent;
a missing endpoint is a logged no-op (never throws) so a mid-rebuild race can't
fault a deploy. Base-interface capability (no surgical sniff, no transitional
default — WP3 wires it explicitly per UNS reference variable).
- SdkAddressSpaceSink forwards to the node manager; DeferredAddressSpaceSink
forwards unconditionally (like EnsureFolder); NullOpcUaAddressSpaceSink no-ops.
- OtOpcUaNodeManager.AddReference: resolve both nodes by full ns-qualified key
(variable/folder/condition via ResolveNodeState), guard both-exist, then wire
the edge bidirectionally (forward Organizes on source, inverse on target) with a
ReferenceExists idempotency guard + ClearChangeMasks on mutated sides. Reference
type resolved by ResolveReferenceType (Organizes default). Added internal
GetNodeReferences test/diagnostic accessor.
- Reflection guard: the exhaustive-forwarding test + the realm-discriminator guard
auto-cover AddReference (it has two AddressSpaceRealm params) — no edit needed.
- New SdkAddressSpaceSinkTests: Organizes UNS→Raw edge created bidirectionally +
idempotent; missing endpoint is a safe no-op.
- All 15 IOpcUaAddressSpaceSink test doubles gain the AddReference no-op so the
solution still builds.
Build: dotnet build ZB.MOM.WW.OtOpcUa.slnx = 0 errors.
Tests: Commons.Tests 310/310; OpcUaServer.Tests 337 passed / 4 pre-existing skips.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Register both v3 namespaces (Raw first, UNS second) on OtOpcUaNodeManager and
thread an AddressSpaceRealm discriminator through every node-naming sink method
so a bare node id is resolved to the correct namespace by realm — never parsed
out of the id string.
- IOpcUaAddressSpaceSink / ISurgicalAddressSpaceSink: every node-naming method
gains `AddressSpaceRealm realm = AddressSpaceRealm.Uns` (transitional default so
un-migrated WP3 call sites still compile; WP3/Wave B makes them explicit and
removes the default). Null + SdkAddressSpaceSink impls updated; DeferredAddress-
SpaceSink forwards realm through every method (the forwarding trap).
- DeferredSinkForwardingReflectionTests: existing exhaustive-forwarding guard
auto-covers the new signatures; added an explicit guard that every node-naming
sink method carries an AddressSpaceRealm parameter (RebuildAddressSpace exempt).
- OtOpcUaNodeManager: register RawNamespaceUri + UnsNamespaceUri; realm->namespace
index via NamespaceIndexForRealm; all node maps (_variables/_folders/
_alarmConditions/_nativeAlarmNodeIds/_historizedTagnames/_eventNotifierSources)
re-keyed by the full ns-qualified NodeId string so Raw and UNS nodes sharing a
bare id stay distinct; _notifierFolders already NodeId-keyed. A historized UNS
reference node registers the SAME historian tagname as its backing raw node
(both NodeIds -> one tagname). Inbound-write hook routes the full ns-qualified
NodeId to the write gateway (realm-aware) and reverts by bare id + realm.
HistoryRead seams resolve via NodeId directly. DefaultNamespaceUri kept as a
transitional alias to UnsNamespaceUri for the SubscriptionSurvivalTests.
- Test doubles across Commons.Tests / OpcUaServer.Tests / Runtime.Tests updated to
the new interface signatures; SdkAddressSpaceSinkTests asserts the UNS namespace
index for default-realm nodes.
Build: dotnet build ZB.MOM.WW.OtOpcUa.slnx = 0 errors.
Tests: Commons.Tests 310/310; OpcUaServer.Tests 335 passed / 4 pre-existing skips.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Resolves the 12 reported build errors (7 CS0535 sink fakes + 5 CLI CS1587).
Runtime.Tests green (74). NOTE: OpcUaServer.Tests still has pre-existing CS7036
errors from the in-progress Galaxy-tag workstream (Phase7Plan/Phase7CompositionResult
new required params) — separate, test-only, not addressed here.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
Phase7Composer now carries UnsAreaProjection + UnsLineProjection lists so
the applier can materialise the full UNS topology in the OPC UA address
space. New IOpcUaAddressSpaceSink.EnsureFolder(folderNodeId, parentNodeId,
displayName) seam (no-op default, recorded in tests, forwarded by
DeferredAddressSpaceSink, implemented by SdkAddressSpaceSink). The SDK-
side OtOpcUaNodeManager gains an EnsureFolder API that creates
FolderState nodes with proper parent linkage; RebuildAddressSpace now
clears folders too so re-applies don't accumulate stale topology.
Phase7Applier.MaterialiseHierarchy walks composition.UnsAreas →
composition.UnsLines → composition.EquipmentNodes, calling EnsureFolder
with the correct parent at each level. Idempotent — calling twice with
the same composition is a no-op. OpcUaPublishActor.HandleRebuild invokes
it after Phase7Applier.Apply so OPC UA clients browsing the server now
see Area/Line/Equipment as proper folders rather than flat tag ids.
DeploymentArtifact.ParseComposition reads UnsAreas + UnsLines from the
JSON snapshot the ControlPlane emits, populating the new fields when
present.
Phase7Composer.Compose now accepts UnsAreas + UnsLines; a 3-arg overload
preserves the old signature for legacy callers + existing tests. The
Phase7CompositionResult convenience ctor likewise keeps the planner
tests working without UNS data.
3 new hierarchy tests (pure unit + boot-verify against a real
OtOpcUaSdkServer); OpcUaServer suite is 48/48 green (was 45, +3),
Runtime 74/74 unchanged.
Closes#85.
OpcUaPublishActor now routes through pluggable seams instead of just
incrementing a counter:
- IOpcUaAddressSpaceSink (Commons.OpcUa) — WriteValue / WriteAlarmState
/ RebuildAddressSpace. OpcUaQuality enum moved here from the actor's
nested type so producers don't have to reference the actor itself.
- IServiceLevelPublisher — Publish(byte). NullServiceLevelPublisher
retains the last level for inspection.
- The actor subscribes to the redundancy-state DPS topic in PreStart
and maps the local node's NodeRedundancyState to a coarse
ServiceLevel (Primary+leader=240, Primary=200, Secondary=100,
Detached=0). This keeps the local SDK's ServiceLevel node honest
without round-tripping back through the admin-singleton calculator.
- ServiceLevelChanged dedupes identical levels so the SDK doesn't see
redundant writes.
- Sink + publisher exceptions are caught and logged; the actor never
crashes its own dispatcher.
- PropsForTests gets optional sink/publisher/localNode params and
skips the DPS subscribe so unit tests stay on a vanilla TestKit
cluster.
Production binding to a real SDK NodeManager + Variable nodes is the
remaining residual — split as F10b. Task 60 still blocked on F10b.
Tests: Runtime 40 -> 46 (+6):
- AttributeValueUpdate routes to sink
- AlarmStateUpdate routes to sink
- RebuildAddressSpace calls sink.Rebuild
- ServiceLevelChanged dedupes
- RedundancyStateChanged for primary-leader publishes 240
- RedundancyStateChanged for secondary publishes 100
All 6 v2 test suites green: 132 tests passing.
Joseph Doherty