Phase 2 Task 5. DriverHostActor and ScriptedAlarmHostActor now subscribe to the
node-local EventStream alongside their existing DPS topic subscriptions, so the
transport flag lives ENTIRELY on the publishing side -- exactly one of the two
ever publishes, and the switch can be flipped or reverted without touching a
single subscriber.
Note the alarm case is not symmetric: the DPS subscribe there still carries the
node-LOCAL publisher (OtOpcUaServerHostedService's OPC UA Part 9 method calls),
which never crosses the boundary and stays on DPS in both modes. Only the
central AdminUI leg moves.
Renamed DriverHostActor._coordinatorOverride to _ackRouter. Under ClusterClient
mode that ref is the NodeCommunicationActor, emphatically not a coordinator,
and the old name would send the next reader looking for one.
Adds two wiring tests, because the unit tests either side of this seam BOTH pass
with the subscription deleted -- the comm actor's test asserts a probe receives
the message, and the host tests drive the actor by direct Tell. Neither notices
a missing PreStart subscribe. Sabotage-verified: deleting either subscription
reddens exactly its own wiring test and nothing else.
Three things went wrong while writing those tests, all worth keeping:
- The first version raced. ActorOf returns before PreStart runs, and an
EventStream.Publish with no subscriber is dropped silently -- no buffering, no
dead letter. Fixed with a warm-up whose ack proves PreStart completed; the
comment says why it is not ceremony.
- The alarm version was VACUOUS: it told the host directly INSIDE the assertion
window alongside the relay, so the direct Tell alone produced the log the
assertion waited for. Split into warm-up (outside) and relay (inside).
- The alarm test needs its own ActorSystem: the shared harness pins
loglevel = WARNING and the only signal an unowned alarm gives is a Debug line.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Layer 3 of #477: a scripted alarm's condition Quality now reflects the WORST
quality across its input tags, mirroring the native OT semantic (#477 L2).
Plumbing (quality was silently discarded twice on the live path):
- VirtualTagActor.DependencyValueChanged gains Quality (defaulted Good); the
DependencyMuxActor forwards the published AttributeValuePublished.Quality it
already carried; ScriptedAlarmHostActor.OnDependencyChanged pushes the real
quality into the engine (was hardcoded 0u/Good).
Engine (Core.ScriptedAlarms):
- ScriptedAlarmEngine computes worst-of-input quality each eval (skipping
not-yet-published inputs, which are a readiness concern, not a quality signal)
and carries it on ScriptedAlarmEvent.WorstInputStatusCode.
- A real transition carries the current worst quality so ToSnapshot's full
snapshot doesn't clobber quality back to Good (e.g. transition while Uncertain).
- A Bad input freezes the condition (no transition), like a comms-lost native
driver; a quality-bucket change with no transition emits the new
EmissionKind.QualityChanged, routed to the existing #477-L2
AlarmQualityUpdate -> WriteAlarmQuality node path (quality only, no /alerts
row, no historian write). ScriptedAlarmSource skips QualityChanged so it never
fabricates a phantom IAlarmSource event.
Host: ToSnapshot maps WorstInputStatusCode -> OpcUaQuality; OnEngineEmission
routes QualityChanged out of band.
Tests (TDD, RED-first): engine worst-carry + Bad/restore QualityChanged +
unchanged-bucket-no-emit; source swallows QualityChanged; mux forwards quality;
host Bad-dep -> AlarmQualityUpdate(no alerts) + transition snapshot carries worst.
Docs: AlarmTracking.md Layer-3 section + design doc.
Closes#478
Part 9 ConditionType.Quality was never assigned; default(StatusCode)==Good
so every native + scripted condition reported Good unconditionally — a
comms-lost device still showed a healthy, inactive, Good condition (a
wrong-VALUE bug, distinct from the null-value #473/#475). Clients (and HMIs
bucketing on IsGood) could not tell "genuinely inactive" from "lost contact".
Layer 1 — make Quality a real, plumbed field:
- AlarmConditionSnapshot gains OpcUaQuality Quality (default Good).
- MaterialiseAlarmCondition sets it (native BadWaitingForInitialData, scripted Good).
- WriteAlarmCondition projects snapshot.Quality; the delta-gate gains a Quality
member so a quality-bucket change fires a Part 9 event.
Layer 2 — drive native quality from driver connectivity (a comms-lost driver
emits no alarm transitions, and an alarm-bearing raw tag has no value variable,
so quality can't come from either existing channel):
- DriverInstanceActor Tells parent ConnectivityChanged on Connected/Reconnecting.
- DriverHostActor fans it to every native condition the driver owns as
OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
- New dedicated IOpcUaAddressSpaceSink.WriteAlarmQuality sets ONLY Quality and
fires only on a bucket change — never touches Active/Acked/Retain (an active
alarm that loses comms stays active). Not a full-snapshot re-projection, so it
can't clobber severity/message and works for a never-fired condition.
Forwarded through DeferredAddressSpaceSink (F10b trap; auto-verified by the
reflection forwarding guard). Ungated by redundancy role; no /alerts row.
Scripted conditions stay Good; worst-of-input quality deferred to #478 (Layer 3).
Tests: node-level (materialise/project/no-clobber/unknown-node no-op),
NativeAlarmProjector, DriverInstanceActor connectivity emission, DriverHostActor
fan-out, OpcUaPublishActor routing, and the wire-level guard
(Condition_event_Quality_tracks_source_connectivity_on_the_wire) — RED-verified
against a simulated pre-fix always-Good server. Existing DriverInstanceActor
parent probes ignore the new ConnectivityChanged.
Docs: docs/AlarmTracking.md §"Condition source-data Quality (#477)";
design doc docs/plans/2026-07-17-alarm-condition-quality-477-design.md.
H1 (HIGH): write-routing key now (AddressSpaceRealm, bareId), not bare-only.
A raw s=<RawPath> and a UNS s=<Area/Line/Equip/Eff> can collide as bare
strings; the bare-only key let a colliding raw+UNS pair route to the WRONG
driver ref (last-writer-wins). The realm the node manager resolves (RealmOf)
is now threaded through IOpcUaNodeWriteGateway.WriteAsync -> RouteNodeWrite ->
_driverRefByNodeId keyed by (realm, bareId). New regression test:
Colliding_raw_and_uns_bare_ids_route_to_their_own_driver_by_realm.
M1 (MEDIUM): discovered-node injection made coherently DORMANT. HandleDiscoveredNodes
hard-short-circuits (single enforcement point; _discoveredByDriver never
populates so the re-inject tail is inert too), with a clear log pointing at the
/raw browse-commit flow. New pin: Discovered_nodes_are_ignored_dormant_in_v3;
the 16+2 v2 injection scenarios re-pointed to an accurate skip reason
(DiscoveryInjectionDormantV3).
M2 (MEDIUM): realm-qualified dual-node self-correction tests —
Failed_uns_write_reverts_uns_node_and_leaves_raw_node_untouched +
Raw_realm_revert_reverts_raw_node_only (the second fails if the realm is dropped).
L1: removed the = AddressSpaceRealm.Uns defaults from the consequential
node-manager mutation methods (WriteValue/WriteAlarmCondition/MaterialiseAlarmCondition/
EnsureFolder/EnsureVariable/UpdateFolderDisplayName/UpdateTagAttributes/
RaiseNodesAddedModelChange/Remove*/RevertOptimisticWriteIfNeeded) + the
AttributeValueUpdate/AlarmStateUpdate records, so the compiler forces explicit
realm; read-only accessors + internal builders retain their defaults.
L3: fixed the stale VirtualTagHostActor class comment (V3NodeIds.Uns, not the
retired EquipmentNodeIds.Variable).
Also: DeploymentArtifactRawUnsParityTests — Raw/UNS node-set byte-parity
round-trip between AddressSpaceComposer.Compose and DeploymentArtifact.ParseComposition.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
T21: add an AdminUI path for acknowledging/shelving alarms that routes
through the admin-pinned AdminOperationsActor cluster singleton, which
republishes onto the same 'alarm-commands' DPS topic the OPC UA method
path (T18) and the engine subscriber (T19) use. The broadcast + the
ScriptedAlarmHostActor ownership filter handle cross-node routing, so
the singleton needs no knowledge of which node owns the alarm.
- Commons: AcknowledgeAlarmCommand/ShelveAlarmCommand (+ result records)
and a shared AlarmCommandsTopic const; ScriptedAlarmHostActor now
re-exports that const (mirrors the DriverControlTopic pattern).
- AdminOperationsActor: two handlers map the control-plane messages to
AlarmCommand (Acknowledge / OneShotShelve / TimedShelve / Unshelve,
threading User/Comment/UnshelveAtUtc) and publish via the DPS mediator.
- IAdminOperationsClient + AdminOperationsClient: typed Acknowledge/Shelve
ask wrappers mirroring StartDeploymentAsync.
- Alerts.razor: per-row DriverOperator-gated Ack/Shelve/Unshelve controls;
operator name from AuthenticationState. Timed-shelve datetime UI deferred.
- 5 TestKit tests (mediator-probe subscribed to alarm-commands) verifying
each kind's mapping + reply; 56/56 ControlPlane tests green.
Validate AddComment up-front (IsNullOrWhiteSpace guard + Warning log) so
a blank-comment command is cleanly rejected before reaching the engine
rather than faulting inside ApplyAddComment and being silently swallowed
by the outer catch. Mirrors the existing TimedShelve missing-UnshelveAtUtc
pattern.
Also fix two stale inline comments: the "async void crash" note on
TimedShelve now correctly says "fault escaping async Task → supervision
restart", and the ownership-filter now documents the benign race with a
concurrent LoadAsync clearing the loaded set.
Tests: AlarmCommand_add_comment_empty_text_is_rejected_not_driven (Theory
— empty string + whitespace) and AlarmCommand_add_comment_nonempty_drives_engine
(positive path, asserts CommentAdded transition on alerts topic).
Subscribe the host to the cluster alarm-commands DPS topic in PreStart and
drive the matching ScriptedAlarmEngine op per inbound AlarmCommand. An
ownership filter (engine.LoadedAlarmIds) ignores commands for alarms this
node does not own; TimedShelve without UnshelveAtUtc and unknown operations
are logged + rejected (never thrown); op failures are caught + logged so a
faulting op can't fault the actor. Re-projection is left to the engine's
existing OnEvent -> OnEngineEmission path.
Handler is a Task-returning ReceiveAsync (the project's AK2003 analyzer
forbids an async-void Receive delegate), giving ordered awaited async on the
actor thread. Adds 3 TestKit tests: ack drives the engine with mapped args,
unowned command ignored, missing-UnshelveAtUtc TimedShelve rejected not
thrown.
Wire the materialised AlarmConditionState method handlers so a client calling
Acknowledge/Confirm/Shelve/AddComment is gated on the AlarmAck data-plane role
and, when allowed, routed back to the scripted-alarm engine via a new
`alarm-commands` DistributedPubSub topic.
- Commons: new AlarmCommand DTO (AlarmId/Operation/User/Comment/UnshelveAtUtc).
- ScriptedAlarmHostActor: add AlarmCommandsTopic const.
- OtOpcUaNodeManager: settable AlarmCommandRouter + wire OnAcknowledge/OnConfirm/
OnAddComment/OnShelve/OnTimedUnshelve. Each resolves the principal off
ISessionOperationContext.UserIdentity as RoleCarryingUserIdentity, fails closed
(BadUserAccessDenied) when the AlarmAck role is absent or no identity, else maps
+ routes an AlarmCommand and returns Good. OnShelve discriminates OneShotShelve/
TimedShelve/Unshelve from the SDK flags; TimedShelve expiry = UtcNow + ms.
No Akka/IActorRef handle — only the Action<AlarmCommand> delegate. T20 de-dup
note left; WriteAlarmCondition untouched.
- OpcUaServer.Security: OpcUaDataPlaneRoles.AlarmAck shared const (the role was a
bare string everywhere; introduced one symbol for the gate + tests).
- OtOpcUaSdkServer: SetAlarmCommandRouter pass-through.
- Host: boot wiring publishes each command via mediator.Tell(Publish(...)) using a
lazy ActorSystem accessor (mirrors DpsScriptLogPublisher).
- Tests: 11 new gate + mapping tests (OpcUaServer.Tests 88->99, all green).
Concrete ITagUpstreamSource the scripted-alarm host actor pushes
DependencyValueChanged values into and ScriptedAlarmEngine reads/subscribes
from. Thread-safe: ConcurrentDictionary value cache + per-path ImmutableList
observer lists with atomic add/remove and capture-then-invoke fan-out.
ReadTag of an unknown path returns a Bad-quality (0x80000000) snapshot stamped
via the injected clock. Adds the Core.ScriptedAlarms project reference Runtime
needs to see the interface.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
OtOpcUaTelemetry (Commons/Observability) centralizes the project's Meter
+ ActivitySource so all instrumentation points emit through a single
named surface. Counters cover the hot paths:
otopcua.deploy.applied (outcome=ack|reject)
otopcua.deploy.apply.duration (s, histogram)
otopcua.driver.lifecycle (event=spawn|spawn_stub|stop|fault)
otopcua.virtualtag.eval (outcome=ok|fail|skip)
otopcua.scriptedalarm.transition (state=activated|acknowledged|cleared)
otopcua.opcua.sink.write (kind=value|alarm|rebuild)
otopcua.redundancy.service_level_change (level=byte)
Plus two ActivitySource spans:
otopcua.deploy.apply wraps DriverHostActor.ApplyAndAck
otopcua.opcua.address_space_rebuild wraps OpcUaPublishActor.HandleRebuild
Instruments are no-op until a listener attaches, so tests + dev hosts
pay nothing for unread telemetry.
Host Program.cs gains AddOtOpcUaObservability() (binds the OtOpcUa Meter
+ ActivitySource to OpenTelemetry, attaches a Prometheus exporter) and
MapOtOpcUaMetrics() (mounts /metrics scrape endpoint). Driver-side
internals + ASP.NET request metrics deliberately stay off — the scrape
payload is scoped to OtOpcUa signals only.
Tests use MeterListener + ActivityListener to verify
VirtualTagActor.eval, OpcUaPublishActor.AttributeValueUpdate, and
RebuildAddressSpace actually emit on the central instruments. Runtime
suite is 72 / 72 green (+3).
Closes#105. Path A (F13b/c/d) complete; next batch options: #85 UNS
folder hierarchy in SDK, or F8b/F9b production engine bindings.
ScriptedAlarmActor now survives actor restart: PreStart loads from
the configured store + restores in-memory state; every Transition()
fires a fire-and-forget save. ActiveState still re-derives from the
evaluator on first tick (Phase 7 decision #14), but Acked state +
lastAckUser persist verbatim so operators don't re-ack across an
outage.
Three pieces:
- IAlarmActorStateStore seam in Commons.Engines, with the
AlarmActorStateSnapshot record (alarmId / state / lastTransitionUtc
/ lastAckUser) and NullAlarmActorStateStore default.
- EfAlarmActorStateStore in Runtime.ScriptedAlarms — production
adapter over the existing ScriptedAlarmState table in ConfigDb.
Maps the actor's 3-state enum to the table's AckedState column
(Active⇒Unacknowledged, Acknowledged⇒Acknowledged, Inactive⇒
Acknowledged). Concurrency conflicts are logged + dropped — the
next transition writes again.
- ScriptedAlarmActor PreStart load (async, piped back as
StateRestored) + Transition save. New Props overload takes the
store; default is NullAlarmActorStateStore so tests stay quiet.
Tests: Runtime 52 -> 57 (+5):
- Transition writes Active then Acknowledged snapshots with
lastAckUser populated
- PreStart with persisted Active state restores so a subsequent
AcknowledgeAlarm fires (not ignored as it would be from Inactive)
- Empty store boots Inactive (AcknowledgeAlarm correctly ignored)
- EfAlarmActorStateStore Save + Load round-trips via in-memory EF
- Load for unknown alarmId returns null
All 6 v2 test suites green: 157 tests passing.
Closes#112. F9 (#80) remaining residual is predicate binding to
Core.ScriptedAlarms.ScriptedAlarmEngine — split as F9b in tasks JSON.
VirtualTagActor and ScriptedAlarmActor now route through pluggable
evaluator interfaces and fan out to the cluster's live-tail topics
shipped in F15.3:
- IVirtualTagEvaluator + NullVirtualTagEvaluator in Commons.Engines.
VirtualTagActor calls evaluator on every DependencyValueChanged,
dedupes unchanged values, forwards EvaluationResult to its parent,
and publishes ScriptLogEntry Warning to the script-logs DPS topic
whenever the evaluator fails.
- IScriptedAlarmEvaluator + NullScriptedAlarmEvaluator. ScriptedAlarmActor
takes an AlarmConfig (id/name/equipment-path/severity/predicate) and
publishes both an AlarmTransitionEvent (alerts topic) and a
ScriptLogEntry (script-logs topic) at every transition. Manual
ConditionMet/Acknowledge/Cleared still flow through the same
Transition() so callers without engine bindings still drive the
state machine; the legacy single-string Props() overload routes
through a default AlarmConfig.
The Null* defaults keep the actors safe when no engine is bound —
unconfigured nodes never spuriously alarm. Production binding to
Core.VirtualTags.VirtualTagEngine and Core.ScriptedAlarms is the
remaining residual (F8b/F9b — split in tasks JSON).
Tests: Runtime 34 -> 40 (+6):
- VirtualTagActorTests x3 (evaluator drives EvaluationResult,
unchanged-value dedup, failure publishes Warning ScriptLogEntry)
- ScriptedAlarmActorTests x3 (engine threshold drives Activated +
Cleared on alerts topic, manual Acknowledge attribution).
All 6 v2 test suites green: 126 tests passing.
Joseph Doherty