lmxopcua/docs/AlarmTracking.md

Alarm tracking — v2 final architecture

This document describes how OtOpcUa surfaces alarms to OPC UA Part 9 clients after the alarms-over-gateway epic (docs/plans/alarms-over-gateway.md) landed. The v1 architecture (Galaxy.Host's COM-side GalaxyAlarmTracker) is preserved at docs/v1/AlarmTracking.md for historical reference.

Three alarm sources, one OPC UA Part 9 surface

Source	Driver capability	Path
Galaxy MxAccess (driver-native)	GalaxyDriver : IAlarmSource	gateway → worker → MxAccess alarm sink → MX_EVENT_FAMILY_ON_ALARM_TRANSITION → EventPump → driver OnAlarmEvent → AlarmConditionService
Galaxy sub-attribute fallback	IWritable writes to $Alarm* sub-attributes	gateway data subscription → driver OnDataChange → DriverNodeManager ConditionSink → AlarmConditionService
Scripted alarms	Phase7Composer	server-side script evaluator → ScriptedAlarmActor transitions → HistorianAdapterActor → IAlarmHistorianSink

All three converge on the alarm-state actor — in v2 the OPC UA Part 9 state machine lives inside ScriptedAlarmActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmActor.cs), which dispatches transitions to the OPC UA condition node managers. Driver-native transitions take precedence over sub-attribute synthesis when both arrive for the same condition — the dedup logic prefers the richer driver-native record because it carries the full operator + raise-time + category metadata that the value-driven path collapses.

v3 Batch 4 — multi-notifier delivery (raw + equipment folders)

Under the v3 dual-namespace address space, a native driver alarm is authored on a raw tag (/raw) and its Part 9 AlarmConditionState materializes once at the raw tag — ConditionId = the tag's RawPath, parent notifier = the raw device/group folder (ns=Raw). Because equipment references raw tags (v3 reference-only UNS), the same condition is wired as an event notifier of every referencing equipment folder (ns=UNS) via the SDK AddNotifier pattern (OtOpcUaNodeManager.WireAlarmNotifiers): alarm.AddNotifier(equipFolder, isInverse:true) + equipFolder.AddNotifier(alarm) + EnsureFolderIsEventNotifier(equipFolder).

• A single ReportEvent fans through the SDK notifier graph to the raw device folder, every referencing equipment folder, and up to the Server object. A subscriber at any one root receives exactly one copy of the transition — the Part 9 shared-InstanceStateSnapshot dedup (MonitoredItem.QueueEvent → IsEventContainedInQueue), never one copy per root. Duplicating the ReportEvent per root is rejected by design (distinct EventIds would break Server-object dedup + Part 9 ack correlation).
• Teardown is symmetric: the node manager tracks the wired notifier pairs and calls RemoveNotifier(..., bidirectional:true) on rebuild / subtree-removal / reference-removal, so inverse-notifier entries never leak across redeploys.
• Ack/confirm/shelve route on ConditionId = RawPath (never SourceNodeId) regardless of which notifier root the operator subscribed at — an ack issued from an equipment-folder subscription resolves to the same raw condition.
• The alerts-topic AlarmTransitionEvent carries the (possibly empty) list of referencing equipment paths, and the AdminUI /alerts page shows one row per condition (primary identity RawPath + condition NodeId) with the equipment list as display metadata.

Condition event identity fields (what a client reads on the wire)

Every condition event — native and scripted — carries the mandatory BaseEventType identity fields, assigned at materialize time in OtOpcUaNodeManager.MaterialiseAlarmCondition. The SDK does not synthesize them on this path (Create builds the children from the type definition but leaves them unset; ReportEvent / InstanceStateSnapshot copy children verbatim), so they are set explicitly. Leaving them unset shipped them as null on every event — see issues #473 (the BaseEventType trio) and #475 (the ConditionType classification pair).

Field	Value	Notes
EventType	the concrete materialized type (TypeDefinitionId)	e.g. OffNormalAlarmType; falls back to AlarmConditionType for an unknown authored type. Readable as a field, not only via an OfType where-clause
SourceNode	the condition's own NodeId — equal to ConditionId	The condition is the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at
SourceName	the same identifying id string: RawPath (native) / ScriptedAlarmId (scripted)	Deliberately the unique id, not the leaf name
ConditionName	the leaf / display name (e.g. HR200)	Where the short human-readable name lives
ConditionClassId	always BaseConditionClassType	Part 9's "no condition class modelled" value. Unset shipped NodeId.Null (#475)
ConditionClassName	always "BaseConditionClass"	Matches ConditionClassId. Unset shipped empty text (#475)
Quality	the condition's source-data quality — native tracks the source's connectivity (Good / Bad); scripted takes the worst of its input tags' qualities (#478)	A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below

Why BaseConditionClassType and not ProcessConditionClassType. We hold no per-alarm classification at the materialize seam, and ConditionClassId is a wire contract clients bucket on. BaseConditionClassType is the honest, spec-conformant report of "this server does not model condition classes" — it fixes the real defect (a null that breaks conformant clients) without asserting a classification we cannot back. ProcessConditionClassType — the SDK sample's pick — was rejected deliberately: it would be actively wrong for a Galaxy alarm whose upstream category is Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm classification is a separate future feature: it needs the driver's alarm category, which today lives only on the runtime AlarmEventArgs transition, carried to the deploy-time authored composition that MaterialiseAlarmCondition sees. Until then the IAlarmSource doc comment claiming the category "maps to ConditionClassName downstream" describes an intent, not the implementation.

Why SourceName is the id, not the leaf name. The leaf is ambiguous across devices (HR200 on two PLCs collides) and is already carried by ConditionName, so the leaf-name option would add no identity while costing uniqueness. Carrying the id makes the SourceNode/SourceName/ConditionId triple mutually consistent and unique. This diverges from the loose OPC UA convention that SourceName mirrors the source node's BrowseName; the divergence is intentional.

Client guidance: key on ConditionId — the condition node's own NodeId, which equals SourceNode, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for scripted ones. It is the identity ack/confirm/shelve route on. SourceName carries the same identifier and is unique, so it is safe to key on by itself — but do not compose it with ConditionName ($"{SourceName}.{ConditionName}"), because SourceName already ends in the condition's leaf name and the result stutters (pymodbus/plc/HR200.HR200).

Wire-level guard: NativeAlarmEventIdentityFieldDeliveryTests asserts the three BaseEventType fields arrive populated on a real subscription using the standard [EventType, SourceNode, SourceName, Time, Message, Severity] select clause, and — in a second test with its own clause — that ConditionClassId / ConditionClassName do too. The two class fields are declared on ConditionType, not BaseEventType, so a client must select them against that type. NodeManagerAlarmSourceFieldsTests guards the node itself across both realms.

Do not correlate live events to HistoryRead on SourceName — the two paths disagree. The HistoryRead events projection (OtOpcUaNodeManager.ProjectEventField) returns Variant.Null for EventType / SourceNode by design: it projects from the historian's HistoricalEvent rows, which do not carry them. It does project SourceName — but the alarm-history writer stamps that field with the EquipmentPath (AlarmEventMapper: SourceName = alarm.EquipmentPath), not the RawPath a live event carries. So SourceName is the one field populated on both paths with different values, and it is not a live↔history join key. Correlate on ConditionId / the RawPath instead. (Pre-existing; the live-path fix above does not change the history path.)

Condition source-data Quality (#477)

ConditionType.Quality reports the quality of the condition's source data. It was never assigned, and because StatusCodes.Good == 0x00000000 an unassigned StatusCode is Good — so every condition reported Good unconditionally (a wrong value, not a null like #473/#475). A native alarm whose device went offline still read Good, so an operator could not tell "genuinely inactive" from "we have lost contact and do not know".

How it is driven now (native alarms). An alarm-bearing raw tag materializes a condition with no sibling value variable, so the value/quality path (WriteValue) never touches it, and a comms-lost driver emits no alarm transitions (the feed goes silent). The quality therefore comes from the driver's connectivity, out of band from alarm transitions:

• DriverInstanceActor Tells its host ConnectivityChanged(driverInstanceId, connected) on every transition into Connected (true) / Reconnecting (false).
• DriverHostActor.OnDriverConnectivityChanged fans that out to every native condition the driver owns as an OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
• OtOpcUaNodeManager.WriteAlarmQuality sets only the condition's Quality and fires a Part 9 event only on a quality-bucket change — it never touches Active/Acked/Severity/Retain (an active alarm that loses comms stays active). This is a dedicated path, not a full-snapshot re-projection, so it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
• A freshly materialized native condition starts BadWaitingForInitialData (the "no driver data yet" convention value variables use); the first Connected confirms it Good.
• The connectivity annotation is ungated by redundancy role (a Secondary keeps its condition quality warm for failover) and publishes no /alerts row — driver comms health already has its own status surface (IDriverHealthPublisher); a row per condition would be alarm-fatigue.

Scripted alarms (Layer 3, #478). A scripted condition's state is computed from one or more input tags, so its Quality is the worst quality across those inputs at evaluation time ("can I trust this condition's state?") — mirroring the native OT semantic:

• The mux now forwards each input's source quality (DependencyValueChanged.Quality), and the scripted host pushes it into the engine's read cache (previously every mux value was treated as Good).
• The ScriptedAlarmEngine computes the worst input quality each evaluation. A real transition carries it on the emitted event → ScriptedAlarmHostActor.ToSnapshot projects it (so a transition fired while an input is Uncertain does not clobber quality back to Good).
• A Bad input freezes the condition (AreInputsReady holds its state — no transition), exactly like a comms-lost native driver. So a quality-bucket change with no transition is emitted as EmissionKind.QualityChanged and routed to the same dedicated AlarmQualityUpdate → WriteAlarmQuality node path native uses (quality only, one Part 9 event on a bucket change, no /alerts row, no historian write). ScriptedAlarmSource skips QualityChanged so it never fabricates a phantom IAlarmSource event.
• An input that has not been published yet (cold start) is not a quality signal (that is the readiness guard's job) — it contributes Good, so scripted conditions don't flash Bad at every deploy. The first actually-Bad published value flips the bucket and annotates.

Coverage boundary (#478 as shipped). Scripted quality tracks input tags whose driver publishes a data change carrying a Bad/Uncertain StatusCode (e.g. an OpcUaClient input forwarding a server's per-item Bad). It does not yet cover a driver comms loss: a poll driver (Modbus/S7) whose device goes unreachable emits only ConnectivityChanged and goes silent on the value feed (see DriverInstanceActor.Reconnecting), so the scripted engine keeps the last-known Good value and the condition stays Good. Bridging driver connectivity into scripted inputs — the symmetric of the native OnDriverConnectivityChanged path above, plus resolving the null-value/cold-start asymmetry (a runtime Bad with a null value is currently indistinguishable from cold start and contributes Good) — is tracked as the Layer-4 follow-up (#481).

Guards: ScriptedAlarmEngineTests (transition carries Uncertain; Bad input with no transition emits QualityChanged(Bad); restore emits QualityChanged(Good); unchanged bucket emits nothing), ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event, DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged, and ScriptedAlarmHostActorTests (Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts, Transition_snapshot_carries_worst_input_quality).

Wire-level guard: NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire subscribes with a [Quality, Message] clause (Quality is declared on ConditionType) and asserts a healthy source reports Good, a comms-lost source reports non-Good, and recovery returns to Good — on a real client subscription. NodeManagerAlarmSourceFieldsTests guards the node itself + the no-clobber / unknown-node-no-op invariants.

Galaxy driver path (driver-native)

Restored in PR B.2 of the epic. GalaxyDriver implements IAlarmSource with these surfaces:

• SubscribeAlarmsAsync(sourceNodeIds) → returns a sentinel handle. The driver doesn't multiplex per source-node-id today; every active handle observes the gateway's alarm-event stream. The server-side AlarmConditionService filters by source-node before raising the OPC UA condition.
• UnsubscribeAlarmsAsync(handle) → symmetric handle removal.
• AcknowledgeAsync(requests) → routes one gateway RPC per acknowledgement through IGalaxyAlarmAcknowledger. Production uses GatewayGalaxyAlarmAcknowledger calling MxGatewayClient.AcknowledgeAlarmAsync (PR E.2 SDK method).
• OnAlarmEvent → bridges EventPump.OnAlarmTransition (PR B.1) onto AlarmEventArgs. Suppressed when no alarm subscription is active so untracked transitions don't leak through.

The proto contract carries the rich payload — alarm full reference, source-object reference, alarm-type-name, transition kind (Raise / Acknowledge / Clear / Retrigger), severity (raw MxAccess scale), original raise timestamp, transition timestamp, operator user, operator comment, alarm category, description. MxAccessSeverityMapper (PR B.1) translates the raw severity onto the four-bucket AlarmSeverity ladder — boundaries match v1's GalaxyAlarmTracker so customers see no surprise re-classification.

The richer fields surface on Core.Abstractions.AlarmEventArgs via the optional properties added in PR E.7 (OperatorComment, OriginalRaiseTimestampUtc, AlarmCategory). Consumers that don't need them are unaffected; consumers that do (Client.UI, Client.CLI verbose mode) read the new fields when present.

Galaxy sub-attribute fallback

For Galaxy templates without $Alarm* extensions, the value-driven path stays in place: DriverNodeManager registers an AlarmConditionState per Galaxy variable that bears alarm-bearing sub-attributes (InAlarm, Acked, Priority, Description), subscribes to those sub-attributes, and synthesizes Part 9 transitions when the values change. This path operated as the only Galaxy alarm path between PR 7.2 and the alarms-over-gateway epic; it remains the fallback today.

When both paths report the same condition, AlarmConditionService.AlarmConditionState keeps the driver-native record and discards the duplicate sub-attribute synthesis. Driver-native transitions are richer (carry operator comment + original raise time) and arrive lower-latency (no publishing-interval delay on the sub-attribute reads), so they win the dedup.

Acknowledge routing — Galaxy / driver alarms

Native alarm acknowledge → AVEVA

When an OPC UA client Acknowledges a native (driver-fed, e.g. Galaxy) AlarmConditionState node, the node manager's OnAcknowledge handler branches on native-ness and routes through a dedicated path — separate from the scripted AlarmCommandRouter:

1. OtOpcUaNodeManager.HandleNativeAlarmAck — gates on the caller's AlarmAck role (fails closed: no role → BadUserAccessDenied), then dispatches a NativeAlarmAck(ConditionNodeId, Comment, OperatorUser) to the NativeAlarmAckRouter seam (fire-and-forget, non-blocking under the node-manager Lock). OperatorUser carries the authenticated session principal's display name.
2. DriverHostActor.HandleRouteNativeAlarmAck — receives a RouteNativeAlarmAck message (the host maps NativeAlarmAck at the wiring boundary to keep Runtime Akka-free of the OPC UA layer). Applied Primary-gate first: a Secondary or Detached node drops the message silently. On Primary, resolves the condition NodeId from the _driverRefByAlarmNodeId inverse map (NodeId → (DriverInstanceId, FullName)) and Tells the owning DriverInstanceActor a RouteAlarmAck(FullName, Comment, OperatorUser).
3. Galaxy driver — DriverInstanceActor calls the driver's IAlarmSource.AcknowledgeAsync with an AlarmAcknowledgeRequest carrying the authored FullName as the ConditionId and the authenticated OperatorUser. The driver forwards this to the Galaxy gateway → AVEVA via GatewayGalaxyAlarmAcknowledger. Fire-and-forget — a failed upstream ack is not surfaced back to the OPC UA client (mirrors the Galaxy write-outcome limitation; the local AlarmConditionState SDK update already committed at step 1).

Only the Acknowledge is routed to the driver. Confirm / AddComment / Shelve operations on a native condition stay on the scripted AlarmCommandRouter path (Phase 3 scope is Acknowledge → AVEVA only).

Best-effort during reconnect. If a native ack arrives while the owning driver is Connecting/Reconnecting (or on a non-Primary node), it is dropped with a debug log and not retried upstream — the OPC UA Part 9 AlarmConditionState was already committed locally at step 1, so the rare outcome is "acked locally but not propagated to AVEVA." Re-issue the ack once the driver is reconnected if upstream propagation is required.

Legacy sub-attribute path

DriverNodeManager picks the acknowledger when registering each condition (PR B.3 logic):

• Driver implements IAlarmSource → DriverAlarmSourceAcknowledger routes the operator comment through IAlarmSource.AcknowledgeAsync via the existing AlarmSurfaceInvoker (Phase 6.1 resilience pipeline; no-retry per decision #143). End-to-end operator-comment fidelity is preserved.
• Driver doesn't implement IAlarmSource → DriverWritableAcknowledger writes the comment into the AckMsgWriteRef sub-attribute via IWritable.WriteAsync. Same resilience pipeline; collapses the comment into a single string write at the wire level.

The OPC UA Part 9 AlarmConditionState.OnAcknowledge delegate already validates the session's AlarmAck role before dispatching, so the gateway-side ack RPC only sees authenticated, authorised calls.

Inbound operator ack/shelve — scripted alarms

Scripted alarms use a separate inbound path that converges on the alarm-commands DPS topic. Two surfaces route onto this topic:

OPC UA Part 9 method path (external OPC UA clients)

OtOpcUaNodeManager wires the Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each scripted-alarm AlarmConditionState node. Every call is gated on the AlarmAck LDAP role — fail-closed: sessions with no role or without AlarmAck group membership receive BadUserAccessDenied immediately. The LDAP-resolved role set is carried past OpcUaApplicationHost by RoleCarryingUserIdentity (a UserIdentity subclass), making it readable inside the method handler at dispatch time.

On allow, the handler publishes a Commons.OpcUa.AlarmCommand onto the alarm-commands DPS topic. The node manager is Akka-free; the dispatch action is a settable Action<AlarmCommand> injected at boot by the hosted service.

OnTimedUnshelve (the SDK's automatic unshelve timer) bypasses the operator gate — it is system-initiated.

WriteAlarmCondition fires the Part 9 condition event only when the incoming state differs from the node's current live state (delta-gate), preventing the double-emit that would otherwise occur when the SDK auto-applies the acked state and the engine re-projection fires a duplicate event immediately after.

AdminUI path

The /alerts page shows per-row Acknowledge / Shelve / Unshelve buttons gated by the DriverOperator AdminUI policy. These route through the AdminOperationsActor cluster singleton (AcknowledgeAlarmCommand / ShelveAlarmCommand), which publishes onto the same alarm-commands topic. The singleton handles cross-node routing — the command always reaches the driver-role node owning the engine regardless of which AdminUI instance the operator is on.

ScriptedAlarmHostActor dispatch

ScriptedAlarmHostActor subscribes to the alarm-commands topic, ownership-filters each command (each node only acts on its own alarms), and dispatches to the matching ScriptedAlarmEngine operation (AcknowledgeAsync / ConfirmAsync / OneShotShelveAsync / TimedShelveAsync / UnshelveAsync / EnableAsync / DisableAsync / AddCommentAsync). The engine's existing OnEvent callback handles the OPC UA node update — no explicit re-projection is required.

The AdminUI /alerts Shelve flow was live-verified on docker-dev 2026-06-11: singleton → topic → host actor → engine → "Shelved" status reflected on /alerts with the operator identity threaded through.

Redundancy deduplication

Under warm/hot redundancy, both cluster nodes run ScriptedAlarmHostActor and the scripted-alarm engine. To prevent duplicate /alerts rows and duplicate historian writes, alarm transition publication to the alerts topic and HistorianAdapterActor historization are Primary-gated: only the node whose RedundancyRole is Primary publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See Redundancy.md §Primary-gated alarm emission and historization.

Historian write-back (non-Galaxy alarms)

Scripted alarms (and any future non-Galaxy IAlarmSource like AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:

• IAlarmHistorianSink is the DI-registered intake contract. The default binding is NullAlarmHistorianSink (registered in ServiceCollectionExtensions.AddOtOpcUaRuntime). Production deployments override it with SqliteStoreAndForwardSink wrapping GatewayAlarmHistorianWriter (the HistorianGateway SendEvent path) — see ServiceHosting.md for the HistorianGateway setup.

• SqliteStoreAndForwardSink queues each transition to a local SQLite database and drains in the background via an IAlarmHistorianWriter. The durability guarantee is bounded: the queue capacity defaults to 1,000,000 rows; under a sustained historian outage, older non-dead-lettered rows are evicted (oldest first) to make room for new events. The HistorianSinkStatus.EvictedCount counter surfaces lifetime eviction events so operators can detect silent data loss without log scraping. The drain cadence, queue capacity, and dead-letter retention are tunable via the AlarmHistorian config section (DrainIntervalSeconds, Capacity, DeadLetterRetentionDays); AlarmHistorianOptions.Validate() logs a startup warning for an empty SharedSecret, a relative DatabasePath, or a non-positive knob.

• HistorianAdapterActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs) subscribes to the cluster alerts DPS topic, translates each AlarmTransitionEvent → AlarmHistorianEvent, and calls EnqueueAsync fire-and-forget. The durable write is gated two ways: (1) Primary-gated — only the Primary node historizes, giving exactly-once writes across a redundant pair; (2) per-alarm — a transition whose HistorizeToAveva is false is skipped (the flag rides on AlarmTransitionEvent as a nullable bool; missing/null/true historize, only an explicit false suppresses, so a cross-version rolling restart defaults to historizing rather than dropping an audit row). Neither gate touches the live alerts publish — the /alerts UI always sees every transition. See AlarmHistorian.md §Configuration for the AlarmHistorian appsettings section that enables the real sink.

  Native alarms (equipment tags carrying an "alarm" object in their TagConfig) support the same HistorizeToAveva opt-out. The field is alarm.historizeToAveva (bool?) and is authored via the "Historize to AVEVA" checkbox in the Tag modal's alarm section. The gate logic is identical (is not false): absent or true historizes; explicit false suppresses the AVEVA write while leaving the live /alerts feed unaffected. See ScriptedAlarms.md §TagConfig alarm fields for the full field reference.

Galaxy-native alarms with $Alarm* extensions reach AVEVA Historian directly via System Platform's HistorizeToAveva toggle on the alarm primitive — no involvement from OtOpcUa. This sidecar path is exclusively for non-Galaxy alarm producers.

Cross-references

• Plan: docs/plans/alarms-over-gateway.md
• v1 archive: docs/v1/AlarmTracking.md
• Galaxy driver: docs/drivers/Galaxy.md
• Phase 7 scripting + alarming: docs/v2/implementation/phase-7-scripting-and-alarming.md
• Security + ACL: docs/security.md