lmxopcua/docs/AlarmTracking.md

Alarm tracking — v2 final architecture

This document describes how OtOpcUa surfaces alarms to OPC UA Part 9 clients after the alarms-over-gateway epic (docs/plans/alarms-over-gateway.md) landed. The v1 architecture (Galaxy.Host's COM-side GalaxyAlarmTracker) is preserved at docs/v1/AlarmTracking.md for historical reference.

Three alarm sources, one OPC UA Part 9 surface

Source	Driver capability	Path
Galaxy MxAccess (driver-native)	GalaxyDriver : IAlarmSource	gateway → worker → MxAccess alarm sink → MX_EVENT_FAMILY_ON_ALARM_TRANSITION → EventPump → driver OnAlarmEvent → AlarmConditionService
MTConnect CONDITION (driver-native, #626)	MTConnectDriver : IAlarmSource	Agent /sample pump → MTConnectConditionTracker → MTConnectConditionAlarmMapper → driver OnAlarmEvent (ConditionId = RawPath) → DriverHostActor.ForwardNativeAlarm
Galaxy sub-attribute fallback	IWritable writes to $Alarm* sub-attributes	gateway data subscription → driver OnDataChange → DriverNodeManager ConditionSink → AlarmConditionService
Scripted alarms	Phase7Composer	server-side script evaluator → ScriptedAlarmActor transitions → HistorianAdapterActor → IAlarmHistorianSink

All three converge on the alarm-state actor — in v2 the OPC UA Part 9 state machine lives inside ScriptedAlarmActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmActor.cs), which dispatches transitions to the OPC UA condition node managers. Driver-native transitions take precedence over sub-attribute synthesis when both arrive for the same condition — the dedup logic prefers the richer driver-native record because it carries the full operator + raise-time + category metadata that the value-driven path collapses.

v3 Batch 4 — multi-notifier delivery (raw + equipment folders)

Under the v3 dual-namespace address space, a native driver alarm is authored on a raw tag (/raw) and its Part 9 AlarmConditionState materializes once at the raw tag — ConditionId = the tag's RawPath, parent notifier = the raw device/group folder (ns=Raw). Because equipment references raw tags (v3 reference-only UNS), the same condition is wired as an event notifier of every referencing equipment folder (ns=UNS) via the SDK AddNotifier pattern (OtOpcUaNodeManager.WireAlarmNotifiers): alarm.AddNotifier(equipFolder, isInverse:true) + equipFolder.AddNotifier(alarm) + EnsureFolderIsEventNotifier(equipFolder).

• A single ReportEvent fans through the SDK notifier graph to the raw device folder, every referencing equipment folder, and to the Server object. A subscriber at any one root receives exactly one copy of the transition — the Part 9 shared-InstanceStateSnapshot dedup (MonitoredItem.QueueEvent → IsEventContainedInQueue), never one copy per root. The Server-object (i=2253) copy is delivered by the condition itself, not by the folders' forward to the core Server object, so it can be filtered per subscriber by the data-plane ACL (Gitea #586 — see docs/security.md). Duplicating the ReportEvent per root is rejected by design (distinct EventIds would break Server-object dedup + Part 9 ack correlation).
• Teardown is symmetric: the node manager tracks the wired notifier pairs and calls RemoveNotifier(..., bidirectional:true) on rebuild / subtree-removal / reference-removal, so inverse-notifier entries never leak across redeploys.
• Ack/confirm/shelve route on ConditionId = RawPath (never SourceNodeId) regardless of which notifier root the operator subscribed at — an ack issued from an equipment-folder subscription resolves to the same raw condition.
• The alerts-topic AlarmTransitionEvent carries the (possibly empty) list of referencing equipment paths, and the AdminUI /alerts page shows one row per condition (primary identity RawPath + condition NodeId) with the equipment list as display metadata.
• Routing is an exact ConditionId == RawPath match (DriverHostActor.ForwardNativeAlarm). A driver that decorates the id has every transition dropped as "no alarm condition" — AbCip did ("{ref}#active", fixed under #641), and so did FOCAS ("{host}#{Type}:{Number}", fixed under #647 — one condition per ALARM: tag, see drivers/FOCAS.md). A Raise on a condition already projected Active with the same message is a re-statement (#647 review): the ack is kept and no second "Activated" row is published — a respawned driver re-reading a still-active alarm sends exactly that. A Raise with a DIFFERENT message is a new statement and un-acks: a respawned driver cannot tell "the same alarm" from "a different one that appeared while it was down", so the host compares what the condition shows. Only Retrigger (or inactive → active, or a changed-message Raise) un-acks. ⚠️ Residual: the comparison is only as good as the source's message determinism. FOCAS and MTConnect messages are deterministic for an unchanged statement; a source whose message carries volatile text (a timestamp, a counter) would un-ack on every respawn — conservative, never hiding an alarm — and a source whose message omits what changed (a new alarm under identical text) would keep a stale ack across a respawn. A native source must also set Kind: an Unspecified kind projects no state change.
• Snapshot / first-sight re-statements (#656). A source that RE-STATES upstream's current state — Galaxy's reconnect snapshot entries, OpcUaClient's first event per condition after a start or deploy respawn — flags AlarmEventArgs.IsRestatement (optional, default false; FOCAS, MTConnect and live Galaxy transitions never set it; AbCip ALMA sets it for the drop-out update below). The flag governs history only, never node state: the projector still projects, but a flagged Retrigger on a condition already Active+unacked with the same message AND severity, a flagged Clear on an already-inactive condition, or a flagged Acknowledge on an already-acknowledged one (#657), writes no /alerts row; so does a flagged Unspecified update on an already-active condition, which moves the node's message/severity (one condition event, active/acked kept) — AbCip ALMA's governing limit dropping out (HH→H). Before this every gateway reconnect wrote a duplicate "Activated" (and "Cleared") row per alarm while clients saw no event. A live Retrigger is a genuine re-occurrence and keeps its row; a changed severity (an MTConnect Warning→Fault that keeps its text) does too. Residual: a Galaxy alarm that genuinely re-occurred during an outage and is still Active+unacked with identical text/severity at reconnect loses that row (indistinguishable upstream; the node shows no change either). A process restart still writes a fresh "Activated" for every already-active alarm — the projector has no memory, correctly.
• AlarmTransitionKind.ActiveAcknowledged (#656) — "active AND acknowledged" in ONE projection (one node write, at most one client event). NativeAlarmProjector.Project(..., out historyKinds) expands it into the rows that actually changed: "Activated" (stamped OriginalRaiseTimestampUtc, no operator comment) only if the condition was not already showing that alarm active, then "Acknowledged" (carrying the comment) only if it was not acked. Galaxy maps a snapshot ActiveAcked entry to it — it used to map to Acknowledge, which kept the host's prior Active = false on a cold host or an inactive prior, so an alarm active in AVEVA projected inactive and was invisible to clients. OpcUaClient emits it for inactive → (active, acked) instead of Raise + Acknowledge, which showed clients an Active+unacked intermediate event.
• AlarmTransitionKind.InactiveUnacknowledged (#657) — its mirror: "inactive AND unacknowledged" (went active and cleared unacked) in ONE projection, never Raise/Retrigger + Clear (which showed clients the condition Active again for an instant). Rows only for what changed, flag or not: "Activated" (raise time when known) + "Cleared" when the condition was acknowledged (an un-ack needs a new occurrence), "Cleared" alone when it was Active+unacked, none when it already showed inactive+unacked. OpcUaClient emits it for every move into inactive+unacked except from Active+unacked (a plain Clear). Its first sight of an inactive condition (after a start or deploy respawn, when the host may still hold ANY state) is InactiveUnacknowledged, or Clear + Acknowledge for inactive+acked — both flagged — so a host already showing that state writes nothing, and a host that retained the condition Active gets a real "Cleared". Pre-#657 a respawn wrote a duplicate "Activated" + "Cleared" pair per inactive+unacked condition, and an inactive+acked first sight went out as Unspecified (an "Activated" row for an inactive alarm; a retained-Active condition was never cleared). Residual (pre-existing, not a regression): a cold host treats an unseen condition as acked, so after a node restart each upstream inactive+unacked condition still writes "Activated" + "Cleared" once. E2E: Runtime.Tests/NativeAlarmRestatementEndToEndTests.

Native condition state survives a full rebuild (Gitea #641)

A full RebuildAddressSpace recreates every condition inactive + acknowledged. For scripted alarms the engine re-asserts every non-normal condition after the reload (#487). Native conditions had nothing equivalent: the host re-subscribes a driver's alarms only when its alarm-ref set changes, and a transition-only source (Galaxy, OpcUaClient) has nothing to replay anyway — so a rebuild that left the alarm set unchanged (adding one UnsTagReference to an alarm-bearing raw tag is enough to force one, as are ChangedEquipment, a scripted-alarm edit, a raw container rename, an alarm severity/type edit, or any surgical fallback to SafeRebuild()) left every active native alarm reading normal until its next device transition.

The node manager now carries native state across the rebuild itself:

1. RebuildAddressSpace captures each native condition's live state off the node — Active, Acked, Confirmed, Severity, Message, Comment + ClientUserId, Quality, Time, ActiveState.TransitionTime — before clearing. Reading the node, not a host-side cache, is what keeps a client's acknowledgement: the SDK applies it to the node and neither the driver nor NativeAlarmProjector ever sees it.
2. MaterialiseAlarmCondition(isNative: true) for the same id in the same apply restores it. Quality is always restored (it is otherwise re-annotated only on a connectivity edge, so even a normal condition would sit at BadWaitingForInitialData); the rest only when the condition was off-normal (active or unacknowledged). A normal condition keeps what the materialise built — message and authored severity included — exactly the #487 rule. Shelving/Enabled are not carried (native conditions refuse them, #587).
3. IOpcUaAddressSpaceSink.CompleteAddressSpaceApply, which OpcUaPublishActor calls at the end of every apply (in a finally), reports one condition event per restored condition — after the notifier wiring, so the raw folder, every referencing equipment folder and i=2253 items all get it, stamped with the original Time, not the deploy time (or the node's newer Time, if a client method call reported in the restore→complete window — that window can yield a second, state-correct event) — and discards every capture no materialise claimed, so a later deploy re-adding the same RawPath gets a fresh node (no resurrection). A kind swap (native ↔ scripted on one id) is never restored. Node-only: no /alerts row, no historian record.

Two companions keep the host and driver in step with the carried node:

• DriverHostActor prunes NativeAlarmProjector to the still-deployed conditions on each apply (RetainOnly) instead of clearing it. Clearing made the next driver Acknowledge of an active alarm project Active=false, and the next Clear of an unacknowledged one project Acked=true — after any deploy, rebuild or not.
• MTConnect's "last told" record is pruned to the conditions that left the subscribed set rather than cleared on any set change; re-raising a surviving (carried) condition would un-acknowledge it. AbCip's ALMD last-seen InAlarm record is likewise driver-owned and pruned to the set, so a re-subscribe (every reconnect / set change) does not re-raise a still-faulted alarm.
• A routed client acknowledgement marks the projector's prior state acknowledged (NativeAlarmProjector.MarkAcknowledged), so a later driver Clear does not project it unacknowledged.

Accepted residual: a condition that clears at the device while the rebuild runs is restored active until the driver's Clear arrives — which it does, because the driver's own record still holds it active. The capture is keyed by RawPath only, so a tag deleted and re-created under the same RawPath in one deploy inherits the old condition's state until its next transition.

Condition event identity fields (what a client reads on the wire)

Every condition event — native and scripted — carries the mandatory BaseEventType identity fields, assigned at materialize time in OtOpcUaNodeManager.MaterialiseAlarmCondition. The SDK does not synthesize them on this path (Create builds the children from the type definition but leaves them unset; ReportEvent / InstanceStateSnapshot copy children verbatim), so they are set explicitly. Leaving them unset shipped them as null on every event — see issues #473 (the BaseEventType trio) and #475 (the ConditionType classification pair).

Field	Value	Notes
EventType	the concrete materialized type (TypeDefinitionId)	e.g. OffNormalAlarmType; falls back to AlarmConditionType for an unknown authored type. Readable as a field, not only via an OfType where-clause
SourceNode	the condition's own NodeId — equal to ConditionId	The condition is the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at
SourceName	the same identifying id string: RawPath (native) / ScriptedAlarmId (scripted)	Deliberately the unique id, not the leaf name
ConditionName	the leaf / display name (e.g. HR200)	Where the short human-readable name lives
ConditionClassId	always BaseConditionClassType	Part 9's "no condition class modelled" value. Unset shipped NodeId.Null (#475)
ConditionClassName	always "BaseConditionClass"	Matches ConditionClassId. Unset shipped empty text (#475)
Quality	the condition's source-data quality — native tracks the source's connectivity (Good / Bad); scripted takes the worst of its input tags' qualities (#478)	A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below

Why BaseConditionClassType and not ProcessConditionClassType. We hold no per-alarm classification at the materialize seam, and ConditionClassId is a wire contract clients bucket on. BaseConditionClassType is the honest, spec-conformant report of "this server does not model condition classes" — it fixes the real defect (a null that breaks conformant clients) without asserting a classification we cannot back. ProcessConditionClassType — the SDK sample's pick — was rejected deliberately: it would be actively wrong for a Galaxy alarm whose upstream category is Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm classification is a separate future feature: it needs the driver's alarm category, which today lives only on the runtime AlarmEventArgs transition, carried to the deploy-time authored composition that MaterialiseAlarmCondition sees. Until then the IAlarmSource doc comment claiming the category "maps to ConditionClassName downstream" describes an intent, not the implementation.

Why SourceName is the id, not the leaf name. The leaf is ambiguous across devices (HR200 on two PLCs collides) and is already carried by ConditionName, so the leaf-name option would add no identity while costing uniqueness. Carrying the id makes the SourceNode/SourceName/ConditionId triple mutually consistent and unique. This diverges from the loose OPC UA convention that SourceName mirrors the source node's BrowseName; the divergence is intentional.

Client guidance: key on ConditionId — the condition node's own NodeId, which equals SourceNode, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for scripted ones. It is the identity ack/confirm/shelve route on. SourceName carries the same identifier and is unique, so it is safe to key on by itself — but do not compose it with ConditionName ($"{SourceName}.{ConditionName}"), because SourceName already ends in the condition's leaf name and the result stutters (pymodbus/plc/HR200.HR200).

Wire-level guard: NativeAlarmEventIdentityFieldDeliveryTests asserts the three BaseEventType fields arrive populated on a real subscription using the standard [EventType, SourceNode, SourceName, Time, Message, Severity] select clause, and — in a second test with its own clause — that ConditionClassId / ConditionClassName do too. The two class fields are declared on ConditionType, not BaseEventType, so a client must select them against that type. NodeManagerAlarmSourceFieldsTests guards the node itself across both realms.

Do not correlate live events to HistoryRead on SourceName — the two paths disagree. The HistoryRead events projection (OtOpcUaNodeManager.ProjectEventField) returns Variant.Null for EventType / SourceNode by design: it projects from the historian's HistoricalEvent rows, which do not carry them. It does project SourceName — but the alarm-history writer stamps that field with the EquipmentPath (AlarmEventMapper: SourceName = alarm.EquipmentPath), not the RawPath a live event carries. So SourceName is the one field populated on both paths with different values, and it is not a live↔history join key. Correlate on ConditionId / the RawPath instead. (Pre-existing; the live-path fix above does not change the history path.)

Condition child nodes are instance-scoped (#629)

Every child of a materialised condition — ActiveState, ActiveState/Id, AckedState/Id, ShelvingState/UnshelveTime, Message, Severity, Retain, the method nodes, … — carries the NodeId {ConditionId}/{browse path} in the condition's own namespace: e.g. ns=<raw>;s=Plant/Modbus/dev1/temp_hi/ShelvingState/UnshelveTime, or ns=<uns>;s=<ScriptedAlarmId>/ActiveState/Id. So TranslateBrowsePathsToNodeIds from the condition resolves, and a direct Read or data-change subscription on a child returns this condition's value.

Before #629 the children kept ns=0 ids: Create(…, assignNodeIds: true) asks the node manager's INodeIdFactory.New, and the SDK default returns the node's current id — the condition type's instance-declaration id (i=9189 for UnshelveTime) or an SDK-generated ns=0 number. Every translate of a child path returned BadNoMatch; a Read of a browsed child id was either BadNodeIdUnknown or served from the type (UnshelveTime read 0 on a condition timed-shelved for a minute). OtOpcUaNodeManager.New now derives the id. Event delivery was never affected (events copy the instance's fields; clients select them by type + browse path), and neither were the Part 9 methods — calling them by the type's method id (MethodIds.AcknowledgeableConditionType_Acknowledge, …) still works, because the SDK matches a method on its declaration id. ⚠️ The ShelvingState methods needed help here: their SDK declaration id is the ShelvedStateMachineType's (i=2948…), and they were only ever found by the AlarmConditionType_ShelvingState_* id (i=9212…) because that WAS their NodeId. New adopts that original id as the method's MethodDeclarationId (a fixed set of those three ids — no address-space lookup, which would take other node managers' locks under ours), so OneShotShelve/TimedShelve/Unshelve by the AlarmConditionType_ShelvingState_* id keep working, as do calls by the condition's own instance method ids ({ConditionId}/ShelvingState/TimedShelve, on the condition or its ShelvingState). Calling them by the ShelvedStateMachineType_* id returns BadMethodInvalid — a pinned trade-off.

• ACL: the slash-joined id makes a child's ACL scope its condition's scope plus the browse path, so with Security:Acl:Enabled a child is governed exactly like its condition (a grant covering the condition covers its children; one that does not, denies them). Before #629 a child was in ns=0 and bypassed the data-plane gate entirely.
• Removal: removing or rebuilding a condition forgets its whole subtree, so a removed condition's children answer BadNodeIdUnknown, not their last value.
• Guarded over the wire by ConditionChildNodeIdTests.

Condition source-data Quality (#477)

ConditionType.Quality reports the quality of the condition's source data. It was never assigned, and because StatusCodes.Good == 0x00000000 an unassigned StatusCode is Good — so every condition reported Good unconditionally (a wrong value, not a null like #473/#475). A native alarm whose device went offline still read Good, so an operator could not tell "genuinely inactive" from "we have lost contact and do not know".

How it is driven now (native alarms). An alarm-bearing raw tag materializes a condition with no sibling value variable, so the value/quality path (WriteValue) never touches it, and a comms-lost driver emits no alarm transitions (the feed goes silent). The quality therefore comes from the driver's connectivity, out of band from alarm transitions:

• DriverInstanceActor Tells its host ConnectivityChanged(driverInstanceId, connected) on every transition into Connected (true) / Reconnecting (false).
• DriverHostActor.OnDriverConnectivityChanged fans that out to every native condition the driver owns as an OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
• OtOpcUaNodeManager.WriteAlarmQuality sets only the condition's Quality and fires a Part 9 event only on a quality-bucket change — it never touches Active/Acked/Severity/Retain (an active alarm that loses comms stays active). This is a dedicated path, not a full-snapshot re-projection, so it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
• A freshly materialized native condition starts BadWaitingForInitialData (the "no driver data yet" convention value variables use); the first Connected confirms it Good.
• The connectivity annotation is ungated by redundancy role (a Secondary keeps its condition quality warm for failover) and publishes no /alerts row — driver comms health already has its own status surface (IDriverHealthPublisher); a row per condition would be alarm-fatigue.

Scripted alarms (Layer 3, #478). A scripted condition's state is computed from one or more input tags, so its Quality is the worst quality across those inputs at evaluation time ("can I trust this condition's state?") — mirroring the native OT semantic:

• The mux now forwards each input's source quality (DependencyValueChanged.Quality), and the scripted host pushes it into the engine's read cache (previously every mux value was treated as Good).
• The ScriptedAlarmEngine computes the worst input quality each evaluation. A real transition carries it on the emitted event → ScriptedAlarmHostActor.ToSnapshot projects it (so a transition fired while an input is Uncertain does not clobber quality back to Good).
• A Bad input freezes the condition (AreInputsReady holds its state — no transition), exactly like a comms-lost native driver. So a quality-bucket change with no transition is emitted as EmissionKind.QualityChanged and routed to the same dedicated AlarmQualityUpdate → WriteAlarmQuality node path native uses (quality only, one Part 9 event on a bucket change, no /alerts row, no historian write). ScriptedAlarmSource skips QualityChanged so it never fabricates a phantom IAlarmSource event.
• An input that has not been published yet (cold start) is not a quality signal (that is the readiness guard's job) — it contributes Good, so scripted conditions don't flash Bad at every deploy. The first actually-Bad published value flips the bucket and annotates.

Comms loss (Layer 4, #481). #478 covered input tags whose driver publishes a data change carrying a Bad/Uncertain StatusCode (e.g. an OpcUaClient input forwarding a server's per-item Bad). It did not cover a driver comms loss: a poll driver (Modbus/S7) whose device goes unreachable emits only ConnectivityChanged and goes silent on the value feed (see DriverInstanceActor.Reconnecting), so the engine kept the last-known Good value, WorstInputStatus stayed Good — and worse, AreInputsReady stayed true, so the predicate went on evaluating a stale value after the source had died.

DriverHostActor.FanCommsLossToDependencyMux closes it, symmetrically with the native OnDriverConnectivityChanged path above: on ConnectivityChanged(false) it pushes a synthetic BadNoCommunication with a null value into the dependency mux for every value ref the driver owns, resolved from the existing _nodeIdByDriverRef map. Alarm-bearing tags are deliberately excluded — a native condition is not a mux value, and its comms quality goes through the Part 9 annotation instead.

Two things worth knowing before changing this:

• Nothing is pushed on reconnect, deliberately — publishing Good would assert a value quality the host has not observed. Recovery rides the driver's own republish: the Connected → Reconnecting edge calls DetachSubscription, so the reconnect's ResubscribeDesired builds a fresh subscription whose PollGroupEngine initial push runs with forceRaise: true and re-publishes every tag with its real quality. That detach is load-bearing — without it the poll engine would compare against its retained LastValues, see an unchanged value, raise nothing, and strand those refs Bad forever.
• Virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and is the correct reading of "this input's source is unreachable", but it is a behaviour change beyond scripted alarms.

The signal this hangs off now fires for a mid-run device loss (#527, closed). The #481 live gate on docker-dev found that DriverInstanceActor.DisconnectObserved had no production sender: the only paths into Reconnecting were a failed initial connect and an operator Force Reconnect. A Modbus driver whose fixture was stopped stayed Connected for >10 minutes publishing per-tag Bad, and did not recover when the fixture came back until the node was restarted. So the fan-out above — and the native #477 Layer 2 annotation it mirrors — were correctly wired to a signal that a dying device never raised. #527 added the comms-loss watchdog on the Connected health tick, which self-sends DisconnectObserved once a driver has gone 90 s without a successful read while reporting itself non-Healthy. See DriverLifecycle.md §"The comms-loss watchdog" for why the trip condition is that conjunction and not simply "non-Healthy for N polls" — the naive rule would reconnect a healthy device's driver every 90 s forever whenever one authored address is unreadable.

Note the partial cover that existed before, because it explains why some of this appeared to work: Modbus publishes Bad per tag rather than going silent, so #478 Layer 3 did carry Bad into scripted input quality for that driver. That was a per-driver accident, not a guarantee, and it did nothing for native conditions or for recovery.

Cold start vs. a driver reporting Bad (#478 Finding 3, closed by #481). WorstInputStatus used to skip every input whose value was null as "no data yet". A comms-lost driver produces exactly that shape — Bad with no value — so the skip was hiding the case that matters most. The upstream source now stamps a never-pushed path BadWaitingForInitialData and nothing else does, so the engine skips that specific code rather than every null. Deleting the null-skip outright would have been the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, which is what the skip existed to prevent. Both failure modes are pinned — see the engine-test pair below, each of which goes red under the opposite mistake.

Guards: ScriptedAlarmEngineTests (transition carries Uncertain; Bad input with no transition emits QualityChanged(Bad); restore emits QualityChanged(Good); unchanged bucket emits nothing; plus the #481 pair — a pushed Bad carrying no value counts, an input never pushed does not), DriverHostActorLiveValueTests (comms loss pushes Bad for every ref the driver owns, reconnect pushes nothing, another driver's loss pushes nothing), DependencyMuxTagUpstreamSourceTests.The_engine_agrees_with_this_source_on_the_never_pushed_status_code, ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event, DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged, and ScriptedAlarmHostActorTests (Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts, Transition_snapshot_carries_worst_input_quality).

Wire-level guard: NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire subscribes with a [Quality, Message] clause (Quality is declared on ConditionType) and asserts a healthy source reports Good, a comms-lost source reports non-Good, and recovery returns to Good — on a real client subscription. NodeManagerAlarmSourceFieldsTests guards the node itself + the no-clobber / unknown-node-no-op invariants.

Galaxy driver path (driver-native)

Restored in PR B.2 of the epic. GalaxyDriver implements IAlarmSource with these surfaces:

• SubscribeAlarmsAsync(sourceNodeIds) → returns a sentinel handle. The driver doesn't multiplex per source-node-id today; every active handle observes the gateway's alarm-event stream. The server-side AlarmConditionService filters by source-node before raising the OPC UA condition.
• UnsubscribeAlarmsAsync(handle) → symmetric handle removal.
• AcknowledgeAsync(requests) → routes one gateway RPC per acknowledgement through IGalaxyAlarmAcknowledger. Production uses GatewayGalaxyAlarmAcknowledger calling MxGatewayClient.AcknowledgeAlarmAsync (PR E.2 SDK method).
• OnAlarmEvent → bridges EventPump.OnAlarmTransition (PR B.1) onto AlarmEventArgs. Suppressed when no alarm subscription is active so untracked transitions don't leak through.

The proto contract carries the rich payload — alarm full reference, source-object reference, alarm-type-name, transition kind (Raise / Acknowledge / Clear / Retrigger), severity (raw MxAccess scale), original raise timestamp, transition timestamp, operator user, operator comment, alarm category, description. MxAccessSeverityMapper (PR B.1) translates the raw severity onto the four-bucket AlarmSeverity ladder — boundaries match v1's GalaxyAlarmTracker so customers see no surprise re-classification.

The richer fields surface on Core.Abstractions.AlarmEventArgs via the optional properties added in PR E.7 (OperatorComment, OriginalRaiseTimestampUtc, AlarmCategory). Consumers that don't need them are unaffected; consumers that do (Client.UI, Client.CLI verbose mode) read the new fields when present.

MTConnect CONDITION path (driver-native, #626)

An MTConnect raw tag bound to a CONDITION DataItem becomes a native condition when its TagConfig carries the standard alarm object (opt-in; without it the tag stays the String state word). Normal → inactive, Warning → active 500, Fault → active 800, Unavailable → state unchanged (an active condition stays active and acked — it is comms loss one hop out, the #477/#481 posture) with BadNoCommunication quality. Two optional AlarmEventArgs fields added for it are generic: OpcUaSeverity (an exact 1..1000 severity that supersedes the four-bucket mapping in NativeAlarmProjector) and QualityStatusCode (projected onto the condition's Quality on a transition; with an Unspecified kind it is a quality-only annotation — DriverHostActor routes it through the same AlarmQualityUpdate → WriteAlarmQuality path as the connectivity annotation, which touches Quality and nothing else, and publishes no /alerts row). Acknowledge is server-side only (MTConnect has no ack; the driver's AcknowledgeAsync is a no-op). The driver diffs against what the server was last told, so a reconnect's replayed /current never re-raises (and so un-acknowledges) an unchanged condition. Full mapping: drivers/MTConnect.md §CONDITION alarms.

Galaxy sub-attribute fallback

For Galaxy templates without $Alarm* extensions, the value-driven path stays in place: DriverNodeManager registers an AlarmConditionState per Galaxy variable that bears alarm-bearing sub-attributes (InAlarm, Acked, Priority, Description), subscribes to those sub-attributes, and synthesizes Part 9 transitions when the values change. This path operated as the only Galaxy alarm path between PR 7.2 and the alarms-over-gateway epic; it remains the fallback today.

When both paths report the same condition, AlarmConditionService.AlarmConditionState keeps the driver-native record and discards the duplicate sub-attribute synthesis. Driver-native transitions are richer (carry operator comment + original raise time) and arrive lower-latency (no publishing-interval delay on the sub-attribute reads), so they win the dedup.

Acknowledge routing — Galaxy / driver alarms

Native alarm acknowledge → AVEVA

When an OPC UA client Acknowledges a native (driver-fed, e.g. Galaxy) AlarmConditionState node, the node manager's OnAcknowledge handler branches on native-ness and routes through a dedicated path — separate from the scripted AlarmCommandRouter:

1. OtOpcUaNodeManager.HandleNativeAlarmAck — gates on the caller's AlarmAck role (fails closed: no role → BadUserAccessDenied), then on the node's redundancy role (not the Primary → BadInvalidState, see §"Acknowledge on a non-Primary node" below), then dispatches a NativeAlarmAck(ConditionNodeId, Comment, OperatorUser) to the NativeAlarmAckRouter seam (fire-and-forget, non-blocking under the node-manager Lock). OperatorUser carries the authenticated session principal's display name.
2. DriverHostActor.HandleRouteNativeAlarmAck — receives a RouteNativeAlarmAck message (the host maps NativeAlarmAck at the wiring boundary to keep Runtime Akka-free of the OPC UA layer). Applied Primary-gate first: a Secondary or Detached node drops the message silently. On Primary, resolves the condition NodeId from the _driverRefByAlarmNodeId inverse map (NodeId → (DriverInstanceId, FullName)) and Tells the owning DriverInstanceActor a RouteAlarmAck(FullName, Comment, OperatorUser).
3. Galaxy driver — DriverInstanceActor calls the driver's IAlarmSource.AcknowledgeAsync with an AlarmAcknowledgeRequest carrying the authored FullName as the ConditionId and the authenticated OperatorUser. The driver forwards this to the Galaxy gateway → AVEVA via GatewayGalaxyAlarmAcknowledger. Fire-and-forget — a failed upstream ack is not surfaced back to the OPC UA client (mirrors the Galaxy write-outcome limitation; the local AlarmConditionState SDK update already committed at step 1).

Only the Acknowledge is routed to the driver. A native condition supports Acknowledge and AddComment only (Gitea #587 review): Confirm, OneShotShelve / TimedShelve / Unshelve and Enable / Disable return BadNotSupported and report no event. There is no upstream path for them and no pair replication of a local state, and the driver projection (NativeAlarmProjector: always Confirmed, Unshelved) would silently undo one at the next transition. AddComment stays on the scripted AlarmCommandRouter path (Phase 3 scope is Acknowledge → AVEVA only) and is applied locally.

A native condition is never Unconfirmed: it starts Confirmed, and because the SDK's own Acknowledge sets ConfirmedState = false on any condition type that supports Confirm, the node manager re-asserts Confirmed = true (and the projection's Retain rule, Active || !Acked) after the SDK handler, before the one ack event. ScadaBridge maps ConfirmedState into its alarm state, so this is client-visible.

Acknowledge on a non-Primary node is refused (Gitea #597). On a node that is not the redundancy Primary, an OPC UA Acknowledge of a native condition returns BadInvalidState ("not the redundancy Primary; acknowledge on the Primary") — nothing is routed, applied or reported. The verdict is the same data-plane one the device-write gate and DriverHostActor's ack route use (PrimaryGatePolicy.ShouldServiceAsPrimary, published to IDriverPrimaryRoleSource on every redundancy snapshot and wired onto the node manager by OtOpcUaServerHostedService). It is checked after authorization, so a caller who may not ack learns nothing about the node's role.

This replaces a documented flap: the Secondary used to apply and report the ack locally while DriverHostActor dropped the route upstream, so the driver never acknowledged, its next transition projected Acked = false, and the client saw acked → unacked. An honest refusal is client-visible — a client (ScadaBridge included) that acks against the Secondary now gets a Bad result instead of a Good that later reverts — so ack against the Primary (the higher ServiceLevel). Scripted conditions are not gated: their engine runs on both nodes, so an ack on the Secondary is real. The published verdict mirrors DriverHostActor's live check from the moment the host starts, not only from the first redundancy snapshot: the host publishes it once at start and again whenever cluster membership changes, so in the boot window of a pair (role still unknown, a driver peer Up) the gate already denies, exactly as the route does (review R1 m2 — previously the view sat on its single-node "allow" seed until the first snapshot, reopening the flap for that window). It mirrors, it does not deny from start: until this node has joined and sees its peer Up, its member count is 0 and both the published verdict and the live route allow (seconds normally; up to the bootstrap guard's 25 s partner probe). The two agree there, so the flap cannot occur, but a native ack in that pre-join window is serviced (review R6 MINOR-2).

Best-effort during reconnect. If a native ack arrives while the owning driver is Connecting/Reconnecting, it is dropped with a debug log and not retried upstream — the OPC UA Part 9 AlarmConditionState was already committed locally at step 1, so the rare outcome is "acked locally but not propagated to AVEVA." Re-issue the ack once the driver is reconnected if upstream propagation is required — a re-ack of a node already showing Acked is still routed to the driver for exactly this reason.

Legacy sub-attribute path

DriverNodeManager picks the acknowledger when registering each condition (PR B.3 logic):

• Driver implements IAlarmSource → DriverAlarmSourceAcknowledger routes the operator comment through IAlarmSource.AcknowledgeAsync via the existing AlarmSurfaceInvoker (Phase 6.1 resilience pipeline; no-retry per decision #143). End-to-end operator-comment fidelity is preserved.
• Driver doesn't implement IAlarmSource → DriverWritableAcknowledger writes the comment into the AckMsgWriteRef sub-attribute via IWritable.WriteAsync. Same resilience pipeline; collapses the comment into a single string write at the wire level.

The OPC UA Part 9 AlarmConditionState.OnAcknowledge delegate already validates the session's AlarmAck role before dispatching, so the gateway-side ack RPC only sees authenticated, authorised calls.

Inbound operator ack/shelve — scripted alarms

Scripted alarms use a separate inbound path that converges on the alarm-commands DPS topic. Two surfaces route onto this topic:

OPC UA Part 9 method path (external OPC UA clients)

OtOpcUaNodeManager wires the Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each scripted-alarm AlarmConditionState node. Every call is gated on the AlarmAck role — fail-closed: sessions with no role or without an AlarmAck-conferring grant receive BadUserAccessDenied immediately. The session's roles come from its API-key principal's NodeAcl grants (or the reserved anonymous principal's; LDAP is AdminUI-only — docs/security.md) and are carried past OpcUaApplicationHost by RoleCarryingUserIdentity (a UserIdentity subclass), making it readable inside the method handler at dispatch time.

On allow, the handler publishes a Commons.OpcUa.AlarmCommand onto the alarm-commands DPS topic. The node manager is Akka-free; the dispatch action is a settable Action<AlarmCommand> injected at boot by the hosted service.

OnTimedUnshelve (the SDK's automatic unshelve timer) is a no-op — it routes nothing (review R1 M2). The engine owns timed-shelve expiry (its 5 s shelving check records AutoUnshelve by system), and its re-projection unshelves the node; see §"UnshelveTime is the real remaining time" below.

One condition event per method call (Gitea #587)

Every accepted Part 9 method call — Acknowledge, Confirm, AddComment, OneShotShelve, TimedShelve, Unshelve, Enable, Disable, on a native or a scripted condition — reports exactly one condition event carrying the new state, to folder subscribers and (through the ACL) Server-object subscribers alike. A refused call (bad status, e.g. a denied gate, an already-acknowledged condition, Enable/Disable on a native condition) reports nothing.

• The SDK does not report it. ConditionState.ReportStateChange is gated on AutoReportStateChanges, which this repo leaves false — so before #587 a client Acknowledge produced no condition event at all (an older comment claimed "the SDK auto-fires its own event"; it never did). Instead the node manager wraps each method's SDK handler (WireMethodTransitionReporting) and, once the handler returns Good, reports the condition itself. That report does not depend on the SDK's AreEventsMonitored bookkeeping, which is imprecise for conditions materialised under an already-subscribed folder.
• The SDK's shelve methods never apply shelving state (they only call OnShelve), so the wrapper applies it — with the same (shelved, oneShot) mapping the projection uses, and a TimedShelve with its requested duration — before reporting.
• The re-projection does not duplicate it. The engine's (or driver's) answer to the routed command reaches WriteAlarmCondition's delta-gate, which compares against the node's current state — already the new state — and suppresses. The whole method call runs under the node-manager lock, so a fast re-projection cannot slip in between the SDK applying the state and the report.
• Only a real change is reported. The condition's state is compared before and after the SDK handler; a call that changed nothing reports nothing. The SDK accepts acknowledging an acknowledged condition and sets Confirmed = false again, so a re-ack also restores the pre-call Confirmed and Retain — silent, and a scripted node never drifts from its engine. AddComment always reports (the comment is the change), and so does a re-ack carrying a comment (Gitea #597): the SDK writes that comment to the node, so it is reported once, with Acked/Confirmed unchanged. Before #597 the comment landed on the node but no subscriber ever saw it. For a scripted condition the comment is also durable (review R1 m1): the routed Acknowledge reaches an engine that is already acknowledged, and Part9StateMachine.ApplyAcknowledge records a commented re-ack exactly as an AddComment (kind AddComment, the caller as user) — persisted and replicated with the condition state, the original ack's LastAck* fields untouched. An uncommented re-ack is still a no-op in the engine. The same applies to a commented re-ack from the AdminUI /alerts page.
• A native re-ack is still routed to the driver (deliberate, #597). The route is fire-and-forget, so a node showing Acked does not prove the upstream system acknowledged — an ack dropped during a reconnect leaves exactly that state — and re-acking is the documented retry (§"Best-effort during reconnect"). The cost is a duplicate ack upstream for an alarm that was already acknowledged there.
• Native conditions: Acknowledge and AddComment only — see §"Native alarm acknowledge → AVEVA" above. The other methods are refused (BadNotSupported, no event), and a native ack keeps the condition Confirmed.
• TimedShelve works (scripted). Create() left MaxTimeShelved at 0, and the SDK rejects any shelving time above it, so every client TimedShelve failed with BadShelvingTimeOutOfRange. It is now int.MaxValue ms (≈24.8 days). The engine owns the expiry and its Unshelved re-projection reports the one expiry event.
• UnshelveTime is the real remaining time (Gitea #597). Both the method path and the projection used to apply a timed shelve with shelvingTime 0, which the SDK reads as MaxTimeShelved — so every timed shelve advertised ≈24.8 days remaining. The method now applies the requested duration, and the engine's projection carries its expiry (AlarmConditionSnapshot.UnshelveAtUtc), so ShelvingState/UnshelveTime counts down to the engine's expiry whichever path shelved it (the AdminUI /alerts timed shelve included). This arms the SDK's own unshelve timer at the real instant, and OnTimedUnshelve — the timer's callback — is deliberately a no-op (review R1 M2): it used to route an operator-style Unshelve by system, which, firing before the engine's 5 s expiry sweep, pre-empted the engine's AutoUnshelve on every expiry, from both pair nodes, rewriting the audit trail as a manual unshelve. The SDK's timer changes no shelving state itself, so the node stays TimedShelved (with UnshelveTime at zero) for up to one sweep after expiry, until the engine's re-projection reports the one expiry event.
• Accepted, not fixed — the SDK's EventsMonitored unit for late conditions (Gitea #597 item 6, from the #586 review). The SDK's AreEventsMonitored counter never reaches a condition materialised under a folder that already had an event subscriber. Our own reporting does not depend on it (above); the one thing it still gates is the SDK's method audit event, and only while the server is auditing — which no configuration in this repo enables. Closed as accepted.
• Method audit events (AuditConditionAcknowledgeEventType, …) are still the SDK's, delivered only while the server is auditing over an encrypted channel.

Engine-driven transitions are unaffected: WriteAlarmCondition still fires whenever the incoming state differs from the node's current state.

AdminUI path

The /alerts page shows per-row Acknowledge / Shelve / Unshelve buttons gated by the DriverOperator AdminUI policy. These route through the AdminOperationsActor cluster singleton (AcknowledgeAlarmCommand / ShelveAlarmCommand), which publishes onto the same alarm-commands topic. The singleton handles cross-node routing — the command always reaches the driver-role node owning the engine regardless of which AdminUI instance the operator is on.

ScriptedAlarmHostActor dispatch

ScriptedAlarmHostActor subscribes to the alarm-commands topic, ownership-filters each command (each node only acts on its own alarms), and dispatches to the matching ScriptedAlarmEngine operation (AcknowledgeAsync / ConfirmAsync / OneShotShelveAsync / TimedShelveAsync / UnshelveAsync / EnableAsync / DisableAsync / AddCommentAsync). The engine's existing OnEvent callback handles the OPC UA node update — no explicit re-projection is required.

The AdminUI /alerts Shelve flow was live-verified on docker-dev 2026-06-11: singleton → topic → host actor → engine → "Shelved" status reflected on /alerts with the operator identity threaded through.

Redundancy deduplication

Under warm/hot redundancy, both cluster nodes run ScriptedAlarmHostActor and the scripted-alarm engine. To prevent duplicate /alerts rows and duplicate historian writes, alarm transition publication to the alerts topic and HistorianAdapterActor historization are Primary-gated: only the node whose RedundancyRole is Primary publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See Redundancy.md §Primary-gated alarm emission and historization.

Historian write-back (non-Galaxy alarms)

Scripted alarms (and any future non-Galaxy IAlarmSource like AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:

• IAlarmHistorianSink is the DI-registered intake contract. The default binding is NullAlarmHistorianSink (registered in ServiceCollectionExtensions.AddOtOpcUaRuntime). Production deployments override it with LocalDbStoreAndForwardSink wrapping GatewayAlarmHistorianWriter (the HistorianGateway SendEvent path) — see ServiceHosting.md for the HistorianGateway setup.

• LocalDbStoreAndForwardSink queues each transition into the node's consolidated LocalDb — where it replicates to the redundant pair peer, so undelivered alarm history survives losing the node holding it — and drains in the background via an IAlarmHistorianWriter. Only the node holding the Primary role drains, since the peer holds a full replica; delivery is consequently at-least-once across a failover, by design. See AlarmHistorian.md. The durability guarantee is bounded: the queue capacity defaults to 1,000,000 rows; under a sustained historian outage, older non-dead-lettered rows are evicted (oldest first) to make room for new events. The HistorianSinkStatus.EvictedCount counter surfaces lifetime eviction events so operators can detect silent data loss without log scraping. The drain cadence, queue capacity, and dead-letter retention are tunable via the AlarmHistorian config section (DrainIntervalSeconds, Capacity, DeadLetterRetentionDays); AlarmHistorianOptions.Validate() logs a startup warning for a non-positive knob. (It no longer warns about SharedSecret or DatabasePath — both keys are gone: the connection comes from ServerHistorian, and the queue lives in the LocalDb.)

• HistorianAdapterActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs) subscribes to the cluster alerts DPS topic, translates each AlarmTransitionEvent → AlarmHistorianEvent, and calls EnqueueAsync fire-and-forget. The durable write is gated two ways: (1) Primary-gated — only the Primary node historizes, giving exactly-once writes across a redundant pair; (2) per-alarm — a transition whose HistorizeToAveva is false is skipped (the flag rides on AlarmTransitionEvent as a nullable bool; missing/null/true historize, only an explicit false suppresses, so a cross-version rolling restart defaults to historizing rather than dropping an audit row). Neither gate touches the live alerts publish — the /alerts UI always sees every transition. See AlarmHistorian.md §Configuration for the AlarmHistorian appsettings section that enables the real sink.

  Native alarms (equipment tags carrying an "alarm" object in their TagConfig) support the same HistorizeToAveva opt-out. The field is alarm.historizeToAveva (bool?) and is authored via the "Historize to AVEVA" checkbox in the Tag modal's alarm section. The gate logic is identical (is not false): absent or true historizes; explicit false suppresses the AVEVA write while leaving the live /alerts feed unaffected. See ScriptedAlarms.md §TagConfig alarm fields for the full field reference.

Galaxy-native alarms with $Alarm* extensions reach AVEVA Historian directly via System Platform's HistorizeToAveva toggle on the alarm primitive — no involvement from OtOpcUa. This sidecar path is exclusively for non-Galaxy alarm producers.

Cross-references

• Plan: docs/plans/alarms-over-gateway.md
• v1 archive: docs/v1/AlarmTracking.md
• Galaxy driver: docs/drivers/Galaxy.md
• Phase 7 scripting + alarming: docs/v2/implementation/phase-7-scripting-and-alarming.md
• Security + ACL: docs/security.md