Files
lmxopcua/docs/AlarmTracking.md
T
Joseph Doherty 5a7a08deca fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths
into Reconnecting were a failed *initial* connect and an operator's Force
Reconnect, so a driver that connected and then lost its device stayed Connected
indefinitely: publishing Bad for every tag, never re-initialising, and NOT
recovering when the device returned. PublishHealthSnapshot read GetHealth() for
reporting only and never acted on a driver reporting itself unhealthy.

Both comms-loss quality paths -- #477's native-condition annotation and #481's
scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were
correctly wired to a signal that never fired.

The Connected health tick now evaluates a watchdog and self-sends
DisconnectObserved when the driver reports a non-Healthy state AND its
LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s).

The conjunction is the design, and the issue's own proposal -- "non-Healthy for
N consecutive polls" -- is wrong. Degraded means *connected but some data is
bad*; several drivers write health per tag inside the read loop
(ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever
against a perfectly reachable device, and a state-only watchdog would reconnect
that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is
the honest evidence: preserved across Degraded/Faulted writes, advanced by any
success. The state check remains the second conjunct so an idle Healthy driver
with nothing subscribed is not declared dead for not reading.

Verified across all 12 drivers that a sustained failure does reach a non-Healthy
state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy
via its reconnect supervisor. CalculationDriver reports Healthy while running
and so can never trip, which is correct: it has no transport.

Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync.
ModbusDriver (_transport) and MqttDriver (_connection) overwrite an
IAsyncDisposable field inside InitializeAsync without disposing the previous
one, so re-initialising without a teardown leaks a socket per attempt -- latent
while only a manual Force Reconnect reached this path, but the watchdog makes it
automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry).

Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence
assertion carrying a positive control (a proven count of processed health-poll
ticks). Falsified five ways -- dropping either conjunct reds exactly the test
written for it, and removing the teardown, the null baseline, or the watchdog
call each reds its own guard.

Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice:
  - fixture stopped -> watchdog fires at ~108s, "declaring comms lost"
  - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)"
    (#481 Layer 4) and the #477 native-condition annotation on a driver
    owning conditions
  - fixture restarted -> reconnected in ~8s with NO node restart; both the raw
    (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
2026-07-31 10:58:38 -04:00

31 KiB

Alarm tracking — v2 final architecture

This document describes how OtOpcUa surfaces alarms to OPC UA Part 9 clients after the alarms-over-gateway epic (docs/plans/alarms-over-gateway.md) landed. The v1 architecture (Galaxy.Host's COM-side GalaxyAlarmTracker) is preserved at docs/v1/AlarmTracking.md for historical reference.

Three alarm sources, one OPC UA Part 9 surface

Source Driver capability Path
Galaxy MxAccess (driver-native) GalaxyDriver : IAlarmSource gateway → worker → MxAccess alarm sink → MX_EVENT_FAMILY_ON_ALARM_TRANSITION → EventPump → driver OnAlarmEvent → AlarmConditionService
Galaxy sub-attribute fallback IWritable writes to $Alarm* sub-attributes gateway data subscription → driver OnDataChange → DriverNodeManager ConditionSink → AlarmConditionService
Scripted alarms Phase7Composer server-side script evaluator → ScriptedAlarmActor transitions → HistorianAdapterActor → IAlarmHistorianSink

All three converge on the alarm-state actor — in v2 the OPC UA Part 9 state machine lives inside ScriptedAlarmActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmActor.cs), which dispatches transitions to the OPC UA condition node managers. Driver-native transitions take precedence over sub-attribute synthesis when both arrive for the same condition — the dedup logic prefers the richer driver-native record because it carries the full operator + raise-time + category metadata that the value-driven path collapses.

v3 Batch 4 — multi-notifier delivery (raw + equipment folders)

Under the v3 dual-namespace address space, a native driver alarm is authored on a raw tag (/raw) and its Part 9 AlarmConditionState materializes once at the raw tag — ConditionId = the tag's RawPath, parent notifier = the raw device/group folder (ns=Raw). Because equipment references raw tags (v3 reference-only UNS), the same condition is wired as an event notifier of every referencing equipment folder (ns=UNS) via the SDK AddNotifier pattern (OtOpcUaNodeManager.WireAlarmNotifiers): alarm.AddNotifier(equipFolder, isInverse:true) + equipFolder.AddNotifier(alarm) + EnsureFolderIsEventNotifier(equipFolder).

  • A single ReportEvent fans through the SDK notifier graph to the raw device folder, every referencing equipment folder, and up to the Server object. A subscriber at any one root receives exactly one copy of the transition — the Part 9 shared-InstanceStateSnapshot dedup (MonitoredItem.QueueEvent → IsEventContainedInQueue), never one copy per root. Duplicating the ReportEvent per root is rejected by design (distinct EventIds would break Server-object dedup + Part 9 ack correlation).
  • Teardown is symmetric: the node manager tracks the wired notifier pairs and calls RemoveNotifier(..., bidirectional:true) on rebuild / subtree-removal / reference-removal, so inverse-notifier entries never leak across redeploys.
  • Ack/confirm/shelve route on ConditionId = RawPath (never SourceNodeId) regardless of which notifier root the operator subscribed at — an ack issued from an equipment-folder subscription resolves to the same raw condition.
  • The alerts-topic AlarmTransitionEvent carries the (possibly empty) list of referencing equipment paths, and the AdminUI /alerts page shows one row per condition (primary identity RawPath + condition NodeId) with the equipment list as display metadata.

Condition event identity fields (what a client reads on the wire)

Every condition event — native and scripted — carries the mandatory BaseEventType identity fields, assigned at materialize time in OtOpcUaNodeManager.MaterialiseAlarmCondition. The SDK does not synthesize them on this path (Create builds the children from the type definition but leaves them unset; ReportEvent / InstanceStateSnapshot copy children verbatim), so they are set explicitly. Leaving them unset shipped them as null on every event — see issues #473 (the BaseEventType trio) and #475 (the ConditionType classification pair).

Field Value Notes
EventType the concrete materialized type (TypeDefinitionId) e.g. OffNormalAlarmType; falls back to AlarmConditionType for an unknown authored type. Readable as a field, not only via an OfType where-clause
SourceNode the condition's own NodeId — equal to ConditionId The condition is the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at
SourceName the same identifying id string: RawPath (native) / ScriptedAlarmId (scripted) Deliberately the unique id, not the leaf name
ConditionName the leaf / display name (e.g. HR200) Where the short human-readable name lives
ConditionClassId always BaseConditionClassType Part 9's "no condition class modelled" value. Unset shipped NodeId.Null (#475)
ConditionClassName always "BaseConditionClass" Matches ConditionClassId. Unset shipped empty text (#475)
Quality the condition's source-data quality — native tracks the source's connectivity (Good / Bad); scripted takes the worst of its input tags' qualities (#478) A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below

Why BaseConditionClassType and not ProcessConditionClassType. We hold no per-alarm classification at the materialize seam, and ConditionClassId is a wire contract clients bucket on. BaseConditionClassType is the honest, spec-conformant report of "this server does not model condition classes" — it fixes the real defect (a null that breaks conformant clients) without asserting a classification we cannot back. ProcessConditionClassType — the SDK sample's pick — was rejected deliberately: it would be actively wrong for a Galaxy alarm whose upstream category is Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm classification is a separate future feature: it needs the driver's alarm category, which today lives only on the runtime AlarmEventArgs transition, carried to the deploy-time authored composition that MaterialiseAlarmCondition sees. Until then the IAlarmSource doc comment claiming the category "maps to ConditionClassName downstream" describes an intent, not the implementation.

Why SourceName is the id, not the leaf name. The leaf is ambiguous across devices (HR200 on two PLCs collides) and is already carried by ConditionName, so the leaf-name option would add no identity while costing uniqueness. Carrying the id makes the SourceNode/SourceName/ConditionId triple mutually consistent and unique. This diverges from the loose OPC UA convention that SourceName mirrors the source node's BrowseName; the divergence is intentional.

Client guidance: key on ConditionId — the condition node's own NodeId, which equals SourceNode, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for scripted ones. It is the identity ack/confirm/shelve route on. SourceName carries the same identifier and is unique, so it is safe to key on by itself — but do not compose it with ConditionName ($"{SourceName}.{ConditionName}"), because SourceName already ends in the condition's leaf name and the result stutters (pymodbus/plc/HR200.HR200).

Wire-level guard: NativeAlarmEventIdentityFieldDeliveryTests asserts the three BaseEventType fields arrive populated on a real subscription using the standard [EventType, SourceNode, SourceName, Time, Message, Severity] select clause, and — in a second test with its own clause — that ConditionClassId / ConditionClassName do too. The two class fields are declared on ConditionType, not BaseEventType, so a client must select them against that type. NodeManagerAlarmSourceFieldsTests guards the node itself across both realms.

Do not correlate live events to HistoryRead on SourceName — the two paths disagree. The HistoryRead events projection (OtOpcUaNodeManager.ProjectEventField) returns Variant.Null for EventType / SourceNode by design: it projects from the historian's HistoricalEvent rows, which do not carry them. It does project SourceName — but the alarm-history writer stamps that field with the EquipmentPath (AlarmEventMapper: SourceName = alarm.EquipmentPath), not the RawPath a live event carries. So SourceName is the one field populated on both paths with different values, and it is not a live↔history join key. Correlate on ConditionId / the RawPath instead. (Pre-existing; the live-path fix above does not change the history path.)

Condition source-data Quality (#477)

ConditionType.Quality reports the quality of the condition's source data. It was never assigned, and because StatusCodes.Good == 0x00000000 an unassigned StatusCode is Good — so every condition reported Good unconditionally (a wrong value, not a null like #473/#475). A native alarm whose device went offline still read Good, so an operator could not tell "genuinely inactive" from "we have lost contact and do not know".

How it is driven now (native alarms). An alarm-bearing raw tag materializes a condition with no sibling value variable, so the value/quality path (WriteValue) never touches it, and a comms-lost driver emits no alarm transitions (the feed goes silent). The quality therefore comes from the driver's connectivity, out of band from alarm transitions:

  • DriverInstanceActor Tells its host ConnectivityChanged(driverInstanceId, connected) on every transition into Connected (true) / Reconnecting (false).
  • DriverHostActor.OnDriverConnectivityChanged fans that out to every native condition the driver owns as an OpcUaPublishActor.AlarmQualityUpdate (Good on connect, Bad on disconnect).
  • OtOpcUaNodeManager.WriteAlarmQuality sets only the condition's Quality and fires a Part 9 event only on a quality-bucket change — it never touches Active/Acked/Severity/Retain (an active alarm that loses comms stays active). This is a dedicated path, not a full-snapshot re-projection, so it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
  • A freshly materialized native condition starts BadWaitingForInitialData (the "no driver data yet" convention value variables use); the first Connected confirms it Good.
  • The connectivity annotation is ungated by redundancy role (a Secondary keeps its condition quality warm for failover) and publishes no /alerts row — driver comms health already has its own status surface (IDriverHealthPublisher); a row per condition would be alarm-fatigue.

Scripted alarms (Layer 3, #478). A scripted condition's state is computed from one or more input tags, so its Quality is the worst quality across those inputs at evaluation time ("can I trust this condition's state?") — mirroring the native OT semantic:

  • The mux now forwards each input's source quality (DependencyValueChanged.Quality), and the scripted host pushes it into the engine's read cache (previously every mux value was treated as Good).
  • The ScriptedAlarmEngine computes the worst input quality each evaluation. A real transition carries it on the emitted event → ScriptedAlarmHostActor.ToSnapshot projects it (so a transition fired while an input is Uncertain does not clobber quality back to Good).
  • A Bad input freezes the condition (AreInputsReady holds its state — no transition), exactly like a comms-lost native driver. So a quality-bucket change with no transition is emitted as EmissionKind.QualityChanged and routed to the same dedicated AlarmQualityUpdate → WriteAlarmQuality node path native uses (quality only, one Part 9 event on a bucket change, no /alerts row, no historian write). ScriptedAlarmSource skips QualityChanged so it never fabricates a phantom IAlarmSource event.
  • An input that has not been published yet (cold start) is not a quality signal (that is the readiness guard's job) — it contributes Good, so scripted conditions don't flash Bad at every deploy. The first actually-Bad published value flips the bucket and annotates.

Comms loss (Layer 4, #481). #478 covered input tags whose driver publishes a data change carrying a Bad/Uncertain StatusCode (e.g. an OpcUaClient input forwarding a server's per-item Bad). It did not cover a driver comms loss: a poll driver (Modbus/S7) whose device goes unreachable emits only ConnectivityChanged and goes silent on the value feed (see DriverInstanceActor.Reconnecting), so the engine kept the last-known Good value, WorstInputStatus stayed Good — and worse, AreInputsReady stayed true, so the predicate went on evaluating a stale value after the source had died.

DriverHostActor.FanCommsLossToDependencyMux closes it, symmetrically with the native OnDriverConnectivityChanged path above: on ConnectivityChanged(false) it pushes a synthetic BadNoCommunication with a null value into the dependency mux for every value ref the driver owns, resolved from the existing _nodeIdByDriverRef map. Alarm-bearing tags are deliberately excluded — a native condition is not a mux value, and its comms quality goes through the Part 9 annotation instead.

Two things worth knowing before changing this:

  • Nothing is pushed on reconnect, deliberately — publishing Good would assert a value quality the host has not observed. Recovery rides the driver's own republish: the Connected → Reconnecting edge calls DetachSubscription, so the reconnect's ResubscribeDesired builds a fresh subscription whose PollGroupEngine initial push runs with forceRaise: true and re-publishes every tag with its real quality. That detach is load-bearing — without it the poll engine would compare against its retained LastValues, see an unchanged value, raise nothing, and strand those refs Bad forever.
  • Virtual tags computed from a comms-lost input now go Bad too. That follows from using the shared mux and is the correct reading of "this input's source is unreachable", but it is a behaviour change beyond scripted alarms.

The signal this hangs off now fires for a mid-run device loss (#527, closed). The #481 live gate on docker-dev found that DriverInstanceActor.DisconnectObserved had no production sender: the only paths into Reconnecting were a failed initial connect and an operator Force Reconnect. A Modbus driver whose fixture was stopped stayed Connected for >10 minutes publishing per-tag Bad, and did not recover when the fixture came back until the node was restarted. So the fan-out above — and the native #477 Layer 2 annotation it mirrors — were correctly wired to a signal that a dying device never raised. #527 added the comms-loss watchdog on the Connected health tick, which self-sends DisconnectObserved once a driver has gone 90 s without a successful read while reporting itself non-Healthy. See DriverLifecycle.md §"The comms-loss watchdog" for why the trip condition is that conjunction and not simply "non-Healthy for N polls" — the naive rule would reconnect a healthy device's driver every 90 s forever whenever one authored address is unreadable.

Note the partial cover that existed before, because it explains why some of this appeared to work: Modbus publishes Bad per tag rather than going silent, so #478 Layer 3 did carry Bad into scripted input quality for that driver. That was a per-driver accident, not a guarantee, and it did nothing for native conditions or for recovery.

Cold start vs. a driver reporting Bad (#478 Finding 3, closed by #481). WorstInputStatus used to skip every input whose value was null as "no data yet". A comms-lost driver produces exactly that shape — Bad with no value — so the skip was hiding the case that matters most. The upstream source now stamps a never-pushed path BadWaitingForInitialData and nothing else does, so the engine skips that specific code rather than every null. Deleting the null-skip outright would have been the obvious fix and is wrong in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, which is what the skip existed to prevent. Both failure modes are pinned — see the engine-test pair below, each of which goes red under the opposite mistake.

Guards: ScriptedAlarmEngineTests (transition carries Uncertain; Bad input with no transition emits QualityChanged(Bad); restore emits QualityChanged(Good); unchanged bucket emits nothing; plus the #481 pair — a pushed Bad carrying no value counts, an input never pushed does not), DriverHostActorLiveValueTests (comms loss pushes Bad for every ref the driver owns, reconnect pushes nothing, another driver's loss pushes nothing), DependencyMuxTagUpstreamSourceTests.The_engine_agrees_with_this_source_on_the_never_pushed_status_code, ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event, DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged, and ScriptedAlarmHostActorTests (Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts, Transition_snapshot_carries_worst_input_quality).

Wire-level guard: NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire subscribes with a [Quality, Message] clause (Quality is declared on ConditionType) and asserts a healthy source reports Good, a comms-lost source reports non-Good, and recovery returns to Good — on a real client subscription. NodeManagerAlarmSourceFieldsTests guards the node itself + the no-clobber / unknown-node-no-op invariants.

Galaxy driver path (driver-native)

Restored in PR B.2 of the epic. GalaxyDriver implements IAlarmSource with these surfaces:

  • SubscribeAlarmsAsync(sourceNodeIds) → returns a sentinel handle. The driver doesn't multiplex per source-node-id today; every active handle observes the gateway's alarm-event stream. The server-side AlarmConditionService filters by source-node before raising the OPC UA condition.
  • UnsubscribeAlarmsAsync(handle) → symmetric handle removal.
  • AcknowledgeAsync(requests) → routes one gateway RPC per acknowledgement through IGalaxyAlarmAcknowledger. Production uses GatewayGalaxyAlarmAcknowledger calling MxGatewayClient.AcknowledgeAlarmAsync (PR E.2 SDK method).
  • OnAlarmEvent → bridges EventPump.OnAlarmTransition (PR B.1) onto AlarmEventArgs. Suppressed when no alarm subscription is active so untracked transitions don't leak through.

The proto contract carries the rich payload — alarm full reference, source-object reference, alarm-type-name, transition kind (Raise / Acknowledge / Clear / Retrigger), severity (raw MxAccess scale), original raise timestamp, transition timestamp, operator user, operator comment, alarm category, description. MxAccessSeverityMapper (PR B.1) translates the raw severity onto the four-bucket AlarmSeverity ladder — boundaries match v1's GalaxyAlarmTracker so customers see no surprise re-classification.

The richer fields surface on Core.Abstractions.AlarmEventArgs via the optional properties added in PR E.7 (OperatorComment, OriginalRaiseTimestampUtc, AlarmCategory). Consumers that don't need them are unaffected; consumers that do (Client.UI, Client.CLI verbose mode) read the new fields when present.

Galaxy sub-attribute fallback

For Galaxy templates without $Alarm* extensions, the value-driven path stays in place: DriverNodeManager registers an AlarmConditionState per Galaxy variable that bears alarm-bearing sub-attributes (InAlarm, Acked, Priority, Description), subscribes to those sub-attributes, and synthesizes Part 9 transitions when the values change. This path operated as the only Galaxy alarm path between PR 7.2 and the alarms-over-gateway epic; it remains the fallback today.

When both paths report the same condition, AlarmConditionService.AlarmConditionState keeps the driver-native record and discards the duplicate sub-attribute synthesis. Driver-native transitions are richer (carry operator comment + original raise time) and arrive lower-latency (no publishing-interval delay on the sub-attribute reads), so they win the dedup.

Acknowledge routing — Galaxy / driver alarms

Native alarm acknowledge → AVEVA

When an OPC UA client Acknowledges a native (driver-fed, e.g. Galaxy) AlarmConditionState node, the node manager's OnAcknowledge handler branches on native-ness and routes through a dedicated path — separate from the scripted AlarmCommandRouter:

  1. OtOpcUaNodeManager.HandleNativeAlarmAck — gates on the caller's AlarmAck role (fails closed: no role → BadUserAccessDenied), then dispatches a NativeAlarmAck(ConditionNodeId, Comment, OperatorUser) to the NativeAlarmAckRouter seam (fire-and-forget, non-blocking under the node-manager Lock). OperatorUser carries the authenticated session principal's display name.
  2. DriverHostActor.HandleRouteNativeAlarmAck — receives a RouteNativeAlarmAck message (the host maps NativeAlarmAck at the wiring boundary to keep Runtime Akka-free of the OPC UA layer). Applied Primary-gate first: a Secondary or Detached node drops the message silently. On Primary, resolves the condition NodeId from the _driverRefByAlarmNodeId inverse map (NodeId → (DriverInstanceId, FullName)) and Tells the owning DriverInstanceActor a RouteAlarmAck(FullName, Comment, OperatorUser).
  3. Galaxy driver — DriverInstanceActor calls the driver's IAlarmSource.AcknowledgeAsync with an AlarmAcknowledgeRequest carrying the authored FullName as the ConditionId and the authenticated OperatorUser. The driver forwards this to the Galaxy gateway → AVEVA via GatewayGalaxyAlarmAcknowledger. Fire-and-forget — a failed upstream ack is not surfaced back to the OPC UA client (mirrors the Galaxy write-outcome limitation; the local AlarmConditionState SDK update already committed at step 1).

Only the Acknowledge is routed to the driver. Confirm / AddComment / Shelve operations on a native condition stay on the scripted AlarmCommandRouter path (Phase 3 scope is Acknowledge → AVEVA only).

Best-effort during reconnect. If a native ack arrives while the owning driver is Connecting/Reconnecting (or on a non-Primary node), it is dropped with a debug log and not retried upstream — the OPC UA Part 9 AlarmConditionState was already committed locally at step 1, so the rare outcome is "acked locally but not propagated to AVEVA." Re-issue the ack once the driver is reconnected if upstream propagation is required.

Legacy sub-attribute path

DriverNodeManager picks the acknowledger when registering each condition (PR B.3 logic):

  • Driver implements IAlarmSource → DriverAlarmSourceAcknowledger routes the operator comment through IAlarmSource.AcknowledgeAsync via the existing AlarmSurfaceInvoker (Phase 6.1 resilience pipeline; no-retry per decision #143). End-to-end operator-comment fidelity is preserved.
  • Driver doesn't implement IAlarmSource → DriverWritableAcknowledger writes the comment into the AckMsgWriteRef sub-attribute via IWritable.WriteAsync. Same resilience pipeline; collapses the comment into a single string write at the wire level.

The OPC UA Part 9 AlarmConditionState.OnAcknowledge delegate already validates the session's AlarmAck role before dispatching, so the gateway-side ack RPC only sees authenticated, authorised calls.

Inbound operator ack/shelve — scripted alarms

Scripted alarms use a separate inbound path that converges on the alarm-commands DPS topic. Two surfaces route onto this topic:

OPC UA Part 9 method path (external OPC UA clients)

OtOpcUaNodeManager wires the Part 9 condition methods (Acknowledge / Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each scripted-alarm AlarmConditionState node. Every call is gated on the AlarmAck LDAP role — fail-closed: sessions with no role or without AlarmAck group membership receive BadUserAccessDenied immediately. The LDAP-resolved role set is carried past OpcUaApplicationHost by RoleCarryingUserIdentity (a UserIdentity subclass), making it readable inside the method handler at dispatch time.

On allow, the handler publishes a Commons.OpcUa.AlarmCommand onto the alarm-commands DPS topic. The node manager is Akka-free; the dispatch action is a settable Action<AlarmCommand> injected at boot by the hosted service.

OnTimedUnshelve (the SDK's automatic unshelve timer) bypasses the operator gate — it is system-initiated.

WriteAlarmCondition fires the Part 9 condition event only when the incoming state differs from the node's current live state (delta-gate), preventing the double-emit that would otherwise occur when the SDK auto-applies the acked state and the engine re-projection fires a duplicate event immediately after.

AdminUI path

The /alerts page shows per-row Acknowledge / Shelve / Unshelve buttons gated by the DriverOperator AdminUI policy. These route through the AdminOperationsActor cluster singleton (AcknowledgeAlarmCommand / ShelveAlarmCommand), which publishes onto the same alarm-commands topic. The singleton handles cross-node routing — the command always reaches the driver-role node owning the engine regardless of which AdminUI instance the operator is on.

ScriptedAlarmHostActor dispatch

ScriptedAlarmHostActor subscribes to the alarm-commands topic, ownership-filters each command (each node only acts on its own alarms), and dispatches to the matching ScriptedAlarmEngine operation (AcknowledgeAsync / ConfirmAsync / OneShotShelveAsync / TimedShelveAsync / UnshelveAsync / EnableAsync / DisableAsync / AddCommentAsync). The engine's existing OnEvent callback handles the OPC UA node update — no explicit re-projection is required.

The AdminUI /alerts Shelve flow was live-verified on docker-dev 2026-06-11: singleton → topic → host actor → engine → "Shelved" status reflected on /alerts with the operator identity threaded through.

Redundancy deduplication

Under warm/hot redundancy, both cluster nodes run ScriptedAlarmHostActor and the scripted-alarm engine. To prevent duplicate /alerts rows and duplicate historian writes, alarm transition publication to the alerts topic and HistorianAdapterActor historization are Primary-gated: only the node whose RedundancyRole is Primary publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See Redundancy.md §Primary-gated alarm emission and historization.

Historian write-back (non-Galaxy alarms)

Scripted alarms (and any future non-Galaxy IAlarmSource like AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:

  • IAlarmHistorianSink is the DI-registered intake contract. The default binding is NullAlarmHistorianSink (registered in ServiceCollectionExtensions.AddOtOpcUaRuntime). Production deployments override it with LocalDbStoreAndForwardSink wrapping GatewayAlarmHistorianWriter (the HistorianGateway SendEvent path) — see ServiceHosting.md for the HistorianGateway setup.

  • LocalDbStoreAndForwardSink queues each transition into the node's consolidated LocalDb — where it replicates to the redundant pair peer, so undelivered alarm history survives losing the node holding it — and drains in the background via an IAlarmHistorianWriter. Only the node holding the Primary role drains, since the peer holds a full replica; delivery is consequently at-least-once across a failover, by design. See AlarmHistorian.md. The durability guarantee is bounded: the queue capacity defaults to 1,000,000 rows; under a sustained historian outage, older non-dead-lettered rows are evicted (oldest first) to make room for new events. The HistorianSinkStatus.EvictedCount counter surfaces lifetime eviction events so operators can detect silent data loss without log scraping. The drain cadence, queue capacity, and dead-letter retention are tunable via the AlarmHistorian config section (DrainIntervalSeconds, Capacity, DeadLetterRetentionDays); AlarmHistorianOptions.Validate() logs a startup warning for a non-positive knob. (It no longer warns about SharedSecret or DatabasePath — both keys are gone: the connection comes from ServerHistorian, and the queue lives in the LocalDb.)

  • HistorianAdapterActor (src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs) subscribes to the cluster alerts DPS topic, translates each AlarmTransitionEvent → AlarmHistorianEvent, and calls EnqueueAsync fire-and-forget. The durable write is gated two ways: (1) Primary-gated — only the Primary node historizes, giving exactly-once writes across a redundant pair; (2) per-alarm — a transition whose HistorizeToAveva is false is skipped (the flag rides on AlarmTransitionEvent as a nullable bool; missing/null/true historize, only an explicit false suppresses, so a cross-version rolling restart defaults to historizing rather than dropping an audit row). Neither gate touches the live alerts publish — the /alerts UI always sees every transition. See AlarmHistorian.md §Configuration for the AlarmHistorian appsettings section that enables the real sink.

    Native alarms (equipment tags carrying an "alarm" object in their TagConfig) support the same HistorizeToAveva opt-out. The field is alarm.historizeToAveva (bool?) and is authored via the "Historize to AVEVA" checkbox in the Tag modal's alarm section. The gate logic is identical (is not false): absent or true historizes; explicit false suppresses the AVEVA write while leaving the live /alerts feed unaffected. See ScriptedAlarms.md §TagConfig alarm fields for the full field reference.

Galaxy-native alarms with $Alarm* extensions reach AVEVA Historian directly via System Platform's HistorizeToAveva toggle on the alarm primitive — no involvement from OtOpcUa. This sidecar path is exclusively for non-Galaxy alarm producers.

Cross-references