lmxopcua/docs/AlarmTracking.md

# Alarm tracking — v2 final architecture

This document describes how OtOpcUa surfaces alarms to OPC UA Part 9
clients after the **alarms-over-gateway** epic
([docs/plans/alarms-over-gateway.md](plans/alarms-over-gateway.md))
landed. The v1 architecture (Galaxy.Host's COM-side `GalaxyAlarmTracker`)
is preserved at [docs/v1/AlarmTracking.md](v1/AlarmTracking.md) for
historical reference.

## Three alarm sources, one OPC UA Part 9 surface

| Source                           | Driver capability        | Path |
|----------------------------------|--------------------------|------|
| **Galaxy MxAccess (driver-native)** | `GalaxyDriver : IAlarmSource` | gateway → worker → MxAccess alarm sink → `MX_EVENT_FAMILY_ON_ALARM_TRANSITION` → `EventPump` → driver `OnAlarmEvent` → `AlarmConditionService` |
| **Galaxy sub-attribute fallback**  | `IWritable` writes to `$Alarm*` sub-attributes | gateway data subscription → driver `OnDataChange` → `DriverNodeManager` ConditionSink → `AlarmConditionService` |
| **Scripted alarms**                | `Phase7Composer` | server-side script evaluator → `ScriptedAlarmActor` transitions → `HistorianAdapterActor` → `IAlarmHistorianSink` |

All three converge on the alarm-state actor — in v2 the OPC UA Part 9 state
machine lives inside `ScriptedAlarmActor`
(`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmActor.cs`),
which dispatches transitions to the OPC UA condition node managers. Driver-native transitions take
precedence over sub-attribute synthesis when both arrive for the same
condition — the dedup logic prefers the richer driver-native record
because it carries the full operator + raise-time + category metadata
that the value-driven path collapses.

## v3 Batch 4 — multi-notifier delivery (raw + equipment folders)

Under the v3 dual-namespace address space, a native driver alarm is authored on a
**raw tag** (`/raw`) and its Part 9 `AlarmConditionState` materializes **once** at the
raw tag — `ConditionId` = the tag's **RawPath**, parent notifier = the raw device/group
folder (`ns=Raw`). Because equipment references raw tags (v3 reference-only UNS), the
same condition is wired as an **event notifier of every referencing equipment folder**
(`ns=UNS`) via the SDK `AddNotifier` pattern (`OtOpcUaNodeManager.WireAlarmNotifiers`):
`alarm.AddNotifier(equipFolder, isInverse:true)` + `equipFolder.AddNotifier(alarm)` +
`EnsureFolderIsEventNotifier(equipFolder)`.

- A **single** `ReportEvent` fans through the SDK notifier graph to the raw device folder,
  every referencing equipment folder, and up to the Server object. A subscriber at any one
  root receives **exactly one** copy of the transition — the Part 9 shared-`InstanceStateSnapshot`
  dedup (`MonitoredItem.QueueEvent` → `IsEventContainedInQueue`), **never** one copy per root.
  Duplicating the `ReportEvent` per root is rejected by design (distinct EventIds would break
  Server-object dedup + Part 9 ack correlation).
- **Teardown is symmetric:** the node manager tracks the wired notifier pairs and calls
  `RemoveNotifier(..., bidirectional:true)` on rebuild / subtree-removal / reference-removal, so
  inverse-notifier entries never leak across redeploys.
- **Ack/confirm/shelve route on `ConditionId = RawPath`** (never `SourceNodeId`) regardless of
  which notifier root the operator subscribed at — an ack issued from an equipment-folder
  subscription resolves to the same raw condition.
- The `alerts`-topic `AlarmTransitionEvent` carries the (possibly empty) **list of referencing
  equipment paths**, and the AdminUI `/alerts` page shows **one row per condition** (primary
  identity RawPath + condition NodeId) with the equipment list as display metadata.

## Condition event identity fields (what a client reads on the wire)

Every condition event — native and scripted — carries the mandatory `BaseEventType` identity
fields, assigned at materialize time in `OtOpcUaNodeManager.MaterialiseAlarmCondition`. The SDK
does **not** synthesize them on this path (`Create` builds the children from the type definition
but leaves them unset; `ReportEvent` / `InstanceStateSnapshot` copy children verbatim), so they
are set explicitly. Leaving them unset shipped them as **null** on every event — see issues #473
(the `BaseEventType` trio) and #475 (the `ConditionType` classification pair).

| Field | Value | Notes |
|---|---|---|
| `EventType` | the **concrete** materialized type (`TypeDefinitionId`) | e.g. `OffNormalAlarmType`; falls back to `AlarmConditionType` for an unknown authored type. Readable as a *field*, not only via an `OfType` where-clause |
| `SourceNode` | the condition's **own NodeId** — equal to `ConditionId` | The condition **is** the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at |
| `SourceName` | the same identifying id string: **RawPath** (native) / **ScriptedAlarmId** (scripted) | Deliberately the *unique* id, **not** the leaf name |
| `ConditionName` | the leaf / display name (e.g. `HR200`) | Where the short human-readable name lives |
| `ConditionClassId` | always **`BaseConditionClassType`** | Part 9's "no condition class modelled" value. Unset shipped `NodeId.Null` (#475) |
| `ConditionClassName` | always **`"BaseConditionClass"`** | Matches `ConditionClassId`. Unset shipped empty text (#475) |

**Why `BaseConditionClassType` and not `ProcessConditionClassType`.** We hold no per-alarm
classification at the materialize seam, and `ConditionClassId` is a wire contract clients bucket on.
`BaseConditionClassType` is the honest, spec-conformant report of *"this server does not model
condition classes"* — it fixes the real defect (a null that breaks conformant clients) without
asserting a classification we cannot back. `ProcessConditionClassType` — the SDK sample's pick —
was rejected deliberately: it would be *actively wrong* for a Galaxy alarm whose upstream category is
Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm
classification is a separate future feature: it needs the driver's alarm category, which today lives
only on the runtime `AlarmEventArgs` transition, carried to the deploy-time authored composition that
`MaterialiseAlarmCondition` sees. Until then the `IAlarmSource` doc comment claiming the category
"maps to `ConditionClassName` downstream" describes an intent, not the implementation.

**Why `SourceName` is the id, not the leaf name.** The leaf is ambiguous across devices (`HR200` on
two PLCs collides) and is already carried by `ConditionName`, so the leaf-name option would add no
identity while costing uniqueness. Carrying the id makes the `SourceNode`/`SourceName`/`ConditionId`
triple mutually consistent and unique. This diverges from the loose OPC UA convention that
`SourceName` mirrors the source node's BrowseName; the divergence is intentional.

**Client guidance:** key on **`ConditionId`** — the condition node's own NodeId, which equals
`SourceNode`, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for
scripted ones. It is the identity ack/confirm/shelve route on. `SourceName` carries the same
identifier and is unique, so it is safe to key on *by itself* — but do **not** compose it with
`ConditionName` (`$"{SourceName}.{ConditionName}"`), because `SourceName` already ends in the
condition's leaf name and the result stutters (`pymodbus/plc/HR200.HR200`).

Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests` asserts the three `BaseEventType`
fields arrive populated on a real subscription using the standard `[EventType, SourceNode,
SourceName, Time, Message, Severity]` select clause, and — in a second test with its own clause —
that `ConditionClassId` / `ConditionClassName` do too. The two class fields are declared on
`ConditionType`, **not** `BaseEventType`, so a client must select them against that type.
`NodeManagerAlarmSourceFieldsTests` guards the node itself across both realms.

> **Do not correlate live events to HistoryRead on `SourceName` — the two paths disagree.**
> The HistoryRead *events* projection (`OtOpcUaNodeManager.ProjectEventField`) returns
> `Variant.Null` for `EventType` / `SourceNode` **by design**: it projects from the historian's
> `HistoricalEvent` rows, which do not carry them. It **does** project `SourceName` — but the
> alarm-history writer stamps that field with the **EquipmentPath**
> (`AlarmEventMapper`: `SourceName = alarm.EquipmentPath`), not the RawPath a live event carries.
> So `SourceName` is the one field populated on both paths **with different values**, and it is not
> a live↔history join key. Correlate on `ConditionId` / the RawPath instead. (Pre-existing; the
> live-path fix above does not change the history path.)

## Galaxy driver path (driver-native)

Restored in PR B.2 of the epic. `GalaxyDriver` implements
`IAlarmSource` with these surfaces:

- `SubscribeAlarmsAsync(sourceNodeIds)` → returns a sentinel handle.
  The driver doesn't multiplex per source-node-id today; every
  active handle observes the gateway's alarm-event stream. The
  server-side `AlarmConditionService` filters by source-node before
  raising the OPC UA condition.
- `UnsubscribeAlarmsAsync(handle)` → symmetric handle removal.
- `AcknowledgeAsync(requests)` → routes one gateway RPC per
  acknowledgement through `IGalaxyAlarmAcknowledger`. Production
  uses `GatewayGalaxyAlarmAcknowledger` calling
  `MxGatewayClient.AcknowledgeAlarmAsync` (PR E.2 SDK method).
- `OnAlarmEvent` → bridges `EventPump.OnAlarmTransition` (PR B.1)
  onto `AlarmEventArgs`. Suppressed when no alarm subscription is
  active so untracked transitions don't leak through.

The proto contract carries the rich payload — alarm full reference,
source-object reference, alarm-type-name, transition kind (Raise /
Acknowledge / Clear / Retrigger), severity (raw MxAccess scale),
original raise timestamp, transition timestamp, operator user,
operator comment, alarm category, description. `MxAccessSeverityMapper`
(PR B.1) translates the raw severity onto the four-bucket
`AlarmSeverity` ladder — boundaries match v1's `GalaxyAlarmTracker`
so customers see no surprise re-classification.

The richer fields surface on `Core.Abstractions.AlarmEventArgs` via
the optional properties added in PR E.7 (`OperatorComment`,
`OriginalRaiseTimestampUtc`, `AlarmCategory`). Consumers that don't
need them are unaffected; consumers that do (Client.UI, Client.CLI
verbose mode) read the new fields when present.

## Galaxy sub-attribute fallback

For Galaxy templates without `$Alarm*` extensions, the value-driven
path stays in place: `DriverNodeManager` registers an
`AlarmConditionState` per Galaxy variable that bears alarm-bearing
sub-attributes (`InAlarm`, `Acked`, `Priority`, `Description`),
subscribes to those sub-attributes, and synthesizes Part 9 transitions
when the values change. This path operated as the only Galaxy alarm
path between PR 7.2 and the alarms-over-gateway epic; it remains the
fallback today.

When both paths report the same condition,
`AlarmConditionService.AlarmConditionState` keeps the
driver-native record and discards the duplicate sub-attribute
synthesis. Driver-native transitions are richer (carry operator
comment + original raise time) and arrive lower-latency (no
publishing-interval delay on the sub-attribute reads), so they win
the dedup.

## Acknowledge routing — Galaxy / driver alarms

### Native alarm acknowledge → AVEVA

When an OPC UA client Acknowledges a **native** (driver-fed, e.g. Galaxy)
`AlarmConditionState` node, the node manager's `OnAcknowledge` handler
branches on native-ness and routes through a dedicated path — separate
from the scripted `AlarmCommandRouter`:

1. **`OtOpcUaNodeManager.HandleNativeAlarmAck`** — gates on the caller's
   `AlarmAck` role (fails closed: no role → `BadUserAccessDenied`), then
   dispatches a `NativeAlarmAck(ConditionNodeId, Comment, OperatorUser)`
   to the `NativeAlarmAckRouter` seam (fire-and-forget, non-blocking under
   the node-manager Lock). `OperatorUser` carries the authenticated
   session principal's display name.
2. **`DriverHostActor.HandleRouteNativeAlarmAck`** — receives a
   `RouteNativeAlarmAck` message (the host maps `NativeAlarmAck` at the
   wiring boundary to keep Runtime Akka-free of the OPC UA layer). Applied
   **Primary-gate first**: a Secondary or Detached node drops the message
   silently. On Primary, resolves the condition NodeId from the
   `_driverRefByAlarmNodeId` inverse map (NodeId → `(DriverInstanceId,
   FullName)`) and Tells the owning `DriverInstanceActor` a
   `RouteAlarmAck(FullName, Comment, OperatorUser)`.
3. **Galaxy driver** — `DriverInstanceActor` calls the driver's
   `IAlarmSource.AcknowledgeAsync` with an `AlarmAcknowledgeRequest`
   carrying the authored `FullName` as the `ConditionId` and the
   authenticated `OperatorUser`. The driver forwards this to the Galaxy
   gateway → AVEVA via `GatewayGalaxyAlarmAcknowledger`.
   **Fire-and-forget** — a failed upstream ack is not surfaced back to the
   OPC UA client (mirrors the Galaxy write-outcome limitation; the local
   `AlarmConditionState` SDK update already committed at step 1).

Only the **Acknowledge** is routed to the driver. `Confirm` / `AddComment`
/ `Shelve` operations on a native condition stay on the scripted
`AlarmCommandRouter` path (Phase 3 scope is Acknowledge → AVEVA only).

> **Best-effort during reconnect.** If a native ack arrives while the owning
> driver is `Connecting`/`Reconnecting` (or on a non-Primary node), it is
> dropped with a debug log and **not** retried upstream — the OPC UA Part 9
> `AlarmConditionState` was already committed locally at step 1, so the rare
> outcome is "acked locally but not propagated to AVEVA." Re-issue the ack once
> the driver is reconnected if upstream propagation is required.

### Legacy sub-attribute path

`DriverNodeManager` picks the acknowledger when registering each
condition (PR B.3 logic):

- Driver implements `IAlarmSource` →
  `DriverAlarmSourceAcknowledger` routes the operator comment
  through `IAlarmSource.AcknowledgeAsync` via the existing
  `AlarmSurfaceInvoker` (Phase 6.1 resilience pipeline; no-retry
  per decision #143). End-to-end operator-comment fidelity is
  preserved.
- Driver doesn't implement `IAlarmSource` →
  `DriverWritableAcknowledger` writes the comment into the
  `AckMsgWriteRef` sub-attribute via `IWritable.WriteAsync`. Same
  resilience pipeline; collapses the comment into a single string
  write at the wire level.

The OPC UA Part 9 `AlarmConditionState.OnAcknowledge` delegate
already validates the session's `AlarmAck` role before dispatching,
so the gateway-side ack RPC only sees authenticated, authorised
calls.

## Inbound operator ack/shelve — scripted alarms

Scripted alarms use a separate inbound path that converges on the
`alarm-commands` DPS topic. Two surfaces route onto this topic:

### OPC UA Part 9 method path (external OPC UA clients)

`OtOpcUaNodeManager` wires the Part 9 condition methods (Acknowledge /
Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each
scripted-alarm `AlarmConditionState` node. Every call is **gated on the
`AlarmAck` LDAP role** — fail-closed: sessions with no role or without
`AlarmAck` group membership receive `BadUserAccessDenied` immediately.
The LDAP-resolved role set is carried past `OpcUaApplicationHost` by
`RoleCarryingUserIdentity` (a `UserIdentity` subclass), making it
readable inside the method handler at dispatch time.

On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` onto the
`alarm-commands` DPS topic. The node manager is Akka-free; the dispatch
action is a settable `Action<AlarmCommand>` injected at boot by the
hosted service.

`OnTimedUnshelve` (the SDK's automatic unshelve timer) bypasses the
operator gate — it is system-initiated.

`WriteAlarmCondition` fires the Part 9 condition event only when the
incoming state differs from the node's current live state (delta-gate),
preventing the double-emit that would otherwise occur when the SDK
auto-applies the acked state and the engine re-projection fires a
duplicate event immediately after.

### AdminUI path

The `/alerts` page shows per-row **Acknowledge / Shelve / Unshelve**
buttons gated by the `DriverOperator` AdminUI policy. These route
through the `AdminOperationsActor` cluster singleton
(`AcknowledgeAlarmCommand` / `ShelveAlarmCommand`), which publishes onto
the same `alarm-commands` topic. The singleton handles cross-node
routing — the command always reaches the driver-role node owning the
engine regardless of which AdminUI instance the operator is on.

### ScriptedAlarmHostActor dispatch

`ScriptedAlarmHostActor` subscribes to the `alarm-commands` topic,
ownership-filters each command (each node only acts on its own alarms),
and dispatches to the matching `ScriptedAlarmEngine` operation
(`AcknowledgeAsync` / `ConfirmAsync` / `OneShotShelveAsync` /
`TimedShelveAsync` / `UnshelveAsync` / `EnableAsync` / `DisableAsync` /
`AddCommentAsync`). The engine's existing `OnEvent` callback handles
the OPC UA node update — no explicit re-projection is required.

The AdminUI `/alerts` Shelve flow was live-verified on docker-dev
2026-06-11: singleton → topic → host actor → engine → "Shelved" status
reflected on `/alerts` with the operator identity threaded through.

## Redundancy deduplication

Under warm/hot redundancy, both cluster nodes run `ScriptedAlarmHostActor` and the scripted-alarm engine. To prevent duplicate `/alerts` rows and duplicate historian writes, alarm transition publication to the `alerts` topic and `HistorianAdapterActor` historization are **Primary-gated**: only the node whose `RedundancyRole` is `Primary` publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See [Redundancy.md §Primary-gated alarm emission and historization](Redundancy.md#primary-gated-alarm-emission-and-historization).

## Historian write-back (non-Galaxy alarms)

Scripted alarms (and any future non-Galaxy `IAlarmSource` like
AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:

- `IAlarmHistorianSink` is the DI-registered intake contract. The
  default binding is `NullAlarmHistorianSink` (registered in
  `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
  deployments override it with `SqliteStoreAndForwardSink` wrapping
  `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
  — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
- `SqliteStoreAndForwardSink` queues each transition to a local
  SQLite database and drains in the background via an
  `IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
  queue capacity defaults to 1,000,000 rows; under a sustained
  historian outage, older non-dead-lettered rows are evicted (oldest
  first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
  counter surfaces lifetime eviction events so operators can detect
  silent data loss without log scraping. The drain cadence, queue
  capacity, and dead-letter retention are tunable via the `AlarmHistorian`
  config section (`DrainIntervalSeconds`, `Capacity`,
  `DeadLetterRetentionDays`); `AlarmHistorianOptions.Validate()` logs a
  startup warning for an empty `SharedSecret`, a relative `DatabasePath`,
  or a non-positive knob.
- `HistorianAdapterActor`
  (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`)
  subscribes to the cluster `alerts` DPS topic, translates each
  `AlarmTransitionEvent` → `AlarmHistorianEvent`, and calls
  `EnqueueAsync` fire-and-forget. The durable write is gated two ways:
  (1) **Primary-gated** — only the Primary node historizes, giving
  exactly-once writes across a redundant pair; (2) **per-alarm** — a
  transition whose `HistorizeToAveva` is `false` is skipped (the flag
  rides on `AlarmTransitionEvent` as a nullable bool; missing/`null`/`true`
  historize, only an explicit `false` suppresses, so a cross-version
  rolling restart defaults to historizing rather than dropping an audit
  row). Neither gate touches the live `alerts` publish — the `/alerts`
  UI always sees every transition. See
  [AlarmHistorian.md §Configuration](AlarmHistorian.md#configuration)
  for the `AlarmHistorian` appsettings section that enables the real sink.

  **Native alarms** (equipment tags carrying an `"alarm"` object in their `TagConfig`)
  support the same `HistorizeToAveva` opt-out. The field is `alarm.historizeToAveva`
  (`bool?`) and is authored via the **"Historize to AVEVA"** checkbox in the Tag modal's
  alarm section. The gate logic is identical (`is not false`): absent or `true` historizes;
  explicit `false` suppresses the AVEVA write while leaving the live `/alerts` feed
  unaffected. See
  [ScriptedAlarms.md §TagConfig alarm fields](ScriptedAlarms.md#tagconfig-alarm-fields)
  for the full field reference.

Galaxy-native alarms with `$Alarm*` extensions reach AVEVA Historian
directly via System Platform's `HistorizeToAveva` toggle on the
alarm primitive — no involvement from OtOpcUa. This sidecar path is
exclusively for non-Galaxy alarm producers.

## Cross-references

- Plan: [docs/plans/alarms-over-gateway.md](plans/alarms-over-gateway.md)
- v1 archive: [docs/v1/AlarmTracking.md](v1/AlarmTracking.md)
- Galaxy driver: [docs/drivers/Galaxy.md](drivers/Galaxy.md)
- Phase 7 scripting + alarming: [docs/v2/implementation/phase-7-scripting-and-alarming.md](v2/implementation/phase-7-scripting-and-alarming.md)
- Security + ACL: [docs/security.md](security.md)