lmxopcua/docs/AlarmTracking.md

# Alarm tracking — v2 final architecture

This document describes how OtOpcUa surfaces alarms to OPC UA Part 9
clients after the **alarms-over-gateway** epic
([docs/plans/alarms-over-gateway.md](plans/alarms-over-gateway.md))
landed. The v1 architecture (Galaxy.Host's COM-side `GalaxyAlarmTracker`)
is preserved at [docs/v1/AlarmTracking.md](v1/AlarmTracking.md) for
historical reference.

## Three alarm sources, one OPC UA Part 9 surface

| Source                           | Driver capability        | Path |
|----------------------------------|--------------------------|------|
| **Galaxy MxAccess (driver-native)** | `GalaxyDriver : IAlarmSource` | gateway → worker → MxAccess alarm sink → `MX_EVENT_FAMILY_ON_ALARM_TRANSITION` → `EventPump` → driver `OnAlarmEvent` → `AlarmConditionService` |
| **MTConnect CONDITION (driver-native, #626)** | `MTConnectDriver : IAlarmSource` | Agent `/sample` pump → `MTConnectConditionTracker` → `MTConnectConditionAlarmMapper` → driver `OnAlarmEvent` (ConditionId = RawPath) → `DriverHostActor.ForwardNativeAlarm` |
| **Galaxy sub-attribute fallback**  | `IWritable` writes to `$Alarm*` sub-attributes | gateway data subscription → driver `OnDataChange` → `DriverNodeManager` ConditionSink → `AlarmConditionService` |
| **Scripted alarms**                | `Phase7Composer` | server-side script evaluator → `ScriptedAlarmActor` transitions → `HistorianAdapterActor` → `IAlarmHistorianSink` |

All three converge on the alarm-state actor — in v2 the OPC UA Part 9 state
machine lives inside `ScriptedAlarmActor`
(`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmActor.cs`),
which dispatches transitions to the OPC UA condition node managers. Driver-native transitions take
precedence over sub-attribute synthesis when both arrive for the same
condition — the dedup logic prefers the richer driver-native record
because it carries the full operator + raise-time + category metadata
that the value-driven path collapses.

## v3 Batch 4 — multi-notifier delivery (raw + equipment folders)

Under the v3 dual-namespace address space, a native driver alarm is authored on a
**raw tag** (`/raw`) and its Part 9 `AlarmConditionState` materializes **once** at the
raw tag — `ConditionId` = the tag's **RawPath**, parent notifier = the raw device/group
folder (`ns=Raw`). Because equipment references raw tags (v3 reference-only UNS), the
same condition is wired as an **event notifier of every referencing equipment folder**
(`ns=UNS`) via the SDK `AddNotifier` pattern (`OtOpcUaNodeManager.WireAlarmNotifiers`):
`alarm.AddNotifier(equipFolder, isInverse:true)` + `equipFolder.AddNotifier(alarm)` +
`EnsureFolderIsEventNotifier(equipFolder)`.

- A **single** `ReportEvent` fans through the SDK notifier graph to the raw device folder,
  every referencing equipment folder, and to the Server object. A subscriber at any one
  root receives **exactly one** copy of the transition — the Part 9 shared-`InstanceStateSnapshot`
  dedup (`MonitoredItem.QueueEvent` → `IsEventContainedInQueue`), **never** one copy per root.
  The Server-object (`i=2253`) copy is delivered by the condition itself, not by the folders'
  forward to the core Server object, so it can be filtered per subscriber by the data-plane ACL
  (Gitea #586 — see `docs/security.md`).
  Duplicating the `ReportEvent` per root is rejected by design (distinct EventIds would break
  Server-object dedup + Part 9 ack correlation).
- **Teardown is symmetric:** the node manager tracks the wired notifier pairs and calls
  `RemoveNotifier(..., bidirectional:true)` on rebuild / subtree-removal / reference-removal, so
  inverse-notifier entries never leak across redeploys.
- **Ack/confirm/shelve route on `ConditionId = RawPath`** (never `SourceNodeId`) regardless of
  which notifier root the operator subscribed at — an ack issued from an equipment-folder
  subscription resolves to the same raw condition.
- The `alerts`-topic `AlarmTransitionEvent` carries the (possibly empty) **list of referencing
  equipment paths**, and the AdminUI `/alerts` page shows **one row per condition** (primary
  identity RawPath + condition NodeId) with the equipment list as display metadata.
- **Routing is an exact `ConditionId == RawPath` match** (`DriverHostActor.ForwardNativeAlarm`). A driver that
  decorates the id has every transition dropped as "no alarm condition" — AbCip did (`"{ref}#active"`, fixed
  under #641); FOCAS still does (see `drivers/FOCAS.md`). A native source must also set `Kind`: an
  `Unspecified` kind projects no state change.

### Native condition state survives a full rebuild (Gitea #641)

A full `RebuildAddressSpace` recreates every condition inactive + acknowledged. For **scripted** alarms the
engine re-asserts every non-normal condition after the reload (#487). **Native** conditions had nothing
equivalent: the host re-subscribes a driver's alarms only when its alarm-ref *set* changes, and a
transition-only source (Galaxy, OpcUaClient) has nothing to replay anyway — so a rebuild that left the alarm
set unchanged (adding one `UnsTagReference` to an alarm-bearing raw tag is enough to force one, as are
`ChangedEquipment`, a scripted-alarm edit, a raw container rename, an alarm severity/type edit, or any
surgical fallback to `SafeRebuild()`) left every active native alarm reading **normal** until its next device
transition.

The node manager now carries native state across the rebuild itself:

1. `RebuildAddressSpace` captures each **native** condition's live state off the **node** — Active, Acked,
   Confirmed, Severity, Message, Comment + ClientUserId, Quality, Time, `ActiveState.TransitionTime` — before clearing. Reading the
   node, not a host-side cache, is what keeps a **client's** acknowledgement: the SDK applies it to the node
   and neither the driver nor `NativeAlarmProjector` ever sees it.
2. `MaterialiseAlarmCondition(isNative: true)` for the same id in the same apply restores it. **Quality is
   always restored** (it is otherwise re-annotated only on a connectivity edge, so even a normal condition
   would sit at `BadWaitingForInitialData`); the rest only when the condition was off-normal (active or
   unacknowledged). A normal condition keeps what the materialise built — message and authored severity
   included — exactly the #487 rule. Shelving/Enabled are not carried (native conditions refuse them, #587).
3. `IOpcUaAddressSpaceSink.CompleteAddressSpaceApply`, which `OpcUaPublishActor` calls at the end of
   **every** apply (in a `finally`), reports **one** condition event per restored condition — after the
   notifier wiring, so the raw folder, every referencing equipment folder and `i=2253` items all get it,
   stamped with the original `Time`, not the deploy time (or the node's newer `Time`, if a client method call
   reported in the restore→complete window — that window can yield a second, state-correct event) — and **discards** every capture no materialise
   claimed, so a later deploy re-adding the same RawPath gets a fresh node (no resurrection). A kind swap
   (native ↔ scripted on one id) is never restored. Node-only: no `/alerts` row, no historian record.

Two companions keep the host and driver in step with the carried node:

- `DriverHostActor` **prunes** `NativeAlarmProjector` to the still-deployed conditions on each apply
  (`RetainOnly`) instead of clearing it. Clearing made the next driver Acknowledge of an active alarm
  project `Active=false`, and the next Clear of an unacknowledged one project `Acked=true` — after *any*
  deploy, rebuild or not.
- MTConnect's "last told" record is pruned to the conditions that **left** the subscribed set rather than
  cleared on any set change; re-raising a surviving (carried) condition would un-acknowledge it. AbCip's ALMD
  last-seen `InFaulted` record is likewise driver-owned and pruned to the set, so a re-subscribe (every
  reconnect / set change) does not re-raise a still-faulted alarm.
- A routed **client** acknowledgement marks the projector's prior state acknowledged
  (`NativeAlarmProjector.MarkAcknowledged`), so a later driver Clear does not project it unacknowledged.

Accepted residual: a condition that clears at the device while the rebuild runs is restored active until
the driver's Clear arrives — which it does, because the driver's own record still holds it active. The capture
is keyed by RawPath only, so a tag deleted and re-created under the same RawPath in one deploy inherits the old
condition's state until its next transition.

## Condition event identity fields (what a client reads on the wire)

Every condition event — native and scripted — carries the mandatory `BaseEventType` identity
fields, assigned at materialize time in `OtOpcUaNodeManager.MaterialiseAlarmCondition`. The SDK
does **not** synthesize them on this path (`Create` builds the children from the type definition
but leaves them unset; `ReportEvent` / `InstanceStateSnapshot` copy children verbatim), so they
are set explicitly. Leaving them unset shipped them as **null** on every event — see issues #473
(the `BaseEventType` trio) and #475 (the `ConditionType` classification pair).

| Field | Value | Notes |
|---|---|---|
| `EventType` | the **concrete** materialized type (`TypeDefinitionId`) | e.g. `OffNormalAlarmType`; falls back to `AlarmConditionType` for an unknown authored type. Readable as a *field*, not only via an `OfType` where-clause |
| `SourceNode` | the condition's **own NodeId** — equal to `ConditionId` | The condition **is** the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at |
| `SourceName` | the same identifying id string: **RawPath** (native) / **ScriptedAlarmId** (scripted) | Deliberately the *unique* id, **not** the leaf name |
| `ConditionName` | the leaf / display name (e.g. `HR200`) | Where the short human-readable name lives |
| `ConditionClassId` | always **`BaseConditionClassType`** | Part 9's "no condition class modelled" value. Unset shipped `NodeId.Null` (#475) |
| `ConditionClassName` | always **`"BaseConditionClass"`** | Matches `ConditionClassId`. Unset shipped empty text (#475) |
| `Quality` | the condition's **source-data quality** — native tracks the source's connectivity (`Good` / `Bad`); scripted takes the worst of its input tags' qualities (#478) | A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below |

**Why `BaseConditionClassType` and not `ProcessConditionClassType`.** We hold no per-alarm
classification at the materialize seam, and `ConditionClassId` is a wire contract clients bucket on.
`BaseConditionClassType` is the honest, spec-conformant report of *"this server does not model
condition classes"* — it fixes the real defect (a null that breaks conformant clients) without
asserting a classification we cannot back. `ProcessConditionClassType` — the SDK sample's pick —
was rejected deliberately: it would be *actively wrong* for a Galaxy alarm whose upstream category is
Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm
classification is a separate future feature: it needs the driver's alarm category, which today lives
only on the runtime `AlarmEventArgs` transition, carried to the deploy-time authored composition that
`MaterialiseAlarmCondition` sees. Until then the `IAlarmSource` doc comment claiming the category
"maps to `ConditionClassName` downstream" describes an intent, not the implementation.

**Why `SourceName` is the id, not the leaf name.** The leaf is ambiguous across devices (`HR200` on
two PLCs collides) and is already carried by `ConditionName`, so the leaf-name option would add no
identity while costing uniqueness. Carrying the id makes the `SourceNode`/`SourceName`/`ConditionId`
triple mutually consistent and unique. This diverges from the loose OPC UA convention that
`SourceName` mirrors the source node's BrowseName; the divergence is intentional.

**Client guidance:** key on **`ConditionId`** — the condition node's own NodeId, which equals
`SourceNode`, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for
scripted ones. It is the identity ack/confirm/shelve route on. `SourceName` carries the same
identifier and is unique, so it is safe to key on *by itself* — but do **not** compose it with
`ConditionName` (`$"{SourceName}.{ConditionName}"`), because `SourceName` already ends in the
condition's leaf name and the result stutters (`pymodbus/plc/HR200.HR200`).

Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests` asserts the three `BaseEventType`
fields arrive populated on a real subscription using the standard `[EventType, SourceNode,
SourceName, Time, Message, Severity]` select clause, and — in a second test with its own clause —
that `ConditionClassId` / `ConditionClassName` do too. The two class fields are declared on
`ConditionType`, **not** `BaseEventType`, so a client must select them against that type.
`NodeManagerAlarmSourceFieldsTests` guards the node itself across both realms.

> **Do not correlate live events to HistoryRead on `SourceName` — the two paths disagree.**
> The HistoryRead *events* projection (`OtOpcUaNodeManager.ProjectEventField`) returns
> `Variant.Null` for `EventType` / `SourceNode` **by design**: it projects from the historian's
> `HistoricalEvent` rows, which do not carry them. It **does** project `SourceName` — but the
> alarm-history writer stamps that field with the **EquipmentPath**
> (`AlarmEventMapper`: `SourceName = alarm.EquipmentPath`), not the RawPath a live event carries.
> So `SourceName` is the one field populated on both paths **with different values**, and it is not
> a live↔history join key. Correlate on `ConditionId` / the RawPath instead. (Pre-existing; the
> live-path fix above does not change the history path.)

### Condition child nodes are instance-scoped (#629)

Every child of a materialised condition — `ActiveState`, `ActiveState/Id`, `AckedState/Id`,
`ShelvingState/UnshelveTime`, `Message`, `Severity`, `Retain`, the method nodes, … — carries the NodeId
**`{ConditionId}/{browse path}`** in the condition's own namespace: e.g.
`ns=<raw>;s=Plant/Modbus/dev1/temp_hi/ShelvingState/UnshelveTime`, or
`ns=<uns>;s=<ScriptedAlarmId>/ActiveState/Id`. So `TranslateBrowsePathsToNodeIds` from the condition
resolves, and a direct Read or data-change subscription on a child returns **this condition's** value.

Before #629 the children kept ns=0 ids: `Create(…, assignNodeIds: true)` asks the node manager's
`INodeIdFactory.New`, and the SDK default returns the node's current id — the condition type's
instance-declaration id (`i=9189` for UnshelveTime) or an SDK-generated ns=0 number. Every translate of
a child path returned `BadNoMatch`; a Read of a browsed child id was either `BadNodeIdUnknown` or served
from the **type** (UnshelveTime read `0` on a condition timed-shelved for a minute). `OtOpcUaNodeManager.New`
now derives the id. Event delivery was never affected (events copy the instance's fields; clients select
them by type + browse path), and neither were the Part 9 methods — calling them by the type's method id
(`MethodIds.AcknowledgeableConditionType_Acknowledge`, …) still works, because the SDK matches a method on
its declaration id. ⚠️ The ShelvingState methods needed help here: their SDK declaration id is the
`ShelvedStateMachineType`'s (`i=2948`…), and they were only ever found by the `AlarmConditionType_ShelvingState_*`
id (`i=9212`…) because that WAS their NodeId. `New` adopts that original id as the method's
`MethodDeclarationId` (a fixed set of those three ids — no address-space lookup, which would take other
node managers' locks under ours), so OneShotShelve/TimedShelve/Unshelve by the
`AlarmConditionType_ShelvingState_*` id keep working, as do calls by the condition's own instance method
ids (`{ConditionId}/ShelvingState/TimedShelve`, on the condition or its ShelvingState). Calling them by the
`ShelvedStateMachineType_*` id returns `BadMethodInvalid` — a pinned trade-off.

- **ACL:** the slash-joined id makes a child's ACL scope its condition's scope plus the browse path, so
  with `Security:Acl:Enabled` a child is governed exactly like its condition (a grant covering the
  condition covers its children; one that does not, denies them). Before #629 a child was in ns=0 and
  bypassed the data-plane gate entirely.
- **Removal:** removing or rebuilding a condition forgets its whole subtree, so a removed condition's
  children answer `BadNodeIdUnknown`, not their last value.
- Guarded over the wire by `ConditionChildNodeIdTests`.

### Condition source-data Quality (#477)

`ConditionType.Quality` reports the quality of the condition's source data. It was never assigned, and
because `StatusCodes.Good == 0x00000000` an unassigned `StatusCode` **is** `Good` — so every condition
reported `Good` unconditionally (a wrong *value*, not a null like #473/#475). A native alarm whose device
went offline still read `Good`, so an operator could not tell *"genuinely inactive"* from *"we have lost
contact and do not know"*.

**How it is driven now (native alarms).** An alarm-bearing raw tag materializes a condition with **no
sibling value variable**, so the value/quality path (`WriteValue`) never touches it, and a comms-lost
driver emits **no alarm transitions** (the feed goes silent). The quality therefore comes from the
**driver's connectivity**, out of band from alarm transitions:

- `DriverInstanceActor` Tells its host `ConnectivityChanged(driverInstanceId, connected)` on every
  transition into `Connected` (`true`) / `Reconnecting` (`false`).
- `DriverHostActor.OnDriverConnectivityChanged` fans that out to **every** native condition the driver
  owns as an `OpcUaPublishActor.AlarmQualityUpdate` (`Good` on connect, `Bad` on disconnect).
- `OtOpcUaNodeManager.WriteAlarmQuality` sets **only** the condition's `Quality` and fires a Part 9
  event **only on a quality-bucket change** — it never touches Active/Acked/Severity/Retain (an active
  alarm that loses comms stays active). This is a dedicated path, *not* a full-snapshot re-projection, so
  it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
- A freshly materialized native condition starts `BadWaitingForInitialData` (the "no driver data yet"
  convention value variables use); the first `Connected` confirms it `Good`.
- The connectivity annotation is **ungated by redundancy role** (a Secondary keeps its condition quality
  warm for failover) and publishes **no `/alerts` row** — driver comms health already has its own status
  surface (`IDriverHealthPublisher`); a row per condition would be alarm-fatigue.

**Scripted alarms (Layer 3, #478).** A scripted condition's state is computed from one or more input tags,
so its `Quality` is the **worst** quality across those inputs at evaluation time ("can I trust this
condition's state?") — mirroring the native OT semantic:

- The mux now forwards each input's source quality (`DependencyValueChanged.Quality`), and the scripted host
  pushes it into the engine's read cache (previously every mux value was treated as `Good`).
- The `ScriptedAlarmEngine` computes the worst input quality each evaluation. A **real transition** carries
  it on the emitted event → `ScriptedAlarmHostActor.ToSnapshot` projects it (so a transition fired while an
  input is `Uncertain` does not clobber quality back to `Good`).
- A **Bad input freezes the condition** (`AreInputsReady` holds its state — no transition), exactly like a
  comms-lost native driver. So a quality-bucket change with no transition is emitted as
  `EmissionKind.QualityChanged` and routed to the **same** dedicated `AlarmQualityUpdate → WriteAlarmQuality`
  node path native uses (quality only, one Part 9 event on a bucket change, **no `/alerts` row**, no
  historian write). `ScriptedAlarmSource` skips `QualityChanged` so it never fabricates a phantom
  `IAlarmSource` event.
- An input that has **not been published yet** (cold start) is *not* a quality signal (that is the readiness
  guard's job) — it contributes `Good`, so scripted conditions don't flash `Bad` at every deploy. The first
  actually-`Bad` published value flips the bucket and annotates.

**Comms loss (Layer 4, #481).** #478 covered input tags whose driver **publishes a data change carrying a
Bad/Uncertain `StatusCode`** (e.g. an OpcUaClient input forwarding a server's per-item Bad). It did not
cover a driver **comms loss**: a poll driver (Modbus/S7) whose device goes unreachable emits only
`ConnectivityChanged` and goes *silent* on the value feed (see `DriverInstanceActor.Reconnecting`), so the
engine kept the last-known Good value, `WorstInputStatus` stayed Good — and worse, `AreInputsReady` stayed
true, so the predicate went on evaluating a stale value after the source had died.

`DriverHostActor.FanCommsLossToDependencyMux` closes it, symmetrically with the native
`OnDriverConnectivityChanged` path above: on `ConnectivityChanged(false)` it pushes a synthetic
`BadNoCommunication` with a **null value** into the dependency mux for every value ref the driver owns,
resolved from the existing `_nodeIdByDriverRef` map. Alarm-bearing tags are deliberately excluded — a
native condition is not a mux value, and its comms quality goes through the Part 9 annotation instead.

Two things worth knowing before changing this:

- **Nothing is pushed on reconnect, deliberately** — publishing Good would assert a value quality the host
  has not observed. Recovery rides the driver's own republish: the `Connected → Reconnecting` edge calls
  `DetachSubscription`, so the reconnect's `ResubscribeDesired` builds a *fresh* subscription whose
  `PollGroupEngine` initial push runs with `forceRaise: true` and re-publishes every tag with its real
  quality. That detach is load-bearing — without it the poll engine would compare against its retained
  `LastValues`, see an unchanged value, raise nothing, and strand those refs Bad forever.
- **Virtual tags computed from a comms-lost input now go Bad too.** That follows from using the shared mux
  and is the correct reading of "this input's source is unreachable", but it is a behaviour change beyond
  scripted alarms.

**The signal this hangs off now fires for a mid-run device loss (#527, closed).** The #481 live gate on
docker-dev found that `DriverInstanceActor.DisconnectObserved` had **no production sender**: the only paths
into `Reconnecting` were a failed *initial* connect and an operator Force Reconnect. A Modbus driver whose
fixture was stopped stayed `Connected` for >10 minutes publishing per-tag Bad, and did not recover when the
fixture came back until the node was restarted. So the fan-out above — and the native #477 Layer 2
annotation it mirrors — were correctly wired to a signal that a dying device never raised. #527 added the
comms-loss watchdog on the Connected health tick, which self-sends `DisconnectObserved` once a driver has
gone 90 s without a successful read *while* reporting itself non-Healthy. See
[`DriverLifecycle.md`](DriverLifecycle.md) §"The comms-loss watchdog" for why the trip condition is that
conjunction and not simply "non-Healthy for N polls" — the naive rule would reconnect a healthy device's
driver every 90 s forever whenever one authored address is unreadable.

Note the partial cover that existed before, because it explains why *some* of this appeared to work: Modbus
publishes Bad *per tag* rather than going silent, so #478 Layer 3 did carry Bad into scripted input quality
for that driver. That was a per-driver accident, not a guarantee, and it did nothing for native conditions
or for recovery.

**Cold start vs. a driver reporting Bad (#478 Finding 3, closed by #481).** `WorstInputStatus` used to skip
every input whose *value* was null as "no data yet". A comms-lost driver produces exactly that shape — Bad
with no value — so the skip was hiding the case that matters most. The upstream source now stamps a
never-pushed path `BadWaitingForInitialData` and nothing else does, so the engine skips *that specific
code* rather than every null. Deleting the null-skip outright would have been the obvious fix and is wrong
in the other direction: it makes every not-yet-published input drag its condition Bad at deploy, which is
what the skip existed to prevent. Both failure modes are pinned — see the engine-test pair below, each of
which goes red under the opposite mistake.

Guards: `ScriptedAlarmEngineTests` (transition carries `Uncertain`; `Bad` input with no transition emits
`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; unchanged bucket emits nothing; plus the #481
pair — a pushed Bad carrying no value counts, an input never pushed does not),
`DriverHostActorLiveValueTests` (comms loss pushes Bad for every ref the driver owns, reconnect pushes
nothing, another driver's loss pushes nothing),
`DependencyMuxTagUpstreamSourceTests.The_engine_agrees_with_this_source_on_the_never_pushed_status_code`,
`ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event`,
`DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged`, and
`ScriptedAlarmHostActorTests` (`Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts`,
`Transition_snapshot_carries_worst_input_quality`).

Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire`
subscribes with a `[Quality, Message]` clause (Quality is declared on `ConditionType`) and asserts a
healthy source reports `Good`, a comms-lost source reports non-`Good`, and recovery returns to `Good` — on
a real client subscription. `NodeManagerAlarmSourceFieldsTests` guards the node itself + the no-clobber /
unknown-node-no-op invariants.

## Galaxy driver path (driver-native)

Restored in PR B.2 of the epic. `GalaxyDriver` implements
`IAlarmSource` with these surfaces:

- `SubscribeAlarmsAsync(sourceNodeIds)` → returns a sentinel handle.
  The driver doesn't multiplex per source-node-id today; every
  active handle observes the gateway's alarm-event stream. The
  server-side `AlarmConditionService` filters by source-node before
  raising the OPC UA condition.
- `UnsubscribeAlarmsAsync(handle)` → symmetric handle removal.
- `AcknowledgeAsync(requests)` → routes one gateway RPC per
  acknowledgement through `IGalaxyAlarmAcknowledger`. Production
  uses `GatewayGalaxyAlarmAcknowledger` calling
  `MxGatewayClient.AcknowledgeAlarmAsync` (PR E.2 SDK method).
- `OnAlarmEvent` → bridges `EventPump.OnAlarmTransition` (PR B.1)
  onto `AlarmEventArgs`. Suppressed when no alarm subscription is
  active so untracked transitions don't leak through.

The proto contract carries the rich payload — alarm full reference,
source-object reference, alarm-type-name, transition kind (Raise /
Acknowledge / Clear / Retrigger), severity (raw MxAccess scale),
original raise timestamp, transition timestamp, operator user,
operator comment, alarm category, description. `MxAccessSeverityMapper`
(PR B.1) translates the raw severity onto the four-bucket
`AlarmSeverity` ladder — boundaries match v1's `GalaxyAlarmTracker`
so customers see no surprise re-classification.

The richer fields surface on `Core.Abstractions.AlarmEventArgs` via
the optional properties added in PR E.7 (`OperatorComment`,
`OriginalRaiseTimestampUtc`, `AlarmCategory`). Consumers that don't
need them are unaffected; consumers that do (Client.UI, Client.CLI
verbose mode) read the new fields when present.

## MTConnect CONDITION path (driver-native, #626)

An MTConnect raw tag bound to a `CONDITION` DataItem becomes a native condition when its `TagConfig`
carries the standard `alarm` object (opt-in; without it the tag stays the String state word). Normal →
inactive, Warning → active 500, Fault → active 800, Unavailable → state unchanged (an active condition stays
active and acked — it is comms loss one hop out, the #477/#481 posture) with `BadNoCommunication`
quality. Two optional `AlarmEventArgs` fields added for it are generic: **`OpcUaSeverity`** (an exact
1..1000 severity that supersedes the four-bucket mapping in `NativeAlarmProjector`) and
**`QualityStatusCode`** (projected onto the condition's `Quality` on a transition; with an `Unspecified`
kind it is a **quality-only annotation** — `DriverHostActor` routes it through the same
`AlarmQualityUpdate` → `WriteAlarmQuality` path as the connectivity annotation, which touches Quality and
nothing else, and publishes no `/alerts` row).
Acknowledge is server-side only (MTConnect has no ack; the driver's `AcknowledgeAsync` is a no-op).
The driver diffs against what the server was last told, so a reconnect's replayed `/current` never
re-raises (and so un-acknowledges) an unchanged condition. Full mapping:
[`drivers/MTConnect.md` §CONDITION alarms](drivers/MTConnect.md#condition-alarms-626).

## Galaxy sub-attribute fallback

For Galaxy templates without `$Alarm*` extensions, the value-driven
path stays in place: `DriverNodeManager` registers an
`AlarmConditionState` per Galaxy variable that bears alarm-bearing
sub-attributes (`InAlarm`, `Acked`, `Priority`, `Description`),
subscribes to those sub-attributes, and synthesizes Part 9 transitions
when the values change. This path operated as the only Galaxy alarm
path between PR 7.2 and the alarms-over-gateway epic; it remains the
fallback today.

When both paths report the same condition,
`AlarmConditionService.AlarmConditionState` keeps the
driver-native record and discards the duplicate sub-attribute
synthesis. Driver-native transitions are richer (carry operator
comment + original raise time) and arrive lower-latency (no
publishing-interval delay on the sub-attribute reads), so they win
the dedup.

## Acknowledge routing — Galaxy / driver alarms

### Native alarm acknowledge → AVEVA

When an OPC UA client Acknowledges a **native** (driver-fed, e.g. Galaxy)
`AlarmConditionState` node, the node manager's `OnAcknowledge` handler
branches on native-ness and routes through a dedicated path — separate
from the scripted `AlarmCommandRouter`:

1. **`OtOpcUaNodeManager.HandleNativeAlarmAck`** — gates on the caller's
   `AlarmAck` role (fails closed: no role → `BadUserAccessDenied`), then on the
   node's redundancy role (not the Primary → `BadInvalidState`, see
   §"Acknowledge on a non-Primary node" below), then dispatches a `NativeAlarmAck(ConditionNodeId, Comment, OperatorUser)`
   to the `NativeAlarmAckRouter` seam (fire-and-forget, non-blocking under
   the node-manager Lock). `OperatorUser` carries the authenticated
   session principal's display name.
2. **`DriverHostActor.HandleRouteNativeAlarmAck`** — receives a
   `RouteNativeAlarmAck` message (the host maps `NativeAlarmAck` at the
   wiring boundary to keep Runtime Akka-free of the OPC UA layer). Applied
   **Primary-gate first**: a Secondary or Detached node drops the message
   silently. On Primary, resolves the condition NodeId from the
   `_driverRefByAlarmNodeId` inverse map (NodeId → `(DriverInstanceId,
   FullName)`) and Tells the owning `DriverInstanceActor` a
   `RouteAlarmAck(FullName, Comment, OperatorUser)`.
3. **Galaxy driver** — `DriverInstanceActor` calls the driver's
   `IAlarmSource.AcknowledgeAsync` with an `AlarmAcknowledgeRequest`
   carrying the authored `FullName` as the `ConditionId` and the
   authenticated `OperatorUser`. The driver forwards this to the Galaxy
   gateway → AVEVA via `GatewayGalaxyAlarmAcknowledger`.
   **Fire-and-forget** — a failed upstream ack is not surfaced back to the
   OPC UA client (mirrors the Galaxy write-outcome limitation; the local
   `AlarmConditionState` SDK update already committed at step 1).

Only the **Acknowledge** is routed to the driver. **A native condition
supports Acknowledge and AddComment only** (Gitea #587 review): `Confirm`,
`OneShotShelve` / `TimedShelve` / `Unshelve` and `Enable` / `Disable` return
`BadNotSupported` and report no event. There is no upstream path for them and
no pair replication of a local state, and the driver projection
(`NativeAlarmProjector`: always `Confirmed`, `Unshelved`) would silently undo
one at the next transition. `AddComment` stays on the scripted
`AlarmCommandRouter` path (Phase 3 scope is Acknowledge → AVEVA only) and is
applied locally.

A native condition is **never Unconfirmed**: it starts Confirmed, and because
the SDK's own Acknowledge sets `ConfirmedState = false` on any condition type
that supports Confirm, the node manager re-asserts `Confirmed = true` (and the
projection's Retain rule, `Active || !Acked`) after the SDK handler, before the
one ack event. ScadaBridge maps `ConfirmedState` into its alarm state, so this
is client-visible.

> **Acknowledge on a non-Primary node is refused** (Gitea #597). On a node
> that is not the redundancy Primary, an OPC UA Acknowledge of a **native**
> condition returns **`BadInvalidState`** ("not the redundancy Primary;
> acknowledge on the Primary") — nothing is routed, applied or reported. The
> verdict is the same data-plane one the device-write gate and
> `DriverHostActor`'s ack route use (`PrimaryGatePolicy.ShouldServiceAsPrimary`,
> published to `IDriverPrimaryRoleSource` on every redundancy snapshot and
> wired onto the node manager by `OtOpcUaServerHostedService`). It is checked
> after authorization, so a caller who may not ack learns nothing about the
> node's role.
>
> This replaces a documented flap: the Secondary used to apply and report the
> ack locally while `DriverHostActor` dropped the route upstream, so the driver
> never acknowledged, its next transition projected `Acked = false`, and the
> client saw acked → unacked. An honest refusal is client-visible — a client
> (ScadaBridge included) that acks against the Secondary now gets a Bad result
> instead of a Good that later reverts — so ack against the Primary (the higher
> ServiceLevel). Scripted conditions are **not** gated: their engine runs on
> both nodes, so an ack on the Secondary is real. The published verdict
> mirrors `DriverHostActor`'s live check from the moment the host starts, not
> only from the first redundancy snapshot: the host publishes it once at start
> and again whenever cluster membership changes, so in the boot window of a
> pair (role still unknown, a driver peer Up) the gate already denies, exactly
> as the route does (review R1 m2 — previously the view sat on its single-node
> "allow" seed until the first snapshot, reopening the flap for that window).
> It mirrors, it does not deny from start: until this node has **joined** and
> sees its peer Up, its member count is 0 and both the published verdict and the
> live route **allow** (seconds normally; up to the bootstrap guard's 25 s
> partner probe). The two agree there, so the flap cannot occur, but a native
> ack in that pre-join window is serviced (review R6 MINOR-2).

> **Best-effort during reconnect.** If a native ack arrives while the owning
> driver is `Connecting`/`Reconnecting`, it is dropped with a debug log and **not**
> retried upstream — the OPC UA Part 9 `AlarmConditionState` was already
> committed locally at step 1, so the rare outcome is "acked locally but not
> propagated to AVEVA." Re-issue the ack once the driver is reconnected if
> upstream propagation is required — a re-ack of a node already showing Acked
> is still routed to the driver for exactly this reason.

### Legacy sub-attribute path

`DriverNodeManager` picks the acknowledger when registering each
condition (PR B.3 logic):

- Driver implements `IAlarmSource` →
  `DriverAlarmSourceAcknowledger` routes the operator comment
  through `IAlarmSource.AcknowledgeAsync` via the existing
  `AlarmSurfaceInvoker` (Phase 6.1 resilience pipeline; no-retry
  per decision #143). End-to-end operator-comment fidelity is
  preserved.
- Driver doesn't implement `IAlarmSource` →
  `DriverWritableAcknowledger` writes the comment into the
  `AckMsgWriteRef` sub-attribute via `IWritable.WriteAsync`. Same
  resilience pipeline; collapses the comment into a single string
  write at the wire level.

The OPC UA Part 9 `AlarmConditionState.OnAcknowledge` delegate
already validates the session's `AlarmAck` role before dispatching,
so the gateway-side ack RPC only sees authenticated, authorised
calls.

## Inbound operator ack/shelve — scripted alarms

Scripted alarms use a separate inbound path that converges on the
`alarm-commands` DPS topic. Two surfaces route onto this topic:

### OPC UA Part 9 method path (external OPC UA clients)

`OtOpcUaNodeManager` wires the Part 9 condition methods (Acknowledge /
Confirm / AddComment / OneShotShelve / TimedShelve / Unshelve) on each
scripted-alarm `AlarmConditionState` node. Every call is **gated on the
`AlarmAck` role** — fail-closed: sessions with no role or without
an `AlarmAck`-conferring grant receive `BadUserAccessDenied` immediately.
The session's roles come from its API-key principal's `NodeAcl` grants (or the
reserved `anonymous` principal's; LDAP is AdminUI-only — `docs/security.md`) and are carried past
`OpcUaApplicationHost` by
`RoleCarryingUserIdentity` (a `UserIdentity` subclass), making it
readable inside the method handler at dispatch time.

On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` onto the
`alarm-commands` DPS topic. The node manager is Akka-free; the dispatch
action is a settable `Action<AlarmCommand>` injected at boot by the
hosted service.

`OnTimedUnshelve` (the SDK's automatic unshelve timer) is a **no-op** — it
routes nothing (review R1 M2). The engine owns timed-shelve expiry (its 5 s
shelving check records `AutoUnshelve` by `system`), and its re-projection
unshelves the node; see §"`UnshelveTime` is the real remaining time" below.

### One condition event per method call (Gitea #587)

Every accepted Part 9 method call — Acknowledge, Confirm, AddComment,
OneShotShelve, TimedShelve, Unshelve, Enable, Disable, on a native or a
scripted condition — reports **exactly one** condition event carrying the
new state, to folder subscribers and (through the ACL) Server-object
subscribers alike. A refused call (bad status, e.g. a denied gate, an
already-acknowledged condition, Enable/Disable on a native condition)
reports nothing.

- **The SDK does not report it.** `ConditionState.ReportStateChange` is
  gated on `AutoReportStateChanges`, which this repo leaves `false` — so
  before #587 a client Acknowledge produced no condition event at all (an
  older comment claimed "the SDK auto-fires its own event"; it never did).
  Instead the node manager wraps each method's SDK handler
  (`WireMethodTransitionReporting`) and, once the handler returns Good,
  reports the condition itself. That report does not depend on the SDK's
  `AreEventsMonitored` bookkeeping, which is imprecise for conditions
  materialised under an already-subscribed folder.
- **The SDK's shelve methods never apply shelving state** (they only call
  `OnShelve`), so the wrapper applies it — with the same `(shelved, oneShot)`
  mapping the projection uses, and a TimedShelve with its requested duration —
  before reporting.
- **The re-projection does not duplicate it.** The engine's (or driver's)
  answer to the routed command reaches `WriteAlarmCondition`'s delta-gate,
  which compares against the node's current state — already the new state —
  and suppresses. The whole method call runs under the node-manager lock, so
  a fast re-projection cannot slip in between the SDK applying the state and
  the report.
- **Only a real change is reported.** The condition's state is compared
  before and after the SDK handler; a call that changed nothing reports
  nothing. The SDK accepts acknowledging an acknowledged condition and sets
  `Confirmed = false` again, so a re-ack also restores the pre-call Confirmed
  and Retain — silent, and a scripted node never drifts from its engine.
  AddComment always reports (the comment is the change), and so does a re-ack
  **carrying a comment** (Gitea #597): the SDK writes that comment to the node,
  so it is reported once, with Acked/Confirmed unchanged. Before #597 the
  comment landed on the node but no subscriber ever saw it. For a **scripted**
  condition the comment is also durable (review R1 m1): the routed Acknowledge
  reaches an engine that is already acknowledged, and
  `Part9StateMachine.ApplyAcknowledge` records a commented re-ack exactly as an
  AddComment (kind `AddComment`, the caller as user) — persisted and replicated
  with the condition state, the original ack's `LastAck*` fields untouched. An
  uncommented re-ack is still a no-op in the engine. The same applies to a
  commented re-ack from the AdminUI `/alerts` page.
- **A native re-ack is still routed to the driver** (deliberate, #597). The
  route is fire-and-forget, so a node showing Acked does not prove the
  upstream system acknowledged — an ack dropped during a reconnect leaves
  exactly that state — and re-acking is the documented retry (§"Best-effort
  during reconnect"). The cost is a duplicate ack upstream for an alarm that
  was already acknowledged there.
- **Native conditions: Acknowledge and AddComment only** — see §"Native alarm
  acknowledge → AVEVA" above. The other methods are refused
  (`BadNotSupported`, no event), and a native ack keeps the condition
  Confirmed.
- **TimedShelve works (scripted).** `Create()` left `MaxTimeShelved` at 0, and
  the SDK rejects any shelving time above it, so every client TimedShelve
  failed with `BadShelvingTimeOutOfRange`. It is now `int.MaxValue` ms (≈24.8
  days). The engine owns the expiry and its Unshelved re-projection reports
  the one expiry event.
- **`UnshelveTime` is the real remaining time** (Gitea #597). Both the method
  path and the projection used to apply a timed shelve with `shelvingTime 0`,
  which the SDK reads as `MaxTimeShelved` — so every timed shelve advertised
  ≈24.8 days remaining. The method now applies the requested duration, and the
  engine's projection carries its expiry (`AlarmConditionSnapshot.UnshelveAtUtc`),
  so `ShelvingState/UnshelveTime` counts down to the engine's expiry whichever
  path shelved it (the AdminUI `/alerts` timed shelve included). This arms the
  SDK's own unshelve timer at the real instant, and `OnTimedUnshelve` — the
  timer's callback — is deliberately a **no-op** (review R1 M2): it used to
  route an operator-style `Unshelve` by `system`, which, firing before the
  engine's 5 s expiry sweep, pre-empted the engine's `AutoUnshelve` on every
  expiry, from both pair nodes, rewriting the audit trail as a manual unshelve.
  The SDK's timer changes no shelving state itself, so the node stays
  TimedShelved (with `UnshelveTime` at zero) for up to one sweep after expiry,
  until the engine's re-projection reports the one expiry event.
- **Accepted, not fixed — the SDK's `EventsMonitored` unit for late conditions**
  (Gitea #597 item 6, from the #586 review). The SDK's `AreEventsMonitored`
  counter never reaches a condition materialised under a folder that already had
  an event subscriber. Our own reporting does not depend on it (above); the one
  thing it still gates is the SDK's method **audit** event, and only while the
  server is auditing — which no configuration in this repo enables. Closed as
  accepted.
- Method **audit** events (`AuditConditionAcknowledgeEventType`, …) are still
  the SDK's, delivered only while the server is auditing over an encrypted
  channel.

Engine-driven transitions are unaffected: `WriteAlarmCondition` still fires
whenever the incoming state differs from the node's current state.

### AdminUI path

The `/alerts` page shows per-row **Acknowledge / Shelve / Unshelve**
buttons gated by the `DriverOperator` AdminUI policy. These route
through the `AdminOperationsActor` cluster singleton
(`AcknowledgeAlarmCommand` / `ShelveAlarmCommand`), which publishes onto
the same `alarm-commands` topic. The singleton handles cross-node
routing — the command always reaches the driver-role node owning the
engine regardless of which AdminUI instance the operator is on.

### ScriptedAlarmHostActor dispatch

`ScriptedAlarmHostActor` subscribes to the `alarm-commands` topic,
ownership-filters each command (each node only acts on its own alarms),
and dispatches to the matching `ScriptedAlarmEngine` operation
(`AcknowledgeAsync` / `ConfirmAsync` / `OneShotShelveAsync` /
`TimedShelveAsync` / `UnshelveAsync` / `EnableAsync` / `DisableAsync` /
`AddCommentAsync`). The engine's existing `OnEvent` callback handles
the OPC UA node update — no explicit re-projection is required.

The AdminUI `/alerts` Shelve flow was live-verified on docker-dev
2026-06-11: singleton → topic → host actor → engine → "Shelved" status
reflected on `/alerts` with the operator identity threaded through.

## Redundancy deduplication

Under warm/hot redundancy, both cluster nodes run `ScriptedAlarmHostActor` and the scripted-alarm engine. To prevent duplicate `/alerts` rows and duplicate historian writes, alarm transition publication to the `alerts` topic and `HistorianAdapterActor` historization are **Primary-gated**: only the node whose `RedundancyRole` is `Primary` publishes externally. OPC UA condition-node writes and inbound ack/shelve processing remain ungated on both nodes so the secondary stays warm for failover. See [Redundancy.md §Primary-gated alarm emission and historization](Redundancy.md#primary-gated-alarm-emission-and-historization).

## Historian write-back (non-Galaxy alarms)

Scripted alarms (and any future non-Galaxy `IAlarmSource` like
AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:

- `IAlarmHistorianSink` is the DI-registered intake contract. The
  default binding is `NullAlarmHistorianSink` (registered in
  `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
  deployments override it with `LocalDbStoreAndForwardSink` wrapping
  `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
  — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
- `LocalDbStoreAndForwardSink` queues each transition into the node's
  consolidated LocalDb — where it **replicates to the redundant pair peer**,
  so undelivered alarm history survives losing the node holding it — and
  drains in the background via an `IAlarmHistorianWriter`. **Only the node
  holding the Primary role drains**, since the peer holds a full replica;
  delivery is consequently at-least-once across a failover, by design. See
  [AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
  queue capacity defaults to 1,000,000 rows; under a sustained
  historian outage, older non-dead-lettered rows are evicted (oldest
  first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
  counter surfaces lifetime eviction events so operators can detect
  silent data loss without log scraping. The drain cadence, queue
  capacity, and dead-letter retention are tunable via the `AlarmHistorian`
  config section (`DrainIntervalSeconds`, `Capacity`,
  `DeadLetterRetentionDays`); `AlarmHistorianOptions.Validate()` logs a
  startup warning for a non-positive knob. (It no longer warns about
  `SharedSecret` or `DatabasePath` — both keys are gone: the connection
  comes from `ServerHistorian`, and the queue lives in the LocalDb.)
- `HistorianAdapterActor`
  (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`)
  subscribes to the cluster `alerts` DPS topic, translates each
  `AlarmTransitionEvent` → `AlarmHistorianEvent`, and calls
  `EnqueueAsync` fire-and-forget. The durable write is gated two ways:
  (1) **Primary-gated** — only the Primary node historizes, giving
  exactly-once writes across a redundant pair; (2) **per-alarm** — a
  transition whose `HistorizeToAveva` is `false` is skipped (the flag
  rides on `AlarmTransitionEvent` as a nullable bool; missing/`null`/`true`
  historize, only an explicit `false` suppresses, so a cross-version
  rolling restart defaults to historizing rather than dropping an audit
  row). Neither gate touches the live `alerts` publish — the `/alerts`
  UI always sees every transition. See
  [AlarmHistorian.md §Configuration](AlarmHistorian.md#configuration)
  for the `AlarmHistorian` appsettings section that enables the real sink.

  **Native alarms** (equipment tags carrying an `"alarm"` object in their `TagConfig`)
  support the same `HistorizeToAveva` opt-out. The field is `alarm.historizeToAveva`
  (`bool?`) and is authored via the **"Historize to AVEVA"** checkbox in the Tag modal's
  alarm section. The gate logic is identical (`is not false`): absent or `true` historizes;
  explicit `false` suppresses the AVEVA write while leaving the live `/alerts` feed
  unaffected. See
  [ScriptedAlarms.md §TagConfig alarm fields](ScriptedAlarms.md#tagconfig-alarm-fields)
  for the full field reference.

Galaxy-native alarms with `$Alarm*` extensions reach AVEVA Historian
directly via System Platform's `HistorizeToAveva` toggle on the
alarm primitive — no involvement from OtOpcUa. This sidecar path is
exclusively for non-Galaxy alarm producers.

## Cross-references

- Plan: [docs/plans/alarms-over-gateway.md](plans/alarms-over-gateway.md)
- v1 archive: [docs/v1/AlarmTracking.md](v1/AlarmTracking.md)
- Galaxy driver: [docs/drivers/Galaxy.md](drivers/Galaxy.md)
- Phase 7 scripting + alarming: [docs/v2/implementation/phase-7-scripting-and-alarming.md](v2/implementation/phase-7-scripting-and-alarming.md)
- Security + ACL: [docs/security.md](security.md)