fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)

A deploy that triggers a FULL address-space rebuild clears `_alarmConditions`
and re-materialises every condition node fresh — inactive, acked, confirmed,
unshelved. The engine reloads its persisted state and re-derives Active from the
predicate, so it ends up correct; but there is no transition to report, so
`LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing
writes the node.

Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that
had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild
reporting Acknowledged with Retain=false — so it disappeared from
ConditionRefresh entirely, while the engine still held Unacknowledged. An
operator's outstanding alarm silently vanishes from the alarm list on deploy.

`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it
reads the new `ScriptedAlarmEngine.GetProjections()` and sends one
`AlarmStateUpdate` per alarm NOT in the Part 9 no-event position.

Load-bearing properties:

- Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts`
  topic, never the telemetry hub. `alerts` is the historization path, so a row
  per alarm per deploy would append a duplicate historian/AVEVA record every
  time anyone deploys. This is why it reads a projection rather than re-emitting:
  `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for
  the emission machinery — or a future refactor — to mistake for a transition.
  The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone
  would have clobbered the node's severity and message, so the projection also
  carries the definition's severity, the resolved message template, and the
  last-observed worst-input quality.
- Only non-normal alarms. One in the no-event position already matches what the
  materialise built, so a never-fired alarm stays byte-identical to a deploy
  without this pass (including its Message, which the materialise seeds with the
  display name), and an all-normal deploy writes nothing at all.
- Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant.
- No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against
  the node's CURRENT state, so a re-assert onto a freshly materialised node is a
  genuine delta (fires once — correct, the rebuild reset what clients see) while
  one onto a surgically-preserved node is suppressed.

Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by
disabling the call — 4 of the 6 host tests go red; the other 2 are absence
guards against an over-broad fix and pass either way by design.

Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image):
- pre-fix: condition absent from ConditionRefresh after a rebuild;
- post-fix: present, Unacknowledged, Retain=True, stamped with its own
  LastTransitionUtc rather than the restart instant;
- /alerts stayed empty across two re-asserts, with the panel proven live by a
  real ACTIVATED transition immediately afterwards — no duplicate history.
This commit is contained in:
Joseph Doherty
2026-07-26 09:06:39 -04:00
parent 28c2866710
commit 549c656489
6 changed files with 575 additions and 14 deletions
+8 -1
View File
@@ -94,7 +94,14 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
## Documented follow-ups (non-blocking)
- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
- **Scripted-alarm redeploy recovery — FIXED (issue #487), see `docs/ScriptedAlarms.md` §"Post-(re)load
condition-node re-assert".** Landed as the proposed shape below: `ScriptedAlarmHostActor.ReassertConditionNodes`
in `OnAlarmsLoaded`, node-only, never the `alerts` topic. Two deviations from the sketch, both deliberate:
it reads a new `ScriptedAlarmEngine.GetProjections()` rather than `GetState(alarmId)` + `LoadedAlarmIds`
(a projection also carries the definition's severity, the resolved message template, and the last-observed
worst-input quality — `GetState` alone would have clobbered severity/message on the node); and it skips
alarms sitting in the Part 9 no-event position, so a deploy where nothing is in alarm writes no condition
nodes at all. The original deferral text follows for context.
The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
`RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields