fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions` and re-materialises every condition node fresh — inactive, acked, confirmed, unshelved. The engine reloads its persisted state and re-derives Active from the predicate, so it ends up correct; but there is no transition to report, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing writes the node. Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild reporting Acknowledged with Retain=false — so it disappeared from ConditionRefresh entirely, while the engine still held Unacknowledged. An operator's outstanding alarm silently vanishes from the alarm list on deploy. `ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it reads the new `ScriptedAlarmEngine.GetProjections()` and sends one `AlarmStateUpdate` per alarm NOT in the Part 9 no-event position. Load-bearing properties: - Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry hub. `alerts` is the historization path, so a row per alarm per deploy would append a duplicate historian/AVEVA record every time anyone deploys. This is why it reads a projection rather than re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission machinery — or a future refactor — to mistake for a transition. The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone would have clobbered the node's severity and message, so the projection also carries the definition's severity, the resolved message template, and the last-observed worst-input quality. - Only non-normal alarms. One in the no-event position already matches what the materialise built, so a never-fired alarm stays byte-identical to a deploy without this pass (including its Message, which the materialise seeds with the display name), and an all-normal deploy writes nothing at all. - Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant. - No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against the node's CURRENT state, so a re-assert onto a freshly materialised node is a genuine delta (fires once — correct, the rebuild reset what clients see) while one onto a surgically-preserved node is suppressed. Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by disabling the call — 4 of the 6 host tests go red; the other 2 are absence guards against an over-broad fix and pass either way by design. Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image): - pre-fix: condition absent from ConditionRefresh after a rebuild; - post-fix: present, Unacknowledged, Retain=True, stamped with its own LastTransitionUtc rather than the restart instant; - /alerts stayed empty across two re-asserts, with the panel proven live by a real ACTIVATED transition immediately afterwards — no duplicate history.
This commit is contained in:
Joseph Doherty
@@ -334,6 +334,48 @@ public sealed class ScriptedAlarmEngine : IDisposable
|
||||
public IReadOnlyCollection<AlarmConditionState> GetAllStates()
|
||||
=> _alarms.Values.Select(a => a.Condition).ToArray();
|
||||
|
||||
/// <summary>
|
||||
/// #487 — the currently-held condition of every loaded alarm, packaged with the definition's
|
||||
/// severity, the rendered message template and the last-observed input quality, i.e. everything
|
||||
/// a caller needs to <b>project</b> the alarm onto an OPC UA condition node without waiting for
|
||||
/// the next transition.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>This is a read, not an emission.</b> It deliberately returns a distinct type rather
|
||||
/// than a <see cref="ScriptedAlarmEvent"/>, and it never touches <see cref="OnEvent"/>. A
|
||||
/// still-active alarm produces <see cref="EmissionKind.None"/> at load (there is no
|
||||
/// transition to report), so the transition stream cannot carry the current state — that is
|
||||
/// exactly why the caller needs this. Routing these through the emission path instead would
|
||||
/// append a duplicate historian / <c>alerts</c> row on every deploy, so the shape here is
|
||||
/// intentionally one the emission machinery cannot consume.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Synchronization.</b> Reads <c>_alarms</c> and <c>_valueCache</c> — both
|
||||
/// <see cref="ConcurrentDictionary{TKey, TValue}"/> — without taking <c>_evalGate</c>, the
|
||||
/// same posture as <see cref="GetState"/> / <see cref="GetAllStates"/>. Each projection is
|
||||
/// individually coherent; the set as a whole is not an atomic snapshot across a concurrent
|
||||
/// re-evaluation. That is sufficient for the node-projection use: an evaluation racing this
|
||||
/// read publishes its own transition through the normal path.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <returns>One projection per loaded alarm; empty before the first <see cref="LoadAsync"/>.</returns>
|
||||
public IReadOnlyList<ScriptedAlarmProjection> GetProjections()
|
||||
{
|
||||
var projections = new List<ScriptedAlarmProjection>(_alarms.Count);
|
||||
foreach (var (alarmId, state) in _alarms)
|
||||
{
|
||||
projections.Add(new ScriptedAlarmProjection(
|
||||
AlarmId: alarmId,
|
||||
Severity: state.Definition.Severity,
|
||||
Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
|
||||
Condition: state.Condition,
|
||||
WorstInputStatusCode: LastWorstStatus(alarmId)));
|
||||
}
|
||||
|
||||
return projections;
|
||||
}
|
||||
|
||||
/// <summary>Acknowledges the specified alarm on behalf of the given user.</summary>
|
||||
/// <param name="alarmId">The alarm identifier.</param>
|
||||
/// <param name="user">The user performing the acknowledgment.</param>
|
||||
@@ -974,6 +1016,26 @@ public sealed record ScriptedAlarmEvent(
|
||||
// the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
|
||||
uint WorstInputStatusCode = 0u);
|
||||
|
||||
/// <summary>
|
||||
/// #487 — a loaded alarm's current condition, ready to be projected onto its OPC UA node.
|
||||
/// Produced by <see cref="ScriptedAlarmEngine.GetProjections"/> on demand; it is <b>not</b> an
|
||||
/// emission and never travels the <see cref="ScriptedAlarmEngine.OnEvent"/> path, so it can never
|
||||
/// become an <c>alerts</c> row or a historian record. Deliberately a separate type from
|
||||
/// <see cref="ScriptedAlarmEvent"/> — it carries no <see cref="EmissionKind"/>, so there is nothing
|
||||
/// for a future refactor to mistake for a transition.
|
||||
/// </summary>
|
||||
/// <param name="AlarmId">The alarm identifier (also its OPC UA condition NodeId).</param>
|
||||
/// <param name="Severity">The definition's severity bucket.</param>
|
||||
/// <param name="Message">The message template resolved against the current value cache.</param>
|
||||
/// <param name="Condition">The Part 9 condition state the engine currently holds.</param>
|
||||
/// <param name="WorstInputStatusCode">The last-observed worst OPC UA StatusCode across the alarm's inputs.</param>
|
||||
public sealed record ScriptedAlarmProjection(
|
||||
string AlarmId,
|
||||
AlarmSeverity Severity,
|
||||
string Message,
|
||||
AlarmConditionState Condition,
|
||||
uint WorstInputStatusCode);
|
||||
|
||||
/// <summary>
|
||||
/// Upstream source abstraction — intentionally identical shape to the virtual-tag
|
||||
/// engine's so Stream G can compose them behind one driver bridge.
|
||||
|
||||
@@ -288,8 +288,101 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
// so a re-Apply with a fresh union simply supersedes the old one — no explicit unregister needed.
|
||||
_mux?.Tell(new DependencyMuxActor.RegisterInterest(msg.DepRefs, Self));
|
||||
_log.Debug("ScriptedAlarmHost: loaded; registered mux interest for {Count} dep refs", msg.DepRefs.Count);
|
||||
|
||||
ReassertConditionNodes();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #487 — re-project every loaded alarm that is NOT in the Part 9 "no-event" position back onto
|
||||
/// its condition node, right after a (re)load.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>The bug this closes.</b> A deploy that triggers a FULL address-space rebuild clears
|
||||
/// <c>_alarmConditions</c> and re-materialises every condition node fresh — inactive, acked,
|
||||
/// confirmed, unshelved. The engine, meanwhile, restores each alarm's persisted state and
|
||||
/// re-derives Active from the predicate, so a still-active alarm ends the reload
|
||||
/// <i>correctly Active in the engine</i> but produces <see cref="EmissionKind.None"/> (there
|
||||
/// is no transition to report) — and <see cref="OnEngineEmission"/> drops None. Nothing then
|
||||
/// writes the node, so an active alarm with static dependencies reads NORMAL to every OPC UA
|
||||
/// client until its next real transition, which may never come. Same shape as the VirtualTag
|
||||
/// redeploy-reset the <c>ReassertValue</c> fix closed.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Node-only, by construction.</b> This sends <see cref="OpcUaPublishActor.AlarmStateUpdate"/>
|
||||
/// and nothing else — it never publishes to the <c>alerts</c> topic and never touches the
|
||||
/// telemetry hub. That is the whole reason it reads <see cref="ScriptedAlarmEngine.GetProjections"/>
|
||||
/// (a plain read returning <see cref="ScriptedAlarmProjection"/>) rather than re-emitting: an
|
||||
/// <c>alerts</c> row per alarm per deploy would append a duplicate historian / AVEVA record
|
||||
/// every time anyone deploys — the alarm analogue of the VirtualTag M1 historian issue.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Ordering.</b> <c>DriverHostActor</c> Tells <c>RebuildAddressSpace</c> to the publish
|
||||
/// actor BEFORE it Tells <see cref="ApplyScriptedAlarms"/> here, and this runs later still —
|
||||
/// after an <c>await</c>ed <see cref="ScriptedAlarmEngine.LoadAsync"/> pipes back. Both Tells
|
||||
/// enqueue synchronously into the publish actor's local mailbox, so the re-materialise is
|
||||
/// already queued ahead of these writes: the node exists by the time they are handled.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why only non-normal alarms.</b> An alarm sitting in the no-event position already
|
||||
/// matches what the materialise just built, so writing it would be a pure no-op on state —
|
||||
/// but it would still overwrite the node's Message (the materialise seeds it with the alarm's
|
||||
/// display name; a projection carries the resolved template). Skipping them keeps a
|
||||
/// never-fired alarm byte-identical to a deploy without this pass, and keeps the write count
|
||||
/// at zero on the overwhelmingly common all-normal deploy.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Duplicate events.</b> None. <c>WriteAlarmCondition</c>'s delta-gate compares the
|
||||
/// incoming snapshot against the node's CURRENT state, so a re-assert onto a freshly
|
||||
/// materialised (normal) node is a genuine delta and fires one Part 9 event — correct, since
|
||||
/// the rebuild reset what clients see — while a re-assert onto a surgically-preserved node
|
||||
/// that already holds the same state is suppressed. This write is ungated by redundancy role,
|
||||
/// like every other node write here: the Secondary keeps its address space warm for failover.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void ReassertConditionNodes()
|
||||
{
|
||||
var reasserted = 0;
|
||||
foreach (var projection in _engine.GetProjections())
|
||||
{
|
||||
if (IsInNoEventPosition(projection.Condition))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(
|
||||
AlarmNodeId: projection.AlarmId,
|
||||
State: ToSnapshot(projection),
|
||||
// The instant the state we are re-asserting actually came about — NOT the deploy time.
|
||||
// A client reading Time/ReceiveTime after a rebuild should still see when the alarm went
|
||||
// active, not when the address space happened to be rebuilt.
|
||||
TimestampUtc: projection.Condition.LastTransitionUtc,
|
||||
Realm: AddressSpaceRealm.Uns));
|
||||
reasserted++;
|
||||
}
|
||||
|
||||
if (reasserted > 0)
|
||||
{
|
||||
_log.Info("ScriptedAlarmHost: re-asserted {Count} non-normal condition node(s) after (re)load", reasserted);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Whether <paramref name="condition"/> sits in the Part 9 "no-event" position — exactly the
|
||||
/// state <see cref="AlarmConditionState.Fresh"/> produces and a freshly materialised condition node
|
||||
/// already carries. Anything else (active, unacknowledged, unconfirmed, disabled, or shelved) is state
|
||||
/// a rebuild would silently lose, so it is what <see cref="ReassertConditionNodes"/> re-projects.</summary>
|
||||
/// <param name="condition">The engine's current condition state.</param>
|
||||
/// <returns><c>true</c> when the condition carries nothing a rebuild could lose.</returns>
|
||||
private static bool IsInNoEventPosition(AlarmConditionState condition)
|
||||
=> condition.Enabled == AlarmEnabledState.Enabled
|
||||
&& condition.Active == AlarmActiveState.Inactive
|
||||
&& condition.Acked == AlarmAckedState.Acknowledged
|
||||
&& condition.Confirmed == AlarmConfirmedState.Confirmed
|
||||
&& condition.Shelving.Kind == ShelvingKind.Unshelved;
|
||||
|
||||
private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg)
|
||||
{
|
||||
// Feed the live value into the upstream the engine subscribes from. #478 — carry the source
|
||||
@@ -594,18 +687,38 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
/// Severity is the OPC UA 1..1000 value <see cref="SeverityToInt"/> derives from the coarse engine
|
||||
/// bucket, cast to the <c>ushort</c> the SDK <c>SetSeverity</c> expects. Shelving's 3-way Core kind
|
||||
/// maps 1:1 onto the Commons <see cref="AlarmShelvingKind"/>.</summary>
|
||||
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmEvent e) => new(
|
||||
Active: e.Condition.Active == AlarmActiveState.Active,
|
||||
Acknowledged: e.Condition.Acked == AlarmAckedState.Acknowledged,
|
||||
Confirmed: e.Condition.Confirmed == AlarmConfirmedState.Confirmed,
|
||||
Enabled: e.Condition.Enabled == AlarmEnabledState.Enabled,
|
||||
Shelving: MapShelving(e.Condition.Shelving.Kind),
|
||||
Severity: (ushort)SeverityToInt(e.Severity),
|
||||
Message: e.Message,
|
||||
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmEvent e)
|
||||
=> ToSnapshot(e.Condition, e.Severity, e.Message, e.WorstInputStatusCode);
|
||||
|
||||
/// <summary>#487 — the same projection for a <see cref="ScriptedAlarmProjection"/>: a state READ used
|
||||
/// to restore a condition node after a rebuild, never an emission. Shares
|
||||
/// <see cref="ToSnapshot(AlarmConditionState, AlarmSeverity, string, uint)"/> with the transition path
|
||||
/// so a re-asserted node is byte-identical to what the alarm's own transition would have written.</summary>
|
||||
/// <param name="p">The engine projection to map.</param>
|
||||
/// <returns>The Commons snapshot the SDK sink projects onto the condition node.</returns>
|
||||
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmProjection p)
|
||||
=> ToSnapshot(p.Condition, p.Severity, p.Message, p.WorstInputStatusCode);
|
||||
|
||||
/// <summary>The single Core → Commons condition projection, shared by the emission and re-assert
|
||||
/// paths.</summary>
|
||||
/// <param name="condition">The Part 9 condition state.</param>
|
||||
/// <param name="severity">The engine's coarse severity bucket.</param>
|
||||
/// <param name="message">The resolved condition message.</param>
|
||||
/// <param name="worstInputStatusCode">The worst OPC UA StatusCode across the script's input tags.</param>
|
||||
/// <returns>The Commons snapshot the SDK sink projects onto the condition node.</returns>
|
||||
private static AlarmConditionSnapshot ToSnapshot(
|
||||
AlarmConditionState condition, AlarmSeverity severity, string message, uint worstInputStatusCode) => new(
|
||||
Active: condition.Active == AlarmActiveState.Active,
|
||||
Acknowledged: condition.Acked == AlarmAckedState.Acknowledged,
|
||||
Confirmed: condition.Confirmed == AlarmConfirmedState.Confirmed,
|
||||
Enabled: condition.Enabled == AlarmEnabledState.Enabled,
|
||||
Shelving: MapShelving(condition.Shelving.Kind),
|
||||
Severity: (ushort)SeverityToInt(severity),
|
||||
Message: message,
|
||||
// #478 — the condition's Quality is the worst quality across the script's input tags at evaluation
|
||||
// time (carried on the event by the engine). A transition fired while an input is Uncertain projects
|
||||
// Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
|
||||
Quality: QualityFromStatus(e.WorstInputStatusCode));
|
||||
Quality: QualityFromStatus(worstInputStatusCode));
|
||||
|
||||
/// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
|
||||
/// mirror (the Commons assembly can't see the Core enum).</summary>
|
||||
|
||||
Reference in New Issue
Block a user