fix(alarms): re-assert non-normal scripted-alarm conditions after a (re)load (#487)
A deploy that triggers a FULL address-space rebuild clears `_alarmConditions` and re-materialises every condition node fresh — inactive, acked, confirmed, unshelved. The engine reloads its persisted state and re-derives Active from the predicate, so it ends up correct; but there is no transition to report, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it. Nothing writes the node. Live-reproduced on the docker-dev rig against the pre-fix image: an alarm that had fired and cleared but was still UNACKNOWLEDGED came back from a rebuild reporting Acknowledged with Retain=false — so it disappeared from ConditionRefresh entirely, while the engine still held Unacknowledged. An operator's outstanding alarm silently vanishes from the alarm list on deploy. `ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load it reads the new `ScriptedAlarmEngine.GetProjections()` and sends one `AlarmStateUpdate` per alarm NOT in the Part 9 no-event position. Load-bearing properties: - Node-only. It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry hub. `alerts` is the historization path, so a row per alarm per deploy would append a duplicate historian/AVEVA record every time anyone deploys. This is why it reads a projection rather than re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission machinery — or a future refactor — to mistake for a transition. The issue's sketch said `GetState(alarmId)` + `LoadedAlarmIds`; that alone would have clobbered the node's severity and message, so the projection also carries the definition's severity, the resolved message template, and the last-observed worst-input quality. - Only non-normal alarms. One in the no-event position already matches what the materialise built, so a never-fired alarm stays byte-identical to a deploy without this pass (including its Message, which the materialise seeds with the display name), and an all-normal deploy writes nothing at all. - Timestamp is the condition's own `LastTransitionUtc`, not the deploy instant. - No duplicate Part 9 events: `WriteAlarmCondition`'s delta-gate compares against the node's CURRENT state, so a re-assert onto a freshly materialised node is a genuine delta (fires once — correct, the rebuild reset what clients see) while one onto a surgically-preserved node is suppressed. Tests: 6 new host tests + 6 new engine tests. Falsifiability checked by disabling the call — 4 of the 6 host tests go red; the other 2 are absence guards against an over-broad fix and pass either way by design. Live gate (docker-dev, central-2, pre-fix image vs. rebuilt image): - pre-fix: condition absent from ConditionRefresh after a rebuild; - post-fix: present, Unacknowledged, Retain=True, stamped with its own LastTransitionUtc rather than the restart instant; - /alerts stayed empty across two re-asserts, with the panel proven live by a real ACTIVATED transition immediately afterwards — no duplicate history.
This commit is contained in:
Joseph Doherty
@@ -102,6 +102,45 @@ Persisted scope per plan decision #14: `Enabled`, `Acked`, `Confirmed`, `Shelvin
|
||||
|
||||
Every mutation the state machine produces is immediately persisted inside the engine's `_evalGate` semaphore, so the store's view is always consistent with the in-memory state.
|
||||
|
||||
### Post-(re)load condition-node re-assert (#487)
|
||||
|
||||
A deploy that triggers a **full address-space rebuild** clears `_alarmConditions` and re-materialises every
|
||||
condition node fresh — inactive, acked, confirmed, unshelved. The engine, reloading in parallel, restores each
|
||||
alarm's persisted state and re-derives `Active` from the predicate, so it ends up *correct*; but a still-active
|
||||
alarm has **no transition to report**, so `LoadAsync` yields `EmissionKind.None` and `OnEngineEmission` drops it.
|
||||
Nothing then writes the node. Before this fix, an active alarm with static dependencies read **normal** to every
|
||||
OPC UA client after such a deploy, until its next real transition — which for a static-dependency alarm may
|
||||
never come.
|
||||
|
||||
`ScriptedAlarmHostActor.ReassertConditionNodes` closes it. After each load completes it reads
|
||||
`ScriptedAlarmEngine.GetProjections()` — a plain read returning `ScriptedAlarmProjection` (condition state +
|
||||
the definition's severity + the resolved message template + last-observed worst-input quality) — and sends one
|
||||
`OpcUaPublishActor.AlarmStateUpdate` per alarm that is **not** in the Part 9 no-event position.
|
||||
|
||||
Four properties are load-bearing:
|
||||
|
||||
- **Node-only.** It sends `AlarmStateUpdate` and nothing else — never the `alerts` topic, never the telemetry
|
||||
hub. `alerts` is the historization path, so an alerts row per alarm per deploy would append a duplicate
|
||||
historian / AVEVA record every time anyone deploys. This is why the re-assert reads a projection instead of
|
||||
re-emitting: `ScriptedAlarmProjection` carries no `EmissionKind`, so there is nothing for the emission
|
||||
machinery — or a future refactor — to mistake for a transition.
|
||||
- **Only non-normal alarms.** An alarm in the no-event position (enabled, inactive, acked, confirmed,
|
||||
unshelved) already matches what the materialise built, so it is skipped. That keeps a never-fired alarm
|
||||
byte-identical to a deploy without this pass — including its Message, which the materialise seeds with the
|
||||
alarm's display name — and writes nothing at all on the common all-normal deploy.
|
||||
- **Ordering.** `DriverHostActor` tells `RebuildAddressSpace` to the publish actor *before* it tells
|
||||
`ApplyScriptedAlarms` to the host, and the re-assert runs later still (after an awaited `LoadAsync` pipes
|
||||
back). Both are local `Tell`s, which enqueue synchronously, so the re-materialise is already queued ahead of
|
||||
these writes.
|
||||
- **No duplicate Part 9 events.** `WriteAlarmCondition`'s delta-gate compares the incoming snapshot against
|
||||
the node's *current* state: a re-assert onto a freshly materialised (normal) node is a genuine delta and
|
||||
fires exactly one condition event — correct, since the rebuild reset what clients see — while a re-assert
|
||||
onto a surgically-preserved node that already holds the same state is suppressed. The write is ungated by
|
||||
redundancy role, like every other node write here; the Secondary keeps its address space warm for failover.
|
||||
|
||||
The timestamp carried is the condition's own `LastTransitionUtc`, **not** the deploy instant, so a client
|
||||
reading `Time` / `ReceiveTime` after a rebuild still sees when the alarm actually went active.
|
||||
|
||||
## Source integration
|
||||
|
||||
`ScriptedAlarmSource` (`ScriptedAlarmSource.cs`) adapts the engine to the driver-agnostic `IAlarmSource` interface. The existing `AlarmSurfaceInvoker` + `GenericDriverNodeManager` fan-out consumes it the same way it consumes Galaxy / AB CIP / FOCAS sources — there is no scripted-alarm-specific code path in the server plumbing. From that point on, the flow into `AlarmConditionState` nodes, the `AlarmAck` session check, and the Historian sink is shared — see [AlarmTracking.md](AlarmTracking.md).
|
||||
|
||||
@@ -94,7 +94,14 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
|
||||
|
||||
## Documented follow-ups (non-blocking)
|
||||
|
||||
- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
|
||||
- **Scripted-alarm redeploy recovery — FIXED (issue #487), see `docs/ScriptedAlarms.md` §"Post-(re)load
|
||||
condition-node re-assert".** Landed as the proposed shape below: `ScriptedAlarmHostActor.ReassertConditionNodes`
|
||||
in `OnAlarmsLoaded`, node-only, never the `alerts` topic. Two deviations from the sketch, both deliberate:
|
||||
it reads a new `ScriptedAlarmEngine.GetProjections()` rather than `GetState(alarmId)` + `LoadedAlarmIds`
|
||||
(a projection also carries the definition's severity, the resolved message template, and the last-observed
|
||||
worst-input quality — `GetState` alone would have clobbered severity/message on the node); and it skips
|
||||
alarms sitting in the Part 9 no-event position, so a deploy where nothing is in alarm writes no condition
|
||||
nodes at all. The original deferral text follows for context.
|
||||
The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
|
||||
`RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
|
||||
fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
|
||||
|
||||
@@ -334,6 +334,48 @@ public sealed class ScriptedAlarmEngine : IDisposable
|
||||
public IReadOnlyCollection<AlarmConditionState> GetAllStates()
|
||||
=> _alarms.Values.Select(a => a.Condition).ToArray();
|
||||
|
||||
/// <summary>
|
||||
/// #487 — the currently-held condition of every loaded alarm, packaged with the definition's
|
||||
/// severity, the rendered message template and the last-observed input quality, i.e. everything
|
||||
/// a caller needs to <b>project</b> the alarm onto an OPC UA condition node without waiting for
|
||||
/// the next transition.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>This is a read, not an emission.</b> It deliberately returns a distinct type rather
|
||||
/// than a <see cref="ScriptedAlarmEvent"/>, and it never touches <see cref="OnEvent"/>. A
|
||||
/// still-active alarm produces <see cref="EmissionKind.None"/> at load (there is no
|
||||
/// transition to report), so the transition stream cannot carry the current state — that is
|
||||
/// exactly why the caller needs this. Routing these through the emission path instead would
|
||||
/// append a duplicate historian / <c>alerts</c> row on every deploy, so the shape here is
|
||||
/// intentionally one the emission machinery cannot consume.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Synchronization.</b> Reads <c>_alarms</c> and <c>_valueCache</c> — both
|
||||
/// <see cref="ConcurrentDictionary{TKey, TValue}"/> — without taking <c>_evalGate</c>, the
|
||||
/// same posture as <see cref="GetState"/> / <see cref="GetAllStates"/>. Each projection is
|
||||
/// individually coherent; the set as a whole is not an atomic snapshot across a concurrent
|
||||
/// re-evaluation. That is sufficient for the node-projection use: an evaluation racing this
|
||||
/// read publishes its own transition through the normal path.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <returns>One projection per loaded alarm; empty before the first <see cref="LoadAsync"/>.</returns>
|
||||
public IReadOnlyList<ScriptedAlarmProjection> GetProjections()
|
||||
{
|
||||
var projections = new List<ScriptedAlarmProjection>(_alarms.Count);
|
||||
foreach (var (alarmId, state) in _alarms)
|
||||
{
|
||||
projections.Add(new ScriptedAlarmProjection(
|
||||
AlarmId: alarmId,
|
||||
Severity: state.Definition.Severity,
|
||||
Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
|
||||
Condition: state.Condition,
|
||||
WorstInputStatusCode: LastWorstStatus(alarmId)));
|
||||
}
|
||||
|
||||
return projections;
|
||||
}
|
||||
|
||||
/// <summary>Acknowledges the specified alarm on behalf of the given user.</summary>
|
||||
/// <param name="alarmId">The alarm identifier.</param>
|
||||
/// <param name="user">The user performing the acknowledgment.</param>
|
||||
@@ -974,6 +1016,26 @@ public sealed record ScriptedAlarmEvent(
|
||||
// the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
|
||||
uint WorstInputStatusCode = 0u);
|
||||
|
||||
/// <summary>
|
||||
/// #487 — a loaded alarm's current condition, ready to be projected onto its OPC UA node.
|
||||
/// Produced by <see cref="ScriptedAlarmEngine.GetProjections"/> on demand; it is <b>not</b> an
|
||||
/// emission and never travels the <see cref="ScriptedAlarmEngine.OnEvent"/> path, so it can never
|
||||
/// become an <c>alerts</c> row or a historian record. Deliberately a separate type from
|
||||
/// <see cref="ScriptedAlarmEvent"/> — it carries no <see cref="EmissionKind"/>, so there is nothing
|
||||
/// for a future refactor to mistake for a transition.
|
||||
/// </summary>
|
||||
/// <param name="AlarmId">The alarm identifier (also its OPC UA condition NodeId).</param>
|
||||
/// <param name="Severity">The definition's severity bucket.</param>
|
||||
/// <param name="Message">The message template resolved against the current value cache.</param>
|
||||
/// <param name="Condition">The Part 9 condition state the engine currently holds.</param>
|
||||
/// <param name="WorstInputStatusCode">The last-observed worst OPC UA StatusCode across the alarm's inputs.</param>
|
||||
public sealed record ScriptedAlarmProjection(
|
||||
string AlarmId,
|
||||
AlarmSeverity Severity,
|
||||
string Message,
|
||||
AlarmConditionState Condition,
|
||||
uint WorstInputStatusCode);
|
||||
|
||||
/// <summary>
|
||||
/// Upstream source abstraction — intentionally identical shape to the virtual-tag
|
||||
/// engine's so Stream G can compose them behind one driver bridge.
|
||||
|
||||
@@ -288,8 +288,101 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
// so a re-Apply with a fresh union simply supersedes the old one — no explicit unregister needed.
|
||||
_mux?.Tell(new DependencyMuxActor.RegisterInterest(msg.DepRefs, Self));
|
||||
_log.Debug("ScriptedAlarmHost: loaded; registered mux interest for {Count} dep refs", msg.DepRefs.Count);
|
||||
|
||||
ReassertConditionNodes();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #487 — re-project every loaded alarm that is NOT in the Part 9 "no-event" position back onto
|
||||
/// its condition node, right after a (re)load.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>The bug this closes.</b> A deploy that triggers a FULL address-space rebuild clears
|
||||
/// <c>_alarmConditions</c> and re-materialises every condition node fresh — inactive, acked,
|
||||
/// confirmed, unshelved. The engine, meanwhile, restores each alarm's persisted state and
|
||||
/// re-derives Active from the predicate, so a still-active alarm ends the reload
|
||||
/// <i>correctly Active in the engine</i> but produces <see cref="EmissionKind.None"/> (there
|
||||
/// is no transition to report) — and <see cref="OnEngineEmission"/> drops None. Nothing then
|
||||
/// writes the node, so an active alarm with static dependencies reads NORMAL to every OPC UA
|
||||
/// client until its next real transition, which may never come. Same shape as the VirtualTag
|
||||
/// redeploy-reset the <c>ReassertValue</c> fix closed.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Node-only, by construction.</b> This sends <see cref="OpcUaPublishActor.AlarmStateUpdate"/>
|
||||
/// and nothing else — it never publishes to the <c>alerts</c> topic and never touches the
|
||||
/// telemetry hub. That is the whole reason it reads <see cref="ScriptedAlarmEngine.GetProjections"/>
|
||||
/// (a plain read returning <see cref="ScriptedAlarmProjection"/>) rather than re-emitting: an
|
||||
/// <c>alerts</c> row per alarm per deploy would append a duplicate historian / AVEVA record
|
||||
/// every time anyone deploys — the alarm analogue of the VirtualTag M1 historian issue.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Ordering.</b> <c>DriverHostActor</c> Tells <c>RebuildAddressSpace</c> to the publish
|
||||
/// actor BEFORE it Tells <see cref="ApplyScriptedAlarms"/> here, and this runs later still —
|
||||
/// after an <c>await</c>ed <see cref="ScriptedAlarmEngine.LoadAsync"/> pipes back. Both Tells
|
||||
/// enqueue synchronously into the publish actor's local mailbox, so the re-materialise is
|
||||
/// already queued ahead of these writes: the node exists by the time they are handled.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why only non-normal alarms.</b> An alarm sitting in the no-event position already
|
||||
/// matches what the materialise just built, so writing it would be a pure no-op on state —
|
||||
/// but it would still overwrite the node's Message (the materialise seeds it with the alarm's
|
||||
/// display name; a projection carries the resolved template). Skipping them keeps a
|
||||
/// never-fired alarm byte-identical to a deploy without this pass, and keeps the write count
|
||||
/// at zero on the overwhelmingly common all-normal deploy.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Duplicate events.</b> None. <c>WriteAlarmCondition</c>'s delta-gate compares the
|
||||
/// incoming snapshot against the node's CURRENT state, so a re-assert onto a freshly
|
||||
/// materialised (normal) node is a genuine delta and fires one Part 9 event — correct, since
|
||||
/// the rebuild reset what clients see — while a re-assert onto a surgically-preserved node
|
||||
/// that already holds the same state is suppressed. This write is ungated by redundancy role,
|
||||
/// like every other node write here: the Secondary keeps its address space warm for failover.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void ReassertConditionNodes()
|
||||
{
|
||||
var reasserted = 0;
|
||||
foreach (var projection in _engine.GetProjections())
|
||||
{
|
||||
if (IsInNoEventPosition(projection.Condition))
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
_publishActor.Tell(new OpcUaPublishActor.AlarmStateUpdate(
|
||||
AlarmNodeId: projection.AlarmId,
|
||||
State: ToSnapshot(projection),
|
||||
// The instant the state we are re-asserting actually came about — NOT the deploy time.
|
||||
// A client reading Time/ReceiveTime after a rebuild should still see when the alarm went
|
||||
// active, not when the address space happened to be rebuilt.
|
||||
TimestampUtc: projection.Condition.LastTransitionUtc,
|
||||
Realm: AddressSpaceRealm.Uns));
|
||||
reasserted++;
|
||||
}
|
||||
|
||||
if (reasserted > 0)
|
||||
{
|
||||
_log.Info("ScriptedAlarmHost: re-asserted {Count} non-normal condition node(s) after (re)load", reasserted);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Whether <paramref name="condition"/> sits in the Part 9 "no-event" position — exactly the
|
||||
/// state <see cref="AlarmConditionState.Fresh"/> produces and a freshly materialised condition node
|
||||
/// already carries. Anything else (active, unacknowledged, unconfirmed, disabled, or shelved) is state
|
||||
/// a rebuild would silently lose, so it is what <see cref="ReassertConditionNodes"/> re-projects.</summary>
|
||||
/// <param name="condition">The engine's current condition state.</param>
|
||||
/// <returns><c>true</c> when the condition carries nothing a rebuild could lose.</returns>
|
||||
private static bool IsInNoEventPosition(AlarmConditionState condition)
|
||||
=> condition.Enabled == AlarmEnabledState.Enabled
|
||||
&& condition.Active == AlarmActiveState.Inactive
|
||||
&& condition.Acked == AlarmAckedState.Acknowledged
|
||||
&& condition.Confirmed == AlarmConfirmedState.Confirmed
|
||||
&& condition.Shelving.Kind == ShelvingKind.Unshelved;
|
||||
|
||||
private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg)
|
||||
{
|
||||
// Feed the live value into the upstream the engine subscribes from. #478 — carry the source
|
||||
@@ -594,18 +687,38 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
|
||||
/// Severity is the OPC UA 1..1000 value <see cref="SeverityToInt"/> derives from the coarse engine
|
||||
/// bucket, cast to the <c>ushort</c> the SDK <c>SetSeverity</c> expects. Shelving's 3-way Core kind
|
||||
/// maps 1:1 onto the Commons <see cref="AlarmShelvingKind"/>.</summary>
|
||||
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmEvent e) => new(
|
||||
Active: e.Condition.Active == AlarmActiveState.Active,
|
||||
Acknowledged: e.Condition.Acked == AlarmAckedState.Acknowledged,
|
||||
Confirmed: e.Condition.Confirmed == AlarmConfirmedState.Confirmed,
|
||||
Enabled: e.Condition.Enabled == AlarmEnabledState.Enabled,
|
||||
Shelving: MapShelving(e.Condition.Shelving.Kind),
|
||||
Severity: (ushort)SeverityToInt(e.Severity),
|
||||
Message: e.Message,
|
||||
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmEvent e)
|
||||
=> ToSnapshot(e.Condition, e.Severity, e.Message, e.WorstInputStatusCode);
|
||||
|
||||
/// <summary>#487 — the same projection for a <see cref="ScriptedAlarmProjection"/>: a state READ used
|
||||
/// to restore a condition node after a rebuild, never an emission. Shares
|
||||
/// <see cref="ToSnapshot(AlarmConditionState, AlarmSeverity, string, uint)"/> with the transition path
|
||||
/// so a re-asserted node is byte-identical to what the alarm's own transition would have written.</summary>
|
||||
/// <param name="p">The engine projection to map.</param>
|
||||
/// <returns>The Commons snapshot the SDK sink projects onto the condition node.</returns>
|
||||
private static AlarmConditionSnapshot ToSnapshot(ScriptedAlarmProjection p)
|
||||
=> ToSnapshot(p.Condition, p.Severity, p.Message, p.WorstInputStatusCode);
|
||||
|
||||
/// <summary>The single Core → Commons condition projection, shared by the emission and re-assert
|
||||
/// paths.</summary>
|
||||
/// <param name="condition">The Part 9 condition state.</param>
|
||||
/// <param name="severity">The engine's coarse severity bucket.</param>
|
||||
/// <param name="message">The resolved condition message.</param>
|
||||
/// <param name="worstInputStatusCode">The worst OPC UA StatusCode across the script's input tags.</param>
|
||||
/// <returns>The Commons snapshot the SDK sink projects onto the condition node.</returns>
|
||||
private static AlarmConditionSnapshot ToSnapshot(
|
||||
AlarmConditionState condition, AlarmSeverity severity, string message, uint worstInputStatusCode) => new(
|
||||
Active: condition.Active == AlarmActiveState.Active,
|
||||
Acknowledged: condition.Acked == AlarmAckedState.Acknowledged,
|
||||
Confirmed: condition.Confirmed == AlarmConfirmedState.Confirmed,
|
||||
Enabled: condition.Enabled == AlarmEnabledState.Enabled,
|
||||
Shelving: MapShelving(condition.Shelving.Kind),
|
||||
Severity: (ushort)SeverityToInt(severity),
|
||||
Message: message,
|
||||
// #478 — the condition's Quality is the worst quality across the script's input tags at evaluation
|
||||
// time (carried on the event by the engine). A transition fired while an input is Uncertain projects
|
||||
// Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
|
||||
Quality: QualityFromStatus(e.WorstInputStatusCode));
|
||||
Quality: QualityFromStatus(worstInputStatusCode));
|
||||
|
||||
/// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
|
||||
/// mirror (the Commons assembly can't see the Core enum).</summary>
|
||||
|
||||
+180
@@ -0,0 +1,180 @@
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// #487 — <see cref="ScriptedAlarmEngine.GetProjections"/>: the read that lets a caller restore an
|
||||
/// alarm's condition node after a full address-space rebuild, without waiting for the alarm's next
|
||||
/// transition (which for a still-active alarm with static dependencies may never come).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The load that motivates this is deliberately the one that emits NOTHING: an alarm whose persisted
|
||||
/// state is already Active and whose predicate is still true produces <see cref="EmissionKind.None"/>,
|
||||
/// so the transition stream cannot carry its state. These tests pin both halves — the projection does
|
||||
/// report the state, and reading it does not fire an emission.
|
||||
/// </remarks>
|
||||
[Trait("Category", "Unit")]
|
||||
public sealed class ScriptedAlarmProjectionTests
|
||||
{
|
||||
private static ScriptedAlarmDefinition Alarm(
|
||||
string id = "HighTemp",
|
||||
string predicate = """return (int)ctx.GetTag("Temp").Value > 100;""",
|
||||
string msg = "Temp is {Temp}",
|
||||
AlarmSeverity sev = AlarmSeverity.High) =>
|
||||
new(AlarmId: id,
|
||||
EquipmentPath: "Plant/Line1/Reactor",
|
||||
AlarmName: id,
|
||||
Kind: AlarmKind.AlarmCondition,
|
||||
Severity: sev,
|
||||
MessageTemplate: msg,
|
||||
PredicateScriptSource: predicate);
|
||||
|
||||
/// <summary>An already-Active persisted state, as a store would hold it across a restart/redeploy.</summary>
|
||||
private static AlarmConditionState PersistedActive(
|
||||
string alarmId = "HighTemp",
|
||||
AlarmAckedState acked = AlarmAckedState.Unacknowledged,
|
||||
DateTime? lastTransitionUtc = null) =>
|
||||
new(alarmId,
|
||||
AlarmEnabledState.Enabled,
|
||||
AlarmActiveState.Active,
|
||||
acked,
|
||||
AlarmConfirmedState.Confirmed,
|
||||
ShelvingState.Unshelved,
|
||||
lastTransitionUtc ?? DateTime.UtcNow,
|
||||
LastActiveUtc: lastTransitionUtc ?? DateTime.UtcNow,
|
||||
LastClearedUtc: null,
|
||||
LastAckUtc: null, LastAckUser: null, LastAckComment: null,
|
||||
LastConfirmUtc: null, LastConfirmUser: null, LastConfirmComment: null,
|
||||
Comments: []);
|
||||
|
||||
private static ScriptedAlarmEngine Build(FakeUpstream up, IAlarmStateStore store)
|
||||
{
|
||||
var logger = new LoggerConfiguration().CreateLogger();
|
||||
return new ScriptedAlarmEngine(up, store, new ScriptLoggerFactory(logger), logger);
|
||||
}
|
||||
|
||||
/// <summary>The core of #487: a reload of a still-active alarm emits NOTHING, yet the projection
|
||||
/// reports it Active — so a caller has a way to restore the node the rebuild reset.</summary>
|
||||
[Fact]
|
||||
public async Task A_still_active_alarm_emits_nothing_on_reload_but_projects_Active()
|
||||
{
|
||||
var up = new FakeUpstream();
|
||||
up.Set("Temp", 150); // predicate still true — no transition to report
|
||||
var store = new InMemoryAlarmStateStore();
|
||||
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
|
||||
|
||||
using var eng = Build(up, store);
|
||||
var emissions = new List<ScriptedAlarmEvent>();
|
||||
eng.OnEvent += (_, e) => { lock (emissions) emissions.Add(e); }; // subscribe BEFORE the load
|
||||
|
||||
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
|
||||
|
||||
// The premise. If this ever starts emitting, the bug's shape has changed and the re-assert
|
||||
// in ScriptedAlarmHostActor may have become a duplicate rather than the only source of truth.
|
||||
lock (emissions)
|
||||
{
|
||||
emissions.ShouldBeEmpty("a still-active alarm has no transition to report on reload");
|
||||
}
|
||||
|
||||
var projection = eng.GetProjections().ShouldHaveSingleItem();
|
||||
projection.AlarmId.ShouldBe("HighTemp");
|
||||
projection.Condition.Active.ShouldBe(AlarmActiveState.Active);
|
||||
projection.Condition.Acked.ShouldBe(AlarmAckedState.Unacknowledged);
|
||||
}
|
||||
|
||||
/// <summary>The projection carries the definition's severity and the message template resolved against
|
||||
/// the live value cache — so a re-asserted node is indistinguishable from one a real transition wrote.</summary>
|
||||
[Fact]
|
||||
public async Task Projection_carries_definition_severity_and_a_resolved_message()
|
||||
{
|
||||
var up = new FakeUpstream();
|
||||
up.Set("Temp", 150);
|
||||
var store = new InMemoryAlarmStateStore();
|
||||
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
|
||||
|
||||
using var eng = Build(up, store);
|
||||
await eng.LoadAsync([Alarm(sev: AlarmSeverity.Critical)], TestContext.Current.CancellationToken);
|
||||
|
||||
var projection = eng.GetProjections().ShouldHaveSingleItem();
|
||||
projection.Severity.ShouldBe(AlarmSeverity.Critical);
|
||||
projection.Message.ShouldBe("Temp is 150", "the template resolves against the current value cache");
|
||||
}
|
||||
|
||||
/// <summary>Reading projections is a pure read — it must never raise <see cref="ScriptedAlarmEngine.OnEvent"/>,
|
||||
/// or every deploy would append a duplicate historian / alerts row.</summary>
|
||||
[Fact]
|
||||
public async Task Reading_projections_never_fires_an_emission()
|
||||
{
|
||||
var up = new FakeUpstream();
|
||||
up.Set("Temp", 150);
|
||||
var store = new InMemoryAlarmStateStore();
|
||||
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
|
||||
|
||||
using var eng = Build(up, store);
|
||||
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
|
||||
|
||||
var emissions = 0;
|
||||
eng.OnEvent += (_, _) => Interlocked.Increment(ref emissions);
|
||||
|
||||
for (var i = 0; i < 5; i++) eng.GetProjections().ShouldNotBeEmpty();
|
||||
|
||||
Volatile.Read(ref emissions).ShouldBe(0);
|
||||
}
|
||||
|
||||
/// <summary>An alarm that is genuinely normal projects the no-event position, so a caller can tell
|
||||
/// "nothing to restore" from "restore this".</summary>
|
||||
[Fact]
|
||||
public async Task A_normal_alarm_projects_the_no_event_position()
|
||||
{
|
||||
var up = new FakeUpstream();
|
||||
up.Set("Temp", 50); // predicate false
|
||||
using var eng = Build(up, new InMemoryAlarmStateStore());
|
||||
|
||||
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
|
||||
|
||||
var projection = eng.GetProjections().ShouldHaveSingleItem();
|
||||
projection.Condition.Active.ShouldBe(AlarmActiveState.Inactive);
|
||||
projection.Condition.Acked.ShouldBe(AlarmAckedState.Acknowledged);
|
||||
projection.Condition.Enabled.ShouldBe(AlarmEnabledState.Enabled);
|
||||
projection.Condition.Shelving.Kind.ShouldBe(ShelvingKind.Unshelved);
|
||||
}
|
||||
|
||||
/// <summary>The worst-input quality rides the projection, so restoring a node whose script inputs are
|
||||
/// Bad does not clobber its Quality back to Good (#478's invariant, on the re-assert path).</summary>
|
||||
[Fact]
|
||||
public async Task Projection_carries_the_last_observed_worst_input_quality()
|
||||
{
|
||||
var up = new FakeUpstream();
|
||||
up.Set("Temp", 150);
|
||||
var store = new InMemoryAlarmStateStore();
|
||||
await store.SaveAsync(PersistedActive(), TestContext.Current.CancellationToken);
|
||||
|
||||
using var eng = Build(up, store);
|
||||
await eng.LoadAsync([Alarm()], TestContext.Current.CancellationToken);
|
||||
eng.GetProjections().ShouldHaveSingleItem().WorstInputStatusCode.ShouldBe(0u, "inputs start Good");
|
||||
|
||||
up.Push("Temp", 150, 0x80000000u); // input goes Bad — condition freezes, quality is annotated
|
||||
|
||||
// The push re-evaluates on a background worker; poll for the tracked bucket to move.
|
||||
var deadline = DateTime.UtcNow.AddSeconds(10);
|
||||
while (eng.GetProjections()[0].WorstInputStatusCode == 0u && DateTime.UtcNow < deadline)
|
||||
{
|
||||
await Task.Delay(20, TestContext.Current.CancellationToken);
|
||||
}
|
||||
|
||||
(eng.GetProjections()[0].WorstInputStatusCode >> 30).ShouldBe(2u, "a Bad input projects Bad quality");
|
||||
}
|
||||
|
||||
/// <summary>Before the first load there is nothing to project — the read is safe, not a throw.</summary>
|
||||
[Fact]
|
||||
public void Projections_are_empty_before_the_first_load()
|
||||
{
|
||||
using var eng = Build(new FakeUpstream(), new InMemoryAlarmStateStore());
|
||||
eng.GetProjections().ShouldBeEmpty();
|
||||
}
|
||||
}
|
||||
+164
-4
@@ -86,20 +86,22 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
|
||||
Retain: false,
|
||||
Enabled: true);
|
||||
|
||||
private static ScriptedAlarmEngine BuildEngine(DependencyMuxTagUpstreamSource upstream)
|
||||
private static ScriptedAlarmEngine BuildEngine(
|
||||
DependencyMuxTagUpstreamSource upstream, IAlarmStateStore? store = null)
|
||||
{
|
||||
var logger = new LoggerConfiguration().CreateLogger();
|
||||
return new ScriptedAlarmEngine(upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
|
||||
return new ScriptedAlarmEngine(
|
||||
upstream, store ?? new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
|
||||
}
|
||||
|
||||
/// <summary>The local node id used by the redundancy-gating tests.</summary>
|
||||
private static readonly NodeId LocalNode = new("node-A");
|
||||
|
||||
private (IActorRef Host, DependencyMuxTagUpstreamSource Upstream) Spawn(
|
||||
TestProbe publish, TestProbe mux, NodeId? localNode = null)
|
||||
TestProbe publish, TestProbe mux, NodeId? localNode = null, IAlarmStateStore? store = null)
|
||||
{
|
||||
var upstream = new DependencyMuxTagUpstreamSource();
|
||||
var engine = BuildEngine(upstream);
|
||||
var engine = BuildEngine(upstream, store);
|
||||
var host = Sys.ActorOf(ScriptedAlarmHostActor.Props(publish.Ref, mux.Ref, upstream, engine, localNode));
|
||||
return (host, upstream);
|
||||
}
|
||||
@@ -794,4 +796,162 @@ public sealed class ScriptedAlarmHostActorTests : RuntimeActorTestBase
|
||||
evt.AlarmId.ShouldBe("alm-1");
|
||||
evt.TransitionKind.ShouldBe("Activated");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// #487 — post-(re)load condition-node re-assert.
|
||||
//
|
||||
// A deploy that triggers a FULL address-space rebuild re-materialises every condition node fresh
|
||||
// (inactive/acked/confirmed/unshelved). The engine reloads its persisted state and re-derives Active
|
||||
// from the predicate, so it ends up CORRECT — but a still-active alarm has no transition to report,
|
||||
// yielding EmissionKind.None, which OnEngineEmission drops. Without the re-assert the node reads
|
||||
// NORMAL to every OPC UA client until the alarm's next real transition — which, for an alarm with
|
||||
// static dependencies, may never come.
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
/// <summary>A persisted condition state as it survives a restart / redeploy, seeded into the store the
|
||||
/// host's engine loads from.</summary>
|
||||
private static AlarmConditionState Persisted(
|
||||
string alarmId = "alm-1",
|
||||
AlarmEnabledState enabled = AlarmEnabledState.Enabled,
|
||||
AlarmActiveState active = AlarmActiveState.Active,
|
||||
AlarmAckedState acked = AlarmAckedState.Unacknowledged,
|
||||
AlarmConfirmedState confirmed = AlarmConfirmedState.Confirmed,
|
||||
ShelvingKind shelving = ShelvingKind.Unshelved,
|
||||
DateTime? lastTransitionUtc = null) =>
|
||||
new(alarmId,
|
||||
enabled,
|
||||
active,
|
||||
acked,
|
||||
confirmed,
|
||||
shelving == ShelvingKind.Unshelved ? ShelvingState.Unshelved : new ShelvingState(shelving, null),
|
||||
lastTransitionUtc ?? DateTime.UtcNow,
|
||||
LastActiveUtc: active == AlarmActiveState.Active ? lastTransitionUtc ?? DateTime.UtcNow : null,
|
||||
LastClearedUtc: null,
|
||||
LastAckUtc: null, LastAckUser: null, LastAckComment: null,
|
||||
LastConfirmUtc: null, LastConfirmUser: null, LastConfirmComment: null,
|
||||
Comments: []);
|
||||
|
||||
/// <summary>Seed <paramref name="state"/> into a fresh store and hand it back for
|
||||
/// <see cref="Spawn"/>. The engine reads it during LoadAsync, exactly as it would after a restart.</summary>
|
||||
private static IAlarmStateStore StoreWith(AlarmConditionState state)
|
||||
{
|
||||
var store = new InMemoryAlarmStateStore();
|
||||
store.SaveAsync(state, CancellationToken.None).GetAwaiter().GetResult();
|
||||
return store;
|
||||
}
|
||||
|
||||
/// <summary>#487, the regression: an alarm whose persisted state is ACTIVE emits nothing on reload
|
||||
/// (no transition to report), but the host must still write its state onto the freshly materialised
|
||||
/// condition node — otherwise an active alarm silently reads normal after a full-rebuild deploy.</summary>
|
||||
[Fact]
|
||||
public void Reload_reasserts_an_active_alarm_onto_its_condition_node()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted()));
|
||||
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(severity: 800) }));
|
||||
|
||||
var state = publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout);
|
||||
state.AlarmNodeId.ShouldBe("alm-1");
|
||||
state.Realm.ShouldBe(AddressSpaceRealm.Uns);
|
||||
state.State.Active.ShouldBeTrue("the persisted state was Active — the rebuilt node must say so");
|
||||
state.State.Acknowledged.ShouldBeFalse();
|
||||
state.State.Enabled.ShouldBeTrue();
|
||||
// The projection carries the definition's severity + resolved message, so a re-asserted node is
|
||||
// indistinguishable from one the alarm's own transition would have written.
|
||||
state.State.Severity.ShouldBe((ushort)1000); // 800 → Critical bucket → 1000
|
||||
state.State.Message.ShouldBe("condition");
|
||||
}
|
||||
|
||||
/// <summary>The re-assert is node-ONLY: it must never reach the <c>alerts</c> topic. That topic is the
|
||||
/// historization path, so an alerts row per alarm per deploy would append a duplicate historian /
|
||||
/// AVEVA record every time anyone deploys.</summary>
|
||||
[Fact]
|
||||
public void Reassert_never_publishes_to_the_alerts_topic()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var alerts = CreateTestProbe();
|
||||
SubscribeToAlerts(alerts);
|
||||
|
||||
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted()));
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
|
||||
|
||||
// Wait for the re-assert to actually happen FIRST — otherwise the absence window below could
|
||||
// elapse before the load even completed and would prove nothing.
|
||||
publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout);
|
||||
|
||||
// Absence assertion: the elapsed time IS the assertion, so this stays deliberately short.
|
||||
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>An alarm sitting in the Part 9 no-event position already matches what the materialise
|
||||
/// built, so it is NOT re-asserted — a deploy where nothing is in alarm writes no condition nodes at
|
||||
/// all, and a never-fired alarm keeps the display name the materialise seeded as its Message.</summary>
|
||||
[Fact]
|
||||
public void A_normal_alarm_is_not_reasserted()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var (host, _) = Spawn(publish, mux); // empty store ⇒ Fresh ⇒ no-event position
|
||||
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
|
||||
|
||||
// Sequence the absence: the re-assert (if any) happens right after the mux registration, so
|
||||
// waiting for that first makes the window below cover the moment the write would have landed.
|
||||
mux.ExpectMsg<DependencyMuxActor.RegisterInterest>(Timeout);
|
||||
publish.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>Not just Active: every field a rebuild would silently lose is restored. A shelved (but
|
||||
/// inactive) alarm is re-asserted too — a rebuilt node would otherwise come back unshelved, so an
|
||||
/// operator's suppression would evaporate on the next deploy.</summary>
|
||||
[Fact]
|
||||
public void Reload_reasserts_a_shelved_but_inactive_alarm()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted(
|
||||
active: AlarmActiveState.Inactive,
|
||||
acked: AlarmAckedState.Acknowledged,
|
||||
shelving: ShelvingKind.OneShot)));
|
||||
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
|
||||
|
||||
var state = publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout);
|
||||
state.State.Active.ShouldBeFalse();
|
||||
state.State.Shelving.ShouldBe(AlarmShelvingKind.OneShot);
|
||||
}
|
||||
|
||||
/// <summary>The re-asserted timestamp is when the state actually came about, NOT the deploy instant —
|
||||
/// a client reading Time/ReceiveTime after a rebuild should still see when the alarm went active.</summary>
|
||||
[Fact]
|
||||
public void Reassert_carries_the_conditions_own_transition_timestamp()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var wentActiveAt = DateTime.UtcNow.AddHours(-3);
|
||||
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted(lastTransitionUtc: wentActiveAt)));
|
||||
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan() }));
|
||||
|
||||
publish.ExpectMsg<OpcUaPublishActor.AlarmStateUpdate>(Timeout)
|
||||
.TimestampUtc.ShouldBe(wentActiveAt);
|
||||
}
|
||||
|
||||
/// <summary>A disabled PLAN exposes no condition node at all (MaterialiseScriptedAlarms skips it) and
|
||||
/// is not loaded into the engine — so it must not be re-asserted either, or the write would target a
|
||||
/// node that does not exist.</summary>
|
||||
[Fact]
|
||||
public void A_disabled_plan_is_not_reasserted()
|
||||
{
|
||||
var publish = CreateTestProbe();
|
||||
var mux = CreateTestProbe();
|
||||
var (host, _) = Spawn(publish, mux, store: StoreWith(Persisted()));
|
||||
|
||||
host.Tell(new ScriptedAlarmHostActor.ApplyScriptedAlarms(new[] { Plan(enabled: false) }));
|
||||
|
||||
publish.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user