feat(alarms): scripted condition Quality from worst-of-input tag quality (#478)
v2-ci / build (pull_request) Successful in 3m48s
v2-ci / unit-tests (pull_request) Failing after 11m0s

Layer 3 of #477: a scripted alarm's condition Quality now reflects the WORST
quality across its input tags, mirroring the native OT semantic (#477 L2).

Plumbing (quality was silently discarded twice on the live path):
- VirtualTagActor.DependencyValueChanged gains Quality (defaulted Good); the
  DependencyMuxActor forwards the published AttributeValuePublished.Quality it
  already carried; ScriptedAlarmHostActor.OnDependencyChanged pushes the real
  quality into the engine (was hardcoded 0u/Good).

Engine (Core.ScriptedAlarms):
- ScriptedAlarmEngine computes worst-of-input quality each eval (skipping
  not-yet-published inputs, which are a readiness concern, not a quality signal)
  and carries it on ScriptedAlarmEvent.WorstInputStatusCode.
- A real transition carries the current worst quality so ToSnapshot's full
  snapshot doesn't clobber quality back to Good (e.g. transition while Uncertain).
- A Bad input freezes the condition (no transition), like a comms-lost native
  driver; a quality-bucket change with no transition emits the new
  EmissionKind.QualityChanged, routed to the existing #477-L2
  AlarmQualityUpdate -> WriteAlarmQuality node path (quality only, no /alerts
  row, no historian write). ScriptedAlarmSource skips QualityChanged so it never
  fabricates a phantom IAlarmSource event.

Host: ToSnapshot maps WorstInputStatusCode -> OpcUaQuality; OnEngineEmission
routes QualityChanged out of band.

Tests (TDD, RED-first): engine worst-carry + Bad/restore QualityChanged +
unchanged-bucket-no-emit; source swallows QualityChanged; mux forwards quality;
host Bad-dep -> AlarmQualityUpdate(no alerts) + transition snapshot carries worst.
Docs: AlarmTracking.md Layer-3 section + design doc.

Closes #478
This commit is contained in:
Joseph Doherty
2026-07-17 15:56:06 -04:00
parent 6dda0549e2
commit 8c5e2be92e
12 changed files with 449 additions and 21 deletions
@@ -377,4 +377,9 @@ public enum EmissionKind
Enabled,
Disabled,
CommentAdded,
/// <summary>#478 — the worst-of-input source quality changed bucket (Good/Uncertain/Bad) with no
/// accompanying Part 9 state transition. Delivered out of band via the dedicated
/// <c>WriteAlarmQuality</c> node path, never through the <c>IAlarmSource</c> fan-out (it is not a
/// state change and must not materialize or historize a condition).</summary>
QualityChanged,
}
@@ -62,6 +62,17 @@ public sealed class ScriptedAlarmEngine : IDisposable
private readonly ConcurrentDictionary<string, AlarmScratch> _scratchByAlarmId =
new(StringComparer.Ordinal);
/// <summary>
/// #478 — last emitted worst-of-input quality bucket per alarm (0 = Good, 1 = Uncertain,
/// 2 = Bad), computed each evaluation over the refilled read cache. A change in this bucket
/// with no accompanying Part 9 state transition drives a standalone
/// <see cref="EmissionKind.QualityChanged"/> emission (a Bad input freezes the condition — no
/// transition — so quality can't ride one, exactly like a comms-lost native driver). Only ever
/// mutated under <c>_evalGate</c>; cleared alongside <see cref="_alarms"/> on load/dispose.
/// </summary>
private readonly ConcurrentDictionary<string, int> _lastQualityBucketByAlarmId =
new(StringComparer.Ordinal);
/// <summary>
/// Compile cache for every alarm predicate. Routes <see cref="LoadAsync"/>'s
/// <see cref="ScriptEvaluator{TContext, TResult}.Compile"/> calls through the
@@ -203,6 +214,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
// have changed (different Inputs, different Logger), so any reuse would be
// unsafe.
_scratchByAlarmId.Clear();
_lastQualityBucketByAlarmId.Clear();
// Dispose every compiled-predicate ALC from the prior generation BEFORE we
// recompile this one. Skipping this is what made the earlier fix a
// no-op in production.
@@ -412,7 +424,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
// OnEvent dispatch until after Release() so a slow subscriber or a
// subscriber that re-enters the engine doesn't block / deadlock.
if (result.Emission != EmissionKind.None)
pending = BuildEmission(state, result.State, result.Emission);
pending = BuildEmission(state, result.State, result.Emission, LastWorstStatus(alarmId));
else if (result.NoOpReason is { } reason)
{
// The Part9StateMachine remarks promise a diagnostic log line for
@@ -513,13 +525,27 @@ public sealed class ScriptedAlarmEngine : IDisposable
_ => new AlarmScratch(state.Inputs, state.Logger, _clock));
RefillReadCache(scratch.ReadCache, state.Inputs);
// #478 — worst OPC UA quality across the alarm's inputs, computed BEFORE the readiness
// short-circuit so an outright-Bad input is still observed. A bucket change with no state
// transition is delivered out of band as a QualityChanged emission (see below).
var worstStatus = WorstInputStatus(scratch.ReadCache);
var qualityBucketChanged = TrackQualityBucket(state.Definition.AlarmId, worstStatus);
// Cold-start guard — skip the predicate when any referenced upstream tag has no
// cached value yet (the upstream subscription hasn't delivered its first push).
// Without this, predicates that cast `(double)ctx.GetTag(path).Value` throw NRE on
// every tick until the cache fills, spamming the log with identical stack traces.
// Bad quality is treated the same: the input isn't available at the predicate's
// expected type, so the only defensible move is to hold the prior condition state.
if (!AreInputsReady(scratch.ReadCache)) return seed;
if (!AreInputsReady(scratch.ReadCache))
{
// The condition is frozen (can't trust its state), but its source quality just changed
// bucket — annotate it out of band so a comms-lost / Bad-input scripted condition reports
// Bad, mirroring the native OT path.
if (qualityBucketChanged)
pendingEmissions.Add(BuildQualityEmission(state, seed, worstStatus));
return seed;
}
var context = scratch.Context;
@@ -544,10 +570,19 @@ public sealed class ScriptedAlarmEngine : IDisposable
}
var result = Part9StateMachine.ApplyPredicate(seed, predicateTrue, nowUtc);
if (result.Emission != EmissionKind.None)
var transition = result.Emission != EmissionKind.None
? BuildEmission(state, result.State, result.Emission, worstStatus)
: null;
if (transition is not null)
{
var evt = BuildEmission(state, result.State, result.Emission);
if (evt is not null) pendingEmissions.Add(evt);
// A real transition carries the current worst quality so the projected full-snapshot
// write doesn't clobber quality back to Good (e.g. a transition while an input is Uncertain).
pendingEmissions.Add(transition);
}
else if (qualityBucketChanged)
{
// No transition (or a Suppressed one) but the quality bucket moved — annotate out of band.
pendingEmissions.Add(BuildQualityEmission(state, result.State, worstStatus));
}
return result.State;
}
@@ -599,7 +634,8 @@ public sealed class ScriptedAlarmEngine : IDisposable
/// done by <see cref="FireEvent(ScriptedAlarmEvent)"/> AFTER the gate is
/// released.
/// </summary>
private ScriptedAlarmEvent? BuildEmission(AlarmState state, AlarmConditionState condition, EmissionKind kind)
private ScriptedAlarmEvent? BuildEmission(
AlarmState state, AlarmConditionState condition, EmissionKind kind, uint worstInputStatus)
{
// Suppressed kind means shelving ate the emission — we don't fire for subscribers
// but the state record still advanced so startup recovery reflects reality.
@@ -629,9 +665,89 @@ public sealed class ScriptedAlarmEngine : IDisposable
// Carry the per-alarm durable-historization opt-out through to subscribers. The historian
// adapter honors it to suppress ONLY the durable sink write; the live alerts fan-out is
// unaffected (it is not gated on this flag).
HistorizeToAveva: state.Definition.HistorizeToAveva);
HistorizeToAveva: state.Definition.HistorizeToAveva,
// #478 — the worst input quality at evaluation time rides the transition so the projected
// full snapshot keeps quality consistent (no clobber-to-Good).
WorstInputStatusCode: worstInputStatus);
}
/// <summary>
/// #478 — build a standalone <see cref="EmissionKind.QualityChanged"/> event carrying the new
/// worst-of-input quality. Emitted when the quality bucket moved but no Part 9 transition fired
/// (a Bad input freezes the condition; a Suppressed/None transition also leaves state unchanged).
/// The host routes it to the dedicated <c>WriteAlarmQuality</c> node path (annotate quality only,
/// no <c>/alerts</c> row, no historian write); the <see cref="IAlarmSource"/> fan-out skips it.
/// </summary>
private ScriptedAlarmEvent BuildQualityEmission(
AlarmState state, AlarmConditionState condition, uint worstInputStatus)
=> new(
AlarmId: state.Definition.AlarmId,
EquipmentPath: state.Definition.EquipmentPath,
AlarmName: state.Definition.AlarmName,
Kind: state.Definition.Kind,
Severity: state.Definition.Severity,
Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
Condition: condition,
Emission: EmissionKind.QualityChanged,
TimestampUtc: _clock(),
HistorizeToAveva: state.Definition.HistorizeToAveva,
WorstInputStatusCode: worstInputStatus);
/// <summary>Worst OPC UA StatusCode across a refilled read cache — the entry with the highest severity
/// bits (top 2). An input with no value yet (null snapshot/value — the cold-start placeholder, or a
/// not-yet-published upstream) is NOT a quality signal: it means "no data", which the
/// <see cref="AreInputsReady"/> guard already handles by holding the condition. Counting it as Bad here
/// would flash every scripted condition Bad at deploy until the first push and would flood the quality
/// path with load-time annotations, so unread inputs are skipped (contribute Good). Empty / all-unread
/// cache ⇒ Good (0).</summary>
private static uint WorstInputStatus(IReadOnlyDictionary<string, DataValueSnapshot> cache)
{
uint worst = 0u;
var worstSeverity = 0u;
foreach (var kv in cache)
{
if (kv.Value is null || kv.Value.Value is null) continue; // no data yet — not a quality signal
var status = kv.Value.StatusCode;
var severity = status >> 30;
if (severity > worstSeverity)
{
worstSeverity = severity;
worst = status;
}
}
return worst;
}
/// <summary>Update the tracked worst-quality bucket for an alarm; return true iff the 3-state bucket
/// (0 = Good, 1 = Uncertain, 2 = Bad) changed from the last observed value. Only called under
/// <c>_evalGate</c>.</summary>
private bool TrackQualityBucket(string alarmId, uint worstStatus)
{
var bucket = QualityBucket(worstStatus);
var prior = _lastQualityBucketByAlarmId.TryGetValue(alarmId, out var b) ? b : 0; // default Good
_lastQualityBucketByAlarmId[alarmId] = bucket;
return bucket != prior;
}
/// <summary>Collapse an OPC UA StatusCode's 2 severity bits (00/01/10/11) to a 3-state quality bucket
/// (0 = Good, 1 = Uncertain, 2 = Bad).</summary>
private static int QualityBucket(uint statusCode)
{
var severity = statusCode >> 30;
return severity >= 2 ? 2 : (int)severity;
}
/// <summary>#478 — a canonical worst StatusCode for an alarm's last-observed quality bucket, used by
/// the operator-command + shelving-timer emission paths (which don't re-read inputs) so an ack /
/// shelve while an input is Bad still carries Bad rather than resetting the condition to Good.</summary>
private uint LastWorstStatus(string alarmId)
=> (_lastQualityBucketByAlarmId.TryGetValue(alarmId, out var bucket) ? bucket : 0) switch
{
2 => 0x80000000u, // Bad
1 => 0x40000000u, // Uncertain
_ => 0u, // Good
};
/// <summary>
/// Invoke the <see cref="OnEvent"/> handler for a built emission. Must be
/// called OUTSIDE <c>_evalGate</c>: a slow subscriber would otherwise
@@ -708,7 +824,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
_alarms[id] = state with { Condition = result.State };
if (result.Emission != EmissionKind.None)
{
var evt = BuildEmission(state, result.State, result.Emission);
var evt = BuildEmission(state, result.State, result.Emission, LastWorstStatus(id));
if (evt is not null) pending.Add(evt);
}
}
@@ -780,6 +896,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
_alarms.Clear();
_alarmsReferencing.Clear();
_scratchByAlarmId.Clear();
_lastQualityBucketByAlarmId.Clear();
// Dispose every compiled-predicate ALC so the engine's shutdown actually
// releases the emitted assemblies. The drain above ensures no evaluator is
// mid-call; CompiledScriptCache.Dispose internally guards against use-after-
@@ -851,7 +968,11 @@ public sealed record ScriptedAlarmEvent(
EmissionKind Emission,
DateTime TimestampUtc,
string? Comment = null,
bool HistorizeToAveva = true);
bool HistorizeToAveva = true,
// #478 — the worst OPC UA StatusCode across the alarm's input tags at evaluation time. A raw uint
// (Core.ScriptedAlarms does not reference Commons/OpcUaQuality); the host maps it to OpcUaQuality by
// the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
uint WorstInputStatusCode = 0u);
/// <summary>
/// Upstream source abstraction — intentionally identical shape to the virtual-tag
@@ -81,6 +81,11 @@ public sealed class ScriptedAlarmSource : IAlarmSource, IDisposable
{
if (_disposed) return;
// #478 — QualityChanged is a source-quality annotation, not a Part 9 state change. It is delivered
// out of band via the dedicated WriteAlarmQuality node path; surfacing it here would fabricate a
// phantom AlarmEventArgs that materializes / historizes a condition. Swallow it.
if (evt.Emission == EmissionKind.QualityChanged) return;
foreach (var sub in _subscriptions.Values)
{
if (!Matches(sub, evt)) continue;
@@ -278,11 +278,31 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg)
{
// Feed the live value into the upstream the engine subscribes from. StatusCode 0 = Good; the
// mux only forwards values it received from a driver publish, so we treat them as Good-quality.
_upstream.Push(msg.TagId, new DataValueSnapshot(msg.Value, 0u, msg.TimestampUtc, msg.TimestampUtc));
// Feed the live value into the upstream the engine subscribes from. #478 — carry the source
// driver's quality (mapped to an OPC UA StatusCode) so the engine can derive a scripted
// condition's worst-of-input quality; a Bad/Uncertain input is no longer silently treated as Good.
_upstream.Push(msg.TagId,
new DataValueSnapshot(msg.Value, StatusFromQuality(msg.Quality), msg.TimestampUtc, msg.TimestampUtc));
}
/// <summary>#478 — map the 3-state <see cref="OpcUaQuality"/> to an OPC UA StatusCode (severity bits)
/// for the engine's read cache. The inverse of <see cref="QualityFromStatus"/>.</summary>
private static uint StatusFromQuality(OpcUaQuality quality) => quality switch
{
OpcUaQuality.Bad => 0x80000000u,
OpcUaQuality.Uncertain => 0x40000000u,
_ => 0u, // Good
};
/// <summary>#478 — collapse an engine <see cref="ScriptedAlarmEvent.WorstInputStatusCode"/> (top-2
/// severity bits) back to the 3-state <see cref="OpcUaQuality"/> the Commons snapshot / node path use.</summary>
private static OpcUaQuality QualityFromStatus(uint statusCode) => (statusCode >> 30) switch
{
0 => OpcUaQuality.Good,
1 => OpcUaQuality.Uncertain,
_ => OpcUaQuality.Bad,
};
private void OnEngineEmission(EngineEmission msg)
{
var e = msg.Event;
@@ -294,6 +314,21 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
return;
}
// #478 — QualityChanged is a source-quality annotation (worst-of-input bucket moved with no Part 9
// state transition — e.g. an input went Bad, freezing the condition). Route it to the dedicated
// WriteAlarmQuality node path (sets ONLY Quality, one Part 9 event on a bucket change): NO full-state
// projection (would clobber severity/message) and NO /alerts row (it is not a state transition, so it
// must not historize). Same out-of-band quality path native comms-loss uses (#477 Layer 2).
if (e.Emission == EmissionKind.QualityChanged)
{
_publishActor.Tell(new OpcUaPublishActor.AlarmQualityUpdate(
AlarmNodeId: e.AlarmId,
Quality: QualityFromStatus(e.WorstInputStatusCode),
TimestampUtc: e.TimestampUtc,
Realm: AddressSpaceRealm.Uns));
return;
}
// Bridge to OPC UA: project the FULL Part 9 condition state (enabled/active/acked/confirmed/
// shelving/severity/message) onto the materialised condition node via the Commons snapshot.
// e.AlarmId is the materialised condition's NodeId (T14 aligned it to the ScriptedAlarmId).
@@ -540,9 +575,10 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
Shelving: MapShelving(e.Condition.Shelving.Kind),
Severity: (ushort)SeverityToInt(e.Severity),
Message: e.Message,
// #477scripted conditions are script-computed and always live in this scope, so they report Good.
// Deriving quality from the worst of a script's input-tag qualities is a deferred follow-up (Layer 3).
Quality: OpcUaQuality.Good);
// #478the condition's Quality is the worst quality across the script's input tags at evaluation
// time (carried on the event by the engine). A transition fired while an input is Uncertain projects
// Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
Quality: QualityFromStatus(e.WorstInputStatusCode));
/// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
/// mirror (the Commons assembly can't see the Core enum).</summary>
@@ -100,7 +100,7 @@ public sealed class DependencyMuxActor : ReceiveActor
// space carries thousands of tags and only a fraction feed virtual-tag expressions.
return;
}
var dep = new VirtualTagActor.DependencyValueChanged(msg.FullReference, msg.Value, msg.TimestampUtc);
var dep = new VirtualTagActor.DependencyValueChanged(msg.FullReference, msg.Value, msg.TimestampUtc, msg.Quality);
foreach (var sub in subscribers)
{
sub.Tell(dep);
@@ -29,7 +29,11 @@ public sealed class VirtualTagActor : ReceiveActor
{
public const string ScriptLogsTopic = "script-logs";
public sealed record DependencyValueChanged(string TagId, object? Value, DateTime TimestampUtc);
/// <summary>A dependency's value changed. <paramref name="Quality"/> (#478) carries the source
/// driver's OPC UA quality so the scripted-alarm host can derive a condition's worst-of-input quality;
/// it defaults to <see cref="OpcUaQuality.Good"/>, and the virtual-tag engine ignores it.</summary>
public sealed record DependencyValueChanged(
string TagId, object? Value, DateTime TimestampUtc, OpcUaQuality Quality = OpcUaQuality.Good);
/// <summary>Re-assert the last emitted value/quality to the parent bridge, bypassing the value dedup.
/// Sent by <see cref="VirtualTagHostActor"/> on every apply (deploy) to a surviving child: a deploy