fix(alarms): #647 review — Raise on an active condition is a re-statement; FOCAS outage quality + health

MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told
record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a
duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active
condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger
(or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for
an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active
snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives.

MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per
outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm
reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an
alarm-only driver trips the #527 comms-loss watchdog.

MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that
fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never
cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 20:18:03 -04:00
parent 482bafdfac
commit 89961af6e6
17 changed files with 516 additions and 66 deletions
+13 -5
View File
@@ -1120,12 +1120,20 @@ host logger factory (wired in `DriverFactoryBootstrap`; the FOCAS and TwinCAT fa
- **FOCAS alarms (#647):** an `ALARM:` address (`*`, `<Type>`, `<Type>/<n>`, `<Type>/<lo>-<hi>`, `T<code>`, `OTHER`) + - **FOCAS alarms (#647):** an `ALARM:` address (`*`, `<Type>`, `<Type>/<n>`, `<Type>/<lo>-<hi>`, `T<code>`, `OTHER`) +
the `alarm` object = ONE condition at the RawPath (either alone is skipped + reported). Overlapping selectors all the `alarm` object = ONE condition at the RawPath (either alone is skipped + reported). Overlapping selectors all
fire; `OTHER` = what no type selector claims. ⚠️ A failed `cnc_rdalmmsg2` THROWS (it used to return `[]` ⇒ mass fire; `OTHER` = what no type selector claims. ⚠️ A failed `cnc_rdalmmsg2` THROWS (it used to return `[]` ⇒ mass
clear + un-acked re-raise); the told record lives on the driver, so reconnect/re-init re-raise nothing. Type names clear + un-acked re-raise) — and annotates its conditions `BadNoCommunication` once per outage (quality-only) and
are the 16i-era table only (30i codes unconfirmed, not invented — use `T<code>` there). Read-only; ack server-side. degrades driver health with `LastSuccessfulRead` frozen, so an alarm-only driver trips the #527 watchdog. The told
`AlarmProjection.Enabled` is a no-op. E2E: `Runtime.Tests/FocasNativeAlarmEndToEndTests`. `docs/drivers/FOCAS.md`. record lives on the driver INSTANCE (reconnect / in-place re-init re-raise nothing); a deploy RESPAWNS the driver,
and its Raise for a still-active alarm is a host-side re-statement (below). Type names are the 16i-era table only
(30i codes unconfirmed, not invented — use `T<code>` there). Read-only; ack server-side. `AlarmProjection.Enabled`
is a no-op. E2E: `Runtime.Tests/FocasNativeAlarmEndToEndTests` (incl. a respawn). `docs/drivers/FOCAS.md`.
- **A Raise on an already-active condition is a RE-STATEMENT (#647 review):** `NativeAlarmProjector` keeps its ack and
`ForwardNativeAlarm` writes no second "Activated" `/alerts` row — every native driver's told-record dies with its
instance, and a deploy respawns the driver. Only `Retrigger` (or inactive → active) un-acks, so a source whose
UPSTREAM is authoritative for the ack sends Retrigger for an unacked re-statement: OpcUaClient on first sight of a
condition (no prior told), Galaxy for an unacked active snapshot entry.
- **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync` - **Re-Initialize without teardown (#630):** FOCAS, AbLegacy, AbCip, TwinCAT, S7 and OpcUaClient `InitializeAsync`
first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip/FOCAS alarm first release the prior device/session state (cancel, ≤ 5 s join, dispose — incl. the AbCip alarm projection;
projections) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes FOCAS's is PAUSED, its told record kept, #647) — the device-state half of `ShutdownAsync` only; the poll engine is kept and the actor re-subscribes
after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw after every successful init. AbCip publishes fresh tag tables by reference swap (a second init used to throw
"tag name collision" on `AdoptConfigDuringInit`); S7 swaps `Plc` under `_gate`. "tag name collision" on `AdoptConfigDuringInit`); S7 swaps `Plc` under `_gate`.
- **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME - **S7 (#632/#638):** STRING/WSTRING/CHAR/WCHAR and DTL/DATE/TOD/TIME/S5TIME codecs (`S7ValueCodecs`; TOD/TIME/S5TIME
+3 -1
View File
@@ -57,7 +57,9 @@ same condition is wired as an **event notifier of every referencing equipment fo
- **Routing is an exact `ConditionId == RawPath` match** (`DriverHostActor.ForwardNativeAlarm`). A driver that - **Routing is an exact `ConditionId == RawPath` match** (`DriverHostActor.ForwardNativeAlarm`). A driver that
decorates the id has every transition dropped as "no alarm condition" — AbCip did (`"{ref}#active"`, fixed decorates the id has every transition dropped as "no alarm condition" — AbCip did (`"{ref}#active"`, fixed
under #641), and so did FOCAS (`"{host}#{Type}:{Number}"`, fixed under #647 — one condition per `ALARM:` tag, see under #641), and so did FOCAS (`"{host}#{Type}:{Number}"`, fixed under #647 — one condition per `ALARM:` tag, see
`drivers/FOCAS.md`). A native source must also set `Kind`: an `drivers/FOCAS.md`). A `Raise` on a condition already projected Active is a **re-statement** (#647 review): the
ack is kept and no second "Activated" row is published — a respawned driver re-reading a still-active alarm sends
exactly that. Only `Retrigger` (or inactive → active) un-acks. A native source must also set `Kind`: an
`Unspecified` kind projects no state change. `Unspecified` kind projects no state change.
### Native condition state survives a full rebuild (Gitea #641) ### Native condition state survives a full rebuild (Gitea #641)
+18 -8
View File
@@ -398,14 +398,24 @@ it, then author `ALARM:T<code>`. Number ranges are series-independent.
**Delivery and failure.** **Delivery and failure.**
- **A failed read is not "no alarms".** `IFocasClient.ReadAlarmsAsync` throws on a down session, a non-OK - **A failed read is not "no alarms".** `IFocasClient.ReadAlarmsAsync` throws on a down session, a non-OK
return code or a wire fault; that device is skipped for the tick and its conditions keep their state. (It return code, a wire fault, or a non-empty reply that fits neither the 76- nor the 80-byte record layout (an
used to answer `[]`, which would have cleared every alarm on a comms blip and re-raised them un-acknowledged empty reply is "no alarms"). That device is skipped for the tick and its conditions keep their Active/Acked
on recovery.) Driver-wide comms loss is the host's `BadNoCommunication` quality annotation (#477) — Active state. (It used to answer `[]`, which would have cleared every alarm on a comms blip and re-raised them
is kept. un-acknowledged on recovery.)
- **Diffed against what the server was TOLD**, recorded only while a handler is attached, and held for the - **…but an unreachable source says so.** The first failed read of an outage annotates every subscribed condition
driver's lifetime: a reconnect (same-set re-subscribe), a re-initialise and a shutdown/init cycle re-raise on that device `BadNoCommunication` through the #626 quality-only path (no state change, no `/alerts` row) and
nothing, so an acknowledgement survives them. A changed subscribed set (an alarm-bearing deploy) prunes only logs ONE Warning naming `cnc_rdalmmsg2`; the next successful read lifts it to Good (before any transition it
the RawPaths that left (#641). also finds, e.g. Good then Clear for an alarm that cleared during the outage). Failed alarm reads also reach
driver health: `Degraded` with `LastSuccessfulRead` frozen (only a successful read advances it), so a driver
whose only tags are `ALARM:` tags trips the comms-loss watchdog (#527) and the host's driver-wide #477
annotation like any other.
- **Diffed against what the server was TOLD**, recorded only while a handler is attached. The record lives on the
driver instance: a reconnect (same-set re-subscribe) and an in-place re-initialise (Force Reconnect, #630) re-raise
nothing. **A deploy that changes the driver's config respawns it** (the planner stops + recreates the driver, a new
instance with an empty record), and its first poll sends `Raise` for every still-active alarm. The host treats a
`Raise` on a condition it already shows Active as a **re-statement**: the acknowledgement is kept and no second
"Activated" `/alerts` row is written (`NativeAlarmProjector`, #647 review). Only `Retrigger` — or inactive →
active — un-acknowledges, which is why a genuinely new CNC alarm in a matched set is sent as `Retrigger`.
- **Polling:** one `cnc_rdalmmsg2` per `AlarmProjection.PollInterval` (default 2 s) for each device that owns a - **Polling:** one `cnc_rdalmmsg2` per `AlarmProjection.PollInterval` (default 2 s) for each device that owns a
**subscribed** alarm tag; devices without one are never polled for alarms. **subscribed** alarm tag; devices without one are never polled for alarms.
- **Opt-in is the tag.** `AlarmProjection.Enabled` is retained so old configs bind, and has no effect. Before - **Opt-in is the tag.** `AlarmProjection.Enabled` is retained so old configs bind, and has no effect. Before
@@ -57,6 +57,22 @@ internal sealed class FocasAlarmProjection : IAsyncDisposable
/// <summary>Devices whose last read came back at the per-read cap. Guarded by <see cref="_alarmLock"/>.</summary> /// <summary>Devices whose last read came back at the per-read cap. Guarded by <see cref="_alarmLock"/>.</summary>
private readonly HashSet<string> _truncationWarned = new(StringComparer.OrdinalIgnoreCase); private readonly HashSet<string> _truncationWarned = new(StringComparer.OrdinalIgnoreCase);
/// <summary>Devices whose alarm read is currently failing (one Warning per episode). Guarded by <see cref="_alarmLock"/>.</summary>
private readonly HashSet<string> _failingDevices = new(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// ConditionIds this projection has annotated <c>BadNoCommunication</c> and not yet lifted. Guarded by
/// <see cref="_alarmLock"/>. Kept apart from <see cref="_told"/>: a quality annotation never touches a
/// condition's Active/Acked state.
/// </summary>
private readonly HashSet<string> _qualityBadTold = new(StringComparer.Ordinal);
/// <summary>Canonical <c>Opc.Ua.StatusCodes.Good</c>.</summary>
private const uint QualityGood = 0x00000000u;
/// <summary>Canonical <c>Opc.Ua.StatusCodes.BadNoCommunication</c>.</summary>
private const uint QualityBadNoCommunication = 0x80310000u;
/// <summary>Serializes loop start/stop.</summary> /// <summary>Serializes loop start/stop.</summary>
private readonly SemaphoreSlim _loopGate = new(1, 1); private readonly SemaphoreSlim _loopGate = new(1, 1);
@@ -95,7 +111,11 @@ internal sealed class FocasAlarmProjection : IAsyncDisposable
lock (_alarmLock) lock (_alarmLock)
{ {
// Forget only the conditions that LEFT the set (#641) — the SAME set (a reconnect) prunes nothing. // Forget only the conditions that LEFT the set (#641) — the SAME set (a reconnect) prunes nothing.
foreach (var gone in _told.Keys.Where(k => !refs.Contains(k)).ToList()) _told.Remove(gone); Volatile.Write(ref _current, sub); foreach (var gone in _told.Keys.Where(k => !refs.Contains(k)).ToList()) _told.Remove(gone);
// A (re)subscribe follows a reconnect, where the host re-annotates every condition Good (#477); forget
// which conditions this projection marked Bad, so a still-failing device re-asserts its Bad.
_qualityBadTold.Clear();
Volatile.Write(ref _current, sub);
} }
await RestartLoopAsync().ConfigureAwait(false); await RestartLoopAsync().ConfigureAwait(false);
return sub.Handle; return sub.Handle;
@@ -225,10 +245,14 @@ internal sealed class FocasAlarmProjection : IAsyncDisposable
catch (OperationCanceledException) when (ct.IsCancellationRequested) { throw; } catch (OperationCanceledException) when (ct.IsCancellationRequested) { throw; }
catch (Exception ex) catch (Exception ex)
{ {
// Not "no alarms": keep every condition on this device exactly as the server was last told. // Not "no alarms": keep every condition on this device exactly as the server was last told, but say
_logger.LogDebug(ex, "FOCAS alarm read failed for {Host}; its conditions keep their state", host); // its source is unreachable — and let the failure reach driver health (an alarm-only driver has no
// other read to trip the comms-loss watchdog with).
_driver.NoteAlarmReadFailed(host, ex);
TickFailed(sub, host, tags, ex);
continue; continue;
} }
_driver.NoteAlarmReadSucceeded();
Tick(sub, host, alarms, tags); Tick(sub, host, alarms, tags);
} }
} }
@@ -266,24 +290,92 @@ internal sealed class FocasAlarmProjection : IAsyncDisposable
var handlers = _driver.AlarmHandlers; var handlers = _driver.AlarmHandlers;
if (handlers is null) return; if (handlers is null) return;
if (_failingDevices.Remove(deviceHostAddress))
_logger.LogInformation("FOCAS alarm read for {Host} recovered", deviceHostAddress);
foreach (var tag in deviceTags.Where(t => sub.References.Contains(t.RawPath))) foreach (var tag in deviceTags.Where(t => sub.References.Contains(t.RawPath)))
{ {
// Lift a Bad this projection put there BEFORE the normal diff, so a recovery that also changed the
// alarm state (cleared during the outage) lands as Good then Clear.
if (_qualityBadTold.Remove(tag.RawPath))
Deliver(handlers, QualityOnly(tag, sub.Handle, QualityGood, now), tag.RawPath);
var view = FocasAlarmMatcher.Evaluate(tag.Selector, alarms, deviceSelectors, truncated); var view = FocasAlarmMatcher.Evaluate(tag.Selector, alarms, deviceSelectors, truncated);
_told.TryGetValue(tag.RawPath, out var told); _told.TryGetValue(tag.RawPath, out var told);
var (evt, nowTold) = FocasAlarmConditionMapper.Transition(told, view, tag, sub.Handle, now); var (evt, nowTold) = FocasAlarmConditionMapper.Transition(told, view, tag, sub.Handle, now);
_told[tag.RawPath] = nowTold; _told[tag.RawPath] = nowTold;
if (evt is null) continue; if (evt is null) continue;
Deliver(handlers, evt, tag.RawPath);
}
}
}
foreach (var registration in handlers.GetInvocationList()) /// <summary>
{ /// A FAILED read of <paramref name="deviceHostAddress"/> (Gitea #647 review): conditions keep their state, but
try { ((EventHandler<AlarmEventArgs>)registration).Invoke(_driver, evt); } /// each subscribed one is annotated <c>BadNoCommunication</c> once per outage through the #626 quality-only
catch (Exception ex) /// path (no Active/Acked change, no <c>/alerts</c> row), and the first failure of the episode logs one
{ /// Warning. The next successful read lifts the Bad in <see cref="Tick"/>.
_logger.LogWarning(ex, /// </summary>
"FOCAS driver caught a fault from an alarm subscriber for {ConditionId}; delivery continues.", /// <param name="sub">The subscription the read was taken for; a superseded one emits nothing.</param>
tag.RawPath); /// <param name="deviceHostAddress">The device whose read failed.</param>
} /// <param name="tags">The driver's alarm-tag table (RawPath → tag).</param>
} /// <param name="error">Why the read failed.</param>
internal void TickFailed(
Subscription sub, string deviceHostAddress, IReadOnlyDictionary<string, FocasAlarmTag> tags, Exception error)
{
var now = DateTime.UtcNow;
lock (_alarmLock)
{
if (!ReferenceEquals(Volatile.Read(ref _current), sub)) return;
if (_failingDevices.Add(deviceHostAddress))
_logger.LogWarning(error,
"FOCAS alarm read (cnc_rdalmmsg2) failed for {Host}; its alarm conditions keep their state and are " +
"marked BadNoCommunication until a read succeeds", deviceHostAddress);
else
_logger.LogDebug(error, "FOCAS alarm read for {Host} still failing", deviceHostAddress);
// As for transitions: with nobody listening, record nothing, so the next failure tells it.
var handlers = _driver.AlarmHandlers;
if (handlers is null) return;
foreach (var tag in tags.Values.Where(t =>
string.Equals(t.DeviceHostAddress, deviceHostAddress, StringComparison.OrdinalIgnoreCase)
&& sub.References.Contains(t.RawPath)))
{
if (!_qualityBadTold.Add(tag.RawPath)) continue;
Deliver(handlers, QualityOnly(tag, sub.Handle, QualityBadNoCommunication, now), tag.RawPath);
}
}
}
/// <summary>A quality-only annotation (#626): <see cref="AlarmTransitionKind.Unspecified"/> + a status code.</summary>
private AlarmEventArgs QualityOnly(FocasAlarmTag tag, IAlarmSubscriptionHandle handle, uint quality, DateTime now)
{
_told.TryGetValue(tag.RawPath, out var told);
return new AlarmEventArgs(handle,
SourceNodeId: tag.DeviceHostAddress,
ConditionId: tag.RawPath,
AlarmType: told?.Winner is { } w ? FocasAlarmConditionMapper.TypeLabel(w.Type) : "CncAlarm",
Message: told?.Message ?? "",
Severity: told?.Severity ?? AlarmSeverity.Low,
SourceTimestampUtc: now,
Kind: AlarmTransitionKind.Unspecified,
OpcUaSeverity: told?.OpcUaSeverity,
QualityStatusCode: quality);
}
/// <summary>Invokes every handler, isolating a faulting one. Caller holds the lock.</summary>
private void Deliver(EventHandler<AlarmEventArgs> handlers, AlarmEventArgs evt, string conditionId)
{
foreach (var registration in handlers.GetInvocationList())
{
try { ((EventHandler<AlarmEventArgs>)registration).Invoke(_driver, evt); }
catch (Exception ex)
{
_logger.LogWarning(ex,
"FOCAS driver caught a fault from an alarm subscriber for {ConditionId}; delivery continues.",
conditionId);
} }
} }
} }
@@ -47,7 +47,9 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
private readonly Dictionary<string, DeviceState> _devices = new(StringComparer.OrdinalIgnoreCase); private readonly Dictionary<string, DeviceState> _devices = new(StringComparer.OrdinalIgnoreCase);
// v3: RawPath → definition, case-sensitive ordinal. Built in InitializeAsync from _options.RawTags // v3: RawPath → definition, case-sensitive ordinal. Built in InitializeAsync from _options.RawTags
// via the pure FocasTagDefinitionFactory mapper (the reference is always a RawPath). // via the pure FocasTagDefinitionFactory mapper (the reference is always a RawPath).
private readonly Dictionary<string, FocasTagDefinition> _tagsByRawPath = new(StringComparer.Ordinal); // Built whole in InitializeAsync and SWAPPED in (Gitea #647 review): the #630-retained poll engine keeps calling
// ReadAsync during a re-initialise, and a table cleared and refilled in place reads BadNodeIdUnknown in between.
private Dictionary<string, FocasTagDefinition> _tagsByRawPath = new(StringComparer.Ordinal);
// Resolves a read/write fullReference (always a RawPath in v3) to a tag definition by a single // Resolves a read/write fullReference (always a RawPath in v3) to a tag definition by a single
// authored-table hit. A miss is a miss — the driver surfaces BadNodeIdUnknown, never a blob-parse. // authored-table hit. A miss is a miss — the driver surfaces BadNodeIdUnknown, never a blob-parse.
private readonly EquipmentTagRefResolver<FocasTagDefinition> _resolver; private readonly EquipmentTagRefResolver<FocasTagDefinition> _resolver;
@@ -56,7 +58,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
// call. ConcurrentDictionary is required because the poll loop // call. ConcurrentDictionary is required because the poll loop
// (ReadAsync) and the host write thread (WriteAsync) both mutate this cache when resolver- // (ReadAsync) and the host write thread (WriteAsync) both mutate this cache when resolver-
// produced equipment tags are encountered for the first time. // produced equipment tags are encountered for the first time.
private readonly ConcurrentDictionary<string, FocasAddress> _parsedAddressesByTagName = private ConcurrentDictionary<string, FocasAddress> _parsedAddressesByTagName =
new(StringComparer.OrdinalIgnoreCase); new(StringComparer.OrdinalIgnoreCase);
/// <summary> /// <summary>
/// Whether this driver instance runs its per-device fixed-tree poll loop, and therefore /// Whether this driver instance runs its per-device fixed-tree poll loop, and therefore
@@ -128,7 +130,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
_logger = logger ?? NullLogger<FocasDriver>.Instance; _logger = logger ?? NullLogger<FocasDriver>.Instance;
_alarmProjection = new FocasAlarmProjection(this, _logger); _alarmProjection = new FocasAlarmProjection(this, _logger);
_resolver = new EquipmentTagRefResolver<FocasTagDefinition>( _resolver = new EquipmentTagRefResolver<FocasTagDefinition>(
r => _tagsByRawPath.TryGetValue(r, out var t) ? t : null); r => Volatile.Read(ref _tagsByRawPath).TryGetValue(r, out var t) ? t : null);
_poll = new PollGroupEngine( _poll = new PollGroupEngine(
reader: ReadAsync, reader: ReadAsync,
onChange: (handle, tagRef, snapshot) => onChange: (handle, tagRef, snapshot) =>
@@ -234,8 +236,9 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
_skippedTags.BeginRebuild(_options.RawTags.Count); _skippedTags.BeginRebuild(_options.RawTags.Count);
// A re-initialise without teardown (#630) rebuilds the tables from scratch: a tag the new config // A re-initialise without teardown (#630) rebuilds the tables from scratch: a tag the new config
// removed — or moved between the value and alarm tables (#647) — must not survive in the old one. // removed — or moved between the value and alarm tables (#647) — must not survive in the old one.
_tagsByRawPath.Clear(); // Built into fresh tables and swapped in below, never cleared in place under a live reader.
_parsedAddressesByTagName.Clear(); var tagsByRawPath = new Dictionary<string, FocasTagDefinition>(StringComparer.Ordinal);
var parsedAddresses = new ConcurrentDictionary<string, FocasAddress>(StringComparer.OrdinalIgnoreCase);
var alarmTags = new Dictionary<string, FocasAlarmTag>(StringComparer.Ordinal); var alarmTags = new Dictionary<string, FocasAlarmTag>(StringComparer.Ordinal);
foreach (var entry in _options.RawTags) foreach (var entry in _options.RawTags)
{ {
@@ -298,7 +301,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
entry.RawPath, why, _driverInstanceId); entry.RawPath, why, _driverInstanceId);
continue; continue;
} }
_tagsByRawPath[entry.RawPath] = def with tagsByRawPath[entry.RawPath] = def with
{ {
DeviceHostAddress = deviceHost, DeviceHostAddress = deviceHost,
WriteIdempotent = entry.WriteIdempotent, WriteIdempotent = entry.WriteIdempotent,
@@ -306,12 +309,14 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
// Cache the parsed FocasAddress so ReadAsync / WriteAsync don't re-parse on every // Cache the parsed FocasAddress so ReadAsync / WriteAsync don't re-parse on every
// hot-path call. The address string has already been validated by FocasAddress.TryParse // hot-path call. The address string has already been validated by FocasAddress.TryParse
// above; reusing the parsed record avoids per-tick allocs on subscription pollers. // above; reusing the parsed record avoids per-tick allocs on subscription pollers.
_parsedAddressesByTagName[entry.RawPath] = parsed; parsedAddresses[entry.RawPath] = parsed;
} }
// Publish the skip tally only once the whole table is built. An unknown-device throw above // Publish the skip tally only once the whole table is built. An unknown-device throw above
// leaves the previous snapshot standing rather than a truncated one that would read as // leaves the previous snapshot standing rather than a truncated one that would read as
// "we now skip fewer tags". // "we now skip fewer tags".
_skippedTags.Commit(); _skippedTags.Commit();
Volatile.Write(ref _parsedAddressesByTagName, parsedAddresses);
Volatile.Write(ref _tagsByRawPath, tagsByRawPath);
Volatile.Write(ref _alarmTags, alarmTags); Volatile.Write(ref _alarmTags, alarmTags);
if (_options.Probe.Enabled) if (_options.Probe.Enabled)
@@ -465,7 +470,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
state.DisposeClient(); state.DisposeClient();
} }
_devices.Clear(); _devices.Clear();
_tagsByRawPath.Clear(); Volatile.Write(ref _tagsByRawPath, new Dictionary<string, FocasTagDefinition>(StringComparer.Ordinal));
Volatile.Write(ref _alarmTags, new Dictionary<string, FocasAlarmTag>(StringComparer.Ordinal)); Volatile.Write(ref _alarmTags, new Dictionary<string, FocasAlarmTag>(StringComparer.Ordinal));
_resolver.Clear(); // no-op in v3 (the resolver reads the live table by closure); kept for symmetry _resolver.Clear(); // no-op in v3 (the resolver reads the live table by closure); kept for symmetry
_parsedAddressesByTagName.Clear(); _parsedAddressesByTagName.Clear();
@@ -1693,6 +1698,35 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
/// <summary>Gets the alarm poll cadence — one <c>cnc_rdalmmsg2</c> per alarm-owning device per tick.</summary> /// <summary>Gets the alarm poll cadence — one <c>cnc_rdalmmsg2</c> per alarm-owning device per tick.</summary>
internal TimeSpan AlarmPollInterval => _options.AlarmProjection.PollInterval; internal TimeSpan AlarmPollInterval => _options.AlarmProjection.PollInterval;
/// <summary>
/// A successful alarm read is a successful read (Gitea #647 review): it advances
/// <see cref="DriverHealth.LastSuccessfulRead"/> exactly as a Good value read does, so an alarm-only driver's
/// health tracks its CNC. Never lifts a <see cref="DriverState.Faulted"/> (a config-level signal).
/// </summary>
internal void NoteAlarmReadSucceeded()
{
var current = Volatile.Read(ref _health);
if (current.State != DriverState.Faulted)
Volatile.Write(ref _health, new DriverHealth(DriverState.Healthy, DateTime.UtcNow, null));
}
/// <summary>
/// A failed alarm read degrades driver health, PRESERVING <see cref="DriverHealth.LastSuccessfulRead"/> — the
/// conjunction the comms-loss watchdog (#527) trips on. Without it a driver whose only tags are <c>ALARM:</c>
/// tags stayed Healthy through a total outage, the watchdog never fired, and the host's #477
/// <c>BadNoCommunication</c> annotation never ran (Gitea #647 review).
/// </summary>
/// <param name="hostAddress">The device whose alarm read failed.</param>
/// <param name="error">Why.</param>
internal void NoteAlarmReadFailed(string hostAddress, Exception error)
{
var current = Volatile.Read(ref _health);
if (current.State != DriverState.Faulted)
Volatile.Write(ref _health, new DriverHealth(DriverState.Degraded, current.LastSuccessfulRead,
$"FOCAS alarm read failed for {hostAddress}: {error.Message}"));
}
/// <summary> /// <summary>
/// Reads one device's active-alarm list. <b>Throws on any failure</b> — an unknown device, a connect that /// Reads one device's active-alarm list. <b>Throws on any failure</b> — an unknown device, a connect that
/// fails, a read the CNC refuses — because the projection must never mistake a failed read for an empty /// fails, a read the CNC refuses — because the projection must never mistake a failed read for an empty
@@ -1101,10 +1101,24 @@ public sealed class FocasWireClient : IAsyncDisposable, IDisposable
return new WireProgramName(name, number); return new WireProgramName(name, number);
} }
private static IReadOnlyList<WireAlarm> ParseAlarms(byte[] payload, short count) /// <summary>
=> payload.Length % 76 == 0 /// Decode a <c>cnc_rdalmmsg2</c> payload: the 76-byte vendor stride, else the 80-byte legacy one. An empty
? ParseVendorAlarms(payload, count) /// payload is "no alarms active". A non-empty payload that fits NEITHER stride is malformed and throws
: ParseLegacyWireAlarms(payload, count); /// (Gitea #647 review): it used to fall into the legacy parser, decode zero records, and so read as "no
/// alarms" — a mass Clear of every condition on the device, then an un-acknowledged re-raise next poll.
/// </summary>
/// <param name="payload">The reply payload.</param>
/// <param name="count">The requested record count.</param>
/// <returns>The decoded alarms.</returns>
/// <exception cref="FocasWireException">The payload length matches neither record layout.</exception>
internal static IReadOnlyList<WireAlarm> ParseAlarms(byte[] payload, short count)
{
if (payload.Length % 76 == 0) return ParseVendorAlarms(payload, count);
if (payload.Length % 80 == 0) return ParseLegacyWireAlarms(payload, count);
throw new FocasWireException(
$"cnc_rdalmmsg2 returned a {payload.Length}-byte payload, which fits neither the 76- nor the 80-byte " +
"record layout; refusing to read it as an empty alarm list.", rc: null);
}
private static IReadOnlyList<WireAlarm> ParseVendorAlarms(byte[] payload, short count) private static IReadOnlyList<WireAlarm> ParseVendorAlarms(byte[] payload, short count)
{ {
@@ -171,14 +171,18 @@ internal sealed class GatewayGalaxyAlarmFeed : IGalaxyAlarmFeed
/// Decode one entry of the initial active-alarm snapshot. Each currently-active /// Decode one entry of the initial active-alarm snapshot. Each currently-active
/// alarm is surfaced as a transition so the OPC UA Part 9 condition layer sees /// alarm is surfaced as a transition so the OPC UA Part 9 condition layer sees
/// the alarm's present state on (re)connect: an unacknowledged active alarm as /// the alarm's present state on (re)connect: an unacknowledged active alarm as
/// a <see cref="GalaxyAlarmTransitionKind.Raise"/>, an acknowledged one as a /// a <see cref="GalaxyAlarmTransitionKind.Retrigger"/>, an acknowledged one as a
/// <see cref="GalaxyAlarmTransitionKind.Acknowledge"/>. /// <see cref="GalaxyAlarmTransitionKind.Acknowledge"/>.
/// <para>Retrigger, not Raise (Gitea #647 review): the host treats a Raise on a condition it already
/// shows Active as a re-statement that KEEPS the acknowledgement. A snapshot re-states upstream state and
/// the upstream's ack state is authoritative, so an unacked entry must un-ack a host condition still
/// holding an ack from before the reconnect.</para>
/// </summary> /// </summary>
private void DispatchSnapshotEntry(ActiveAlarmSnapshot snapshot) private void DispatchSnapshotEntry(ActiveAlarmSnapshot snapshot)
{ {
var kind = snapshot.CurrentState switch var kind = snapshot.CurrentState switch
{ {
AlarmConditionState.Active => GalaxyAlarmTransitionKind.Raise, AlarmConditionState.Active => GalaxyAlarmTransitionKind.Retrigger,
AlarmConditionState.ActiveAcked => GalaxyAlarmTransitionKind.Acknowledge, AlarmConditionState.ActiveAcked => GalaxyAlarmTransitionKind.Acknowledge,
AlarmConditionState.Inactive => GalaxyAlarmTransitionKind.Clear, AlarmConditionState.Inactive => GalaxyAlarmTransitionKind.Clear,
_ => GalaxyAlarmTransitionKind.Unspecified, _ => GalaxyAlarmTransitionKind.Unspecified,
@@ -1982,12 +1982,11 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
// Fresh prior = the host projector's own default (inactive, acked). // Fresh prior = the host projector's own default (inactive, acked).
var told = new AlarmToldState(target.Active, target.Acked, e.Message, e.OpcUaSeverity); var told = new AlarmToldState(target.Active, target.Acked, e.Message, e.OpcUaSeverity);
var prior = _alarmStateByRawPath.TryGetValue(e.ConditionId, out var p) var priorKnown = _alarmStateByRawPath.TryGetValue(e.ConditionId, out var p);
? p var prior = priorKnown ? p! : new AlarmToldState(false, true, null, null);
: new AlarmToldState(false, true, null, null);
_alarmStateByRawPath[e.ConditionId] = told; _alarmStateByRawPath[e.ConditionId] = told;
var kinds = AlarmTransitions((prior.Active, prior.Acked), target); var kinds = AlarmTransitions((prior.Active, prior.Acked), target, priorKnown);
if (kinds.Count == 0) if (kinds.Count == 0)
{ {
// Same state: a message/severity change still reaches the host; an identical copy does not. // Same state: a message/severity change still reaches the host; an identical copy does not.
@@ -2106,18 +2105,34 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
/// <summary> /// <summary>
/// #619 review — the <see cref="AlarmTransitionKind"/> sequence that moves the host's /// #619 review — the <see cref="AlarmTransitionKind"/> sequence that moves the host's
/// <c>NativeAlarmProjector</c> (which knows only kinds: Raise ⇒ active+unacked, Acknowledge ⇒ acked, /// <c>NativeAlarmProjector</c> (which knows only kinds: Raise ⇒ active+unacked — or, on an already-active condition, a re-statement keeping the ack (#647) — Acknowledge ⇒ acked,
/// Clear ⇒ inactive keeping acked) from <paramref name="prior"/> to <paramref name="target"/>. A single /// Clear ⇒ inactive keeping acked) from <paramref name="prior"/> to <paramref name="target"/>. A single
/// upstream event can need two (an alarm that went active and was acknowledged between two notifications). /// upstream event can need two (an alarm that went active and was acknowledged between two notifications).
/// No state change ⇒ empty (the caller decides whether a message/severity change warrants an Unspecified). /// No state change ⇒ empty (the caller decides whether a message/severity change warrants an Unspecified).
/// </summary> /// </summary>
/// <param name="prior">The state last told to the host.</param> /// <param name="prior">The state last told to the host.</param>
/// <param name="target">The upstream event's ActiveState/Id + AckedState/Id.</param> /// <param name="target">The upstream event's ActiveState/Id + AckedState/Id.</param>
/// <param name="priorKnown">
/// False when this driver instance has told the host nothing about the condition yet (first sight after a
/// start or a deploy respawn), so <paramref name="prior"/> is only the projector's default. Gitea #647 review:
/// the host now treats a Raise on an already-active condition as a re-statement that KEEPS its ack, so an
/// upstream that says <i>unacknowledged</i> must then be told with <see cref="AlarmTransitionKind.Retrigger"/>
/// — the upstream's ack state is authoritative, and the host may be holding an ack from before the respawn.
/// </param>
/// <returns>The kinds to emit, in order.</returns> /// <returns>The kinds to emit, in order.</returns>
internal static IReadOnlyList<AlarmTransitionKind> AlarmTransitions( internal static IReadOnlyList<AlarmTransitionKind> AlarmTransitions(
(bool Active, bool Acked) prior, (bool Active, bool Acked) target) (bool Active, bool Acked) prior, (bool Active, bool Acked) target, bool priorKnown = true)
{ {
if (prior == target) return []; if (prior == target) return [];
var kinds = Transitions(prior, target);
return priorKnown || target.Acked
? kinds
: [.. kinds.Select(k => k == AlarmTransitionKind.Raise ? AlarmTransitionKind.Retrigger : k)];
}
private static IReadOnlyList<AlarmTransitionKind> Transitions(
(bool Active, bool Acked) prior, (bool Active, bool Acked) target)
{
return target switch return target switch
{ {
// Active+unacked: a fresh raise, or a re-arm of an active-acked condition. // Active+unacked: a fresh raise, or a re-arm of an active-acked condition.
@@ -1877,12 +1877,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
foreach (var n in nodeIds) foreach (var n in nodeIds)
{ {
var nodeId = n.NodeId; var nodeId = n.NodeId;
var snapshot = _nativeAlarmProjector.Project(nodeId, msg.Args); var snapshot = _nativeAlarmProjector.Project(nodeId, msg.Args, out var restatement);
_opcUaPublishActor.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AlarmStateUpdate( _opcUaPublishActor.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AlarmStateUpdate(
nodeId, snapshot, msg.Args.SourceTimestampUtc, n.Realm)); nodeId, snapshot, msg.Args.SourceTimestampUtc, n.Realm));
// Only the Primary publishes the cluster-wide alerts transition (see the gate above). // Only the Primary publishes the cluster-wide alerts transition (see the gate above). A re-statement
if (!serviceAlertsAsPrimary) continue; // (Gitea #647 review: a Raise on an already-active condition, sent by a driver a deploy respawned) is
// not a transition — the condition was already Active, so a second "Activated" row would be a duplicate.
if (!serviceAlertsAsPrimary || restatement) continue;
var meta = _alarmMetaByNodeId.TryGetValue(nodeId, out var m) var meta = _alarmMetaByNodeId.TryGetValue(nodeId, out var m)
? m : (EquipmentId: nodeId, Name: nodeId, AlarmType: "AlarmCondition", HistorizeToAveva: (bool?)null, ? m : (EquipmentId: nodeId, Name: nodeId, AlarmType: "AlarmCondition", HistorizeToAveva: (bool?)null,
@@ -18,13 +18,29 @@ public sealed class NativeAlarmProjector
/// <param name="nodeId">The materialised condition node's id (the projection's state key).</param> /// <param name="nodeId">The materialised condition node's id (the projection's state key).</param>
/// <param name="e">The native alarm transition.</param> /// <param name="e">The native alarm transition.</param>
/// <returns>The full Part 9 condition snapshot to write to the node.</returns> /// <returns>The full Part 9 condition snapshot to write to the node.</returns>
public AlarmConditionSnapshot Project(string nodeId, AlarmEventArgs e) public AlarmConditionSnapshot Project(string nodeId, AlarmEventArgs e) => Project(nodeId, e, out _);
/// <summary>
/// Project an alarm transition, reporting whether it was a <b>re-statement</b> (Gitea #647 review): a
/// <see cref="AlarmTransitionKind.Raise"/> on a condition already projected Active. That is what a driver
/// whose told-record died with its instance sends after a deploy respawns it — the alarm it re-reads is the
/// one the server already shows, so the acknowledgement is KEPT and the host writes no second "Activated"
/// row. Only <see cref="AlarmTransitionKind.Retrigger"/> (or inactive → active) un-acknowledges; a source
/// that knows the alarm is a genuinely new statement must send Retrigger.
/// </summary>
/// <param name="nodeId">The materialised condition node's id (the projection's state key).</param>
/// <param name="e">The native alarm transition.</param>
/// <param name="restatement">True when <paramref name="e"/> re-stated an already-active condition.</param>
/// <returns>The full Part 9 condition snapshot to write to the node.</returns>
public AlarmConditionSnapshot Project(string nodeId, AlarmEventArgs e, out bool restatement)
{ {
var prev = _prior.TryGetValue(nodeId, out var p) ? p : (Active: false, Acked: true); var prev = _prior.TryGetValue(nodeId, out var p) ? p : (Active: false, Acked: true);
// #626: a source with a finer scale than the four buckets supplies the exact Part 9 severity. // #626: a source with a finer scale than the four buckets supplies the exact Part 9 severity.
var sev = e.OpcUaSeverity is { } exact ? (ushort)Math.Clamp(exact, 1, 1000) : MapSeverity(e.Severity); var sev = e.OpcUaSeverity is { } exact ? (ushort)Math.Clamp(exact, 1, 1000) : MapSeverity(e.Severity);
restatement = e.Kind == AlarmTransitionKind.Raise && prev.Active;
var (active, acked) = e.Kind switch var (active, acked) = e.Kind switch
{ {
AlarmTransitionKind.Raise when prev.Active => (true, prev.Acked),
AlarmTransitionKind.Raise or AlarmTransitionKind.Retrigger => (true, false), AlarmTransitionKind.Raise or AlarmTransitionKind.Retrigger => (true, false),
AlarmTransitionKind.Acknowledge => (prev.Active, true), AlarmTransitionKind.Acknowledge => (prev.Active, true),
AlarmTransitionKind.Clear => (false, prev.Acked), AlarmTransitionKind.Clear => (false, prev.Acked),
@@ -62,10 +62,14 @@ public sealed class FocasAlarmProjectionTests
private static void Tick(FocasDriver drv, params FocasActiveAlarm[] alarms) => private static void Tick(FocasDriver drv, params FocasActiveAlarm[] alarms) =>
drv.AlarmProjection.Tick(drv.AlarmProjection.Current!, Host, alarms, drv.AlarmTags); drv.AlarmProjection.Tick(drv.AlarmProjection.Current!, Host, alarms, drv.AlarmTags);
private static List<AlarmEventArgs> Capture(FocasDriver drv) private static List<AlarmEventArgs> Capture(FocasDriver drv, bool transitionsOnly = false)
{ {
var events = new List<AlarmEventArgs>(); var events = new List<AlarmEventArgs>();
drv.OnAlarmEvent += (_, e) => { lock (events) events.Add(e); }; drv.OnAlarmEvent += (_, e) =>
{
if (transitionsOnly && e.IsQualityOnly) return;
lock (events) events.Add(e);
};
return events; return events;
} }
@@ -277,7 +281,8 @@ public sealed class FocasAlarmProjectionTests
{ {
var (drv, factory) = NewDriver(Tag(ServoPath, "ALARM:Servo")); var (drv, factory) = NewDriver(Tag(ServoPath, "ALARM:Servo"));
await drv.InitializeAsync("{}", CancellationToken.None); await drv.InitializeAsync("{}", CancellationToken.None);
var events = Capture(drv); // Transitions only: the outage's quality-only BadNoCommunication annotation is pinned separately.
var events = Capture(drv, transitionsOnly: true);
await drv.SubscribeAlarmsAsync([ServoPath], CancellationToken.None); await drv.SubscribeAlarmsAsync([ServoPath], CancellationToken.None);
await WaitFor(() => factory.Clients.Count > 0); await WaitFor(() => factory.Clients.Count > 0);
var client = ClientOf(factory); var client = ClientOf(factory);
@@ -421,6 +426,125 @@ public sealed class FocasAlarmProjectionTests
await drv.ShutdownAsync(CancellationToken.None); await drv.ShutdownAsync(CancellationToken.None);
} }
// ---- Gitea #647 review MAJOR-2: an unreachable alarm source must say so ----------------------------------
private const uint BadNoCommunication = 0x80310000u;
private static void TickFailed(FocasDriver drv) =>
drv.AlarmProjection.TickFailed(drv.AlarmProjection.Current!, Host, drv.AlarmTags, new InvalidOperationException("down"));
/// <summary>
/// A failed read marks each subscribed condition BadNoCommunication ONCE per outage, quality-only (no Clear,
/// no state change); recovery lifts it to Good and, with the alarm unchanged, re-raises nothing.
/// </summary>
[Fact]
public async Task A_failed_read_annotates_quality_once_and_recovery_lifts_it_without_a_re_raise()
{
var (drv, factory) = NewDriver(Tag(ServoPath, "ALARM:Servo"), Tag(AnyPath, "ALARM:*"));
await InitAndSubscribeForTicksAsync(drv, factory, ServoPath, AnyPath);
var events = Capture(drv);
Tick(drv, Servo401());
events.Clear();
TickFailed(drv);
TickFailed(drv);
events.Count.ShouldBe(2, "one Bad per condition for the whole outage");
events.ShouldAllBe(e => e.IsQualityOnly && e.QualityStatusCode == BadNoCommunication);
events.Select(e => e.ConditionId).Order().ToArray().ShouldBe(new[] { AnyPath, ServoPath });
events.Clear();
Tick(drv, Servo401());
events.Count.ShouldBe(2);
events.ShouldAllBe(e => e.IsQualityOnly && e.QualityStatusCode == 0u, "recovery lifts the Bad and says nothing else");
}
[Fact]
public async Task A_recovery_that_finds_the_alarm_cleared_lifts_quality_then_clears()
{
var (drv, factory) = NewDriver(Tag(ServoPath, "ALARM:Servo"));
await InitAndSubscribeForTicksAsync(drv, factory, ServoPath);
var events = Capture(drv);
Tick(drv, Servo401());
TickFailed(drv);
events.Clear();
Tick(drv);
events.Select(e => (e.Kind, e.QualityStatusCode)).ToArray().ShouldBe(new (AlarmTransitionKind, uint?)[]
{
(AlarmTransitionKind.Unspecified, 0u),
(AlarmTransitionKind.Clear, null),
});
}
[Fact]
public async Task A_persistently_failing_alarm_read_warns_once_per_outage()
{
var logger = new ListLogger();
var factory = new FakeFocasClientFactory();
var drv = new FocasDriver(new FocasDriverOptions
{
Devices = [new FocasDeviceOptions(Host, DeviceName: "m1")],
RawTags = [Tag(ServoPath, "ALARM:Servo")],
Probe = new FocasProbeOptions { Enabled = false },
}, "drv-1", factory, logger);
await InitAndSubscribeForTicksAsync(drv, factory, ServoPath);
Capture(drv);
var before = logger.Warnings();
TickFailed(drv);
TickFailed(drv);
TickFailed(drv);
logger.Warnings().ShouldBe(before + 1);
Tick(drv); // recovery ends the episode…
TickFailed(drv); // …so the next outage warns again
logger.Warnings().ShouldBe(before + 2);
}
/// <summary>
/// An alarm-only driver must be able to trip the comms-loss watchdog (#527): sustained alarm-read failure
/// goes non-Healthy with LastSuccessfulRead FROZEN; a successful alarm read advances it.
/// </summary>
[Fact]
public async Task Sustained_alarm_read_failure_degrades_health_with_LastSuccessfulRead_frozen()
{
var (drv, factory) = NewDriver(Tag(ServoPath, "ALARM:Servo"));
factory.Customise = () => new FakeFocasClient { ThrowOnReadAlarms = true };
await drv.InitializeAsync("{}", CancellationToken.None);
var atInit = drv.GetHealth().LastSuccessfulRead;
await drv.SubscribeAlarmsAsync([ServoPath], CancellationToken.None);
await WaitFor(() => factory.Clients.Count > 0 && factory.Clients[0].AlarmReadCount >= 3);
var failing = drv.GetHealth();
failing.State.ShouldBe(DriverState.Degraded);
failing.LastSuccessfulRead.ShouldBe(atInit, "a failed read never advances LastSuccessfulRead");
factory.Clients[0].ThrowOnReadAlarms = false;
await WaitFor(() => drv.GetHealth().State == DriverState.Healthy);
drv.GetHealth().LastSuccessfulRead!.Value.ShouldBeGreaterThan(atInit!.Value);
await drv.ShutdownAsync(CancellationToken.None);
}
private sealed class ListLogger : Microsoft.Extensions.Logging.ILogger<FocasDriver>
{
private int _warnings;
public int Warnings() => Volatile.Read(ref _warnings);
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => true;
public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId,
TState state, Exception? exception, Func<TState, Exception?, string> formatter)
{
if (logLevel == Microsoft.Extensions.Logging.LogLevel.Warning
&& formatter(state, exception).Contains("cnc_rdalmmsg2", StringComparison.Ordinal))
Interlocked.Increment(ref _warnings);
}
}
private static int Count(List<AlarmEventArgs> events) private static int Count(List<AlarmEventArgs> events)
{ {
lock (events) return events.Count; lock (events) return events.Count;
@@ -133,6 +133,20 @@ public sealed class FocasAlarmSelectorTests
c.Alarm.ShouldBeNull(); c.Alarm.ShouldBeNull();
} }
/// <summary>
/// Gitea #647 review MINOR-5 — a non-empty alarm payload that fits neither record stride is malformed, not
/// "no alarms" (which would mass-clear every condition on the device). Empty stays empty.
/// </summary>
[Fact]
public void A_malformed_alarm_payload_throws_while_an_empty_one_is_no_alarms()
{
FocasWireClient.ParseAlarms([], 32).ShouldBeEmpty();
FocasWireClient.ParseAlarms(new byte[76], 32).Count.ShouldBe(1);
FocasWireClient.ParseAlarms(new byte[160], 32).Count.ShouldBe(2);
Should.Throw<FocasWireException>(() => FocasWireClient.ParseAlarms(new byte[77], 32));
Should.Throw<FocasWireException>(() => FocasWireClient.ParseAlarms(new byte[12], 32));
}
/// <summary> /// <summary>
/// Gitea #647 defect 4 — a read that FAILED must not look like "no alarms": the projection would diff it /// Gitea #647 defect 4 — a read that FAILED must not look like "no alarms": the projection would diff it
/// to a mass clear and re-raise everything un-acknowledged on recovery. A disconnected wire client throws. /// to a mass clear and re-raise everything un-acknowledged on recovery. A disconnected wire client throws.
@@ -58,9 +58,9 @@ public sealed class GatewayGalaxyAlarmFeedTests
observed.Count.ShouldBe(3); observed.Count.ShouldBe(3);
// Active snapshot entry → Raise. // Active (unacked) snapshot entry → Retrigger (#647 review: forces the upstream's unack on the host).
observed[0].AlarmFullReference.ShouldBe("Tank01.Level.HiHi"); observed[0].AlarmFullReference.ShouldBe("Tank01.Level.HiHi");
observed[0].TransitionKind.ShouldBe(GalaxyAlarmTransitionKind.Raise); observed[0].TransitionKind.ShouldBe(GalaxyAlarmTransitionKind.Retrigger);
observed[0].SeverityBucket.ShouldBe(AlarmSeverity.Critical); observed[0].SeverityBucket.ShouldBe(AlarmSeverity.Critical);
observed[0].RawMxAccessSeverity.ShouldBe(750); observed[0].RawMxAccessSeverity.ShouldBe(750);
@@ -232,6 +232,21 @@ public sealed class OpcUaClientAlarmTests
OpcUaClientDriver.AlarmTransitions((priorActive, priorAcked), (active, acked)).ShouldBe(expected); OpcUaClientDriver.AlarmTransitions((priorActive, priorAcked), (active, acked)).ShouldBe(expected);
} }
/// <summary>
/// Gitea #647 review — with no prior this driver told the host (first sight after start or a deploy respawn),
/// an unacknowledged upstream target goes out as Retrigger, never Raise: the host keeps an existing ack across
/// a Raise on an already-active condition, so only a Retrigger makes the host match the upstream's unack.
/// An acknowledged target keeps its Raise (a re-statement there is exactly right).
/// </summary>
[Theory]
[InlineData(true, false, new[] { AlarmTransitionKind.Retrigger })]
[InlineData(false, false, new[] { AlarmTransitionKind.Retrigger, AlarmTransitionKind.Clear })]
[InlineData(true, true, new[] { AlarmTransitionKind.Raise, AlarmTransitionKind.Acknowledge })]
public void An_unknown_prior_tells_an_unacked_upstream_with_Retrigger(bool active, bool acked, AlarmTransitionKind[] expected)
{
OpcUaClientDriver.AlarmTransitions((false, true), (active, acked), priorKnown: false).ShouldBe(expected);
}
// ---- #619 re-review: the transition table must reach EVERY target from EVERY prior. ---- // ---- #619 re-review: the transition table must reach EVERY target from EVERY prior. ----
/// <summary>The host projector's semantics, restated: the only thing the driver's kinds act upon.</summary> /// <summary>The host projector's semantics, restated: the only thing the driver's kinds act upon.</summary>
@@ -357,7 +372,9 @@ public sealed class OpcUaClientAlarmTests
drv.RouteAlarmEvent(ConditionFields(true, false), TripIndex, handle); drv.RouteAlarmEvent(ConditionFields(true, false), TripIndex, handle);
drv.RouteAlarmEvent(ConditionFields(true, true), TripIndex, handle); drv.RouteAlarmEvent(ConditionFields(true, true), TripIndex, handle);
seen.Select(e => e.Kind).ShouldBe([AlarmTransitionKind.Raise, AlarmTransitionKind.Acknowledge]); // First sight of an UNACKED upstream condition is a Retrigger (#647 review: the host keeps an ack across a
// Raise on an already-active condition, and only the upstream knows it is unacknowledged).
seen.Select(e => e.Kind).ShouldBe([AlarmTransitionKind.Retrigger, AlarmTransitionKind.Acknowledge]);
drv.RouteAlarmEvent(ConditionFields(true, true), TripIndex, handle); // the refresh copy drv.RouteAlarmEvent(ConditionFields(true, true), TripIndex, handle); // the refresh copy
seen.Count.ShouldBe(2); seen.Count.ShouldBe(2);
@@ -257,7 +257,7 @@ public sealed class OpcUaClientDriverWireTests
/// <summary> /// <summary>
/// #619 — ConditionRefresh over the wire: a condition already active+unacked BEFORE the driver subscribes is /// #619 — ConditionRefresh over the wire: a condition already active+unacked BEFORE the driver subscribes is
/// reported (Raise, keyed by the RawPath) right after subscribe, not only on its next transition. A second /// reported (Retrigger since #647, keyed by the RawPath) right after subscribe, not only on its next transition. A second
/// subscribe of the same set refreshes again and must emit nothing (no re-raise); a positive control then /// subscribe of the same set refreshes again and must emit nothing (no re-raise); a positive control then
/// proves the feed is still live by acknowledging upstream and seeing exactly the Acknowledge arrive. /// proves the feed is still live by acknowledging upstream and seeing exactly the Acknowledge arrive.
/// </summary> /// </summary>
@@ -293,9 +293,11 @@ public sealed class OpcUaClientDriverWireTests
AlarmEventArgs[] Seen() { lock (seen) return [.. seen]; } AlarmEventArgs[] Seen() { lock (seen) return [.. seen]; }
var first = await drv.SubscribeAlarmsAsync([AlarmRef], ct); var first = await drv.SubscribeAlarmsAsync([AlarmRef], ct);
await RecordedEvents.WaitUntilAsync(() => Seen().Any(e => e.Kind == AlarmTransitionKind.Raise)); // First sight of an active+UNACKED upstream condition is a Retrigger (#647 review): the host keeps an ack
// across a Raise on an already-active condition, so only a Retrigger forces the upstream's unack.
await RecordedEvents.WaitUntilAsync(() => Seen().Any(e => e.Kind == AlarmTransitionKind.Retrigger));
var raise = Seen().Single(e => e.Kind == AlarmTransitionKind.Raise); var raise = Seen().Single(e => e.Kind == AlarmTransitionKind.Retrigger);
raise.ConditionId.ShouldBe(AlarmRef); raise.ConditionId.ShouldBe(AlarmRef);
raise.Message.ShouldBe("already hot"); raise.Message.ShouldBe("already hot");
raise.OpcUaSeverity.ShouldBe(800); raise.OpcUaSeverity.ShouldBe(800);
@@ -1,10 +1,12 @@
using System.Text.Json; using System.Text.Json;
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.TestKit; using Akka.TestKit;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
using Shouldly; using Shouldly;
using Xunit; using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Engines; using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa; using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types; using ZB.MOM.WW.OtOpcUa.Commons.Types;
@@ -15,6 +17,7 @@ using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.FOCAS; using ZB.MOM.WW.OtOpcUa.Driver.FOCAS;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers; using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa; using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness; using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers; namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
@@ -44,8 +47,8 @@ public sealed class FocasNativeAlarmEndToEndTests : RuntimeActorTestBase
{ {
var db = NewInMemoryDbFactory(); var db = NewInMemoryDbFactory();
var blob = BuildArtifact(); var blob = BuildArtifact();
var deploymentId = Seed(db, blob); var deploymentId = Seed(db, blob, Rev);
var (host, publish) = SpawnHostAndApply(db, deploymentId); var (host, publish, _) = SpawnHostAndApply(db, deploymentId);
// The raw tags exactly as the driver receives them: the artifact's merged DriverConfig RawTags array. // The raw tags exactly as the driver receives them: the artifact's merged DriverConfig RawTags array.
var rawTags = RawTagsFor(blob, DriverId); var rawTags = RawTagsFor(blob, DriverId);
@@ -89,6 +92,76 @@ public sealed class FocasNativeAlarmEndToEndTests : RuntimeActorTestBase
} }
} }
/// <summary>
/// Gitea #647 review (MAJOR-1) — a deploy that changes ANY of the driver's config respawns it, and the fresh
/// instance has no record of what the server was told. Its first poll re-reads the still-active alarm and
/// sends Raise; the host must treat that as a re-statement: the operator's acknowledgement survives and no
/// second "Activated" alerts row is written. Driven through the real host with a FRESH driver instance.
/// </summary>
[Fact]
public async Task A_respawned_driver_re_reading_an_acknowledged_active_alarm_keeps_the_ack()
{
var db = NewInMemoryDbFactory();
var blob = BuildArtifact();
var alerts = CreateTestProbe();
DistributedPubSub.Get(Sys).Mediator.Tell(new Subscribe(ScriptedAlarmHostActor.AlertsTopic, alerts.Ref), alerts.Ref);
alerts.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
var (host, publish, coordinator) = SpawnHostAndApply(db, Seed(db, blob, Rev));
var rawTags = RawTagsFor(blob, DriverId);
var cnc = new AlarmOnlyFocasClient();
cnc.SetAlarms(new FocasActiveAlarm(401, FocasAlarmType.Servo, 1, "SERVO V-READY OFF"));
var first = await StartDriverAsync(rawTags, cnc, host);
try
{
publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(_ => true, TimeSpan.FromSeconds(5))
.State.Acknowledged.ShouldBeFalse();
alerts.ExpectMsg<AlarmTransitionEvent>(TimeSpan.FromSeconds(5));
}
finally
{
await first.ShutdownAsync(CancellationToken.None);
}
// The operator acknowledges, then a deploy changes the driver's config => stop + respawn.
host.Tell(new DriverHostActor.RouteNativeAlarmAck(ExpectedRawPath, "seen", "op"));
var revB = RevisionHash.Parse(new string('e', 64));
host.Tell(new DispatchDeployment(Seed(db, blob, revB), revB, CorrelationId.NewId()));
coordinator.FishForMessage<ApplyAck>(a => a.Outcome == ApplyAckOutcome.Applied, TimeSpan.FromSeconds(5));
publish.FishForMessage<OpcUaPublishActor.RebuildAddressSpace>(_ => true, TimeSpan.FromSeconds(5));
var respawned = await StartDriverAsync(rawTags, cnc, host);
try
{
var restated = publish.FishForMessage<OpcUaPublishActor.AlarmStateUpdate>(_ => true, TimeSpan.FromSeconds(5));
restated.AlarmNodeId.ShouldBe(ExpectedRawPath);
restated.State.Active.ShouldBeTrue();
restated.State.Acknowledged.ShouldBeTrue("a respawned driver's re-statement must not un-acknowledge");
alerts.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
}
finally
{
await respawned.ShutdownAsync(CancellationToken.None);
}
}
private static async Task<FocasDriver> StartDriverAsync(List<RawTagEntry> rawTags, IFocasClient cnc, IActorRef host)
{
var driver = new FocasDriver(new FocasDriverOptions
{
Devices = [new FocasDeviceOptions(Host, DeviceName: "m1")],
RawTags = rawTags,
Probe = new FocasProbeOptions { Enabled = false },
AlarmProjection = new FocasAlarmProjectionOptions { PollInterval = TimeSpan.FromMilliseconds(20) },
}, DriverId, new SingleClientFactory(cnc));
driver.OnAlarmEvent += (_, e) => host.Tell(new DriverInstanceActor.AttributeAlarmPublished(DriverId, e));
await driver.InitializeAsync("{}", CancellationToken.None);
#pragma warning disable OTOPCUA0001 // the routing is under test, not resilience wrapping
await driver.SubscribeAlarmsAsync([.. rawTags.Select(t => t.RawPath)], CancellationToken.None);
#pragma warning restore OTOPCUA0001
return driver;
}
/// <summary>The seeded artifact's FOCAS driver raw tags, decoded through the production artifact reader.</summary> /// <summary>The seeded artifact's FOCAS driver raw tags, decoded through the production artifact reader.</summary>
private static List<RawTagEntry> RawTagsFor(byte[] blob, string driverInstanceId) private static List<RawTagEntry> RawTagsFor(byte[] blob, string driverInstanceId)
{ {
@@ -133,14 +206,14 @@ public sealed class FocasNativeAlarmEndToEndTests : RuntimeActorTestBase
}, },
}); });
private static DeploymentId Seed(IDbContextFactory<OtOpcUaConfigDbContext> db, byte[] blob) private static DeploymentId Seed(IDbContextFactory<OtOpcUaConfigDbContext> db, byte[] blob, RevisionHash rev)
{ {
var id = DeploymentId.NewId(); var id = DeploymentId.NewId();
using var ctx = db.CreateDbContext(); using var ctx = db.CreateDbContext();
ctx.Deployments.Add(new Deployment ctx.Deployments.Add(new Deployment
{ {
DeploymentId = id.Value, DeploymentId = id.Value,
RevisionHash = Rev.Value, RevisionHash = rev.Value,
Status = DeploymentStatus.Sealed, Status = DeploymentStatus.Sealed,
CreatedBy = "test", CreatedBy = "test",
SealedAtUtc = DateTime.UtcNow, SealedAtUtc = DateTime.UtcNow,
@@ -150,7 +223,7 @@ public sealed class FocasNativeAlarmEndToEndTests : RuntimeActorTestBase
return id; return id;
} }
private (IActorRef Actor, TestProbe Publish) SpawnHostAndApply( private (IActorRef Actor, TestProbe Publish, TestProbe Coordinator) SpawnHostAndApply(
IDbContextFactory<OtOpcUaConfigDbContext> db, DeploymentId deploymentId) IDbContextFactory<OtOpcUaConfigDbContext> db, DeploymentId deploymentId)
{ {
var coordinator = CreateTestProbe(); var coordinator = CreateTestProbe();
@@ -166,7 +239,7 @@ public sealed class FocasNativeAlarmEndToEndTests : RuntimeActorTestBase
actor.Tell(new DispatchDeployment(deploymentId, Rev, CorrelationId.NewId())); actor.Tell(new DispatchDeployment(deploymentId, Rev, CorrelationId.NewId()));
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied); coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied);
publish.ExpectMsg<OpcUaPublishActor.RebuildAddressSpace>(TimeSpan.FromSeconds(5)); publish.ExpectMsg<OpcUaPublishActor.RebuildAddressSpace>(TimeSpan.FromSeconds(5));
return (actor, publish); return (actor, publish, coordinator);
} }
private sealed class SingleClientFactory(IFocasClient client) : IFocasClientFactory private sealed class SingleClientFactory(IFocasClient client) : IFocasClientFactory
@@ -224,6 +224,29 @@ public class NativeAlarmProjectorTests
snap.StatusCode.ShouldBe(OpcUaStatus.BadNoCommunication); snap.StatusCode.ShouldBe(OpcUaStatus.BadNoCommunication);
} }
/// <summary>
/// Gitea #647 review — a Raise on a condition that is ALREADY active is a re-statement (a respawned driver
/// re-reading a still-active alarm it never told the server about): it keeps the acknowledgement and is
/// flagged so the host writes no duplicate "Activated" row. Only a Retrigger (or inactive → active) un-acks.
/// </summary>
[Fact]
public void A_Raise_on_an_already_active_condition_is_a_restatement_that_keeps_the_ack()
{
var sut = new NativeAlarmProjector();
sut.Project("n1", Evt(AlarmTransitionKind.Raise), out var first).Active.ShouldBeTrue();
first.ShouldBeFalse("inactive → active is a real transition");
sut.MarkAcknowledged("n1");
var restated = sut.Project("n1", Evt(AlarmTransitionKind.Raise, msg: "again"), out var restatement);
restatement.ShouldBeTrue();
restated.Active.ShouldBeTrue();
restated.Acknowledged.ShouldBeTrue("a re-statement must not un-acknowledge");
var retriggered = sut.Project("n1", Evt(AlarmTransitionKind.Retrigger), out var retrigger);
retrigger.ShouldBeFalse();
retriggered.Acknowledged.ShouldBeFalse("a Retrigger is a new statement and un-acks");
}
private static AlarmEventArgs Evt( private static AlarmEventArgs Evt(
AlarmTransitionKind kind, AlarmTransitionKind kind,
AlarmSeverity sev = AlarmSeverity.High, AlarmSeverity sev = AlarmSeverity.High,