fix(alarms): #647 review — Raise on an active condition is a re-statement; FOCAS outage quality + health

MAJOR-1: a deploy that changes a driver's config respawns it, and the fresh instance's empty told
record re-sent Raise for every still-active alarm, un-acking acknowledged conditions and writing a
duplicate "Activated" alerts row. NativeAlarmProjector now treats Raise on an already-active
condition as a re-statement (keeps Acked; ForwardNativeAlarm skips the alerts row); only Retrigger
(or inactive->active) un-acks. Sources whose upstream ack state is authoritative send Retrigger for
an unacked re-statement: OpcUaClient on first sight (no prior told), Galaxy for an unacked active
snapshot entry. Runtime E2E drives a FRESH FOCAS driver after a redeploy and asserts the ack survives.

MAJOR-2: a failed FOCAS alarm read annotates each subscribed condition BadNoCommunication once per
outage (#626 quality-only path) with one Warning (MINOR-6), lifted to Good on recovery; failed alarm
reads degrade driver health with LastSuccessfulRead frozen (successful ones advance it), so an
alarm-only driver trips the #527 comms-loss watchdog.

MINOR-3 CLAUDE.md #630 bullet; MINOR-4 split line; MINOR-5 a non-empty cnc_rdalmmsg2 payload that
fits neither stride throws; MINOR-7 value-tag + parsed-address tables are built and swapped, never
cleared in place under the retained poll engine. Docs corrected (FOCAS.md, AlarmTracking.md, CLAUDE.md).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 20:18:03 -04:00
parent 482bafdfac
commit 89961af6e6
17 changed files with 516 additions and 66 deletions
@@ -47,7 +47,9 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
private readonly Dictionary<string, DeviceState> _devices = new(StringComparer.OrdinalIgnoreCase);
// v3: RawPath → definition, case-sensitive ordinal. Built in InitializeAsync from _options.RawTags
// via the pure FocasTagDefinitionFactory mapper (the reference is always a RawPath).
private readonly Dictionary<string, FocasTagDefinition> _tagsByRawPath = new(StringComparer.Ordinal);
// Built whole in InitializeAsync and SWAPPED in (Gitea #647 review): the #630-retained poll engine keeps calling
// ReadAsync during a re-initialise, and a table cleared and refilled in place reads BadNodeIdUnknown in between.
private Dictionary<string, FocasTagDefinition> _tagsByRawPath = new(StringComparer.Ordinal);
// Resolves a read/write fullReference (always a RawPath in v3) to a tag definition by a single
// authored-table hit. A miss is a miss — the driver surfaces BadNodeIdUnknown, never a blob-parse.
private readonly EquipmentTagRefResolver<FocasTagDefinition> _resolver;
@@ -56,7 +58,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
// call. ConcurrentDictionary is required because the poll loop
// (ReadAsync) and the host write thread (WriteAsync) both mutate this cache when resolver-
// produced equipment tags are encountered for the first time.
private readonly ConcurrentDictionary<string, FocasAddress> _parsedAddressesByTagName =
private ConcurrentDictionary<string, FocasAddress> _parsedAddressesByTagName =
new(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Whether this driver instance runs its per-device fixed-tree poll loop, and therefore
@@ -128,7 +130,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
_logger = logger ?? NullLogger<FocasDriver>.Instance;
_alarmProjection = new FocasAlarmProjection(this, _logger);
_resolver = new EquipmentTagRefResolver<FocasTagDefinition>(
r => _tagsByRawPath.TryGetValue(r, out var t) ? t : null);
r => Volatile.Read(ref _tagsByRawPath).TryGetValue(r, out var t) ? t : null);
_poll = new PollGroupEngine(
reader: ReadAsync,
onChange: (handle, tagRef, snapshot) =>
@@ -234,8 +236,9 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
_skippedTags.BeginRebuild(_options.RawTags.Count);
// A re-initialise without teardown (#630) rebuilds the tables from scratch: a tag the new config
// removed — or moved between the value and alarm tables (#647) — must not survive in the old one.
_tagsByRawPath.Clear();
_parsedAddressesByTagName.Clear();
// Built into fresh tables and swapped in below, never cleared in place under a live reader.
var tagsByRawPath = new Dictionary<string, FocasTagDefinition>(StringComparer.Ordinal);
var parsedAddresses = new ConcurrentDictionary<string, FocasAddress>(StringComparer.OrdinalIgnoreCase);
var alarmTags = new Dictionary<string, FocasAlarmTag>(StringComparer.Ordinal);
foreach (var entry in _options.RawTags)
{
@@ -298,7 +301,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
entry.RawPath, why, _driverInstanceId);
continue;
}
_tagsByRawPath[entry.RawPath] = def with
tagsByRawPath[entry.RawPath] = def with
{
DeviceHostAddress = deviceHost,
WriteIdempotent = entry.WriteIdempotent,
@@ -306,12 +309,14 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
// Cache the parsed FocasAddress so ReadAsync / WriteAsync don't re-parse on every
// hot-path call. The address string has already been validated by FocasAddress.TryParse
// above; reusing the parsed record avoids per-tick allocs on subscription pollers.
_parsedAddressesByTagName[entry.RawPath] = parsed;
parsedAddresses[entry.RawPath] = parsed;
}
// Publish the skip tally only once the whole table is built. An unknown-device throw above
// leaves the previous snapshot standing rather than a truncated one that would read as
// "we now skip fewer tags".
_skippedTags.Commit();
Volatile.Write(ref _parsedAddressesByTagName, parsedAddresses);
Volatile.Write(ref _tagsByRawPath, tagsByRawPath);
Volatile.Write(ref _alarmTags, alarmTags);
if (_options.Probe.Enabled)
@@ -465,7 +470,7 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
state.DisposeClient();
}
_devices.Clear();
_tagsByRawPath.Clear();
Volatile.Write(ref _tagsByRawPath, new Dictionary<string, FocasTagDefinition>(StringComparer.Ordinal));
Volatile.Write(ref _alarmTags, new Dictionary<string, FocasAlarmTag>(StringComparer.Ordinal));
_resolver.Clear(); // no-op in v3 (the resolver reads the live table by closure); kept for symmetry
_parsedAddressesByTagName.Clear();
@@ -1693,6 +1698,35 @@ public sealed class FocasDriver : IDriver, IReadable, IWritable, ITagDiscovery,
/// <summary>Gets the alarm poll cadence — one <c>cnc_rdalmmsg2</c> per alarm-owning device per tick.</summary>
internal TimeSpan AlarmPollInterval => _options.AlarmProjection.PollInterval;
/// <summary>
/// A successful alarm read is a successful read (Gitea #647 review): it advances
/// <see cref="DriverHealth.LastSuccessfulRead"/> exactly as a Good value read does, so an alarm-only driver's
/// health tracks its CNC. Never lifts a <see cref="DriverState.Faulted"/> (a config-level signal).
/// </summary>
internal void NoteAlarmReadSucceeded()
{
var current = Volatile.Read(ref _health);
if (current.State != DriverState.Faulted)
Volatile.Write(ref _health, new DriverHealth(DriverState.Healthy, DateTime.UtcNow, null));
}
/// <summary>
/// A failed alarm read degrades driver health, PRESERVING <see cref="DriverHealth.LastSuccessfulRead"/> — the
/// conjunction the comms-loss watchdog (#527) trips on. Without it a driver whose only tags are <c>ALARM:</c>
/// tags stayed Healthy through a total outage, the watchdog never fired, and the host's #477
/// <c>BadNoCommunication</c> annotation never ran (Gitea #647 review).
/// </summary>
/// <param name="hostAddress">The device whose alarm read failed.</param>
/// <param name="error">Why.</param>
internal void NoteAlarmReadFailed(string hostAddress, Exception error)
{
var current = Volatile.Read(ref _health);
if (current.State != DriverState.Faulted)
Volatile.Write(ref _health, new DriverHealth(DriverState.Degraded, current.LastSuccessfulRead,
$"FOCAS alarm read failed for {hostAddress}: {error.Message}"));
}
/// <summary>
/// Reads one device's active-alarm list. <b>Throws on any failure</b> — an unknown device, a connect that
/// fails, a read the CNC refuses — because the projection must never mistake a failed read for an empty