feat(alarms): structural degraded-status signal for truncated alarm snapshots

The truncation-cliff fix made alarm transitions truncation-safe but silent:
when GetXmlCurrentAlarms2 returns exactly maxAlmCnt records the worker
suppresses absence-implies-Clear inference and says so only in a rate-limited
stderr warning. No client and no operator could tell a complete active set
from a capped one.

Two additive proto3 booleans carry the verdict out:

- QueryActiveAlarmsReplyPayload.snapshot_truncated = 2 (worker IPC reply)
- ActiveAlarmSnapshot.from_truncated_snapshot = 16 (per record)

The per-record field is not an aesthetic choice. QueryActiveAlarms returns a
bare `stream ActiveAlarmSnapshot` with no envelope, header, or trailer, so a
per-record boolean is the only carrier that stays wire-compatible; an envelope
message would change every existing client's stream element type. The reply
payload states it too because a prefix filter can leave zero records and a
truncated fetch with nothing to report still has to say so. The flag means
"this set may be incomplete", never "this record is unreliable" — it is
independent of the subtag-fallback `degraded` field.

Detection is deliberately UNCHANGED: IsTruncatedFetch remains
`fetchedRecordCount >= maxAlarmsPerFetch`. The live probe (docs/AlarmProbeFindings.md,
ce5d8ae) could not verify whether ALARM_RECORDS/@COUNT reports the total active
count or only the records in the reply, so @COUNT is not parsed for detection;
switching to it stays blocked on probe evidence. The probe's comment
annotations in WnWrapAlarmConsumer.cs are preserved.

Reset semantics: not latched. WnWrapAlarmConsumer.FoldFetch replaces the
verdict on every poll under the same lock as the snapshot merge, so the first
sub-cap fetch clears it; GatewayAlarmMonitor.ClearCache drops it with the cache
generation it describes. A caveat that never turns off is one operators learn
to ignore.

Flow: WnWrapAlarmConsumer.LastSnapshotTruncated -> AlarmDispatcher (stamps every
record) / IAlarmCommandHandler (payload) -> MxAccessCommandExecutor reply ->
GatewayAlarmMonitor._snapshotTruncated -> IGatewayAlarmService.SnapshotTruncated
-> DashboardAlarmQueryResult -> AlarmsPage warning banner (render-side only; the
poll loop and DisposeAsync drain are untouched). The public QueryActiveAlarms
RPC forwards worker snapshots unmodified, so the per-record flag needed no
mapper change — a test pins that.

Parity: this describes OUR fetch mechanics — additive gateway metadata — not
MXAccess provider behavior. No event is synthesized and no MXAccess-observable
semantics change, so it is not a parity deviation.

Tests: worker LastSnapshotTruncated set/reset/consecutive-burst (windev-run);
gateway end-to-end truncated reply -> monitor -> public stream, with the
complete-reply control as the load-bearing assertion; AlarmsPage banner
present/absent. Docs: gateway.md alarm surface, docs/DesignDecisions.md entry.
This commit is contained in:
Joseph Doherty
2026-08-17 04:18:34 -04:00
parent b8b7b69ba0
commit 693a78db7d
41 changed files with 2217 additions and 309 deletions
@@ -371,6 +371,9 @@ public sealed class AlarmCommandExecutorTests
/// <summary>Gets the last alarm filter prefix.</summary>
public string? LastFilterPrefix { get; private set; }
/// <summary>Gets or sets the truncation verdict the executor stamps onto the reply payload.</summary>
public bool LastSnapshotTruncated { get; set; }
/// <inheritdoc />
public void Subscribe(SubscribeAlarmsCommand command, string sessionId)
{
@@ -473,6 +473,9 @@ public sealed class AlarmCommandHandlerTests
/// <inheritdoc />
public IReadOnlyList<MxAlarmSnapshotRecord> SnapshotActiveAlarms() => SnapshotResult;
/// <summary>Gets or sets the truncation verdict reported for the last fetch.</summary>
public bool LastSnapshotTruncated { get; set; }
/// <summary>Gets the number of times polled.</summary>
public int PollCount { get; private set; }
@@ -438,6 +438,9 @@ public sealed class AlarmDispatcherTests
return SnapshotResult;
}
/// <summary>Gets or sets the truncation verdict the dispatcher stamps onto snapshots.</summary>
public bool LastSnapshotTruncated { get; set; }
/// <summary>Gets the count of poll operations.</summary>
public int PollCount { get; private set; }
@@ -80,6 +80,9 @@ public sealed class FailoverAlarmConsumerTests
/// <inheritdoc />
public IReadOnlyList<MxAlarmSnapshotRecord> SnapshotActiveAlarms() => Array.Empty<MxAlarmSnapshotRecord>();
/// <summary>Gets or sets the truncation verdict this child reports, so delegation is observable.</summary>
public bool LastSnapshotTruncated { get; set; }
/// <inheritdoc />
public void Dispose() { }
@@ -132,6 +135,9 @@ public sealed class FailoverAlarmConsumerTests
return Array.Empty<MxAlarmSnapshotRecord>();
}
/// <summary>Gets or sets the truncation verdict this child reports, so delegation is observable.</summary>
public bool LastSnapshotTruncated { get; set; }
/// <inheritdoc />
public void Dispose() { }
@@ -643,6 +643,9 @@ public sealed class MxAccessStaSessionTests
get { lock (gate) return lastPollThreadId; }
}
/// <summary>Gets or sets the truncation verdict reported for the last fetch.</summary>
public bool LastSnapshotTruncated { get; set; }
/// <inheritdoc />
public void Subscribe(SubscribeAlarmsCommand command, string sessionId)
{
@@ -688,6 +688,88 @@ public sealed class WnWrapAlarmConsumerXmlTests
Assert.Equal(MxAlarmStateKind.UnackAlm, record.State);
}
// -------------------------------------------------------------------------
// Degraded-status signal. The truncation guard above keeps a capped fetch
// from broadcasting phantom Clears, but it does so silently: the retained
// snapshot simply stops shrinking. LastSnapshotTruncated is what makes that
// suppression visible to the QueryActiveAlarms reply and, through it, the
// dashboard banner — so its set/reset behaviour is the contract, not detail.
// -------------------------------------------------------------------------
/// <summary>
/// A capped fetch sets the retained truncation verdict. Without this the
/// signal never leaves the consumer and the reply builder stamps a
/// complete-looking snapshot over a capped one.
/// </summary>
[Fact]
public void FoldFetch_WhenFetchTruncated_SetsLastSnapshotTruncated()
{
const int Cap = 8;
using WnWrapAlarmConsumer consumer = new WnWrapAlarmConsumer(Cap);
Assert.False(consumer.LastSnapshotTruncated);
Dictionary<Guid, MxAlarmSnapshotRecord> next =
WnWrapAlarmConsumer.ParseSnapshotXml(BuildAlarmXml(Cap), out int fetchedRecordCount);
Assert.True(WnWrapAlarmConsumer.IsTruncatedFetch(fetchedRecordCount, Cap));
consumer.FoldFetch(next, truncated: true, out int retainedCount);
Assert.True(consumer.LastSnapshotTruncated);
Assert.Equal(Cap, retainedCount);
}
/// <summary>
/// THE reset test. A sub-cap fetch is complete, so it restores absence
/// authority and must clear the verdict. Latching it instead would leave
/// the operator banner asserting "snapshot may be incomplete" forever
/// after a single burst above the cap, which trains operators to ignore
/// it — the opposite of what the signal is for.
/// </summary>
[Fact]
public void FoldFetch_AfterTruncatedFetch_SubCapFetchClearsLastSnapshotTruncated()
{
const int Cap = 8;
using WnWrapAlarmConsumer consumer = new WnWrapAlarmConsumer(Cap);
Dictionary<Guid, MxAlarmSnapshotRecord> capped =
WnWrapAlarmConsumer.ParseSnapshotXml(BuildAlarmXml(Cap), out _);
consumer.FoldFetch(capped, truncated: true, out _);
Assert.True(consumer.LastSnapshotTruncated);
Dictionary<Guid, MxAlarmSnapshotRecord> complete =
WnWrapAlarmConsumer.ParseSnapshotXml(BuildAlarmXml(Cap - 1), out int fetchedRecordCount);
Assert.False(WnWrapAlarmConsumer.IsTruncatedFetch(fetchedRecordCount, Cap));
consumer.FoldFetch(complete, truncated: false, out int retainedCount);
Assert.False(consumer.LastSnapshotTruncated);
// The complete fetch also replaced the snapshot wholesale, which is what
// makes it authoritative about absence — pinned here so a future change
// cannot clear the verdict while keeping the merge semantics.
Assert.Equal(Cap - 1, retainedCount);
}
/// <summary>
/// Consecutive capped fetches keep the verdict set. It is per-fetch state,
/// not an edge-triggered one-shot: an operator arriving mid-burst must
/// still see the caveat.
/// </summary>
[Fact]
public void FoldFetch_WithConsecutiveTruncatedFetches_KeepsLastSnapshotTruncatedSet()
{
const int Cap = 8;
using WnWrapAlarmConsumer consumer = new WnWrapAlarmConsumer(Cap);
for (int pass = 0; pass < 3; pass++)
{
Dictionary<Guid, MxAlarmSnapshotRecord> capped =
WnWrapAlarmConsumer.ParseSnapshotXml(BuildAlarmXml(Cap), out _);
consumer.FoldFetch(capped, truncated: true, out _);
Assert.True(consumer.LastSnapshotTruncated);
}
}
private static MxAlarmSnapshotRecord NewRecord(Guid guid, MxAlarmStateKind state)
{
return new MxAlarmSnapshotRecord