fix(alarms): fetch/poll ceilings; truncation-semantics docs; log-format conformance

This commit is contained in:
Joseph Doherty
2026-08-15 17:19:41 -04:00
parent 7c9add3d73
commit b5ea6bb461
11 changed files with 254 additions and 40 deletions
+24 -4
View File
@@ -172,6 +172,19 @@ Consequences, and how this sits with the existing failover/reconcile design:
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
back. Consumers already apply transitions as "set this alarm to this state",
so a repeated or delayed Clear is absorbed.
- **Under *sustained* truncation, some intermediate history is lost — end state
is not.** For an alarm that stays outside the fetch window, a full
clear→re-raise cycle that begins and ends between two sightings emits **no
transitions at all**: the retained record is identical before and after, so
the diff sees nothing to report. Consumers that render current state are
correct; consumers that *count occurrences* lose an event. Likewise, an
operator acknowledgement of an out-of-window alarm does not reach the feed
until that alarm re-enters a fetch window, at which point the reconcile
repairs the acked state. This is a strictly better failure than the
pre-guard behaviour (which fabricated a Clear for every out-of-window alarm
on every poll), but it is not lossless, and it is another reason a
persistently truncating deployment is a configuration defect to fix rather
than a mode to run in.
- **It does not synthesize anything.** Suppressing an inference is the opposite
of inventing an event; no transition is fabricated on a truncated poll.
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
@@ -179,10 +192,17 @@ Consequences, and how this sits with the existing failover/reconcile design:
consumer's own snapshot bookkeeping and changes neither the failure counting
that triggers failover nor the subtag standby's snapshot, which is built from
a bounded watch-list and has no per-fetch cap to hit.
- **Operators get told.** A truncated poll logs a rate-limited (once per
minute) `AlarmSnapshotTruncated` warning carrying the cap, the record counts,
and the running truncated-fetch total — identifiers and counts only, never
tag names, values, limits, or comments. A galaxy that truncates persistently
- **Operators get told, weakly.** A truncated poll logs a rate-limited (once
per minute) `AlarmSnapshotTruncated` warning carrying the cap, the record
counts, and the running truncated-fetch total — identifiers and counts only,
never tag names, values, limits, or comments. Be honest about its reach: it
goes to the worker's console/stderr, which is captured on dev hosts but is
not a metric, not a dashboard tile, and not part of any session-status or
alarm-feed payload, so a production deployment can truncate indefinitely
without anyone noticing. Surfacing truncation as a **structural** degraded
status (a field on the alarm-provider mode/status surface the dashboard and
`StreamAlarms` consumers already read) is filed as a follow-up; until it
lands, the log line is the only signal. A galaxy that truncates persistently
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
## Session-Resilience Epic Scope