fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently

The subtag fallback's coverage was fixed at SubscribeAlarms time and never
re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository
outage and returns whatever configuration contributed, so a discovery failure at
startup left the monitor subscribed with an empty watch-list for the rest of its
life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong
until the failover that needs the standby finds it covering zero alarms — with a
single LogWarning emitted possibly weeks earlier as the only evidence.

ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery
was attempted and whether it failed, with the failure message), which is what the
caller needed to tell "discovery produced nothing" from "nothing was configured".
GatewayAlarmMonitor grades that resolution against the effective mode and folds
the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge
and every late-joining subscriber:

- zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with
  SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed
  and the message. ForceAlarmManager builds no subtag consumer and is not graded.
- degraded is now the OR of "subtag is serving" and "the watch-list covers
  nothing", so the Auto case where alarmmgr is healthy and the standby is empty
  is visible. DashboardAlarmProviderStatus renders that combination as the amber
  "Alarm Manager (no subtag standby)" badge rather than the failover label, which
  would have named the wrong live provider.
- on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the
  monitor re-resolves BEFORE publishing the status and reports any difference as
  SubtagWatchListStale with both counts.

The re-resolution is deliberately NOT re-sent. The worker rejects a second
SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only
expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds
FailoverAlarmConsumer, which restarts on the primary that just failed and
discards the warm standby state, so it would flap the feed rather than repair it.
The finding's other alternative (treat empty coverage as a lifecycle failure) was
rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage
and take the working alarmmgr feed down with it. A monitor restart re-resolves and
re-sends, and the status now says so.

Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale
claims (the resolver method name, and "re-runs discovery on its reconcile cadence
and pushes an updated watch-list", which never happened); gateway.md gains the
standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the
consequence on the UseGalaxyRepository and Mode rows.

Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith-
Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift
drive the fake-worker monitor; two resolver cases pin the new provenance fields.
The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
This commit is contained in:
Joseph Doherty
2026-08-20 09:38:46 -04:00
parent 282a0e70ad
commit 193221773d
16 changed files with 635 additions and 44 deletions
+16
View File
@@ -249,6 +249,22 @@ monitoring (forced)") when subtag mode is the configured `Fallback:Mode=ForceSub
as a fault. Metrics: `mxgateway.alarms.provider_mode` gauge (1 = alarmmgr,
2 = subtag) and `mxgateway.alarms.provider_switches` counter.
**Standby-coverage visibility:** `degraded` also covers the case where the alarm
manager is serving normally but the subtag standby behind it could not cover
anything — a watch-list that resolved to zero targets, or Galaxy Repository
discovery that was enabled and failed. That condition used to be a single startup
log line, invisible until the failover that needed the standby found it empty; it
is now a standing `AlarmProviderStatus` reason
(`AlarmProviderReasons.SubtagWatchListEmpty` / `SubtagDiscoveryFailed`) carried on
the feed and rendered as the amber `Alarm Manager (no subtag standby)` badge, which
keeps the failover label from naming the wrong live provider. The watch-list the
worker holds is fixed for the life of the subscription, so on the failover edge the
monitor re-resolves and reports a difference
(`AlarmProviderReasons.SubtagWatchListStale`, with both counts) rather than
re-sending it — re-sending would rebuild the failover consumer on the primary that
just failed. A monitor restart re-resolves and re-sends. See
`docs/AlarmClientDiscovery.md`.
**Truncated-snapshot visibility:** `GetXmlCurrentAlarms2` caps its reply at
`MxGateway:Alarms:MaxAlarmsPerFetch` and offers no confirmed "more available"
flag, so a reply holding exactly the cap is treated as truncated. On such a