fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently
The subtag fallback's coverage was fixed at SubscribeAlarms time and never re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository outage and returns whatever configuration contributed, so a discovery failure at startup left the monitor subscribed with an empty watch-list for the rest of its life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong until the failover that needs the standby finds it covering zero alarms — with a single LogWarning emitted possibly weeks earlier as the only evidence. ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery was attempted and whether it failed, with the failure message), which is what the caller needed to tell "discovery produced nothing" from "nothing was configured". GatewayAlarmMonitor grades that resolution against the effective mode and folds the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge and every late-joining subscriber: - zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed and the message. ForceAlarmManager builds no subtag consumer and is not graded. - degraded is now the OR of "subtag is serving" and "the watch-list covers nothing", so the Auto case where alarmmgr is healthy and the standby is empty is visible. DashboardAlarmProviderStatus renders that combination as the amber "Alarm Manager (no subtag standby)" badge rather than the failover label, which would have named the wrong live provider. - on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the monitor re-resolves BEFORE publishing the status and reports any difference as SubtagWatchListStale with both counts. The re-resolution is deliberately NOT re-sent. The worker rejects a second SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds FailoverAlarmConsumer, which restarts on the primary that just failed and discards the warm standby state, so it would flap the feed rather than repair it. The finding's other alternative (treat empty coverage as a lifecycle failure) was rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage and take the working alarmmgr feed down with it. A monitor restart re-resolves and re-sends, and the status now says so. Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale claims (the resolver method name, and "re-runs discovery on its reconcile cadence and pushes an updated watch-list", which never happened); gateway.md gains the standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the consequence on the UseGalaxyRepository and Mode rows. Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith- Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift drive the fake-worker monitor; two resolver cases pin the new provenance fields. The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
This commit is contained in:
@@ -15,4 +15,32 @@ public static class AlarmProviderReasons
|
||||
/// distinguish a forced subtag mode from an unexpected degraded failover.
|
||||
/// </summary>
|
||||
public const string ForcedSubtag = "Forced subtag mode (configuration)";
|
||||
|
||||
/// <summary>
|
||||
/// Reason prefix set when the effective mode can need the subtag watch-list
|
||||
/// (<c>Auto</c> or <c>ForceSubtag</c>) but the list resolved to zero targets, so the
|
||||
/// subtag provider covers no alarms (GWC-37). In <c>Auto</c> this is a warning about the
|
||||
/// standby, not about the currently-serving alarm manager, which is why the dashboard
|
||||
/// renders it against the <c>ALARMMGR</c> mode as a coverage badge rather than a failover.
|
||||
/// </summary>
|
||||
public const string SubtagWatchListEmpty =
|
||||
"Subtag watch-list is empty: the subtag provider covers no alarms";
|
||||
|
||||
/// <summary>
|
||||
/// Reason prefix set when Galaxy Repository alarm discovery was enabled but failed, so
|
||||
/// the watch-list carries only what configuration contributed and may be incomplete
|
||||
/// (GWC-37). Distinct from <see cref="SubtagWatchListEmpty"/>: discovery failing with a
|
||||
/// usable config-only list is a completeness warning, not a total loss of coverage.
|
||||
/// </summary>
|
||||
public const string SubtagDiscoveryFailed =
|
||||
"Galaxy Repository alarm discovery failed: the subtag watch-list may be incomplete";
|
||||
|
||||
/// <summary>
|
||||
/// Reason prefix set when a re-resolution at the failover edge produced a different
|
||||
/// watch-list than the one sent at <c>SubscribeAlarms</c> (GWC-37). The worker keeps the
|
||||
/// list it was given for the life of the subscription, so the difference is reported
|
||||
/// rather than applied; a monitor restart re-resolves and re-sends.
|
||||
/// </summary>
|
||||
public const string SubtagWatchListStale =
|
||||
"Subtag watch-list differs from the one the worker is using";
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user