Joseph Doherty
193221773d
The subtag fallback's coverage was fixed at SubscribeAlarms time and never re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository outage and returns whatever configuration contributed, so a discovery failure at startup left the monitor subscribed with an empty watch-list for the rest of its life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong until the failover that needs the standby finds it covering zero alarms — with a single LogWarning emitted possibly weeks earlier as the only evidence. ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery was attempted and whether it failed, with the failure message), which is what the caller needed to tell "discovery produced nothing" from "nothing was configured". GatewayAlarmMonitor grades that resolution against the effective mode and folds the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge and every late-joining subscriber: - zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed and the message. ForceAlarmManager builds no subtag consumer and is not graded. - degraded is now the OR of "subtag is serving" and "the watch-list covers nothing", so the Auto case where alarmmgr is healthy and the standby is empty is visible. DashboardAlarmProviderStatus renders that combination as the amber "Alarm Manager (no subtag standby)" badge rather than the failover label, which would have named the wrong live provider. - on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the monitor re-resolves BEFORE publishing the status and reports any difference as SubtagWatchListStale with both counts. The re-resolution is deliberately NOT re-sent. The worker rejects a second SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds FailoverAlarmConsumer, which restarts on the primary that just failed and discards the warm standby state, so it would flap the feed rather than repair it. The finding's other alternative (treat empty coverage as a lifecycle failure) was rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage and take the working alarmmgr feed down with it. A monitor restart re-resolves and re-sends, and the status now says so. Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale claims (the resolver method name, and "re-runs discovery on its reconcile cadence and pushes an updated watch-list", which never happened); gateway.md gains the standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the consequence on the UseGalaxyRepository and Mode rows. Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith- Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift drive the fake-worker monitor; two resolver cases pin the new provenance fields. The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
30 lines
1.4 KiB
C#
30 lines
1.4 KiB
C#
using ZB.MOM.WW.MxGateway.Contracts.Proto;
|
|
|
|
namespace ZB.MOM.WW.MxGateway.Server.Alarms;
|
|
|
|
/// <summary>
|
|
/// The outcome of one subtag watch-list resolution: the composed targets plus enough
|
|
/// provenance for the caller to tell an empty list that means "nothing was configured"
|
|
/// from one that means "discovery was configured and silently produced nothing"
|
|
/// (GWC-37). <see cref="IAlarmWatchListResolver.ResolveAsync"/> deliberately never throws
|
|
/// on a discovery failure, so without this the two are indistinguishable at the call site
|
|
/// and a Galaxy Repository outage at startup can only be found in a log line.
|
|
/// </summary>
|
|
/// <param name="Targets">The resolved watch-list, in order; possibly empty.</param>
|
|
/// <param name="DiscoveryAttempted">
|
|
/// Galaxy Repository discovery was enabled for this resolution
|
|
/// (<c>Fallback:Discovery:UseGalaxyRepository</c>).
|
|
/// </param>
|
|
/// <param name="DiscoveryFailed">
|
|
/// Discovery was attempted and failed, so <paramref name="Targets"/> holds only what
|
|
/// configuration contributed.
|
|
/// </param>
|
|
/// <param name="DiscoveryFailureMessage">
|
|
/// The discovery failure's message, or <c>null</c> when discovery did not fail.
|
|
/// </param>
|
|
public sealed record AlarmWatchListResolution(
|
|
IReadOnlyList<AlarmSubtagTarget> Targets,
|
|
bool DiscoveryAttempted,
|
|
bool DiscoveryFailed,
|
|
string? DiscoveryFailureMessage);
|