Files
mxaccessgw/src/ZB.MOM.WW.MxGateway.Server/Alarms/AlarmWatchListResolver.cs
T
Joseph Doherty 193221773d fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently
The subtag fallback's coverage was fixed at SubscribeAlarms time and never
re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository
outage and returns whatever configuration contributed, so a discovery failure at
startup left the monitor subscribed with an empty watch-list for the rest of its
life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong
until the failover that needs the standby finds it covering zero alarms — with a
single LogWarning emitted possibly weeks earlier as the only evidence.

ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery
was attempted and whether it failed, with the failure message), which is what the
caller needed to tell "discovery produced nothing" from "nothing was configured".
GatewayAlarmMonitor grades that resolution against the effective mode and folds
the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge
and every late-joining subscriber:

- zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with
  SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed
  and the message. ForceAlarmManager builds no subtag consumer and is not graded.
- degraded is now the OR of "subtag is serving" and "the watch-list covers
  nothing", so the Auto case where alarmmgr is healthy and the standby is empty
  is visible. DashboardAlarmProviderStatus renders that combination as the amber
  "Alarm Manager (no subtag standby)" badge rather than the failover label, which
  would have named the wrong live provider.
- on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the
  monitor re-resolves BEFORE publishing the status and reports any difference as
  SubtagWatchListStale with both counts.

The re-resolution is deliberately NOT re-sent. The worker rejects a second
SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only
expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds
FailoverAlarmConsumer, which restarts on the primary that just failed and
discards the warm standby state, so it would flap the feed rather than repair it.
The finding's other alternative (treat empty coverage as a lifecycle failure) was
rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage
and take the working alarmmgr feed down with it. A monitor restart re-resolves and
re-sends, and the status now says so.

Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale
claims (the resolver method name, and "re-runs discovery on its reconcile cadence
and pushes an updated watch-list", which never happened); gateway.md gains the
standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the
consequence on the UseGalaxyRepository and Mode rows.

Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith-
Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift
drive the fake-worker monitor; two resolver cases pin the new provenance fields.
The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
2026-08-20 09:38:46 -04:00

183 lines
8.2 KiB
C#

using ZB.MOM.WW.GalaxyRepository;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
using ZB.MOM.WW.MxGateway.Server.Configuration;
namespace ZB.MOM.WW.MxGateway.Server.Alarms;
/// <summary>
/// Default <see cref="IAlarmWatchListResolver"/>. Merges Galaxy Repository
/// alarm-attribute discovery with the configured include/exclude overrides
/// and composes the per-attribute subtag item addresses from the configured
/// subtag names.
/// </summary>
public sealed class AlarmWatchListResolver : IAlarmWatchListResolver
{
private const string ProviderLiteral = "Galaxy";
private const string DefaultActiveSubtag = "InAlarm";
private const string DefaultAckedSubtag = "Acked";
private readonly IGalaxyRepository _repository;
private readonly ILogger<AlarmWatchListResolver> _logger;
/// <summary>Initializes the watch-list resolver.</summary>
/// <param name="repository">Galaxy Repository used for alarm-attribute discovery.</param>
/// <param name="logger">Diagnostic logger.</param>
public AlarmWatchListResolver(
IGalaxyRepository repository,
ILogger<AlarmWatchListResolver> logger)
{
_repository = repository ?? throw new ArgumentNullException(nameof(repository));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
/// <inheritdoc />
public async Task<AlarmWatchListResolution> ResolveAsync(
AlarmsOptions options,
CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(options);
AlarmDiscoveryOptions discovery = options.Fallback.Discovery;
// GWC-37: the caller cannot grade an empty watch-list without knowing whether discovery
// was asked to contribute and whether it failed while doing so, because the failure is
// deliberately swallowed below.
string? discoveryFailureMessage = null;
// Config fallback area used only for explicit IncludeAttributes entries (which
// carry no discovered area): discovery area, else the default area (may be empty).
string configFallbackArea = string.IsNullOrEmpty(discovery.Area) ? options.DefaultArea : discovery.Area;
// 1. Build the ordered, de-duplicated attribute reference set.
// Each entry carries the reference, the source-object reference, and the
// per-entry area used to compose the canonical reference. GR rows contribute
// the object's real Galaxy area; config includes contribute the config
// fallback area (Discovery.Area else DefaultArea).
List<(string Reference, string SourceObject, string Area)> ordered = [];
HashSet<string> seen = new(StringComparer.OrdinalIgnoreCase);
if (discovery.UseGalaxyRepository)
{
List<GalaxyAlarmAttributeRow> rows;
try
{
rows = await _repository.GetAlarmAttributesAsync(cancellationToken).ConfigureAwait(false);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
// Discovery being unavailable must not crash the resolver: log and
// continue with an empty discovery set. The caller decides what to
// do with the (possibly config-only) result. Cancellation is the one
// exception — an OperationCanceledException propagates per the
// IAlarmWatchListResolver contract so the caller unwinds promptly.
_logger.LogWarning(
ex,
"Galaxy Repository alarm-attribute discovery failed; continuing with configuration-only watch-list.");
discoveryFailureMessage = ex.Message;
rows = [];
}
foreach (GalaxyAlarmAttributeRow row in rows)
{
if (string.IsNullOrEmpty(row.FullTagReference) || !seen.Add(row.FullTagReference))
{
continue;
}
ordered.Add((row.FullTagReference, row.SourceObjectReference, row.Area));
}
}
foreach (string include in discovery.IncludeAttributes)
{
if (string.IsNullOrEmpty(include) || !seen.Add(include))
{
continue;
}
ordered.Add((include, DeriveSourceObject(include), configFallbackArea));
}
// Remove excluded references (case-insensitive), but only when GR discovery
// is active. ExcludeAttributes is documented as "Ignored when
// UseGalaxyRepository is false" (AlarmDiscoveryOptions.ExcludeAttributes).
// Whitespace-only entries are skipped, consistent with the include guard above.
if (discovery.UseGalaxyRepository)
{
HashSet<string> excluded = new(
discovery.ExcludeAttributes.Where(e => !string.IsNullOrWhiteSpace(e)),
StringComparer.OrdinalIgnoreCase);
if (excluded.Count > 0)
{
ordered.RemoveAll(e => excluded.Contains(e.Reference));
}
}
// 2. Resolve subtag names with safe fallbacks.
string active = string.IsNullOrEmpty(options.Fallback.Subtags.Active)
? DefaultActiveSubtag
: options.Fallback.Subtags.Active;
string acked = string.IsNullOrEmpty(options.Fallback.Subtags.Acked)
? DefaultAckedSubtag
: options.Fallback.Subtags.Acked;
string priority = options.Fallback.Subtags.Priority;
string ackComment = options.Fallback.Subtags.AckComment;
// 3. Compose one target per reference, using the PER-ENTRY area: the GR row's
// real Galaxy area (matching the alarmmgr group), or the config fallback for
// explicit includes.
List<AlarmSubtagTarget> targets = new(ordered.Count);
foreach ((string reference, string sourceObject, string area) in ordered)
{
targets.Add(new AlarmSubtagTarget
{
AlarmFullReference = ComposeFullReference(area, reference),
SourceObjectReference = sourceObject,
ActiveSubtag = $"{reference}.{active}",
AckedSubtag = $"{reference}.{acked}",
PrioritySubtag = string.IsNullOrEmpty(priority) ? string.Empty : $"{reference}.{priority}",
AckCommentSubtag = string.IsNullOrEmpty(ackComment) ? string.Empty : $"{reference}.{ackComment}",
});
}
// 4. Report the resolved count; warn when subtag mode was expected to cover
// something (GR enabled, or explicit includes were configured) but resolved
// to nothing. Only emit the Debug line when there is at least one target,
// to avoid a confusing "0 target(s)" noise line.
if (targets.Count == 0 && (discovery.UseGalaxyRepository || discovery.IncludeAttributes.Length > 0))
{
_logger.LogWarning(
"Alarm subtag watch-list resolved to zero targets; subtag-polling fallback will cover no alarms.");
}
else if (targets.Count > 0)
{
_logger.LogDebug("Resolved alarm subtag watch-list with {TargetCount} target(s).", targets.Count);
}
return new AlarmWatchListResolution(
targets,
DiscoveryAttempted: discovery.UseGalaxyRepository,
DiscoveryFailed: discoveryFailureMessage is not null,
discoveryFailureMessage);
}
/// <summary>
/// Derives the source-object reference for a configuration entry: the
/// substring before the first '.', or the whole string when there is no dot.
/// </summary>
private static string DeriveSourceObject(string reference)
{
int dot = reference.IndexOf('.', StringComparison.Ordinal);
return dot < 0 ? reference : reference[..dot];
}
/// <summary>
/// Composes the canonical alarm full reference: <c>Galaxy!{area}.{reference}</c>
/// when an area is set, otherwise <c>Galaxy!{reference}</c>.
/// </summary>
private static string ComposeFullReference(string area, string reference) =>
string.IsNullOrEmpty(area)
? $"{ProviderLiteral}!{reference}"
: $"{ProviderLiteral}!{area}.{reference}";
}