fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently
The subtag fallback's coverage was fixed at SubscribeAlarms time and never re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository outage and returns whatever configuration contributed, so a discovery failure at startup left the monitor subscribed with an empty watch-list for the rest of its life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong until the failover that needs the standby finds it covering zero alarms — with a single LogWarning emitted possibly weeks earlier as the only evidence. ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery was attempted and whether it failed, with the failure message), which is what the caller needed to tell "discovery produced nothing" from "nothing was configured". GatewayAlarmMonitor grades that resolution against the effective mode and folds the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge and every late-joining subscriber: - zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed and the message. ForceAlarmManager builds no subtag consumer and is not graded. - degraded is now the OR of "subtag is serving" and "the watch-list covers nothing", so the Auto case where alarmmgr is healthy and the standby is empty is visible. DashboardAlarmProviderStatus renders that combination as the amber "Alarm Manager (no subtag standby)" badge rather than the failover label, which would have named the wrong live provider. - on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the monitor re-resolves BEFORE publishing the status and reports any difference as SubtagWatchListStale with both counts. The re-resolution is deliberately NOT re-sent. The worker rejects a second SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds FailoverAlarmConsumer, which restarts on the primary that just failed and discards the warm standby state, so it would flap the feed rather than repair it. The finding's other alternative (treat empty coverage as a lifecycle failure) was rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage and take the working alarmmgr feed down with it. A monitor restart re-resolves and re-sends, and the status now says so. Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale claims (the resolver method name, and "re-runs discovery on its reconcile cadence and pushes an updated watch-list", which never happened); gateway.md gains the standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the consequence on the UseGalaxyRepository and Mode rows. Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith- Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift drive the fake-worker monitor; two resolver cases pin the new provenance fields. The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
This commit is contained in:
Joseph Doherty
@@ -623,6 +623,184 @@ public sealed class GatewayAlarmMonitorProviderModeTests
|
||||
await monitor.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-37: a Galaxy Repository outage at startup used to leave one <c>LogWarning</c> and an
|
||||
/// empty subtag watch-list that stayed empty for the monitor's whole life — invisible until
|
||||
/// a failover found the standby covering nothing. The condition is now a standing verdict
|
||||
/// on the provider status, so it reaches the alarm feed, the dashboard badge and every
|
||||
/// late-joining subscriber. The mode stays <c>ALARMMGR</c>: the live provider is healthy,
|
||||
/// it is the standby behind it that is unsound.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWithReason()
|
||||
{
|
||||
using GatewayMetrics metrics = new();
|
||||
FakeSessionManager sessions = new();
|
||||
ScriptedWatchListResolver resolver = new(DiscoveryFailedResolution());
|
||||
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, resolver);
|
||||
|
||||
using CancellationTokenSource cts = new();
|
||||
await monitor.StartAsync(cts.Token);
|
||||
await sessions.WaitForSubscribeAsync(WaitTimeout);
|
||||
|
||||
// The empty list still went to the worker — the verdict is a report, not a behaviour change.
|
||||
Assert.Empty(sessions.LastSubscribeCommand!.WatchList);
|
||||
|
||||
AlarmProviderStatus status = await ReadFirstProviderStatusAsync(monitor);
|
||||
|
||||
Assert.Equal(AlarmProviderMode.Alarmmgr, status.Mode);
|
||||
Assert.True(status.Degraded);
|
||||
Assert.Contains(AlarmProviderReasons.SubtagWatchListEmpty, status.Reason, StringComparison.Ordinal);
|
||||
Assert.Contains(AlarmProviderReasons.SubtagDiscoveryFailed, status.Reason, StringComparison.Ordinal);
|
||||
Assert.Contains("gr offline", status.Reason, StringComparison.Ordinal);
|
||||
|
||||
await cts.CancelAsync();
|
||||
await monitor.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GWC-37: the failover edge is the moment the watch-list stops being theoretical, so the
|
||||
/// monitor re-resolves it there and publishes the fresh verdict on the same status frame
|
||||
/// the failover produces. Discovery that has recovered since <c>SubscribeAlarms</c> is
|
||||
/// reported as drift rather than applied — the worker rejects a second <c>SubscribeAlarms</c>
|
||||
/// while subscribed, and the only way to replace the list rebuilds the failover consumer on
|
||||
/// the primary that just failed. The status names both the fresh count and the sent count so
|
||||
/// an operator knows a monitor restart will pick the new list up.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift()
|
||||
{
|
||||
using GatewayMetrics metrics = new();
|
||||
FakeSessionManager sessions = new();
|
||||
ScriptedWatchListResolver resolver = new(
|
||||
DiscoveryFailedResolution(),
|
||||
new AlarmWatchListResolution(
|
||||
[Target("Galaxy!Area.Tank01.Level.HiHi"), Target("Galaxy!Area.Tank02.Level.HiHi")],
|
||||
DiscoveryAttempted: true,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
using GatewayAlarmMonitor monitor = CreateMonitor(sessions, metrics, resolver);
|
||||
|
||||
using CancellationTokenSource cts = new();
|
||||
await monitor.StartAsync(cts.Token);
|
||||
await sessions.WaitForSubscribeAsync(WaitTimeout);
|
||||
Assert.Equal(1, resolver.Calls);
|
||||
|
||||
List<AlarmFeedMessage> received = [];
|
||||
TaskCompletionSource baselineReceived = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
using CancellationTokenSource streamCts = new();
|
||||
Task reader = Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
await foreach (AlarmFeedMessage message in monitor.StreamAsync(null, streamCts.Token))
|
||||
{
|
||||
lock (received)
|
||||
{
|
||||
received.Add(message);
|
||||
if (received.Count == 1)
|
||||
{
|
||||
baselineReceived.TrySetResult();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Expected when the test cancels the stream.
|
||||
}
|
||||
});
|
||||
|
||||
await baselineReceived.Task.WaitAsync(WaitTimeout);
|
||||
|
||||
sessions.EmitEvent(new MxEvent
|
||||
{
|
||||
OnAlarmProviderModeChanged = new OnAlarmProviderModeChangedEvent
|
||||
{
|
||||
Mode = AlarmProviderMode.Subtag,
|
||||
Reason = "alarmmgr failed",
|
||||
At = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow),
|
||||
},
|
||||
});
|
||||
|
||||
AlarmFeedMessage degraded = await WaitForAsync(
|
||||
received,
|
||||
m => m.PayloadCase == AlarmFeedMessage.PayloadOneofCase.ProviderStatus
|
||||
&& m.ProviderStatus.Mode == AlarmProviderMode.Subtag,
|
||||
WaitTimeout);
|
||||
|
||||
// The re-resolve happened on the failover edge, before the status was published.
|
||||
Assert.Equal(2, resolver.Calls);
|
||||
|
||||
string reason = degraded.ProviderStatus.Reason;
|
||||
Assert.Contains("alarmmgr failed", reason, StringComparison.Ordinal);
|
||||
Assert.Contains(AlarmProviderReasons.SubtagWatchListStale, reason, StringComparison.Ordinal);
|
||||
Assert.Contains("2 target(s) now resolve against 0 sent", reason, StringComparison.Ordinal);
|
||||
|
||||
// The stale-list warning replaced the empty/failed pair: discovery works again, so the only
|
||||
// remaining problem is that the worker is still using the list it was handed.
|
||||
Assert.DoesNotContain(AlarmProviderReasons.SubtagWatchListEmpty, reason, StringComparison.Ordinal);
|
||||
Assert.DoesNotContain(AlarmProviderReasons.SubtagDiscoveryFailed, reason, StringComparison.Ordinal);
|
||||
|
||||
await streamCts.CancelAsync();
|
||||
await reader;
|
||||
await cts.CancelAsync();
|
||||
await monitor.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
/// <summary>Reads the provider-status frame every stream opens with, then closes the stream.</summary>
|
||||
/// <param name="monitor">The monitor to read from.</param>
|
||||
/// <returns>The provider status carried by the first frame.</returns>
|
||||
private static async Task<AlarmProviderStatus> ReadFirstProviderStatusAsync(GatewayAlarmMonitor monitor)
|
||||
{
|
||||
using CancellationTokenSource streamCts = new(WaitTimeout);
|
||||
await foreach (AlarmFeedMessage message in monitor.StreamAsync(null, streamCts.Token))
|
||||
{
|
||||
if (message.PayloadCase == AlarmFeedMessage.PayloadOneofCase.ProviderStatus)
|
||||
{
|
||||
return message.ProviderStatus;
|
||||
}
|
||||
}
|
||||
|
||||
throw new TimeoutException("No provider-status message was received in time.");
|
||||
}
|
||||
|
||||
/// <summary>A resolution standing for "Galaxy Repository discovery was on and it failed".</summary>
|
||||
/// <returns>The resolution.</returns>
|
||||
private static AlarmWatchListResolution DiscoveryFailedResolution() => new(
|
||||
[],
|
||||
DiscoveryAttempted: true,
|
||||
DiscoveryFailed: true,
|
||||
DiscoveryFailureMessage: "gr offline");
|
||||
|
||||
private static AlarmSubtagTarget Target(string fullReference) => new()
|
||||
{
|
||||
AlarmFullReference = fullReference,
|
||||
SourceObjectReference = fullReference,
|
||||
ActiveSubtag = $"{fullReference}.InAlarm",
|
||||
AckedSubtag = $"{fullReference}.Acked",
|
||||
};
|
||||
|
||||
private static GatewayAlarmMonitor CreateMonitor(
|
||||
FakeSessionManager sessions,
|
||||
GatewayMetrics metrics,
|
||||
IAlarmWatchListResolver resolver)
|
||||
{
|
||||
AlarmsOptions options = new()
|
||||
{
|
||||
Enabled = true,
|
||||
SubscriptionExpression = @"\\NODE\Galaxy!Area",
|
||||
};
|
||||
return new GatewayAlarmMonitor(
|
||||
sessions,
|
||||
resolver,
|
||||
metrics,
|
||||
Microsoft.Extensions.Options.Options.Create(new GatewayOptions { Alarms = options }),
|
||||
NullLogger<GatewayAlarmMonitor>.Instance);
|
||||
}
|
||||
|
||||
private static GatewayAlarmMonitor CreateMonitor(FakeSessionManager sessions, GatewayMetrics metrics)
|
||||
{
|
||||
AlarmsOptions options = new()
|
||||
@@ -693,13 +871,42 @@ public sealed class GatewayAlarmMonitorProviderModeTests
|
||||
throw new TimeoutException("Condition was not met in time.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="IAlarmWatchListResolver"/> that hands out a scripted sequence of
|
||||
/// resolutions — one per call, the last repeating — and counts the calls, so a test can
|
||||
/// drive "discovery was down at subscribe and back up by the failover" and assert that the
|
||||
/// second resolution happened at all (GWC-37).
|
||||
/// </summary>
|
||||
/// <param name="script">Resolutions to return, in call order.</param>
|
||||
private sealed class ScriptedWatchListResolver(params AlarmWatchListResolution[] script) : IAlarmWatchListResolver
|
||||
{
|
||||
private int _calls;
|
||||
|
||||
/// <summary>Gets the number of times <see cref="ResolveAsync"/> has been called.</summary>
|
||||
public int Calls => Volatile.Read(ref _calls);
|
||||
|
||||
/// <inheritdoc />
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
int index = Interlocked.Increment(ref _calls) - 1;
|
||||
return Task.FromResult(script[Math.Min(index, script.Length - 1)]);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary><see cref="IAlarmWatchListResolver"/> that returns a fixed watch-list.</summary>
|
||||
private sealed class StubWatchListResolver(IReadOnlyList<AlarmSubtagTarget> targets) : IAlarmWatchListResolver
|
||||
{
|
||||
/// <inheritdoc />
|
||||
public Task<IReadOnlyList<AlarmSubtagTarget>> ResolveAsync(
|
||||
public Task<AlarmWatchListResolution> ResolveAsync(
|
||||
AlarmsOptions options,
|
||||
CancellationToken cancellationToken = default) => Task.FromResult(targets);
|
||||
CancellationToken cancellationToken = default) =>
|
||||
Task.FromResult(new AlarmWatchListResolution(
|
||||
targets,
|
||||
DiscoveryAttempted: false,
|
||||
DiscoveryFailed: false,
|
||||
DiscoveryFailureMessage: null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
Reference in New Issue
Block a user