fix(GWC-37): grade the alarm subtag watch-list instead of latching it silently

The subtag fallback's coverage was fixed at SubscribeAlarms time and never re-examined. AlarmWatchListResolver deliberately swallows a Galaxy Repository outage and returns whatever configuration contributed, so a discovery failure at startup left the monitor subscribed with an empty watch-list for the rest of its life. In Auto mode the alarmmgr primary keeps working, so nothing looks wrong until the failover that needs the standby finds it covering zero alarms — with a single LogWarning emitted possibly weeks earlier as the only evidence. ResolveAsync now returns an AlarmWatchListResolution (targets + whether discovery was attempted and whether it failed, with the failure message), which is what the caller needed to tell "discovery produced nothing" from "nothing was configured". GatewayAlarmMonitor grades that resolution against the effective mode and folds the verdict into AlarmProviderStatus, so it reaches the feed, the dashboard badge and every late-joining subscriber: - zero targets while the mode can need them (Auto/ForceSubtag) -> degraded with SubtagWatchListEmpty; discovery enabled but failing adds SubtagDiscoveryFailed and the message. ForceAlarmManager builds no subtag consumer and is not graded. - degraded is now the OR of "subtag is serving" and "the watch-list covers nothing", so the Auto case where alarmmgr is healthy and the standby is empty is visible. DashboardAlarmProviderStatus renders that combination as the amber "Alarm Manager (no subtag standby)" badge rather than the failover label, which would have named the wrong live provider. - on the failover edge (ApplyProviderModeChangeAsync, toMode == Subtag) the monitor re-resolves BEFORE publishing the status and reports any difference as SubtagWatchListStale with both counts. The re-resolution is deliberately NOT re-sent. The worker rejects a second SubscribeAlarms while subscribed (AlarmCommandHandler.Subscribe), and the only expressible replacement — UnsubscribeAlarms then SubscribeAlarms — rebuilds FailoverAlarmConsumer, which restarts on the primary that just failed and discards the warm standby state, so it would flap the feed rather than repair it. The finding's other alternative (treat empty coverage as a lifecycle failure) was rejected too: it would crash-loop the monitor every 5 s on a persistent GR outage and take the working alarmmgr feed down with it. A monitor restart re-resolves and re-sends, and the status now says so. Docs: docs/AlarmClientDiscovery.md gains the grading rules and corrects two stale claims (the resolver method name, and "re-runs discovery on its reconcile cadence and pushes an updated watch-list", which never happened); gateway.md gains the standby-coverage visibility paragraph; docs/GatewayConfiguration.md records the consequence on the UseGalaxyRepository and Mode rows. Tests: SubtagCoverage_WhenDiscoveryFailsAtSubscribe_ProviderStatusIsDegradedWith- Reason and SubtagCoverage_WhenDiscoveryRecoversByFailover_ReResolvesAndReportsDrift drive the fake-worker monitor; two resolver cases pin the new provenance fields. The pre-existing dashboard case for degraded+ALARMMGR now asserts the new label.
2026-08-20 09:38:46 -04:00
parent 282a0e70ad
commit 193221773d
16 changed files with 635 additions and 44 deletions
@@ -835,7 +835,7 @@ carries `degraded = true`.
 ### Watch-list discovery

 `GatewayAlarmMonitor` resolves the subtag watch-list at subscribe time by
-calling `IAlarmWatchListResolver.GetAlarmAttributesAsync`. The resolver merges:
+calling `IAlarmWatchListResolver.ResolveAsync`. The resolver merges:

 1. Galaxy Repository SQL (`GetAlarmAttributesAsync`) — objects that have alarm
   extensions in the configured area.
@@ -847,8 +847,42 @@ The resolved list is a set of `AlarmSubtagTarget` messages sent to the worker
 inside `SubscribeAlarmsCommand.watch_list`. Each target carries the composed
 MXAccess item addresses for the `InAlarm`, `Acked`, `AckMsg`, and `Priority`
 subtags (confirmed AVEVA `AlarmExtension` field names, verified against the live
-ZB Galaxy `attribute_definition` rows). The gateway re-runs discovery on its
-reconcile cadence and pushes an updated watch-list when the model changes.
+ZB Galaxy `attribute_definition` rows).
+
+**The list is fixed for the life of the subscription.** The worker builds its
+subtag consumer from the targets `SubscribeAlarms` carried and keeps them until
+the subscription ends; a second `SubscribeAlarms` on a subscribed session is
+rejected, and the only way to replace the list — `UnsubscribeAlarms` then
+`SubscribeAlarms` — rebuilds `FailoverAlarmConsumer`, restarting it on the
+primary and discarding the warm standby state. Re-sending on a failover would
+therefore flap the feed rather than repair it. A monitor restart (a worker fault,
+or a gateway restart) re-resolves and re-sends, and that is the supported way to
+pick up Galaxy objects added since the monitor started.
+
+Because the list is fixed, the resolution is **graded** rather than merely
+logged (GWC-37). `ResolveAsync` returns an `AlarmWatchListResolution` carrying
+whether Galaxy Repository discovery was attempted and whether it failed, and the
+monitor folds the verdict into the alarm feed's `AlarmProviderStatus`:
+
+- Zero targets while the effective mode can need them (`Auto` or `ForceSubtag`)
+  sets `degraded = true` with reason `AlarmProviderReasons.SubtagWatchListEmpty`
+  — the subtag provider covers no alarms.
+- Discovery enabled but failing adds
+  `AlarmProviderReasons.SubtagDiscoveryFailed` and the failure message, so a
+  Galaxy Repository outage at startup is a standing status rather than one log
+  line emitted possibly weeks before the failover that needs the list.
+- On the failover edge (`ApplyProviderModeChangeAsync` with `toMode == Subtag`)
+  the monitor re-resolves and compares against what it sent. A difference is
+  reported as `AlarmProviderReasons.SubtagWatchListStale` with both counts, on
+  the same status frame as the failover, because it cannot be applied.
+
+`degraded` on the status is therefore the OR of "the subtag provider is the one
+serving" and "the subtag watch-list cannot cover anything". In `Auto` the second
+can be true while the mode is still `ALARMMGR`: the live feed is healthy and the
+standby behind it is empty. The dashboard renders that combination as the amber
+`Alarm Manager (no subtag standby)` badge rather than the failover label, so the
+badge never names the wrong live provider. `ForceAlarmManager` never builds a
+subtag consumer, so its watch-list is not graded at all.

 Each target's canonical `AlarmFullReference` is composed as
 `Galaxy!{area}.{reference}` (literal `Galaxy` provider). The `{area}` is the
@@ -445,11 +445,11 @@ native wnwrap alarm-manager provider and the subtag-monitoring fallback.

 | Option | Default | Description |
 |--------|---------|-------------|
-| `MxGateway:Alarms:Fallback:Mode` | `Auto` | Provider selection mode. `Auto` uses the alarm manager as primary and fails over to subtag monitoring after consecutive COM failures, then fails back automatically. `ForceAlarmManager` disables failover. `ForceSubtag` forces subtag monitoring on from startup. Values are case-insensitive. |
+| `MxGateway:Alarms:Fallback:Mode` | `Auto` | Provider selection mode. `Auto` uses the alarm manager as primary and fails over to subtag monitoring after consecutive COM failures, then fails back automatically. `ForceAlarmManager` disables failover (and is the one mode whose subtag watch-list is never graded, because no subtag consumer is built). `ForceSubtag` forces subtag monitoring on from startup. Values are case-insensitive. |
 | `MxGateway:Alarms:Fallback:ConsecutiveFailureThreshold` | `3` | Number of consecutive wnwrap COM failures (`COMException` or failure HRESULT from `Subscribe` / `GetXmlCurrentAlarms2`) before the monitor switches to subtag mode. Floored at 1. |
 | `MxGateway:Alarms:Fallback:FailbackProbeIntervalSeconds` | `30` | While in subtag mode, how often (in seconds) the monitor probes the wnwrap provider to detect recovery. Floored at 1. |
 | `MxGateway:Alarms:Fallback:FailbackStableProbes` | `3` | Number of consecutive clean wnwrap probes required before the monitor switches back to the alarm manager. Floored at 1. |
-| `MxGateway:Alarms:Fallback:Discovery:UseGalaxyRepository` | `true` | When `true`, the monitor queries the Galaxy Repository SQL database to build the subtag watch-list for the configured area. |
+| `MxGateway:Alarms:Fallback:Discovery:UseGalaxyRepository` | `true` | When `true`, the monitor queries the Galaxy Repository SQL database to build the subtag watch-list for the configured area. A failed query is not fatal — the monitor continues with whatever configuration contributed — but with `Mode` `Auto` or `ForceSubtag` the resulting coverage gap is reported on the alarm feed's `AlarmProviderStatus` (`degraded = true` with a `Subtag watch-list is empty` / `Galaxy Repository alarm discovery failed` reason) and shown on the dashboard as the amber `Alarm Manager (no subtag standby)` badge, instead of only a startup log line. |
 | `MxGateway:Alarms:Fallback:Discovery:Area` | _(empty)_ | Galaxy area to scope the Repository query to. Falls back to `MxGateway:Alarms:DefaultArea` when empty. Ignored when `UseGalaxyRepository` is `false`. This area is **not** used to compose a Repository-derived alarm's canonical `Galaxy!{area}.{reference}`: each discovered alarm uses its object's real Galaxy area (discovered via `gobject.area_gobject_id`), so the reference's group matches what the native alarmmgr emits. `Discovery:Area` / `DefaultArea` is used as the composition area only for explicit `IncludeAttributes` entries, which carry no discovered area. |
 | `MxGateway:Alarms:Fallback:Discovery:IncludeAttributes` | _(empty)_ | Explicit MXAccess attribute paths to add to the subtag watch-list, supplementing (or replacing, when `UseGalaxyRepository` is `false`) the Repository-derived list. |
 | `MxGateway:Alarms:Fallback:Discovery:ExcludeAttributes` | _(empty)_ | Attribute paths to remove from the merged watch-list (case-insensitive). The exclude is applied after the Repository-derived rows and the explicit `IncludeAttributes` entries are combined, so an exclude that matches an explicit include suppresses it too — excludes win. Ignored when `UseGalaxyRepository` is `false`. |