Merge fix/c3-session-lifecycle: GWC-32 WorkerClient dispose is terminal, GWC-34 attach-after-dispose throws SessionNotReady, GWC-37 subtag watch-list coverage graded on provider status + re-resolved at failover edge
This commit is contained in:
@@ -835,7 +835,7 @@ carries `degraded = true`.
|
||||
### Watch-list discovery
|
||||
|
||||
`GatewayAlarmMonitor` resolves the subtag watch-list at subscribe time by
|
||||
calling `IAlarmWatchListResolver.GetAlarmAttributesAsync`. The resolver merges:
|
||||
calling `IAlarmWatchListResolver.ResolveAsync`. The resolver merges:
|
||||
|
||||
1. Galaxy Repository SQL (`GetAlarmAttributesAsync`) — objects that have alarm
|
||||
extensions in the configured area.
|
||||
@@ -847,8 +847,42 @@ The resolved list is a set of `AlarmSubtagTarget` messages sent to the worker
|
||||
inside `SubscribeAlarmsCommand.watch_list`. Each target carries the composed
|
||||
MXAccess item addresses for the `InAlarm`, `Acked`, `AckMsg`, and `Priority`
|
||||
subtags (confirmed AVEVA `AlarmExtension` field names, verified against the live
|
||||
ZB Galaxy `attribute_definition` rows). The gateway re-runs discovery on its
|
||||
reconcile cadence and pushes an updated watch-list when the model changes.
|
||||
ZB Galaxy `attribute_definition` rows).
|
||||
|
||||
**The list is fixed for the life of the subscription.** The worker builds its
|
||||
subtag consumer from the targets `SubscribeAlarms` carried and keeps them until
|
||||
the subscription ends; a second `SubscribeAlarms` on a subscribed session is
|
||||
rejected, and the only way to replace the list — `UnsubscribeAlarms` then
|
||||
`SubscribeAlarms` — rebuilds `FailoverAlarmConsumer`, restarting it on the
|
||||
primary and discarding the warm standby state. Re-sending on a failover would
|
||||
therefore flap the feed rather than repair it. A monitor restart (a worker fault,
|
||||
or a gateway restart) re-resolves and re-sends, and that is the supported way to
|
||||
pick up Galaxy objects added since the monitor started.
|
||||
|
||||
Because the list is fixed, the resolution is **graded** rather than merely
|
||||
logged (GWC-37). `ResolveAsync` returns an `AlarmWatchListResolution` carrying
|
||||
whether Galaxy Repository discovery was attempted and whether it failed, and the
|
||||
monitor folds the verdict into the alarm feed's `AlarmProviderStatus`:
|
||||
|
||||
- Zero targets while the effective mode can need them (`Auto` or `ForceSubtag`)
|
||||
sets `degraded = true` with reason `AlarmProviderReasons.SubtagWatchListEmpty`
|
||||
— the subtag provider covers no alarms.
|
||||
- Discovery enabled but failing adds
|
||||
`AlarmProviderReasons.SubtagDiscoveryFailed` and the failure message, so a
|
||||
Galaxy Repository outage at startup is a standing status rather than one log
|
||||
line emitted possibly weeks before the failover that needs the list.
|
||||
- On the failover edge (`ApplyProviderModeChangeAsync` with `toMode == Subtag`)
|
||||
the monitor re-resolves and compares against what it sent. A difference is
|
||||
reported as `AlarmProviderReasons.SubtagWatchListStale` with both counts, on
|
||||
the same status frame as the failover, because it cannot be applied.
|
||||
|
||||
`degraded` on the status is therefore the OR of "the subtag provider is the one
|
||||
serving" and "the subtag watch-list cannot cover anything". In `Auto` the second
|
||||
can be true while the mode is still `ALARMMGR`: the live feed is healthy and the
|
||||
standby behind it is empty. The dashboard renders that combination as the amber
|
||||
`Alarm Manager (no subtag standby)` badge rather than the failover label, so the
|
||||
badge never names the wrong live provider. `ForceAlarmManager` never builds a
|
||||
subtag consumer, so its watch-list is not graded at all.
|
||||
|
||||
Each target's canonical `AlarmFullReference` is composed as
|
||||
`Galaxy!{area}.{reference}` (literal `Galaxy` provider). The `{area}` is the
|
||||
|
||||
@@ -453,11 +453,11 @@ native wnwrap alarm-manager provider and the subtag-monitoring fallback.
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Alarms:Fallback:Mode` | `Auto` | Provider selection mode. `Auto` uses the alarm manager as primary and fails over to subtag monitoring after consecutive COM failures, then fails back automatically. `ForceAlarmManager` disables failover. `ForceSubtag` forces subtag monitoring on from startup. Values are case-insensitive. |
|
||||
| `MxGateway:Alarms:Fallback:Mode` | `Auto` | Provider selection mode. `Auto` uses the alarm manager as primary and fails over to subtag monitoring after consecutive COM failures, then fails back automatically. `ForceAlarmManager` disables failover (and is the one mode whose subtag watch-list is never graded, because no subtag consumer is built). `ForceSubtag` forces subtag monitoring on from startup. Values are case-insensitive. |
|
||||
| `MxGateway:Alarms:Fallback:ConsecutiveFailureThreshold` | `3` | Number of consecutive wnwrap COM failures (`COMException` or failure HRESULT from `Subscribe` / `GetXmlCurrentAlarms2`) before the monitor switches to subtag mode. Floored at 1. |
|
||||
| `MxGateway:Alarms:Fallback:FailbackProbeIntervalSeconds` | `30` | While in subtag mode, how often (in seconds) the monitor probes the wnwrap provider to detect recovery. Floored at 1. |
|
||||
| `MxGateway:Alarms:Fallback:FailbackStableProbes` | `3` | Number of consecutive clean wnwrap probes required before the monitor switches back to the alarm manager. Floored at 1. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:UseGalaxyRepository` | `true` | When `true`, the monitor queries the Galaxy Repository SQL database to build the subtag watch-list for the configured area. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:UseGalaxyRepository` | `true` | When `true`, the monitor queries the Galaxy Repository SQL database to build the subtag watch-list for the configured area. A failed query is not fatal — the monitor continues with whatever configuration contributed — but with `Mode` `Auto` or `ForceSubtag` the resulting coverage gap is reported on the alarm feed's `AlarmProviderStatus` (`degraded = true` with a `Subtag watch-list is empty` / `Galaxy Repository alarm discovery failed` reason) and shown on the dashboard as the amber `Alarm Manager (no subtag standby)` badge, instead of only a startup log line. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:Area` | _(empty)_ | Galaxy area to scope the Repository query to. Falls back to `MxGateway:Alarms:DefaultArea` when empty. Ignored when `UseGalaxyRepository` is `false`. This area is **not** used to compose a Repository-derived alarm's canonical `Galaxy!{area}.{reference}`: each discovered alarm uses its object's real Galaxy area (discovered via `gobject.area_gobject_id`), so the reference's group matches what the native alarmmgr emits. `Discovery:Area` / `DefaultArea` is used as the composition area only for explicit `IncludeAttributes` entries, which carry no discovered area. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:IncludeAttributes` | _(empty)_ | Explicit MXAccess attribute paths to add to the subtag watch-list, supplementing (or replacing, when `UseGalaxyRepository` is `false`) the Repository-derived list. |
|
||||
| `MxGateway:Alarms:Fallback:Discovery:ExcludeAttributes` | _(empty)_ | Attribute paths to remove from the merged watch-list (case-insensitive). The exclude is applied after the Repository-derived rows and the explicit `IncludeAttributes` entries are combined, so an exclude that matches an explicit include suppresses it too — excludes win. Ignored when `UseGalaxyRepository` is `false`. |
|
||||
|
||||
@@ -213,6 +213,8 @@ The repair transitions the monitor's reconcile broadcasts on the alarm feed (Rai
|
||||
|
||||
`AttachInternalEventSubscriber` enforces the same readiness gate as `AttachEventSubscriber` — a session (or worker) that is not `Ready` throws `SessionNotReady` *before* the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime.
|
||||
|
||||
The same one-distributor-per-session rule is enforced at the other end of the lifecycle. Session disposal disposes the distributor and latches the fact, so an attach that passed its readiness gate and then lost the CPU until the session was torn down throws `SessionNotReady` rather than constructing a replacement distributor. A replacement can never be pumped — the pump-start flag is a one-way latch — so its subscriber would block on an empty channel that is never completed until the client's own deadline expired, with no error, log or metric to explain it. Every attach entry point (`AttachEventSubscriber`, `AttachEventSubscriberWithReplay`, `AttachInternalEventSubscriber`) fails this way, and each rolls back the subscriber-count slot it took, so a caller that loses the race to teardown sees the same failure it would have seen a moment later from the readiness gate.
|
||||
|
||||
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
|
||||
|
||||
#### Teardown parallelism
|
||||
|
||||
Reference in New Issue
Block a user