fix(alarms): truncation-safe transitions; perf: single-pass alarm parse; configurable poll cadence

This commit is contained in:
Joseph Doherty
2026-08-15 17:04:06 -04:00
parent 94fdc18c3c
commit f3e1de5f37
10 changed files with 910 additions and 33 deletions
+50
View File
@@ -135,6 +135,56 @@ alarm state is gateway-wide, not session-scoped — every client wants the same
current set plus updates, and forcing each to own a worker would multiply AVEVA
polling load for no benefit.
### Alarms — a capped snapshot fetch never implies a clear
Decision (2026-08-15): when the worker's `GetXmlCurrentAlarms2` fetch comes back
holding exactly `MxGateway:Alarms:MaxAlarmsPerFetch` records, the worker treats
the snapshot as **truncated** and merges it into the retained snapshot instead
of replacing it. Alarms the capped reply did carry update normally; alarms it
had no room to mention are retained untouched.
The COM API caps its reply at `maxAlmCnt` and exposes no "more available" flag,
so a reply sitting exactly on the cap is indistinguishable from a galaxy that
happens to hold exactly that many active alarms. Both are treated as truncated,
because the two error directions are not symmetric.
Nothing in the worker emits a Clear transition. The clear is an **inference**:
`WnWrapAlarmConsumer.ComputeTransitions` produces no transition for an alarm
that disappears from the snapshot, and `GatewayAlarmMonitor.ApplyReconcile`
later diffs its cache against `SnapshotActiveAlarms()` and broadcasts a Clear
for every cached alarm the worker no longer reports. Before this decision, a
capped fetch shrank that snapshot, so every alarm past the cap was broadcast as
cleared while still standing — a silent, galaxy-wide false clear on exactly the
alarm floods where the cap is reached.
Consequences, and how this sits with the existing failover/reconcile design:
- **The suppression is an eviction guard, not a transition filter.** It lives in
the snapshot update inside `PollOnce`, not in `ComputeTransitions`, which was
never going to emit anything for a disappearance. The reconcile/dedup
machinery (`_clearedByReconcile` tombstones, the NEXT-03 duplicate-Clear
suppression) is untouched: it still sees the same shape of snapshot, only
with the truncated poll's unmentionable alarms still present.
- **It preserves at-least-once, idempotent application.** The failure mode
becomes bounded staleness — a genuinely cleared alarm can linger until the
first sub-cap fetch evicts it, and the reconcile then broadcasts its Clear
late. A late Clear is repaired by the next complete poll; a Clear that never
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
back. Consumers already apply transitions as "set this alarm to this state",
so a repeated or delayed Clear is absorbed.
- **It does not synthesize anything.** Suppressing an inference is the opposite
of inventing an event; no transition is fabricated on a truncated poll.
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
`IMxAccessAlarmConsumer` is live; the guard is internal to the wnwrap
consumer's own snapshot bookkeeping and changes neither the failure counting
that triggers failover nor the subtag standby's snapshot, which is built from
a bounded watch-list and has no per-fetch cap to hit.
- **Operators get told.** A truncated poll logs a rate-limited (once per
minute) `AlarmSnapshotTruncated` warning carrying the cap, the record counts,
and the running truncated-fetch total — identifiers and counts only, never
tag names, values, limits, or comments. A galaxy that truncates persistently
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
## Session-Resilience Epic Scope
Decision (2026-07-09, archreview TST-04; migrated here 2026-08-07 from the retired
+2
View File
@@ -417,6 +417,8 @@ behavior.
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be `>= 100`: every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or unusable value leaves the worker on the 500 ms default rather than failing the session. |
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be `>= 64`. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning (identifiers and counts only). Raise this on galaxies whose steady-state active-alarm count approaches the cap — a galaxy permanently above it holds stale entries in the snapshot until a sub-cap poll. Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or unusable value leaves the worker on the 1024 default. |
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
`StreamAlarms` are session-less RPCs served by the monitor.