fix(alarms): fetch/poll ceilings; truncation-semantics docs; log-format conformance
This commit is contained in:
@@ -140,6 +140,26 @@ Two viable A.2 designs given the probe data:
|
||||
poll period; modest CPU floor because the call is cheap. Matches
|
||||
the heartbeat-style WM 0xC275 semantics — AVEVA itself runs a
|
||||
poll loop internally.
|
||||
|
||||
As shipped, this is the chosen design, and the cadence is **no
|
||||
longer fixed at 500 ms**: it is the 500 ms *default* of
|
||||
`MxGateway:Alarms:PollIntervalMilliseconds` (range 100 ms – 1 h),
|
||||
which the gateway hands the worker through the
|
||||
`MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable. The
|
||||
per-fetch cap is likewise configurable
|
||||
(`MxGateway:Alarms:MaxAlarmsPerFetch`, default 1024).
|
||||
|
||||
One snapshot rule matters when reading the capture below: a fetch
|
||||
that returns exactly the cap is treated as **truncated**, and the
|
||||
worker *merges* it into the retained snapshot instead of replacing
|
||||
it. `GetXmlCurrentAlarms2` caps its reply with no "more available"
|
||||
flag, so a capped reply is authoritative about presence only —
|
||||
alarms it had no room to mention are retained rather than allowed
|
||||
to vanish, because their disappearance is what the gateway's
|
||||
reconcile pass reads as a clear. Only a sub-cap fetch replaces the
|
||||
snapshot wholesale and can therefore clear alarms. See
|
||||
`docs/DesignDecisions.md`, "Alarms — a capped snapshot fetch never
|
||||
implies a clear".
|
||||
2. **Hook AVEVA's internal window.** Discover AVEVA's own window
|
||||
(`hwnd=0x18032E` in the probe), `SetWindowsHookEx` or
|
||||
`SetWindowSubclass` on it, and intercept WM 0xC275 on AVEVA's
|
||||
|
||||
+24
-4
@@ -172,6 +172,19 @@ Consequences, and how this sits with the existing failover/reconcile design:
|
||||
happened is broadcast to every `StreamAlarms` subscriber and cannot be taken
|
||||
back. Consumers already apply transitions as "set this alarm to this state",
|
||||
so a repeated or delayed Clear is absorbed.
|
||||
- **Under *sustained* truncation, some intermediate history is lost — end state
|
||||
is not.** For an alarm that stays outside the fetch window, a full
|
||||
clear→re-raise cycle that begins and ends between two sightings emits **no
|
||||
transitions at all**: the retained record is identical before and after, so
|
||||
the diff sees nothing to report. Consumers that render current state are
|
||||
correct; consumers that *count occurrences* lose an event. Likewise, an
|
||||
operator acknowledgement of an out-of-window alarm does not reach the feed
|
||||
until that alarm re-enters a fetch window, at which point the reconcile
|
||||
repairs the acked state. This is a strictly better failure than the
|
||||
pre-guard behaviour (which fabricated a Clear for every out-of-window alarm
|
||||
on every poll), but it is not lossless, and it is another reason a
|
||||
persistently truncating deployment is a configuration defect to fix rather
|
||||
than a mode to run in.
|
||||
- **It does not synthesize anything.** Suppressing an inference is the opposite
|
||||
of inventing an event; no transition is fabricated on a truncated poll.
|
||||
- **Failover is unaffected.** `FailoverAlarmConsumer` selects which
|
||||
@@ -179,10 +192,17 @@ Consequences, and how this sits with the existing failover/reconcile design:
|
||||
consumer's own snapshot bookkeeping and changes neither the failure counting
|
||||
that triggers failover nor the subtag standby's snapshot, which is built from
|
||||
a bounded watch-list and has no per-fetch cap to hit.
|
||||
- **Operators get told.** A truncated poll logs a rate-limited (once per
|
||||
minute) `AlarmSnapshotTruncated` warning carrying the cap, the record counts,
|
||||
and the running truncated-fetch total — identifiers and counts only, never
|
||||
tag names, values, limits, or comments. A galaxy that truncates persistently
|
||||
- **Operators get told, weakly.** A truncated poll logs a rate-limited (once
|
||||
per minute) `AlarmSnapshotTruncated` warning carrying the cap, the record
|
||||
counts, and the running truncated-fetch total — identifiers and counts only,
|
||||
never tag names, values, limits, or comments. Be honest about its reach: it
|
||||
goes to the worker's console/stderr, which is captured on dev hosts but is
|
||||
not a metric, not a dashboard tile, and not part of any session-status or
|
||||
alarm-feed payload, so a production deployment can truncate indefinitely
|
||||
without anyone noticing. Surfacing truncation as a **structural** degraded
|
||||
status (a field on the alarm-provider mode/status surface the dashboard and
|
||||
`StreamAlarms` consumers already read) is filed as a follow-up; until it
|
||||
lands, the log line is the only signal. A galaxy that truncates persistently
|
||||
is a configuration problem: raise `MxGateway:Alarms:MaxAlarmsPerFetch`.
|
||||
|
||||
## Session-Resilience Epic Scope
|
||||
|
||||
@@ -418,8 +418,8 @@ behavior.
|
||||
| `MxGateway:Alarms:SubscriptionExpression` | _(empty)_ | AVEVA alarm-subscription expression the monitor subscribes on startup, in canonical `\\<machine>\Galaxy!<area>` form. The literal `Galaxy` provider is correct regardless of the Galaxy database name. When empty and `Enabled` is `true`, the gateway falls back to `\\<MachineName>\Galaxy!<DefaultArea>` if `DefaultArea` is set. |
|
||||
| `MxGateway:Alarms:DefaultArea` | _(empty)_ | Area name used to compose a default subscription when `SubscriptionExpression` is empty. If both are empty while `Enabled` is `true`, the monitor faults with a configuration diagnostic. |
|
||||
| `MxGateway:Alarms:ReconcileIntervalSeconds` | `30` | How often the monitor reconciles its in-process alarm cache against the worker's authoritative active-alarm snapshot, catching transitions the live poll-and-diff feed missed. Floored at 5 seconds. |
|
||||
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be `>= 100`: every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or unusable value leaves the worker on the 500 ms default rather than failing the session. |
|
||||
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be `>= 64`. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning (identifiers and counts only). Raise this on galaxies whose steady-state active-alarm count approaches the cap — a galaxy permanently above it holds stale entries in the snapshot until a sub-cap poll. Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or unusable value leaves the worker on the 1024 default. |
|
||||
| `MxGateway:Alarms:PollIntervalMilliseconds` | `500` | Cadence at which the worker's STA polls the AVEVA alarm consumer (`GetXmlCurrentAlarms2`) for the active-alarm snapshot the live feed diffs. Must be between `100` and `3600000` (one hour): every poll is a COM call plus an XML parse on the same STA that serves reads and writes, so a tighter cadence starves the command path, while a value above an hour stops being a cadence and silently disables alarm polling. The gateway conveys the value to the worker via the `MXGATEWAY_ALARM_POLL_INTERVAL_MS` environment variable; a missing or out-of-range value leaves the worker on the 500 ms default rather than failing the session. |
|
||||
| `MxGateway:Alarms:MaxAlarmsPerFetch` | `1024` | Cap the worker passes to `GetXmlCurrentAlarms2`'s `maxAlmCnt`. Must be between `64` and `65536` — the worker is a 32-bit process that materializes each reply as one BSTR plus a full `XmlDocument`, so an unbounded cap faults the STA with an out-of-memory rather than merely slowing it. It doubles as the **truncation threshold**: a fetch returning exactly this many records is treated as truncated, because the COM API caps its reply with no "more available" flag. On a truncated poll the worker retains the alarms the capped reply could not mention instead of letting their absence read as a clear, and logs a rate-limited `AlarmSnapshotTruncated` warning to its stderr (identifiers and counts only). **Remediation when you see that warning: raise this value** so the steady-state active-alarm count fits inside one fetch. A galaxy permanently above the cap holds stale entries in the snapshot until a sub-cap poll, and loses clear→re-raise cycles that happen entirely out of window (see `docs/DesignDecisions.md`). Conveyed to the worker via the `MXGATEWAY_ALARM_MAX_ALARMS_PER_FETCH` environment variable; a missing or out-of-range value leaves the worker on the 1024 default. |
|
||||
|
||||
The alarm monitor is independent of client sessions: `AcknowledgeAlarm` and
|
||||
`StreamAlarms` are session-less RPCs served by the monitor.
|
||||
|
||||
Reference in New Issue
Block a user