dohertj2/mxaccessgw
1
0
Fork 0
Code Issues Pull Requests 2 Actions 7 Packages Projects Releases Wiki Activity

fix(WRK-01): STA pump step refreshes activity to prevent false StaHung

Browse Source 
A long legitimate ReadBulk pumped Windows messages without refreshing
LastStaActivityUtc, so the watchdog false-positived StaHung past
HeartbeatStuckCeiling and then silently dropped every reply. PumpPendingMessages()
now calls MarkActivity() after pumping; a genuinely stuck STA (no pumping)
still accrues staleness and faults correctly. No MXAccess parity change.

archreview: WRK-01 (P0). Verified on the Windows host (x86): worker builds
clean, StaRuntimeTests + WorkerPipeSessionTests 33/33 pass.
This commit is contained in:
Joseph Doherty
2026-07-09 05:51:45 -04:00
parent 5faef6c012
commit 31eec41456
4 changed files with 161 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/MxAccessWorkerInstanceDesign.md
+21 -9
View File
@@ -656,11 +656,15 @@ the event queue implementation owns those counters.
The STA watchdog currently emits a `WorkerFault` with
`WorkerFaultCategory.StaHung` when `LastStaActivityUtc` is older than
`WorkerPipeSessionOptions.HeartbeatGrace` **and no command is in flight**.
`StaRuntime.ProcessQueuedCommands` calls `MarkActivity()` only immediately
before and after each work item, so a synchronously long-running STA command
(for example a `ReadBulk` waiting `timeout_ms` for the first `OnDataChange`)
legitimately freezes `LastStaActivityUtc` for the duration of the wait while
the worker is healthy. The watchdog is therefore suppressed while the
`StaRuntime.ProcessQueuedCommands` calls `MarkActivity()` immediately before
and after each work item, so a synchronously long-running STA command that
neither completes work items nor pumps would freeze `LastStaActivityUtc` for
the duration of the wait while the worker is healthy. Commands that hold the
STA to wait for COM events (for example a `ReadBulk` waiting `timeout_ms` for
the first `OnDataChange`) avoid this: they pump via
`StaRuntime.PumpPendingMessages()`, which now refreshes `LastStaActivityUtc`
on every iteration (see the `HeartbeatStuckCeiling` discussion below). The
watchdog is additionally suppressed while the
heartbeat snapshot's `CurrentCommandCorrelationId` is non-empty: the worker is
busy executing a command, not hung, and the heartbeat already surfaces the
in-flight correlation id so the gateway can apply its own per-command timeout
@@ -684,10 +688,18 @@ session and only the gateway's per-command timeout would catch the hang —
losing the worker-originated diagnostic (`StaHung` fault category, the
stale-by interval) from the gateway audit trail. Once `LastStaActivityUtc`
has been stale for longer than `HeartbeatStuckCeiling`, the watchdog fires
`StaHung` regardless of whether a command is in flight, on the assumption
that no legitimate STA command should run that long without periodically
refreshing activity. Deployments that legitimately run very long bulk
operations should raise the ceiling rather than disable it.
`StaHung` regardless of whether a command is in flight. This is now safe for
healthy long-running commands: `StaRuntime.PumpPendingMessages()` refreshes
`LastStaActivityUtc` (via `MarkActivity()`) every time it runs, and long-hold
STA commands invoke it on every wait iteration (`ReadBulk` routes its
per-tag wait through the `pumpStep` wired from `StaRuntime.PumpPendingMessages`).
A command that keeps pumping therefore keeps its activity timestamp fresh and
never reaches the ceiling, while a genuinely stuck STA — one that has stopped
pumping — accrues staleness and faults correctly. The ceiling is thus the
backstop for a command that both holds the thread and stops pumping, not a
guillotine for slow-but-healthy work. Deployments that legitimately run very
long bulk operations should still be able to raise the ceiling rather than
disable it.
## Shutdown