fix(worker): review follow-up — ghost cache entries, ReadBulk dedupe/pumping, fault hygiene
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m15s
ci / portable (push) Failing after 4m19s
ci / windows-x86 (push) Successful in 6m16s

- AddItem/AddItem2/AddBufferedItem drop stale cache entries for a reused handle id
- ReadBulk: dedupe handles within a batch (advise/teardown once, fan out result);
  pump every 16 items in add/advise and teardown phases
- alarm poll stops without recording a fault once StaRuntime.LoopFault is set
- StaRuntime skips the loop-fault handler during a requested shutdown
- StaHung loop fault diagnostic carries type + HRESULT only
- gateway.md documents ReadBulk's shared deadline
This commit is contained in:
Joseph Doherty
2026-10-22 08:23:05 -04:00
parent 4ea3672b1e
commit 38fbbde191
9 changed files with 375 additions and 21 deletions
+24 -3
View File
@@ -283,8 +283,11 @@ If the STA loop itself throws after `Start` has returned (a failed
`StaRuntime` records the exception as `LoopFault`, fails every later
`InvokeAsync` with an `InvalidOperationException` wrapping it (never the
graceful `StaRuntimeShutdownException`), cancels queued work, and invokes the
handler registered through `RegisterLoopFaultHandler`. `MxAccessStaSession`
registers a handler that records a `StaHung` `WorkerFault` on the event queue,
handler registered through `RegisterLoopFaultHandler` — unless a requested
shutdown is already in progress, in which case the death is part of a
deliberate teardown and is not reported. `MxAccessStaSession` registers a
handler that records a `StaHung` `WorkerFault` on the event queue (diagnostic
text carries the exception type and HRESULT only, never the message),
so the IPC drain loop sends the fault to the gateway and ends the pipe session
and the worker process. Before this, a post-start loop death was stored in the
startup-exception field that only `Start` reads, leaving a live worker whose
@@ -354,6 +357,13 @@ handle) failing only that tag, and the `ReadBulk timed out after <n> ms waiting
for first OnDataChange.` message for each tag that never delivered. The cache
probes all run before any snapshot `AddItem`, so a later tag in the same
request can never be served from an earlier tag's snapshot subscription.
When MXAccess returns the same item handle for two tags in one batch (a
duplicate address), the handle is advised, unadvised, and removed once, and
every duplicate receives the shared outcome. Phases 2 and 4 make one or two
COM calls per tag without waiting, so they call the pump step every
`ReadBulkPumpInterval` (16) tags; a large batch therefore keeps dispatching
inbound callbacks and refreshing `LastStaActivityUtc` instead of drifting
toward `HeartbeatStuckCeiling`.
The write wait's budget is `MxGateway:Worker:WriteCompletionWaitMilliseconds`
(default 1500). It applies to `Write`, `Write2`, `WriteSecured`, and
@@ -712,7 +722,14 @@ caches the event sink fills: `RemoveItem` drops the handle pair's
`Unregister` drops every entry recorded under the server handle (MXAccess
removes the server's items with it, so the per-item eviction never runs for
them). Without this, an `AddItem` → `Write` → `RemoveItem` churn grew the
completion cache by one entry per cycle for the life of the worker. Failed advice and cleanup calls
completion cache by one entry per cycle for the life of the worker.
`AddItem`, `AddItem2`, and `AddBufferedItem` also drop both cache entries for
the handle they receive when the registry does not already track it. An
`OnDataChange` or `OnWriteComplete` that was queued when `RemoveItem` or
`Unregister` ran is dispatched after the eviction and recreates the entry;
when MXAccess later reuses that item id, `ReadBulk`'s cached path would
otherwise serve the stale value as `was_cached = true`. A handle the
registry still tracks belongs to a live item, so its entries are kept. Failed advice and cleanup calls
leave registry state unchanged so diagnostics continue to reflect the last
successful MXAccess-owned state transition.
@@ -958,6 +975,10 @@ can echo alarm or tag names. A single transient COM failure in
`GetXmlCurrentAlarms2` used to fault the whole session, taking every data
subscription with it; a persistent failure, including an STA-affinity
regression that fails every tick, still faults within three poll intervals.
A poll failure seen after the STA loop itself has died (`StaRuntime.LoopFault`
is set) ends the poll loop without recording anything: the loop-fault handler
has already recorded the `StaHung` fault, and the failure is only the
fast-fail `InvokeAsync` returns afterwards.
The in-flight-command suppression itself is bounded by
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
+1 -1
View File
@@ -111,7 +111,7 @@ return workItem.Task;
### Loop failure after start
An exception thrown by the loop after `Start()` returned — `WaitForWorkOrMessages` reporting `WAIT_FAILED`, or an exception escaping a dispatched message — is not a startup failure, and nothing reads `startupException` after `Start()`. `ThreadMain` instead stores it as `LoopFault` under `gate`, so every later `InvokeAsync` fails immediately with an `InvalidOperationException` whose inner exception is the loop fault, and work already queued is cancelled by the `finally` block. It then invokes the handler registered with `RegisterLoopFaultHandler`; a handler registered after the loop died runs immediately on the registering thread. `MxAccessStaSession` registers a handler that records a `StaHung` `WorkerFault` on the event queue, which the IPC drain loop sends to the gateway before ending the pipe session and the worker process.
An exception thrown by the loop after `Start()` returned — `WaitForWorkOrMessages` reporting `WAIT_FAILED`, or an exception escaping a dispatched message — is not a startup failure, and nothing reads `startupException` after `Start()`. `ThreadMain` instead stores it as `LoopFault` under `gate`, so every later `InvokeAsync` fails immediately with an `InvalidOperationException` whose inner exception is the loop fault, and work already queued is cancelled by the `finally` block. It then invokes the handler registered with `RegisterLoopFaultHandler`; a handler registered after the loop died runs immediately on the registering thread. Neither happens when `Shutdown` was already requested, because a loop that throws during a deliberate teardown is not a worker fault. `MxAccessStaSession` registers a handler that records a `StaHung` `WorkerFault` on the event queue, which the IPC drain loop sends to the gateway before ending the pipe session and the worker process.
## Command Dispatch