fix(WRK-22,WRK-24,WRK-25,WRK-27,IPC-26): worker write-seam hardening
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext skips cancelled, claims the frame it returns) so a cancelled write never reaches the wire unless already claimed mid-write (documented residual). WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained event batch through it, so a burst of N events costs one flush not N. IPC-30 oversized-event structured fault preserved via FindOversizedEvent. WRK-24: reject a below-1024 negotiated frame maximum at the handshake (MinNegotiableFrameBytes, matching GatewayOptionsValidator floor). WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot so the watchdog suppresses to the ceiling, not the grace. Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the 2026-07-12 remediation registers/change-log updated in the same commit.
This commit is contained in:
@@ -747,6 +747,23 @@ heartbeat fields until dedicated thresholds own those warnings. The worker
|
||||
reports stale STA activity, but the gateway owns the final kill decision
|
||||
through its existing heartbeat and worker lifecycle policy.
|
||||
|
||||
The alarm poll runs outside the command dispatcher — `RunAlarmPollLoopAsync`
|
||||
invokes `PollOnce` directly on the STA rather than through
|
||||
`StaCommandDispatcher`, so it does not inflate `PendingCommandCount` or perturb
|
||||
dispatch ordering for real gateway commands. Because it is not a dispatched
|
||||
command it has no `CurrentCommandCorrelationId`, so a healthy-but-slow poll (a
|
||||
large `GetXmlCurrentAlarms2` against a busy provider) blocking the STA past
|
||||
`HeartbeatGrace` would otherwise fault a healthy session at 15 s while a
|
||||
dispatched command gets the 75 s ceiling. To close that asymmetry (WRK-27) the
|
||||
poll advertises itself on the heartbeat snapshot's `StaCallInProgress` flag —
|
||||
set on the STA thread for exactly the span of the COM call — and the watchdog
|
||||
suppression honors that flag alongside `CurrentCommandCorrelationId`. The poll
|
||||
therefore receives the same grace-to-ceiling treatment as a dispatched command:
|
||||
suppressed up to `HeartbeatStuckCeiling`, faulted past it (a poll that blocks
|
||||
the STA more than 75 s without pumping *should* fault — that is the ceiling's
|
||||
contract). The flag is named generically so any future non-dispatcher STA work
|
||||
reuses it.
|
||||
|
||||
The in-flight-command suppression itself is bounded by
|
||||
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
|
||||
`HeartbeatGrace`). The motivating case for the suppression is a legitimately
|
||||
|
||||
+33
-13
@@ -29,6 +29,17 @@ default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
|
||||
means "use the worker's built-in default". This keeps both ends framing to the
|
||||
same limit rather than depending on matched compile-time constants.
|
||||
|
||||
The worker accepts a negotiated value in the closed range [1024, 256 MiB]
|
||||
(`MinNegotiableFrameBytes` .. `MaxNegotiableFrameBytes`); 0 keeps the default.
|
||||
A value outside that range is rejected at the handshake with a fault frame
|
||||
rather than adopted, because a nonsensical maximum — a gateway bug or a
|
||||
foreign/old peer — would otherwise leave a session that handshakes cleanly and
|
||||
then fails every subsequent frame with per-frame size errors, the worst
|
||||
diagnostic shape for an operator. The 1024-byte floor matches the gateway's own
|
||||
`GatewayOptionsValidator.MinimumMaxMessageBytes`, so the worker never rejects a
|
||||
value the gateway's validator accepts as legal configuration, and 1024 still
|
||||
guarantees hellos, heartbeats, acks, and faults fit.
|
||||
|
||||
Every worker-to-gateway frame must serialize within this limit, control replies
|
||||
included, so reply builders truncate to fit rather than emit a frame the writer
|
||||
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
|
||||
@@ -112,20 +123,29 @@ runs after the whole batch, and only then does every successfully-written
|
||||
frame's completion resolve — so a caller's `WriteAsync` still does not
|
||||
complete until its bytes are both written *and* flushed, but a batch that
|
||||
happened to contain several queued frames pays one flush instead of one per
|
||||
frame. In practice this coalescing currently engages only when multiple
|
||||
frames are queued at the moment a lock-holder starts draining. The event
|
||||
drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`) awaits each drained
|
||||
event's `WriteAsync` individually before writing the next, so today at most
|
||||
one event frame is queued per drain pass and each event still costs its own
|
||||
flush; a dedicated batch write entry point that submits a whole drained
|
||||
event batch under one lock acquisition is designed but not yet landed, so a
|
||||
burst of N events currently costs N flushes on the event hot path, not one.
|
||||
frame. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
|
||||
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
|
||||
every frame under one `_gate` acquisition, takes the write lock once, and
|
||||
drains them together, so a burst of N events costs one flush rather than N —
|
||||
the coalescing the batch machinery was built for now engages on the event hot
|
||||
path, not only when independent producers happen to queue behind a blocked
|
||||
write. Intra-batch order is preserved (FIFO enqueue under one lock), and a
|
||||
concurrently queued control frame is still drained ahead of the batch. A
|
||||
per-frame rejection inside a batch (for example one oversized event) surfaces
|
||||
from the batch's awaited completions as that frame's
|
||||
`WorkerFrameProtocolException`; the remaining completions are still observed
|
||||
so none faults unobserved.
|
||||
|
||||
Cancellation semantics for a `WriteAsync` call that is still waiting for the
|
||||
write lock when its token fires are not yet defined at this layer — pending
|
||||
a fix that will tombstone the queued frame so a cancelled call is guaranteed
|
||||
never to reach the wire. Until that lands, a cancelled caller may still see
|
||||
its frame written by whichever caller next holds the lock.
|
||||
Cancellation of a `WriteAsync`/`WriteBatchAsync` call that is still waiting
|
||||
for the write lock when its token fires tombstones the queued frame: the
|
||||
cancelled caller marks its frame under `_gate`, and the draining lock-holder's
|
||||
`DequeueNext` skips any tombstoned frame, so a cancelled call is guaranteed
|
||||
never to reach the wire — *unless* a lock-holder has already claimed the frame
|
||||
to write it. Claiming and cancelling are interlocked under `_gate`, so exactly
|
||||
one wins; a frame already claimed is mid-write and can no longer be recalled,
|
||||
so the caller observes `OperationCanceledException` while that one frame still
|
||||
reaches the wire. That residual window is by design: blocking the canceller
|
||||
behind the very write it is abandoning would defeat the point of cancellation.
|
||||
|
||||
## Verification
|
||||
|
||||
|
||||
Reference in New Issue
Block a user