fix(WRK-22,WRK-24,WRK-25,WRK-27,IPC-26): worker write-seam hardening
ci / java (push) Successful in 2m7s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m13s
ci / portable (push) Failing after 4m6s

WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).

WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.

WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).

WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.

Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
This commit is contained in:
Joseph Doherty
2026-08-07 07:50:38 -04:00
parent 10534ec906
commit 8df35cd63a
14 changed files with 912 additions and 58 deletions
@@ -371,27 +371,57 @@ public sealed class WorkerPipeSession
continue;
}
foreach (WorkerEvent workerEvent in events)
// Submit the whole drained batch through the writer's batch entry point under one lock
// acquisition so the burst pays a single flush instead of one per event (WRK-25). Events
// are the low-priority frame class: the writer holds them behind any pending control frame
// (reply, fault, heartbeat, shutdown ack) so those are not delayed behind an event backlog,
// and intra-batch order is preserved.
WorkerEnvelope[] envelopes = new WorkerEnvelope[events.Count];
for (int index = 0; index < events.Count; index++)
{
// Events are the low-priority frame class: the writer holds them behind any pending
// control frame (reply, fault, heartbeat, shutdown ack) so those are not delayed
// behind an event backlog.
try
{
await _writer
.WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
.ConfigureAwait(false);
}
catch (WorkerFrameProtocolException exception)
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
{
await FaultOnOversizedEventAsync(workerEvent, exception, cancellationToken)
.ConfigureAwait(false);
}
envelopes[index] = CreateEnvelope(events[index]);
}
try
{
await _writer
.WriteBatchAsync(envelopes, WorkerFrameWritePriority.Event, cancellationToken)
.ConfigureAwait(false);
}
catch (WorkerFrameProtocolException exception)
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
{
// A single oversized event surfaces from the batch's awaited completions; the death is
// still IPC-30's structured, event-naming fault. Map the rejection back to the first
// event in batch order whose envelope overshoots the negotiated maximum — the same
// frame the writer rejected first.
await FaultOnOversizedEventAsync(
FindOversizedEvent(events, envelopes),
exception,
cancellationToken)
.ConfigureAwait(false);
}
}
}
private WorkerEvent FindOversizedEvent(
IReadOnlyList<WorkerEvent> events,
WorkerEnvelope[] envelopes)
{
for (int index = 0; index < envelopes.Length; index++)
{
if (envelopes[index].CalculateSize() > _options.MaxMessageBytes)
{
return events[index];
}
}
// Unreachable in practice: WriteBatchAsync surfaced MessageTooLarge, so at least one envelope
// exceeded the negotiated maximum. Fall back to the first event so the fault still names a
// concrete event rather than throwing a second, less useful exception from the fault path.
return events[0];
}
/// <summary>
/// Ends the session on an event that cannot be framed, but deliberately and diagnosably
/// (IPC-30). An event above the negotiated frame maximum is undeliverable end to end — the
@@ -1085,16 +1115,16 @@ public sealed class WorkerPipeSession
return;
}
if (!string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId)
if ((!string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId) || snapshot.StaCallInProgress)
&& staleFor <= _sessionOptions.HeartbeatStuckCeiling)
{
// A command is in flight and we are still within the defensive
// suppression ceiling — the STA is busy executing it, not
// hung. The next MarkActivity() in StaRuntime.ProcessQueuedCommands
// will refresh LastActivityUtc once the command returns, at which
// point this branch stops being taken. The heartbeat already
// surfaces the in-flight correlation id so the gateway can apply
// its own per-command timeout if it considers the command too slow.
// A command is in flight, or an STA call outside the dispatcher (the alarm poll, WRK-27) is
// executing, and we are still within the defensive suppression ceiling — the STA is busy
// doing that work, not hung. The next MarkActivity() in StaRuntime.ProcessQueuedCommands
// will refresh LastActivityUtc once the work returns, at which point this branch stops
// being taken. The heartbeat already surfaces the in-flight correlation id so the gateway
// can apply its own per-command timeout if it considers the command too slow; a poll that
// blocks the STA past the ceiling still faults, which is the ceiling's contract.
return;
}