fix(WRK-22,WRK-24,WRK-25,WRK-27,IPC-26): worker write-seam hardening
ci / java (push) Successful in 2m7s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m13s
ci / portable (push) Failing after 4m6s

WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).

WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.

WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).

WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.

Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
This commit is contained in:
Joseph Doherty
2026-08-07 07:50:38 -04:00
parent 10534ec906
commit 8df35cd63a
14 changed files with 912 additions and 58 deletions
File diff suppressed because one or more lines are too long
@@ -17,12 +17,12 @@ members, no positional records). The worker builds and tests only on the Windows
| ID | Sev | Tier | Eff | Dep | Status | Title | | ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------| |----|-----|------|-----|-----|--------|-------|
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events | | WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
| WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later | | WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Done | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps | | WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps |
| WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check | | WRK-24 | Low | — | S | — | Done | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
| WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Not started | WRK-12 flush coalescing never engages on the event hot path | | WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Done | WRK-12 flush coalescing never engages on the event hot path |
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change | | WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) | | WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract | | WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
--- ---
@@ -15,7 +15,7 @@ All `path:line` citations were re-verified against the working tree at `4f5371f`
| IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) | | IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes | | IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes |
| IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them | | IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them |
| IPC-26 | Low | P2 | S¹ | WRK-22 | Not started | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) | | IPC-26 | Low | P2 | S¹ | WRK-22 | Done (mechanics landed in WRK-22) | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
| IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract | | IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract |
| IPC-28 | Low | — | S | — | Done | `docs/Grpc.md` omits the `CommandTooLarge``ResourceExhausted` mapping | | IPC-28 | Low | — | S | — | Done | `docs/Grpc.md` omits the `CommandTooLarge``ResourceExhausted` mapping |
| IPC-29 | Low | — | S | — | Done (discharged by WRK-26) | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc | | IPC-29 | Low | — | S | — | Done (discharged by WRK-26) | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
+17
View File
@@ -747,6 +747,23 @@ heartbeat fields until dedicated thresholds own those warnings. The worker
reports stale STA activity, but the gateway owns the final kill decision reports stale STA activity, but the gateway owns the final kill decision
through its existing heartbeat and worker lifecycle policy. through its existing heartbeat and worker lifecycle policy.
The alarm poll runs outside the command dispatcher — `RunAlarmPollLoopAsync`
invokes `PollOnce` directly on the STA rather than through
`StaCommandDispatcher`, so it does not inflate `PendingCommandCount` or perturb
dispatch ordering for real gateway commands. Because it is not a dispatched
command it has no `CurrentCommandCorrelationId`, so a healthy-but-slow poll (a
large `GetXmlCurrentAlarms2` against a busy provider) blocking the STA past
`HeartbeatGrace` would otherwise fault a healthy session at 15 s while a
dispatched command gets the 75 s ceiling. To close that asymmetry (WRK-27) the
poll advertises itself on the heartbeat snapshot's `StaCallInProgress` flag —
set on the STA thread for exactly the span of the COM call — and the watchdog
suppression honors that flag alongside `CurrentCommandCorrelationId`. The poll
therefore receives the same grace-to-ceiling treatment as a dispatched command:
suppressed up to `HeartbeatStuckCeiling`, faulted past it (a poll that blocks
the STA more than 75 s without pumping *should* fault — that is the ceiling's
contract). The flag is named generically so any future non-dispatcher STA work
reuses it.
The in-flight-command suppression itself is bounded by The in-flight-command suppression itself is bounded by
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 × `WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
`HeartbeatGrace`). The motivating case for the suppression is a legitimately `HeartbeatGrace`). The motivating case for the suppression is a legitimately
+33 -13
View File
@@ -29,6 +29,17 @@ default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
means "use the worker's built-in default". This keeps both ends framing to the means "use the worker's built-in default". This keeps both ends framing to the
same limit rather than depending on matched compile-time constants. same limit rather than depending on matched compile-time constants.
The worker accepts a negotiated value in the closed range [1024, 256 MiB]
(`MinNegotiableFrameBytes` .. `MaxNegotiableFrameBytes`); 0 keeps the default.
A value outside that range is rejected at the handshake with a fault frame
rather than adopted, because a nonsensical maximum — a gateway bug or a
foreign/old peer — would otherwise leave a session that handshakes cleanly and
then fails every subsequent frame with per-frame size errors, the worst
diagnostic shape for an operator. The 1024-byte floor matches the gateway's own
`GatewayOptionsValidator.MinimumMaxMessageBytes`, so the worker never rejects a
value the gateway's validator accepts as legal configuration, and 1024 still
guarantees hellos, heartbeats, acks, and faults fit.
Every worker-to-gateway frame must serialize within this limit, control replies Every worker-to-gateway frame must serialize within this limit, control replies
included, so reply builders truncate to fit rather than emit a frame the writer included, so reply builders truncate to fit rather than emit a frame the writer
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
@@ -112,20 +123,29 @@ runs after the whole batch, and only then does every successfully-written
frame's completion resolve — so a caller's `WriteAsync` still does not frame's completion resolve — so a caller's `WriteAsync` still does not
complete until its bytes are both written *and* flushed, but a batch that complete until its bytes are both written *and* flushed, but a batch that
happened to contain several queued frames pays one flush instead of one per happened to contain several queued frames pays one flush instead of one per
frame. In practice this coalescing currently engages only when multiple frame. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
frames are queued at the moment a lock-holder starts draining. The event submits a whole drained event batch through `WriteBatchAsync`, which enqueues
drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`) awaits each drained every frame under one `_gate` acquisition, takes the write lock once, and
event's `WriteAsync` individually before writing the next, so today at most drains them together, so a burst of N events costs one flush rather than N —
one event frame is queued per drain pass and each event still costs its own the coalescing the batch machinery was built for now engages on the event hot
flush; a dedicated batch write entry point that submits a whole drained path, not only when independent producers happen to queue behind a blocked
event batch under one lock acquisition is designed but not yet landed, so a write. Intra-batch order is preserved (FIFO enqueue under one lock), and a
burst of N events currently costs N flushes on the event hot path, not one. concurrently queued control frame is still drained ahead of the batch. A
per-frame rejection inside a batch (for example one oversized event) surfaces
from the batch's awaited completions as that frame's
`WorkerFrameProtocolException`; the remaining completions are still observed
so none faults unobserved.
Cancellation semantics for a `WriteAsync` call that is still waiting for the Cancellation of a `WriteAsync`/`WriteBatchAsync` call that is still waiting
write lock when its token fires are not yet defined at this layer — pending for the write lock when its token fires tombstones the queued frame: the
a fix that will tombstone the queued frame so a cancelled call is guaranteed cancelled caller marks its frame under `_gate`, and the draining lock-holder's
never to reach the wire. Until that lands, a cancelled caller may still see `DequeueNext` skips any tombstoned frame, so a cancelled call is guaranteed
its frame written by whichever caller next holds the lock. never to reach the wire — *unless* a lock-holder has already claimed the frame
to write it. Claiming and cancelling are interlocked under `_gate`, so exactly
one wins; a frame already claimed is mid-write and can no longer be recalled,
so the caller observes `OperationCanceledException` while that one frame still
reaches the wire. That residual window is by design: blocking the canceller
behind the very write it is abandoning would defeat the point of cancellation.
## Verification ## Verification
@@ -481,6 +481,192 @@ public sealed class WorkerFrameProtocolTests
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode); Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
} }
/// <summary>
/// Verifies a negotiated frame maximum below the worker floor is rejected as
/// <c>InvalidConfiguration</c> (WRK-24), and that exactly the floor is adopted. The floor matches
/// the gateway's own <c>GatewayOptionsValidator.MinimumMaxMessageBytes</c>, so the worker never
/// rejects a value the gateway's validator accepts, yet a nonsensical tiny value faults at the
/// handshake instead of leaving a session that fails every later frame.
/// </summary>
[Fact]
public void AdoptNegotiatedMaxMessageBytes_BelowFloor_ThrowsInvalidConfiguration()
{
WorkerFrameProtocolOptions belowFloor = CreateOptions();
WorkerFrameProtocolException exception = Assert.Throws<WorkerFrameProtocolException>(
() => belowFloor.AdoptNegotiatedMaxMessageBytes(512));
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
// Boundary: exactly the floor is accepted.
WorkerFrameProtocolOptions atFloor = CreateOptions();
atFloor.AdoptNegotiatedMaxMessageBytes((uint)WorkerFrameProtocolOptions.MinNegotiableFrameBytes);
Assert.Equal(WorkerFrameProtocolOptions.MinNegotiableFrameBytes, atFloor.MaxMessageBytes);
}
/// <summary>
/// WRK-22 / IPC-26. A <c>WriteAsync</c> cancelled while it waits for the write lock must never
/// have its frame written by the next lock-holder. Writer A holds the lock mid-write (blocked in
/// the stream); an event write is queued and then cancelled; when A is released and a later
/// control frame drains, the wire carries A's frame and the control frame only — the cancelled
/// event envelope is tombstoned and skipped.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteAsync_CancelledWhileWaitingForLock_FrameIsNeverWritten()
{
WorkerFrameProtocolOptions options = CreateOptions();
using GatedWriteStream stream = new();
WorkerFrameWriter writer = new(stream, options);
// Writer A occupies the writer and blocks inside the stream, holding the write lock.
Task firstWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
// Queue an event write with its own CTS while the lock is held, then cancel it.
using CancellationTokenSource cts = new();
Task cancelledWrite = writer.WriteAsync(CreateEventEnvelope(), WorkerFrameWritePriority.Event, cts.Token);
await Task.Delay(50);
cts.Cancel();
await Assert.ThrowsAnyAsync<OperationCanceledException>(async () => await cancelledWrite);
// Release A, then drive a fresh control write.
stream.ReleaseFirstWrite();
await AwaitWithTimeoutAsync(firstWrite);
await writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
stream.Position = 0;
WorkerFrameReader reader = new(stream, options);
WorkerEnvelope frame1 = await reader.ReadAsync();
WorkerEnvelope frame2 = await reader.ReadAsync();
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, frame1.BodyCase);
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, frame2.BodyCase);
// The cancelled event never reached the wire — no third frame, and sequences stay contiguous.
Assert.Equal(stream.Length, stream.Position);
Assert.Equal(1UL, frame1.Sequence);
Assert.Equal(2UL, frame2.Sequence);
}
/// <summary>
/// WRK-22 / IPC-26, the review's shutdown scenario. A cancelled event frame queued before a
/// shutdown-ack control frame must not trail the ack on the wire: the tombstone rule plus the
/// control-before-event scheduler keeps the ack the last frame written.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteAsync_CancelledEventFrame_DoesNotTrailShutdownAck()
{
WorkerFrameProtocolOptions options = CreateOptions();
using GatedWriteStream stream = new();
WorkerFrameWriter writer = new(stream, options);
Task firstWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
using CancellationTokenSource cts = new();
Task cancelledEvent = writer.WriteAsync(CreateEventEnvelope(), WorkerFrameWritePriority.Event, cts.Token);
await Task.Delay(50);
cts.Cancel();
await Assert.ThrowsAnyAsync<OperationCanceledException>(async () => await cancelledEvent);
// The shutdown ack (a control frame) is queued behind the still-blocked first write.
Task ackWrite = writer.WriteAsync(CreateShutdownAckEnvelope(), WorkerFrameWritePriority.Control);
await Task.Delay(50);
stream.ReleaseFirstWrite();
await AwaitWithTimeoutAsync(Task.WhenAll(firstWrite, ackWrite));
stream.Position = 0;
WorkerFrameReader reader = new(stream, options);
WorkerEnvelope frame1 = await reader.ReadAsync();
WorkerEnvelope frame2 = await reader.ReadAsync();
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, frame1.BodyCase);
// The ack is the last frame — the cancelled event did not trail it.
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerShutdownAck, frame2.BodyCase);
Assert.Equal(stream.Length, stream.Position);
}
/// <summary>
/// WRK-25. The batch entry point enqueues a whole event burst under one lock acquisition and
/// drains it together, so N events cost exactly one flush and reach the wire in batch order with
/// monotonic sequences — the coalescing WRK-12 shipped, now on the event hot path.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteBatchAsync_FlushesOnceAndPreservesOrder()
{
const int count = 8;
WorkerFrameProtocolOptions options = CreateOptions();
using FlushCountingStream stream = new();
WorkerFrameWriter writer = new(stream, options);
WorkerEnvelope[] batch = new WorkerEnvelope[count];
for (int index = 0; index < count; index++)
{
batch[index] = CreateEventEnvelope(workerSequence: (ulong)(100 + index));
}
await writer.WriteBatchAsync(batch, WorkerFrameWritePriority.Event);
// The whole batch was queued before the single lock wait, so it drained in one pass => one flush.
Assert.Equal(1, stream.FlushCount);
stream.Position = 0;
WorkerFrameReader reader = new(stream, options);
for (int index = 0; index < count; index++)
{
WorkerEnvelope frame = await reader.ReadAsync();
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, frame.BodyCase);
// Wire order matches batch order.
Assert.Equal((ulong)(100 + index), frame.WorkerEvent.Event.WorkerSequence);
// Write-time stamped sequence is monotonic 1..count.
Assert.Equal((ulong)(index + 1), frame.Sequence);
}
}
/// <summary>
/// WRK-25. Control-before-event still holds mid-batch: a control frame queued while a batch is
/// draining jumps ahead of the batch's remaining events.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task WriteBatchAsync_ControlFrameQueuedDuringBatch_JumpsRemainingEvents()
{
WorkerFrameProtocolOptions options = CreateOptions();
using GatedWriteStream stream = new();
WorkerFrameWriter writer = new(stream, options);
WorkerEnvelope[] batch = new[]
{
CreateEventEnvelope(),
CreateEventEnvelope(),
CreateEventEnvelope(),
};
// The batch takes the lock and blocks writing its first event frame inside the stream.
Task batchWrite = writer.WriteBatchAsync(batch, WorkerFrameWritePriority.Event);
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
// A control frame queued mid-drain must jump the batch's remaining events.
Task controlWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
await Task.Delay(50);
stream.ReleaseFirstWrite();
await AwaitWithTimeoutAsync(Task.WhenAll(batchWrite, controlWrite));
stream.Position = 0;
WorkerFrameReader reader = new(stream, options);
WorkerEnvelope f1 = await reader.ReadAsync();
WorkerEnvelope f2 = await reader.ReadAsync();
WorkerEnvelope f3 = await reader.ReadAsync();
WorkerEnvelope f4 = await reader.ReadAsync();
// First event was already writing when the control frame queued; the control frame then jumps
// ahead of the two remaining events.
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, f1.BodyCase);
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, f2.BodyCase);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, f3.BodyCase);
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, f4.BodyCase);
}
private static WorkerEnvelope CreateGatewayHelloEnvelope(ulong sequence = 1) private static WorkerEnvelope CreateGatewayHelloEnvelope(ulong sequence = 1)
{ {
return new WorkerEnvelope return new WorkerEnvelope
@@ -522,6 +708,47 @@ public sealed class WorkerFrameProtocolTests
}; };
} }
private static WorkerEnvelope CreateEventEnvelope(ulong workerSequence)
{
WorkerEnvelope envelope = CreateEventEnvelope();
envelope.WorkerEvent.Event.WorkerSequence = workerSequence;
return envelope;
}
private static WorkerEnvelope CreateShutdownAckEnvelope()
{
return new WorkerEnvelope
{
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
SessionId = SessionId,
WorkerShutdownAck = new WorkerShutdownAck
{
Status = new ProtocolStatus
{
Code = ProtocolStatusCode.Ok,
Message = "OK",
},
},
};
}
// A MemoryStream that counts FlushAsync calls without gating any write, so a batch write can be
// asserted to flush exactly once.
private sealed class FlushCountingStream : MemoryStream
{
private int _flushCount;
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
public int FlushCount => Volatile.Read(ref _flushCount);
/// <inheritdoc />
public override Task FlushAsync(CancellationToken cancellationToken)
{
Interlocked.Increment(ref _flushCount);
return base.FlushAsync(cancellationToken);
}
}
// A MemoryStream whose first WriteAsync blocks until released, so a test can queue additional frames // A MemoryStream whose first WriteAsync blocks until released, so a test can queue additional frames
// behind an in-progress write and observe the writer's priority ordering. // behind an in-progress write and observe the writer's priority ordering.
private sealed class GatedWriteStream : MemoryStream private sealed class GatedWriteStream : MemoryStream
@@ -1112,6 +1112,184 @@ public sealed class WorkerPipeSessionTests
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token); await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
} }
/// <summary>
/// WRK-27. An STA call outside the command dispatcher (the alarm poll) advertises itself on
/// the heartbeat snapshot's <c>StaCallInProgress</c> flag, and the watchdog grants it the same
/// grace-to-ceiling suppression as a dispatched command: stale STA activity within the ceiling
/// does not fault while the flag is set, but stale activity beyond the ceiling faults anyway.
/// This closes the 15 s-vs-75 s asymmetry between polls and commands.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Watchdog_StaCallInProgress_SuppressedUntilCeiling()
{
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(10));
// Phase 1 — within the ceiling: stale beyond grace, empty correlation id, StaCallInProgress
// set. The default 75 s ceiling is far beyond the 5 s staleness, so the watchdog must suppress.
using (PipePair pipePair = await PipePair.CreateAsync(cancellation.Token))
{
FakeRuntimeSession runtime = new();
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow - TimeSpan.FromSeconds(5),
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId: string.Empty,
staCallInProgress: true));
WorkerPipeSession session = CreatePipeSession(
pipePair.WorkerStream,
runtime,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
HeartbeatGrace = TimeSpan.FromMilliseconds(50),
});
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
const int framesToInspect = 6;
int heartbeatsObserved = 0;
for (int index = 0; index < framesToInspect; index++)
{
WorkerEnvelope envelope = await pipePair.GatewayReader.ReadAsync(cancellation.Token);
Assert.NotEqual(WorkerEnvelope.BodyOneofCase.WorkerFault, envelope.BodyCase);
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerHeartbeat)
{
heartbeatsObserved++;
}
}
Assert.True(
heartbeatsObserved >= 2,
$"Expected multiple heartbeats during the in-progress STA-call window; observed {heartbeatsObserved}.");
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
// Phase 2 — beyond the ceiling: same StaCallInProgress flag, but staleness (5 s) exceeds the
// 200 ms ceiling, so the watchdog must fire even with the poll in progress.
using (PipePair pipePair = await PipePair.CreateAsync(cancellation.Token))
{
FakeRuntimeSession runtime = new();
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset.UtcNow - TimeSpan.FromSeconds(5),
pendingCommandCount: 0,
outboundEventQueueDepth: 0,
lastEventSequence: 0,
currentCommandCorrelationId: string.Empty,
staCallInProgress: true));
WorkerPipeSession session = CreatePipeSession(
pipePair.WorkerStream,
runtime,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
HeartbeatGrace = TimeSpan.FromMilliseconds(50),
HeartbeatStuckCeiling = TimeSpan.FromMilliseconds(200),
});
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
WorkerEnvelope fault = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerFault,
cancellation.Token);
Assert.Equal(WorkerFaultCategory.StaHung, fault.WorkerFault.Category);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
}
/// <summary>
/// WRK-25. The event drain loop submits a whole drained batch through the writer's batch entry
/// point, so a burst of 128 events costs one flush, not 128 — the assertion the WRK-12
/// tracking claim needed to actually hold on the event hot path.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task EventBurst_DrainLoopCoalescesFlushes()
{
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new();
// A far-off heartbeat interval keeps heartbeat flushes out of the measurement window.
FlushCountingPassthroughStream countingStream = new(pipePair.WorkerStream);
WorkerPipeSession session = CreatePipeSession(
countingStream,
runtime,
new WorkerPipeSessionOptions
{
HeartbeatInterval = TimeSpan.FromMinutes(5),
HeartbeatGrace = TimeSpan.FromSeconds(30),
});
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
// Let the idle drain loop settle (no events yet → no flushes) and record the baseline.
await Task.Delay(100, cancellation.Token);
int baselineFlushes = countingStream.FlushCount;
// Enqueue a full 128-event batch atomically so the drain loop sees it as one batch.
const int burst = 128;
List<WorkerEvent> batch = new(burst);
for (int index = 0; index < burst; index++)
{
batch.Add(CreateWorkerEvent(sequence: (ulong)(index + 1)));
}
runtime.EnqueueEvents(batch);
// Drain all 128 events off the gateway side.
for (int index = 0; index < burst; index++)
{
await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerEvent,
cancellation.Token);
}
// The whole burst cost exactly one additional flush.
Assert.Equal(1, countingStream.FlushCount - baselineFlushes);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
/// <summary>
/// WRK-24. A GatewayHello negotiating a frame maximum below the worker floor faults at the
/// handshake with a fault frame rather than being adopted — mirroring the above-ceiling
/// handshake behavior — so a nonsensical tiny value never leaves a session that fails every
/// later frame. No message loop is entered.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task Handshake_GatewayHelloWithTinyMaxFrameBytes_FaultsAtHandshake()
{
WorkerFrameProtocolOptions options = CreateOptions();
using MemoryStream inbound = new();
await new WorkerFrameWriter(inbound, options)
.WriteAsync(CreateGatewayHelloEnvelope(maxFrameBytes: 512));
inbound.Position = 0;
using MemoryStream outbound = new();
WorkerPipeSession session = CreateSession(inbound, outbound, options);
bool initialized = false;
WorkerFrameProtocolException exception =
await Assert.ThrowsAsync<WorkerFrameProtocolException>(
async () => await session.CompleteStartupHandshakeAsync(
_ =>
{
initialized = true;
return Task.CompletedTask;
}));
Assert.False(initialized);
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
WorkerEnvelope fault = Assert.Single(ReadWrittenFrames(outbound, options));
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerFault, fault.BodyCase);
}
/// <summary> /// <summary>
/// Regression test: a long in-flight STA command that keeps pumping /// Regression test: a long in-flight STA command that keeps pumping
/// must NOT self-fault as <c>StaHung</c>, and its reply must still be /// must NOT self-fault as <c>StaHung</c>, and its reply must still be
@@ -1976,6 +2154,78 @@ public sealed class WorkerPipeSessionTests
} }
} }
// Wraps the worker side of the pipe and counts FlushAsync calls so a test can assert the event
// drain loop coalesces a burst into a single flush. Delegates every other operation to the inner
// stream; does not own the inner stream's lifetime (PipePair disposes it).
private sealed class FlushCountingPassthroughStream : Stream
{
private readonly Stream inner;
private int flushCount;
/// <summary>Initializes the passthrough over the given inner stream.</summary>
/// <param name="inner">The stream to delegate to.</param>
public FlushCountingPassthroughStream(Stream inner)
{
this.inner = inner;
}
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
public int FlushCount => Volatile.Read(ref flushCount);
/// <inheritdoc />
public override bool CanRead => inner.CanRead;
/// <inheritdoc />
public override bool CanSeek => inner.CanSeek;
/// <inheritdoc />
public override bool CanWrite => inner.CanWrite;
/// <inheritdoc />
public override long Length => inner.Length;
/// <inheritdoc />
public override long Position
{
get => inner.Position;
set => inner.Position = value;
}
/// <inheritdoc />
public override void Flush()
{
Interlocked.Increment(ref flushCount);
inner.Flush();
}
/// <inheritdoc />
public override Task FlushAsync(CancellationToken cancellationToken)
{
Interlocked.Increment(ref flushCount);
return inner.FlushAsync(cancellationToken);
}
/// <inheritdoc />
public override int Read(byte[] buffer, int offset, int count) => inner.Read(buffer, offset, count);
/// <inheritdoc />
public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
=> inner.ReadAsync(buffer, offset, count, cancellationToken);
/// <inheritdoc />
public override long Seek(long offset, SeekOrigin origin) => inner.Seek(offset, origin);
/// <inheritdoc />
public override void SetLength(long value) => inner.SetLength(value);
/// <inheritdoc />
public override void Write(byte[] buffer, int offset, int count) => inner.Write(buffer, offset, count);
/// <inheritdoc />
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
=> inner.WriteAsync(buffer, offset, count, cancellationToken);
}
private sealed class PipePair : IDisposable private sealed class PipePair : IDisposable
{ {
private readonly NamedPipeServerStream gatewayStream; private readonly NamedPipeServerStream gatewayStream;
@@ -438,6 +438,53 @@ public sealed class MxAccessStaSessionTests
Assert.Contains("alarm poll failed", fault.DiagnosticMessage, StringComparison.OrdinalIgnoreCase); Assert.Contains("alarm poll failed", fault.DiagnosticMessage, StringComparison.OrdinalIgnoreCase);
} }
/// <summary>
/// WRK-27. While the alarm poll's PollOnce is executing on the STA, a heartbeat captured mid-poll
/// must report <see cref="WorkerRuntimeHeartbeatSnapshot.StaCallInProgress"/> so the watchdog
/// grants the poll the same grace-to-ceiling suppression as a dispatched command; once the poll
/// returns the flag clears. PollOnce is blocked on a gate so the heartbeat can be captured while
/// the STA call is genuinely in flight.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task CaptureHeartbeat_DuringAlarmPoll_ReportsStaCallInProgress()
{
FakeAlarmCommandHandler handler = new() { BlockPoll = true };
FakeMxAccessComObjectFactory factory = new();
FakeMxAccessEventSink eventSink = new();
using StaRuntime runtime = CreateRuntime();
using MxAccessStaSession session = new(
runtime,
factory,
eventSink,
new MxAccessEventQueue(),
(_eq, _affinity, _comFactory) => handler);
await session.StartAsync("session-1", workerProcessId: 1);
// Wait until PollOnce is blocked mid-call on the STA thread.
Assert.True(
handler.WaitForPollEntered(TimeSpan.FromSeconds(5)),
"Expected the alarm poll to start within 5 seconds.");
// Captured mid-poll, the heartbeat advertises the in-progress STA call.
Assert.True(session.CaptureHeartbeat().StaCallInProgress);
// Release the poll and stop blocking; the flag clears once the poll returns.
handler.BlockPoll = false;
handler.ReleasePoll();
using CancellationTokenSource timeout = new(TimeSpan.FromSeconds(5));
while (session.CaptureHeartbeat().StaCallInProgress && !timeout.IsCancellationRequested)
{
await Task.Delay(25, CancellationToken.None);
}
Assert.False(
session.CaptureHeartbeat().StaCallInProgress,
"Expected StaCallInProgress to clear once the alarm poll returned.");
}
/// <summary> /// <summary>
/// The STA-affinity guard throws when an /// The STA-affinity guard throws when an
/// IMxAccessAlarmConsumer call is attempted off the thread that created /// IMxAccessAlarmConsumer call is attempted off the thread that created
@@ -472,6 +519,8 @@ public sealed class MxAccessStaSessionTests
private sealed class FakeAlarmCommandHandler : IAlarmCommandHandler private sealed class FakeAlarmCommandHandler : IAlarmCommandHandler
{ {
private readonly object gate = new object(); private readonly object gate = new object();
private readonly ManualResetEventSlim pollEntered = new(false);
private readonly ManualResetEventSlim releasePoll = new(false);
private int pollCount; private int pollCount;
private int? lastPollThreadId; private int? lastPollThreadId;
@@ -484,6 +533,17 @@ public sealed class MxAccessStaSessionTests
/// <summary>Exception thrown by PollOnce; null to succeed.</summary> /// <summary>Exception thrown by PollOnce; null to succeed.</summary>
public Exception? PollException { get; set; } public Exception? PollException { get; set; }
/// <summary>When set, <see cref="PollOnce"/> blocks until <see cref="ReleasePoll"/> is called.</summary>
public bool BlockPoll { get; set; }
/// <summary>Waits until a blocking <see cref="PollOnce"/> has entered and is blocked.</summary>
/// <param name="timeout">Maximum time to wait.</param>
/// <returns>True if a poll entered within the timeout.</returns>
public bool WaitForPollEntered(TimeSpan timeout) => pollEntered.Wait(timeout);
/// <summary>Releases a <see cref="PollOnce"/> blocked on the gate.</summary>
public void ReleasePoll() => releasePoll.Set();
/// <summary>Gets the count of PollOnce calls.</summary> /// <summary>Gets the count of PollOnce calls.</summary>
public int PollCount public int PollCount
{ {
@@ -533,6 +593,12 @@ public sealed class MxAccessStaSessionTests
lastPollThreadId = Thread.CurrentThread.ManagedThreadId; lastPollThreadId = Thread.CurrentThread.ManagedThreadId;
} }
if (BlockPoll)
{
pollEntered.Set();
releasePoll.Wait(TimeSpan.FromSeconds(10));
}
if (PollException is not null) if (PollException is not null)
{ {
throw PollException; throw PollException;
@@ -540,6 +606,10 @@ public sealed class MxAccessStaSessionTests
} }
/// <inheritdoc /> /// <inheritdoc />
public void Dispose() { } public void Dispose()
{
pollEntered.Dispose();
releasePoll.Dispose();
}
} }
} }
@@ -372,6 +372,23 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
} }
} }
/// <summary>
/// Enqueues a batch of worker events atomically under one lock so the drain loop cannot
/// observe a partial batch. Lets a test assert the drain loop coalesces a whole batch into one
/// flush (WRK-25) without racing a mid-enqueue drain that would split the batch.
/// </summary>
/// <param name="workerEvents">The events to enqueue in order.</param>
public void EnqueueEvents(IEnumerable<WorkerEvent> workerEvents)
{
lock (gate)
{
foreach (WorkerEvent workerEvent in workerEvents)
{
events.Enqueue(workerEvent);
}
}
}
/// <inheritdoc /> /// <inheritdoc />
public void Dispose() public void Dispose()
{ {
@@ -18,6 +18,17 @@ public sealed class WorkerFrameProtocolOptions
/// </summary> /// </summary>
public const int MaxNegotiableFrameBytes = 256 * 1024 * 1024; public const int MaxNegotiableFrameBytes = 256 * 1024 * 1024;
/// <summary>
/// Lower floor the worker will accept for a gateway-negotiated frame maximum
/// (<c>GatewayHello.max_frame_bytes</c>). Matches the gateway's own
/// <c>GatewayOptionsValidator.MinimumMaxMessageBytes</c> validation floor so the worker never
/// rejects a value the gateway's own validator accepts as legal configuration, yet a nonsensical
/// tiny value (a gateway bug or a foreign/old peer) is rejected at the handshake rather than
/// leaving a session that handshakes cleanly and then fails every subsequent frame with
/// per-frame size errors. 1024 bytes still guarantees hellos, heartbeats, acks, and faults fit.
/// </summary>
public const int MinNegotiableFrameBytes = 1024;
/// <summary>Initializes a new instance of the WorkerFrameProtocolOptions class from WorkerOptions.</summary> /// <summary>Initializes a new instance of the WorkerFrameProtocolOptions class from WorkerOptions.</summary>
/// <param name="options">Worker initialization options.</param> /// <param name="options">Worker initialization options.</param>
public WorkerFrameProtocolOptions(WorkerOptions options) public WorkerFrameProtocolOptions(WorkerOptions options)
@@ -118,7 +129,9 @@ public sealed class WorkerFrameProtocolOptions
/// <summary> /// <summary>
/// Adopts the gateway-negotiated frame maximum conveyed in <c>GatewayHello.max_frame_bytes</c>. /// Adopts the gateway-negotiated frame maximum conveyed in <c>GatewayHello.max_frame_bytes</c>.
/// A value of 0 (an older gateway that never set the field) is ignored and the /// A value of 0 (an older gateway that never set the field) is ignored and the
/// constructor default is kept. A value above <see cref="MaxNegotiableFrameBytes"/> is rejected. /// constructor default is kept. A value outside the accepted range
/// [<see cref="MinNegotiableFrameBytes"/>, <see cref="MaxNegotiableFrameBytes"/>] is rejected so a
/// nonsensical negotiated value faults at the handshake rather than mid-session.
/// </summary> /// </summary>
/// <param name="negotiatedMaxFrameBytes">The gateway-negotiated maximum, or 0 for "keep default".</param> /// <param name="negotiatedMaxFrameBytes">The gateway-negotiated maximum, or 0 for "keep default".</param>
internal void AdoptNegotiatedMaxMessageBytes(uint negotiatedMaxFrameBytes) internal void AdoptNegotiatedMaxMessageBytes(uint negotiatedMaxFrameBytes)
@@ -128,6 +141,14 @@ public sealed class WorkerFrameProtocolOptions
return; return;
} }
if (negotiatedMaxFrameBytes < MinNegotiableFrameBytes)
{
throw new WorkerFrameProtocolException(
WorkerFrameProtocolErrorCode.InvalidConfiguration,
$"GatewayHello negotiated frame maximum {negotiatedMaxFrameBytes} is below the worker floor "
+ $"of {MinNegotiableFrameBytes} bytes.");
}
if (negotiatedMaxFrameBytes > MaxNegotiableFrameBytes) if (negotiatedMaxFrameBytes > MaxNegotiableFrameBytes)
{ {
throw new WorkerFrameProtocolException( throw new WorkerFrameProtocolException(
@@ -1,6 +1,7 @@
using System; using System;
using System.Collections.Generic; using System.Collections.Generic;
using System.IO; using System.IO;
using System.Runtime.ExceptionServices;
using System.Threading; using System.Threading;
using System.Threading.Tasks; using System.Threading.Tasks;
using Google.Protobuf; using Google.Protobuf;
@@ -33,6 +34,16 @@ public sealed class WorkerFrameWriter
/// <summary>Gets the completion source signaled once the frame has been written or has failed.</summary> /// <summary>Gets the completion source signaled once the frame has been written or has failed.</summary>
public TaskCompletionSource<bool> Completion { get; } public TaskCompletionSource<bool> Completion { get; }
/// <summary>
/// Set to <c>true</c> by <see cref="DequeueNext"/> — under <c>_gate</c> — at the instant the
/// draining lock-holder takes ownership of this frame to write it. A cancelled caller
/// tombstones its frame only while it is still unclaimed, so a claim and a cancel can never
/// both win: the flag is the interlock between the two. A frame already claimed is mid-write
/// and can no longer be recalled (see <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>).
/// Mutated only under <c>_gate</c>.
/// </summary>
public bool Claimed;
} }
private readonly WorkerFrameProtocolOptions _options; private readonly WorkerFrameProtocolOptions _options;
@@ -75,6 +86,13 @@ public sealed class WorkerFrameWriter
/// <param name="priority">Scheduling priority; control frames are written ahead of event frames.</param> /// <param name="priority">Scheduling priority; control frames are written ahead of event frames.</param>
/// <param name="cancellationToken">Token to cancel waiting for the write lock.</param> /// <param name="cancellationToken">Token to cancel waiting for the write lock.</param>
/// <returns>A task that completes when the frame has been written and flushed.</returns> /// <returns>A task that completes when the frame has been written and flushed.</returns>
/// <remarks>
/// Cancellation contract (WRK-22): if the token fires while this call is waiting for the write
/// lock, the frame is tombstoned so it is never written — unless a draining lock-holder has
/// already claimed it, in which case the frame may still reach the wire even though this call
/// observes <see cref="OperationCanceledException"/>. That residual window is by design: blocking
/// the canceller behind the very write it is abandoning would defeat the point of cancellation.
/// </remarks>
public async Task WriteAsync( public async Task WriteAsync(
WorkerEnvelope envelope, WorkerEnvelope envelope,
WorkerFrameWritePriority priority, WorkerFrameWritePriority priority,
@@ -101,7 +119,19 @@ public sealed class WorkerFrameWriter
// Contend for the single writer: whoever wins drains every currently-queued frame in priority // Contend for the single writer: whoever wins drains every currently-queued frame in priority
// order, so this frame is written by this call or by a concurrent caller that got the lock // order, so this frame is written by this call or by a concurrent caller that got the lock
// first. Either way it completes via its own TaskCompletionSource. // first. Either way it completes via its own TaskCompletionSource.
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false); try
{
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Tombstone the queued frame so DequeueNext skips it — but only if a draining lock-holder
// has not already claimed it. If it is claimed it is mid-write and cannot be recalled; the
// caller still observes cancellation while the frame reaches the wire (documented above).
TombstoneIfUnclaimed(frame, cancellationToken);
throw;
}
try try
{ {
await DrainQueuedFramesAsync().ConfigureAwait(false); await DrainQueuedFramesAsync().ConfigureAwait(false);
@@ -114,6 +144,123 @@ public sealed class WorkerFrameWriter
await frame.Completion.Task.ConfigureAwait(false); await frame.Completion.Task.ConfigureAwait(false);
} }
/// <summary>
/// Queues a whole batch of envelopes at one priority under a single lock acquisition and drains
/// it, so a burst of frames — the event drain loop's hot path — pays one flush for the batch
/// rather than one per frame (WRK-25, realizing the WRK-12 coalescing on the path it was built
/// for). Intra-batch order is preserved because the enqueue is atomic under <c>_gate</c> and each
/// class queue is FIFO; the control-before-event guarantee still holds because any concurrently
/// queued control frame is drained ahead of this batch by <see cref="DequeueNext"/>. Every frame's
/// "written and flushed before completion" contract is unchanged.
/// </summary>
/// <param name="envelopes">Envelopes to write, in order.</param>
/// <param name="priority">Scheduling priority for the whole batch.</param>
/// <param name="cancellationToken">Token to cancel waiting for the write lock.</param>
/// <returns>A task that completes when every frame in the batch has been written and flushed.</returns>
/// <remarks>
/// A per-frame rejection inside the batch (for example one oversized event) surfaces from the
/// awaited completions as its <see cref="WorkerFrameProtocolException"/>; the remaining frames are
/// still observed so none faults unobserved. Cancellation while waiting for the lock tombstones
/// every still-unclaimed frame in the batch, per the WRK-22 contract on
/// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>.
/// </remarks>
public async Task WriteBatchAsync(
IReadOnlyList<WorkerEnvelope> envelopes,
WorkerFrameWritePriority priority,
CancellationToken cancellationToken = default)
{
if (envelopes is null)
{
throw new ArgumentNullException(nameof(envelopes));
}
if (envelopes.Count == 0)
{
return;
}
PendingFrame[] frames = new PendingFrame[envelopes.Count];
for (int index = 0; index < envelopes.Count; index++)
{
WorkerEnvelope envelope = envelopes[index]
?? throw new ArgumentException("Batch envelopes must not contain null.", nameof(envelopes));
frames[index] = new PendingFrame(envelope);
}
lock (_gate)
{
Queue<PendingFrame> queue = priority == WorkerFrameWritePriority.Event ? _eventFrames : _controlFrames;
foreach (PendingFrame frame in frames)
{
queue.Enqueue(frame);
}
}
try
{
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
TombstoneUnclaimed(frames, cancellationToken);
throw;
}
try
{
await DrainQueuedFramesAsync().ConfigureAwait(false);
}
finally
{
_writeLock.Release();
}
// Await every completion so no per-frame rejection faults unobserved, but surface the first
// failure (in batch order) to the caller — the drain loop maps it back to the offending event.
Exception? firstFailure = null;
foreach (PendingFrame frame in frames)
{
try
{
await frame.Completion.Task.ConfigureAwait(false);
}
catch (Exception exception)
{
firstFailure ??= exception;
}
}
if (firstFailure is not null)
{
ExceptionDispatchInfo.Capture(firstFailure).Throw();
}
}
private void TombstoneIfUnclaimed(PendingFrame frame, CancellationToken cancellationToken)
{
lock (_gate)
{
if (!frame.Claimed)
{
frame.Completion.TrySetCanceled(cancellationToken);
}
}
}
private void TombstoneUnclaimed(PendingFrame[] frames, CancellationToken cancellationToken)
{
lock (_gate)
{
foreach (PendingFrame frame in frames)
{
if (!frame.Claimed)
{
frame.Completion.TrySetCanceled(cancellationToken);
}
}
}
}
// Runs only under _writeLock. Drains control frames before event frames, stamping and writing each. // Runs only under _writeLock. Drains control frames before event frames, stamping and writing each.
// The stream write itself is not cancellable: a frame is written atomically or fails, never left // The stream write itself is not cancellable: a frame is written atomically or fails, never left
// half-written on the pipe because a caller gave up waiting. // half-written on the pipe because a caller gave up waiting.
@@ -198,18 +345,36 @@ public sealed class WorkerFrameWriter
or WorkerFrameProtocolErrorCode.SessionMismatch; or WorkerFrameProtocolErrorCode.SessionMismatch;
} }
// Returns the next frame to write, control frames first, skipping any frame a cancelled caller
// tombstoned while it waited for the lock (WRK-22). The frame actually returned is marked Claimed
// under _gate in the same critical section that checks the tombstone, so a claim and a concurrent
// cancel are mutually exclusive: whichever acquires _gate first wins.
private PendingFrame? DequeueNext() private PendingFrame? DequeueNext()
{ {
lock (_gate) lock (_gate)
{ {
if (_controlFrames.Count > 0) while (_controlFrames.Count > 0)
{ {
return _controlFrames.Dequeue(); PendingFrame frame = _controlFrames.Dequeue();
if (frame.Completion.Task.IsCanceled)
{
continue;
}
frame.Claimed = true;
return frame;
} }
if (_eventFrames.Count > 0) while (_eventFrames.Count > 0)
{ {
return _eventFrames.Dequeue(); PendingFrame frame = _eventFrames.Dequeue();
if (frame.Completion.Task.IsCanceled)
{
continue;
}
frame.Claimed = true;
return frame;
} }
return null; return null;
@@ -371,27 +371,57 @@ public sealed class WorkerPipeSession
continue; continue;
} }
foreach (WorkerEvent workerEvent in events) // Submit the whole drained batch through the writer's batch entry point under one lock
// acquisition so the burst pays a single flush instead of one per event (WRK-25). Events
// are the low-priority frame class: the writer holds them behind any pending control frame
// (reply, fault, heartbeat, shutdown ack) so those are not delayed behind an event backlog,
// and intra-batch order is preserved.
WorkerEnvelope[] envelopes = new WorkerEnvelope[events.Count];
for (int index = 0; index < events.Count; index++)
{ {
// Events are the low-priority frame class: the writer holds them behind any pending envelopes[index] = CreateEnvelope(events[index]);
// control frame (reply, fault, heartbeat, shutdown ack) so those are not delayed }
// behind an event backlog.
try try
{ {
await _writer await _writer
.WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken) .WriteBatchAsync(envelopes, WorkerFrameWritePriority.Event, cancellationToken)
.ConfigureAwait(false); .ConfigureAwait(false);
} }
catch (WorkerFrameProtocolException exception) catch (WorkerFrameProtocolException exception)
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge) when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
{ {
await FaultOnOversizedEventAsync(workerEvent, exception, cancellationToken) // A single oversized event surfaces from the batch's awaited completions; the death is
.ConfigureAwait(false); // still IPC-30's structured, event-naming fault. Map the rejection back to the first
} // event in batch order whose envelope overshoots the negotiated maximum — the same
// frame the writer rejected first.
await FaultOnOversizedEventAsync(
FindOversizedEvent(events, envelopes),
exception,
cancellationToken)
.ConfigureAwait(false);
} }
} }
} }
private WorkerEvent FindOversizedEvent(
IReadOnlyList<WorkerEvent> events,
WorkerEnvelope[] envelopes)
{
for (int index = 0; index < envelopes.Length; index++)
{
if (envelopes[index].CalculateSize() > _options.MaxMessageBytes)
{
return events[index];
}
}
// Unreachable in practice: WriteBatchAsync surfaced MessageTooLarge, so at least one envelope
// exceeded the negotiated maximum. Fall back to the first event so the fault still names a
// concrete event rather than throwing a second, less useful exception from the fault path.
return events[0];
}
/// <summary> /// <summary>
/// Ends the session on an event that cannot be framed, but deliberately and diagnosably /// Ends the session on an event that cannot be framed, but deliberately and diagnosably
/// (IPC-30). An event above the negotiated frame maximum is undeliverable end to end — the /// (IPC-30). An event above the negotiated frame maximum is undeliverable end to end — the
@@ -1085,16 +1115,16 @@ public sealed class WorkerPipeSession
return; return;
} }
if (!string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId) if ((!string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId) || snapshot.StaCallInProgress)
&& staleFor <= _sessionOptions.HeartbeatStuckCeiling) && staleFor <= _sessionOptions.HeartbeatStuckCeiling)
{ {
// A command is in flight and we are still within the defensive // A command is in flight, or an STA call outside the dispatcher (the alarm poll, WRK-27) is
// suppression ceiling — the STA is busy executing it, not // executing, and we are still within the defensive suppression ceiling — the STA is busy
// hung. The next MarkActivity() in StaRuntime.ProcessQueuedCommands // doing that work, not hung. The next MarkActivity() in StaRuntime.ProcessQueuedCommands
// will refresh LastActivityUtc once the command returns, at which // will refresh LastActivityUtc once the work returns, at which point this branch stops
// point this branch stops being taken. The heartbeat already // being taken. The heartbeat already surfaces the in-flight correlation id so the gateway
// surfaces the in-flight correlation id so the gateway can apply // can apply its own per-command timeout if it considers the command too slow; a poll that
// its own per-command timeout if it considers the command too slow. // blocks the STA past the ceiling still faults, which is the ceiling's contract.
return; return;
} }
@@ -24,6 +24,14 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
private CancellationTokenSource? alarmPollCts; private CancellationTokenSource? alarmPollCts;
private Task? alarmPollTask; private Task? alarmPollTask;
private int? alarmConsumerThreadId; private int? alarmConsumerThreadId;
// True on the STA thread exactly around the alarm PollOnce COM call. The alarm poll runs outside
// the StaCommandDispatcher (so it does not inflate PendingCommandCount or perturb command dispatch
// ordering), which means CaptureHeartbeat would otherwise see no in-flight activity during a long
// poll and the watchdog would fault the session at the 15 s grace instead of the 75 s ceiling
// granted to dispatched commands. Surfacing the poll on the heartbeat closes that asymmetry
// (WRK-27). Volatile: written on the STA thread, read on the heartbeat thread.
private volatile bool staAlarmPollInProgress;
private bool disposed; private bool disposed;
/// <summary> /// <summary>
@@ -247,8 +255,20 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
await staRuntime.InvokeAsync( await staRuntime.InvokeAsync(
() => () =>
{ {
EnsureOnAlarmConsumerThread(); // Advertise the poll to the watchdog for exactly the span of the COM call
handler.PollOnce(); // (WRK-27): set on the STA thread immediately before the affinity check and
// PollOnce, clear in the finally so a heartbeat captured mid-poll reports
// StaCallInProgress and one captured after does not.
staAlarmPollInProgress = true;
try
{
EnsureOnAlarmConsumerThread();
handler.PollOnce();
}
finally
{
staAlarmPollInProgress = false;
}
}, },
cancellationToken).ConfigureAwait(false); cancellationToken).ConfigureAwait(false);
} }
@@ -377,7 +397,8 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
pendingCommandCount, pendingCommandCount,
(uint)eventQueue.Count, (uint)eventQueue.Count,
eventQueue.LastEventSequence, eventQueue.LastEventSequence,
currentCommandCorrelationId); currentCommandCorrelationId,
staAlarmPollInProgress);
} }
/// <inheritdoc /> /// <inheritdoc />
@@ -10,18 +10,26 @@ public sealed class WorkerRuntimeHeartbeatSnapshot
/// <param name="outboundEventQueueDepth">Current depth of the worker event queue.</param> /// <param name="outboundEventQueueDepth">Current depth of the worker event queue.</param>
/// <param name="lastEventSequence">Sequence number of the most recent event.</param> /// <param name="lastEventSequence">Sequence number of the most recent event.</param>
/// <param name="currentCommandCorrelationId">Correlation ID of the in-flight command.</param> /// <param name="currentCommandCorrelationId">Correlation ID of the in-flight command.</param>
/// <param name="staCallInProgress">
/// True while an STA call outside the command dispatcher is executing (currently the alarm poll,
/// WRK-27). The watchdog treats this like an in-flight command: it suppresses the stale-STA fault
/// up to the stuck ceiling instead of the shorter grace, so a healthy-but-slow poll does not fault
/// a healthy session. Named generically so any future non-dispatcher STA work reuses it.
/// </param>
public WorkerRuntimeHeartbeatSnapshot( public WorkerRuntimeHeartbeatSnapshot(
DateTimeOffset lastStaActivityUtc, DateTimeOffset lastStaActivityUtc,
uint pendingCommandCount, uint pendingCommandCount,
uint outboundEventQueueDepth, uint outboundEventQueueDepth,
ulong lastEventSequence, ulong lastEventSequence,
string currentCommandCorrelationId) string currentCommandCorrelationId,
bool staCallInProgress = false)
{ {
LastStaActivityUtc = lastStaActivityUtc; LastStaActivityUtc = lastStaActivityUtc;
PendingCommandCount = pendingCommandCount; PendingCommandCount = pendingCommandCount;
OutboundEventQueueDepth = outboundEventQueueDepth; OutboundEventQueueDepth = outboundEventQueueDepth;
LastEventSequence = lastEventSequence; LastEventSequence = lastEventSequence;
CurrentCommandCorrelationId = currentCommandCorrelationId ?? string.Empty; CurrentCommandCorrelationId = currentCommandCorrelationId ?? string.Empty;
StaCallInProgress = staCallInProgress;
} }
/// <summary>Gets the last STA activity timestamp in UTC.</summary> /// <summary>Gets the last STA activity timestamp in UTC.</summary>
@@ -38,4 +46,11 @@ public sealed class WorkerRuntimeHeartbeatSnapshot
/// <summary>Gets the correlation ID of the in-flight command.</summary> /// <summary>Gets the correlation ID of the in-flight command.</summary>
public string CurrentCommandCorrelationId { get; } public string CurrentCommandCorrelationId { get; }
/// <summary>
/// Gets a value indicating whether an STA call outside the command dispatcher (the alarm poll) is
/// executing. When true the watchdog grants the poll the same grace-to-ceiling suppression as a
/// dispatched command (WRK-27).
/// </summary>
public bool StaCallInProgress { get; }
} }