fix(WRK-22,WRK-24,WRK-25,WRK-27,IPC-26): worker write-seam hardening
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext skips cancelled, claims the frame it returns) so a cancelled write never reaches the wire unless already claimed mid-write (documented residual). WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained event batch through it, so a burst of N events costs one flush not N. IPC-30 oversized-event structured fault preserved via FindOversizedEvent. WRK-24: reject a below-1024 negotiated frame maximum at the handshake (MinNegotiableFrameBytes, matching GatewayOptionsValidator floor). WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot so the watchdog suppresses to the ceiling, not the grace. Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the 2026-07-12 remediation registers/change-log updated in the same commit.
This commit is contained in:
File diff suppressed because one or more lines are too long
@@ -17,12 +17,12 @@ members, no positional records). The worker builds and tests only on the Windows
|
|||||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|----|-----|------|-----|-----|--------|-------|
|
|----|-----|------|-----|-----|--------|-------|
|
||||||
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
|
| WRK-21 | Medium | P0 | M | IPC-23 (same defect, fix owned here); WRK-28 (same lines) | Done | DrainEvents bound is count-based only; an oversized reply still kills the session and loses the drained events |
|
||||||
| WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
|
| WRK-22 | Low | — | S | IPC-26 (same defect, fix owned here) | Done | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
|
||||||
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps |
|
| WRK-23 | Low | — | S | WRK-21 (rejection path becomes backstop-only) | Done | Rejected frames consume sequence numbers, producing wire gaps |
|
||||||
| WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
|
| WRK-24 | Low | — | S | — | Done | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
|
||||||
| WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Not started | WRK-12 flush coalescing never engages on the event hot path |
|
| WRK-25 | Low | P2 | S | WRK-22 (both touch enqueue/dequeue) | Done | WRK-12 flush coalescing never engages on the event hot path |
|
||||||
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
|
| WRK-26 | Low | P1 | S | WRK-23 (soft — sequence prose); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
|
||||||
| WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
|
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
|
||||||
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
|
| WRK-28 | Low | — | S | WRK-21 (land in the same commit cluster) | Done | 10,000 drain cap is a duplicated magic constant with a comment-only sync contract |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ All `path:line` citations were re-verified against the working tree at `4f5371f`
|
|||||||
| IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
|
| IPC-23 | Medium | P0 | S¹ | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents bound is count-based only; byte-heavy queue still builds a session-killing reply frame (contract requirements here; fix mechanics in WRK-21) |
|
||||||
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes |
|
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real generated-code drift for message-level proto changes |
|
||||||
| IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them |
|
| IPC-25 | Medium | P0 | M | — | Not started | Committed Go/Python worker bindings are stale at HEAD; no guard covers them |
|
||||||
| IPC-26 | Low | P2 | S¹ | WRK-22 | Not started | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
|
| IPC-26 | Low | P2 | S¹ | WRK-22 | Done (mechanics landed in WRK-22) | Cancelled write leaves a ghost frame that is still written (contract requirement here; fix mechanics in WRK-22) |
|
||||||
| IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract |
|
| IPC-27 | Low | P2 | S | — | Not started | Descriptor freshness test blind to enums, enum values, services/methods, and the Galaxy contract |
|
||||||
| IPC-28 | Low | — | S | — | Done | `docs/Grpc.md` omits the `CommandTooLarge` → `ResourceExhausted` mapping |
|
| IPC-28 | Low | — | S | — | Done | `docs/Grpc.md` omits the `CommandTooLarge` → `ResourceExhausted` mapping |
|
||||||
| IPC-29 | Low | — | S | — | Done (discharged by WRK-26) | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
|
| IPC-29 | Low | — | S | — | Done (discharged by WRK-26) | Worker writer priority scheduling and write-time sequence stamping undocumented in the frame-protocol doc |
|
||||||
|
|||||||
@@ -747,6 +747,23 @@ heartbeat fields until dedicated thresholds own those warnings. The worker
|
|||||||
reports stale STA activity, but the gateway owns the final kill decision
|
reports stale STA activity, but the gateway owns the final kill decision
|
||||||
through its existing heartbeat and worker lifecycle policy.
|
through its existing heartbeat and worker lifecycle policy.
|
||||||
|
|
||||||
|
The alarm poll runs outside the command dispatcher — `RunAlarmPollLoopAsync`
|
||||||
|
invokes `PollOnce` directly on the STA rather than through
|
||||||
|
`StaCommandDispatcher`, so it does not inflate `PendingCommandCount` or perturb
|
||||||
|
dispatch ordering for real gateway commands. Because it is not a dispatched
|
||||||
|
command it has no `CurrentCommandCorrelationId`, so a healthy-but-slow poll (a
|
||||||
|
large `GetXmlCurrentAlarms2` against a busy provider) blocking the STA past
|
||||||
|
`HeartbeatGrace` would otherwise fault a healthy session at 15 s while a
|
||||||
|
dispatched command gets the 75 s ceiling. To close that asymmetry (WRK-27) the
|
||||||
|
poll advertises itself on the heartbeat snapshot's `StaCallInProgress` flag —
|
||||||
|
set on the STA thread for exactly the span of the COM call — and the watchdog
|
||||||
|
suppression honors that flag alongside `CurrentCommandCorrelationId`. The poll
|
||||||
|
therefore receives the same grace-to-ceiling treatment as a dispatched command:
|
||||||
|
suppressed up to `HeartbeatStuckCeiling`, faulted past it (a poll that blocks
|
||||||
|
the STA more than 75 s without pumping *should* fault — that is the ceiling's
|
||||||
|
contract). The flag is named generically so any future non-dispatcher STA work
|
||||||
|
reuses it.
|
||||||
|
|
||||||
The in-flight-command suppression itself is bounded by
|
The in-flight-command suppression itself is bounded by
|
||||||
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
|
`WorkerPipeSessionOptions.HeartbeatStuckCeiling` (default 75 seconds = 5 ×
|
||||||
`HeartbeatGrace`). The motivating case for the suppression is a legitimately
|
`HeartbeatGrace`). The motivating case for the suppression is a legitimately
|
||||||
|
|||||||
+33
-13
@@ -29,6 +29,17 @@ default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
|
|||||||
means "use the worker's built-in default". This keeps both ends framing to the
|
means "use the worker's built-in default". This keeps both ends framing to the
|
||||||
same limit rather than depending on matched compile-time constants.
|
same limit rather than depending on matched compile-time constants.
|
||||||
|
|
||||||
|
The worker accepts a negotiated value in the closed range [1024, 256 MiB]
|
||||||
|
(`MinNegotiableFrameBytes` .. `MaxNegotiableFrameBytes`); 0 keeps the default.
|
||||||
|
A value outside that range is rejected at the handshake with a fault frame
|
||||||
|
rather than adopted, because a nonsensical maximum — a gateway bug or a
|
||||||
|
foreign/old peer — would otherwise leave a session that handshakes cleanly and
|
||||||
|
then fails every subsequent frame with per-frame size errors, the worst
|
||||||
|
diagnostic shape for an operator. The 1024-byte floor matches the gateway's own
|
||||||
|
`GatewayOptionsValidator.MinimumMaxMessageBytes`, so the worker never rejects a
|
||||||
|
value the gateway's validator accepts as legal configuration, and 1024 still
|
||||||
|
guarantees hellos, heartbeats, acks, and faults fit.
|
||||||
|
|
||||||
Every worker-to-gateway frame must serialize within this limit, control replies
|
Every worker-to-gateway frame must serialize within this limit, control replies
|
||||||
included, so reply builders truncate to fit rather than emit a frame the writer
|
included, so reply builders truncate to fit rather than emit a frame the writer
|
||||||
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
|
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
|
||||||
@@ -112,20 +123,29 @@ runs after the whole batch, and only then does every successfully-written
|
|||||||
frame's completion resolve — so a caller's `WriteAsync` still does not
|
frame's completion resolve — so a caller's `WriteAsync` still does not
|
||||||
complete until its bytes are both written *and* flushed, but a batch that
|
complete until its bytes are both written *and* flushed, but a batch that
|
||||||
happened to contain several queued frames pays one flush instead of one per
|
happened to contain several queued frames pays one flush instead of one per
|
||||||
frame. In practice this coalescing currently engages only when multiple
|
frame. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
|
||||||
frames are queued at the moment a lock-holder starts draining. The event
|
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
|
||||||
drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`) awaits each drained
|
every frame under one `_gate` acquisition, takes the write lock once, and
|
||||||
event's `WriteAsync` individually before writing the next, so today at most
|
drains them together, so a burst of N events costs one flush rather than N —
|
||||||
one event frame is queued per drain pass and each event still costs its own
|
the coalescing the batch machinery was built for now engages on the event hot
|
||||||
flush; a dedicated batch write entry point that submits a whole drained
|
path, not only when independent producers happen to queue behind a blocked
|
||||||
event batch under one lock acquisition is designed but not yet landed, so a
|
write. Intra-batch order is preserved (FIFO enqueue under one lock), and a
|
||||||
burst of N events currently costs N flushes on the event hot path, not one.
|
concurrently queued control frame is still drained ahead of the batch. A
|
||||||
|
per-frame rejection inside a batch (for example one oversized event) surfaces
|
||||||
|
from the batch's awaited completions as that frame's
|
||||||
|
`WorkerFrameProtocolException`; the remaining completions are still observed
|
||||||
|
so none faults unobserved.
|
||||||
|
|
||||||
Cancellation semantics for a `WriteAsync` call that is still waiting for the
|
Cancellation of a `WriteAsync`/`WriteBatchAsync` call that is still waiting
|
||||||
write lock when its token fires are not yet defined at this layer — pending
|
for the write lock when its token fires tombstones the queued frame: the
|
||||||
a fix that will tombstone the queued frame so a cancelled call is guaranteed
|
cancelled caller marks its frame under `_gate`, and the draining lock-holder's
|
||||||
never to reach the wire. Until that lands, a cancelled caller may still see
|
`DequeueNext` skips any tombstoned frame, so a cancelled call is guaranteed
|
||||||
its frame written by whichever caller next holds the lock.
|
never to reach the wire — *unless* a lock-holder has already claimed the frame
|
||||||
|
to write it. Claiming and cancelling are interlocked under `_gate`, so exactly
|
||||||
|
one wins; a frame already claimed is mid-write and can no longer be recalled,
|
||||||
|
so the caller observes `OperationCanceledException` while that one frame still
|
||||||
|
reaches the wire. That residual window is by design: blocking the canceller
|
||||||
|
behind the very write it is abandoning would defeat the point of cancellation.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
|
|||||||
@@ -481,6 +481,192 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
|
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies a negotiated frame maximum below the worker floor is rejected as
|
||||||
|
/// <c>InvalidConfiguration</c> (WRK-24), and that exactly the floor is adopted. The floor matches
|
||||||
|
/// the gateway's own <c>GatewayOptionsValidator.MinimumMaxMessageBytes</c>, so the worker never
|
||||||
|
/// rejects a value the gateway's validator accepts, yet a nonsensical tiny value faults at the
|
||||||
|
/// handshake instead of leaving a session that fails every later frame.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void AdoptNegotiatedMaxMessageBytes_BelowFloor_ThrowsInvalidConfiguration()
|
||||||
|
{
|
||||||
|
WorkerFrameProtocolOptions belowFloor = CreateOptions();
|
||||||
|
WorkerFrameProtocolException exception = Assert.Throws<WorkerFrameProtocolException>(
|
||||||
|
() => belowFloor.AdoptNegotiatedMaxMessageBytes(512));
|
||||||
|
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
|
||||||
|
|
||||||
|
// Boundary: exactly the floor is accepted.
|
||||||
|
WorkerFrameProtocolOptions atFloor = CreateOptions();
|
||||||
|
atFloor.AdoptNegotiatedMaxMessageBytes((uint)WorkerFrameProtocolOptions.MinNegotiableFrameBytes);
|
||||||
|
Assert.Equal(WorkerFrameProtocolOptions.MinNegotiableFrameBytes, atFloor.MaxMessageBytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-22 / IPC-26. A <c>WriteAsync</c> cancelled while it waits for the write lock must never
|
||||||
|
/// have its frame written by the next lock-holder. Writer A holds the lock mid-write (blocked in
|
||||||
|
/// the stream); an event write is queued and then cancelled; when A is released and a later
|
||||||
|
/// control frame drains, the wire carries A's frame and the control frame only — the cancelled
|
||||||
|
/// event envelope is tombstoned and skipped.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task WriteAsync_CancelledWhileWaitingForLock_FrameIsNeverWritten()
|
||||||
|
{
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
using GatedWriteStream stream = new();
|
||||||
|
WorkerFrameWriter writer = new(stream, options);
|
||||||
|
|
||||||
|
// Writer A occupies the writer and blocks inside the stream, holding the write lock.
|
||||||
|
Task firstWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
|
||||||
|
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
|
||||||
|
|
||||||
|
// Queue an event write with its own CTS while the lock is held, then cancel it.
|
||||||
|
using CancellationTokenSource cts = new();
|
||||||
|
Task cancelledWrite = writer.WriteAsync(CreateEventEnvelope(), WorkerFrameWritePriority.Event, cts.Token);
|
||||||
|
await Task.Delay(50);
|
||||||
|
cts.Cancel();
|
||||||
|
await Assert.ThrowsAnyAsync<OperationCanceledException>(async () => await cancelledWrite);
|
||||||
|
|
||||||
|
// Release A, then drive a fresh control write.
|
||||||
|
stream.ReleaseFirstWrite();
|
||||||
|
await AwaitWithTimeoutAsync(firstWrite);
|
||||||
|
await writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
|
||||||
|
|
||||||
|
stream.Position = 0;
|
||||||
|
WorkerFrameReader reader = new(stream, options);
|
||||||
|
WorkerEnvelope frame1 = await reader.ReadAsync();
|
||||||
|
WorkerEnvelope frame2 = await reader.ReadAsync();
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, frame1.BodyCase);
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, frame2.BodyCase);
|
||||||
|
// The cancelled event never reached the wire — no third frame, and sequences stay contiguous.
|
||||||
|
Assert.Equal(stream.Length, stream.Position);
|
||||||
|
Assert.Equal(1UL, frame1.Sequence);
|
||||||
|
Assert.Equal(2UL, frame2.Sequence);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-22 / IPC-26, the review's shutdown scenario. A cancelled event frame queued before a
|
||||||
|
/// shutdown-ack control frame must not trail the ack on the wire: the tombstone rule plus the
|
||||||
|
/// control-before-event scheduler keeps the ack the last frame written.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task WriteAsync_CancelledEventFrame_DoesNotTrailShutdownAck()
|
||||||
|
{
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
using GatedWriteStream stream = new();
|
||||||
|
WorkerFrameWriter writer = new(stream, options);
|
||||||
|
|
||||||
|
Task firstWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
|
||||||
|
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
|
||||||
|
|
||||||
|
using CancellationTokenSource cts = new();
|
||||||
|
Task cancelledEvent = writer.WriteAsync(CreateEventEnvelope(), WorkerFrameWritePriority.Event, cts.Token);
|
||||||
|
await Task.Delay(50);
|
||||||
|
cts.Cancel();
|
||||||
|
await Assert.ThrowsAnyAsync<OperationCanceledException>(async () => await cancelledEvent);
|
||||||
|
|
||||||
|
// The shutdown ack (a control frame) is queued behind the still-blocked first write.
|
||||||
|
Task ackWrite = writer.WriteAsync(CreateShutdownAckEnvelope(), WorkerFrameWritePriority.Control);
|
||||||
|
await Task.Delay(50);
|
||||||
|
|
||||||
|
stream.ReleaseFirstWrite();
|
||||||
|
await AwaitWithTimeoutAsync(Task.WhenAll(firstWrite, ackWrite));
|
||||||
|
|
||||||
|
stream.Position = 0;
|
||||||
|
WorkerFrameReader reader = new(stream, options);
|
||||||
|
WorkerEnvelope frame1 = await reader.ReadAsync();
|
||||||
|
WorkerEnvelope frame2 = await reader.ReadAsync();
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, frame1.BodyCase);
|
||||||
|
// The ack is the last frame — the cancelled event did not trail it.
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerShutdownAck, frame2.BodyCase);
|
||||||
|
Assert.Equal(stream.Length, stream.Position);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-25. The batch entry point enqueues a whole event burst under one lock acquisition and
|
||||||
|
/// drains it together, so N events cost exactly one flush and reach the wire in batch order with
|
||||||
|
/// monotonic sequences — the coalescing WRK-12 shipped, now on the event hot path.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task WriteBatchAsync_FlushesOnceAndPreservesOrder()
|
||||||
|
{
|
||||||
|
const int count = 8;
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
using FlushCountingStream stream = new();
|
||||||
|
WorkerFrameWriter writer = new(stream, options);
|
||||||
|
|
||||||
|
WorkerEnvelope[] batch = new WorkerEnvelope[count];
|
||||||
|
for (int index = 0; index < count; index++)
|
||||||
|
{
|
||||||
|
batch[index] = CreateEventEnvelope(workerSequence: (ulong)(100 + index));
|
||||||
|
}
|
||||||
|
|
||||||
|
await writer.WriteBatchAsync(batch, WorkerFrameWritePriority.Event);
|
||||||
|
|
||||||
|
// The whole batch was queued before the single lock wait, so it drained in one pass => one flush.
|
||||||
|
Assert.Equal(1, stream.FlushCount);
|
||||||
|
|
||||||
|
stream.Position = 0;
|
||||||
|
WorkerFrameReader reader = new(stream, options);
|
||||||
|
for (int index = 0; index < count; index++)
|
||||||
|
{
|
||||||
|
WorkerEnvelope frame = await reader.ReadAsync();
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, frame.BodyCase);
|
||||||
|
// Wire order matches batch order.
|
||||||
|
Assert.Equal((ulong)(100 + index), frame.WorkerEvent.Event.WorkerSequence);
|
||||||
|
// Write-time stamped sequence is monotonic 1..count.
|
||||||
|
Assert.Equal((ulong)(index + 1), frame.Sequence);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-25. Control-before-event still holds mid-batch: a control frame queued while a batch is
|
||||||
|
/// draining jumps ahead of the batch's remaining events.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task WriteBatchAsync_ControlFrameQueuedDuringBatch_JumpsRemainingEvents()
|
||||||
|
{
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
using GatedWriteStream stream = new();
|
||||||
|
WorkerFrameWriter writer = new(stream, options);
|
||||||
|
|
||||||
|
WorkerEnvelope[] batch = new[]
|
||||||
|
{
|
||||||
|
CreateEventEnvelope(),
|
||||||
|
CreateEventEnvelope(),
|
||||||
|
CreateEventEnvelope(),
|
||||||
|
};
|
||||||
|
|
||||||
|
// The batch takes the lock and blocks writing its first event frame inside the stream.
|
||||||
|
Task batchWrite = writer.WriteBatchAsync(batch, WorkerFrameWritePriority.Event);
|
||||||
|
await AwaitWithTimeoutAsync(stream.FirstWriteStarted);
|
||||||
|
|
||||||
|
// A control frame queued mid-drain must jump the batch's remaining events.
|
||||||
|
Task controlWrite = writer.WriteAsync(CreateGatewayHelloEnvelope(), WorkerFrameWritePriority.Control);
|
||||||
|
await Task.Delay(50);
|
||||||
|
|
||||||
|
stream.ReleaseFirstWrite();
|
||||||
|
await AwaitWithTimeoutAsync(Task.WhenAll(batchWrite, controlWrite));
|
||||||
|
|
||||||
|
stream.Position = 0;
|
||||||
|
WorkerFrameReader reader = new(stream, options);
|
||||||
|
WorkerEnvelope f1 = await reader.ReadAsync();
|
||||||
|
WorkerEnvelope f2 = await reader.ReadAsync();
|
||||||
|
WorkerEnvelope f3 = await reader.ReadAsync();
|
||||||
|
WorkerEnvelope f4 = await reader.ReadAsync();
|
||||||
|
|
||||||
|
// First event was already writing when the control frame queued; the control frame then jumps
|
||||||
|
// ahead of the two remaining events.
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, f1.BodyCase);
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.GatewayHello, f2.BodyCase);
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, f3.BodyCase);
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerEvent, f4.BodyCase);
|
||||||
|
}
|
||||||
|
|
||||||
private static WorkerEnvelope CreateGatewayHelloEnvelope(ulong sequence = 1)
|
private static WorkerEnvelope CreateGatewayHelloEnvelope(ulong sequence = 1)
|
||||||
{
|
{
|
||||||
return new WorkerEnvelope
|
return new WorkerEnvelope
|
||||||
@@ -522,6 +708,47 @@ public sealed class WorkerFrameProtocolTests
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static WorkerEnvelope CreateEventEnvelope(ulong workerSequence)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = CreateEventEnvelope();
|
||||||
|
envelope.WorkerEvent.Event.WorkerSequence = workerSequence;
|
||||||
|
return envelope;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static WorkerEnvelope CreateShutdownAckEnvelope()
|
||||||
|
{
|
||||||
|
return new WorkerEnvelope
|
||||||
|
{
|
||||||
|
ProtocolVersion = GatewayContractInfo.WorkerProtocolVersion,
|
||||||
|
SessionId = SessionId,
|
||||||
|
WorkerShutdownAck = new WorkerShutdownAck
|
||||||
|
{
|
||||||
|
Status = new ProtocolStatus
|
||||||
|
{
|
||||||
|
Code = ProtocolStatusCode.Ok,
|
||||||
|
Message = "OK",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// A MemoryStream that counts FlushAsync calls without gating any write, so a batch write can be
|
||||||
|
// asserted to flush exactly once.
|
||||||
|
private sealed class FlushCountingStream : MemoryStream
|
||||||
|
{
|
||||||
|
private int _flushCount;
|
||||||
|
|
||||||
|
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
|
||||||
|
public int FlushCount => Volatile.Read(ref _flushCount);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override Task FlushAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref _flushCount);
|
||||||
|
return base.FlushAsync(cancellationToken);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// A MemoryStream whose first WriteAsync blocks until released, so a test can queue additional frames
|
// A MemoryStream whose first WriteAsync blocks until released, so a test can queue additional frames
|
||||||
// behind an in-progress write and observe the writer's priority ordering.
|
// behind an in-progress write and observe the writer's priority ordering.
|
||||||
private sealed class GatedWriteStream : MemoryStream
|
private sealed class GatedWriteStream : MemoryStream
|
||||||
|
|||||||
@@ -1112,6 +1112,184 @@ public sealed class WorkerPipeSessionTests
|
|||||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-27. An STA call outside the command dispatcher (the alarm poll) advertises itself on
|
||||||
|
/// the heartbeat snapshot's <c>StaCallInProgress</c> flag, and the watchdog grants it the same
|
||||||
|
/// grace-to-ceiling suppression as a dispatched command: stale STA activity within the ceiling
|
||||||
|
/// does not fault while the flag is set, but stale activity beyond the ceiling faults anyway.
|
||||||
|
/// This closes the 15 s-vs-75 s asymmetry between polls and commands.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task Watchdog_StaCallInProgress_SuppressedUntilCeiling()
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(10));
|
||||||
|
|
||||||
|
// Phase 1 — within the ceiling: stale beyond grace, empty correlation id, StaCallInProgress
|
||||||
|
// set. The default 75 s ceiling is far beyond the 5 s staleness, so the watchdog must suppress.
|
||||||
|
using (PipePair pipePair = await PipePair.CreateAsync(cancellation.Token))
|
||||||
|
{
|
||||||
|
FakeRuntimeSession runtime = new();
|
||||||
|
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow - TimeSpan.FromSeconds(5),
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId: string.Empty,
|
||||||
|
staCallInProgress: true));
|
||||||
|
WorkerPipeSession session = CreatePipeSession(
|
||||||
|
pipePair.WorkerStream,
|
||||||
|
runtime,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
|
||||||
|
HeartbeatGrace = TimeSpan.FromMilliseconds(50),
|
||||||
|
});
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
|
||||||
|
const int framesToInspect = 6;
|
||||||
|
int heartbeatsObserved = 0;
|
||||||
|
for (int index = 0; index < framesToInspect; index++)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = await pipePair.GatewayReader.ReadAsync(cancellation.Token);
|
||||||
|
Assert.NotEqual(WorkerEnvelope.BodyOneofCase.WorkerFault, envelope.BodyCase);
|
||||||
|
if (envelope.BodyCase == WorkerEnvelope.BodyOneofCase.WorkerHeartbeat)
|
||||||
|
{
|
||||||
|
heartbeatsObserved++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.True(
|
||||||
|
heartbeatsObserved >= 2,
|
||||||
|
$"Expected multiple heartbeats during the in-progress STA-call window; observed {heartbeatsObserved}.");
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Phase 2 — beyond the ceiling: same StaCallInProgress flag, but staleness (5 s) exceeds the
|
||||||
|
// 200 ms ceiling, so the watchdog must fire even with the poll in progress.
|
||||||
|
using (PipePair pipePair = await PipePair.CreateAsync(cancellation.Token))
|
||||||
|
{
|
||||||
|
FakeRuntimeSession runtime = new();
|
||||||
|
runtime.SetSnapshot(new WorkerRuntimeHeartbeatSnapshot(
|
||||||
|
DateTimeOffset.UtcNow - TimeSpan.FromSeconds(5),
|
||||||
|
pendingCommandCount: 0,
|
||||||
|
outboundEventQueueDepth: 0,
|
||||||
|
lastEventSequence: 0,
|
||||||
|
currentCommandCorrelationId: string.Empty,
|
||||||
|
staCallInProgress: true));
|
||||||
|
WorkerPipeSession session = CreatePipeSession(
|
||||||
|
pipePair.WorkerStream,
|
||||||
|
runtime,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromMilliseconds(20),
|
||||||
|
HeartbeatGrace = TimeSpan.FromMilliseconds(50),
|
||||||
|
HeartbeatStuckCeiling = TimeSpan.FromMilliseconds(200),
|
||||||
|
});
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
|
||||||
|
WorkerEnvelope fault = await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerFault,
|
||||||
|
cancellation.Token);
|
||||||
|
|
||||||
|
Assert.Equal(WorkerFaultCategory.StaHung, fault.WorkerFault.Category);
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-25. The event drain loop submits a whole drained batch through the writer's batch entry
|
||||||
|
/// point, so a burst of 128 events costs one flush, not 128 — the assertion the WRK-12
|
||||||
|
/// tracking claim needed to actually hold on the event hot path.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task EventBurst_DrainLoopCoalescesFlushes()
|
||||||
|
{
|
||||||
|
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(15));
|
||||||
|
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||||
|
FakeRuntimeSession runtime = new();
|
||||||
|
// A far-off heartbeat interval keeps heartbeat flushes out of the measurement window.
|
||||||
|
FlushCountingPassthroughStream countingStream = new(pipePair.WorkerStream);
|
||||||
|
WorkerPipeSession session = CreatePipeSession(
|
||||||
|
countingStream,
|
||||||
|
runtime,
|
||||||
|
new WorkerPipeSessionOptions
|
||||||
|
{
|
||||||
|
HeartbeatInterval = TimeSpan.FromMinutes(5),
|
||||||
|
HeartbeatGrace = TimeSpan.FromSeconds(30),
|
||||||
|
});
|
||||||
|
Task runTask = session.RunAsync(cancellation.Token);
|
||||||
|
await CompleteGatewayHandshakeAsync(pipePair, cancellation.Token);
|
||||||
|
|
||||||
|
// Let the idle drain loop settle (no events yet → no flushes) and record the baseline.
|
||||||
|
await Task.Delay(100, cancellation.Token);
|
||||||
|
int baselineFlushes = countingStream.FlushCount;
|
||||||
|
|
||||||
|
// Enqueue a full 128-event batch atomically so the drain loop sees it as one batch.
|
||||||
|
const int burst = 128;
|
||||||
|
List<WorkerEvent> batch = new(burst);
|
||||||
|
for (int index = 0; index < burst; index++)
|
||||||
|
{
|
||||||
|
batch.Add(CreateWorkerEvent(sequence: (ulong)(index + 1)));
|
||||||
|
}
|
||||||
|
|
||||||
|
runtime.EnqueueEvents(batch);
|
||||||
|
|
||||||
|
// Drain all 128 events off the gateway side.
|
||||||
|
for (int index = 0; index < burst; index++)
|
||||||
|
{
|
||||||
|
await ReadUntilAsync(
|
||||||
|
pipePair.GatewayReader,
|
||||||
|
WorkerEnvelope.BodyOneofCase.WorkerEvent,
|
||||||
|
cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The whole burst cost exactly one additional flush.
|
||||||
|
Assert.Equal(1, countingStream.FlushCount - baselineFlushes);
|
||||||
|
|
||||||
|
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-24. A GatewayHello negotiating a frame maximum below the worker floor faults at the
|
||||||
|
/// handshake with a fault frame rather than being adopted — mirroring the above-ceiling
|
||||||
|
/// handshake behavior — so a nonsensical tiny value never leaves a session that fails every
|
||||||
|
/// later frame. No message loop is entered.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task Handshake_GatewayHelloWithTinyMaxFrameBytes_FaultsAtHandshake()
|
||||||
|
{
|
||||||
|
WorkerFrameProtocolOptions options = CreateOptions();
|
||||||
|
using MemoryStream inbound = new();
|
||||||
|
await new WorkerFrameWriter(inbound, options)
|
||||||
|
.WriteAsync(CreateGatewayHelloEnvelope(maxFrameBytes: 512));
|
||||||
|
inbound.Position = 0;
|
||||||
|
using MemoryStream outbound = new();
|
||||||
|
WorkerPipeSession session = CreateSession(inbound, outbound, options);
|
||||||
|
bool initialized = false;
|
||||||
|
|
||||||
|
WorkerFrameProtocolException exception =
|
||||||
|
await Assert.ThrowsAsync<WorkerFrameProtocolException>(
|
||||||
|
async () => await session.CompleteStartupHandshakeAsync(
|
||||||
|
_ =>
|
||||||
|
{
|
||||||
|
initialized = true;
|
||||||
|
return Task.CompletedTask;
|
||||||
|
}));
|
||||||
|
|
||||||
|
Assert.False(initialized);
|
||||||
|
Assert.Equal(WorkerFrameProtocolErrorCode.InvalidConfiguration, exception.ErrorCode);
|
||||||
|
WorkerEnvelope fault = Assert.Single(ReadWrittenFrames(outbound, options));
|
||||||
|
Assert.Equal(WorkerEnvelope.BodyOneofCase.WorkerFault, fault.BodyCase);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Regression test: a long in-flight STA command that keeps pumping
|
/// Regression test: a long in-flight STA command that keeps pumping
|
||||||
/// must NOT self-fault as <c>StaHung</c>, and its reply must still be
|
/// must NOT self-fault as <c>StaHung</c>, and its reply must still be
|
||||||
@@ -1976,6 +2154,78 @@ public sealed class WorkerPipeSessionTests
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Wraps the worker side of the pipe and counts FlushAsync calls so a test can assert the event
|
||||||
|
// drain loop coalesces a burst into a single flush. Delegates every other operation to the inner
|
||||||
|
// stream; does not own the inner stream's lifetime (PipePair disposes it).
|
||||||
|
private sealed class FlushCountingPassthroughStream : Stream
|
||||||
|
{
|
||||||
|
private readonly Stream inner;
|
||||||
|
private int flushCount;
|
||||||
|
|
||||||
|
/// <summary>Initializes the passthrough over the given inner stream.</summary>
|
||||||
|
/// <param name="inner">The stream to delegate to.</param>
|
||||||
|
public FlushCountingPassthroughStream(Stream inner)
|
||||||
|
{
|
||||||
|
this.inner = inner;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Gets the number of <see cref="FlushAsync"/> calls observed so far.</summary>
|
||||||
|
public int FlushCount => Volatile.Read(ref flushCount);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override bool CanRead => inner.CanRead;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override bool CanSeek => inner.CanSeek;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override bool CanWrite => inner.CanWrite;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override long Length => inner.Length;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override long Position
|
||||||
|
{
|
||||||
|
get => inner.Position;
|
||||||
|
set => inner.Position = value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override void Flush()
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref flushCount);
|
||||||
|
inner.Flush();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override Task FlushAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
Interlocked.Increment(ref flushCount);
|
||||||
|
return inner.FlushAsync(cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override int Read(byte[] buffer, int offset, int count) => inner.Read(buffer, offset, count);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
|
||||||
|
=> inner.ReadAsync(buffer, offset, count, cancellationToken);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override long Seek(long offset, SeekOrigin origin) => inner.Seek(offset, origin);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override void SetLength(long value) => inner.SetLength(value);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override void Write(byte[] buffer, int offset, int count) => inner.Write(buffer, offset, count);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
|
||||||
|
=> inner.WriteAsync(buffer, offset, count, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
private sealed class PipePair : IDisposable
|
private sealed class PipePair : IDisposable
|
||||||
{
|
{
|
||||||
private readonly NamedPipeServerStream gatewayStream;
|
private readonly NamedPipeServerStream gatewayStream;
|
||||||
|
|||||||
@@ -438,6 +438,53 @@ public sealed class MxAccessStaSessionTests
|
|||||||
Assert.Contains("alarm poll failed", fault.DiagnosticMessage, StringComparison.OrdinalIgnoreCase);
|
Assert.Contains("alarm poll failed", fault.DiagnosticMessage, StringComparison.OrdinalIgnoreCase);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// WRK-27. While the alarm poll's PollOnce is executing on the STA, a heartbeat captured mid-poll
|
||||||
|
/// must report <see cref="WorkerRuntimeHeartbeatSnapshot.StaCallInProgress"/> so the watchdog
|
||||||
|
/// grants the poll the same grace-to-ceiling suppression as a dispatched command; once the poll
|
||||||
|
/// returns the flag clears. PollOnce is blocked on a gate so the heartbeat can be captured while
|
||||||
|
/// the STA call is genuinely in flight.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task CaptureHeartbeat_DuringAlarmPoll_ReportsStaCallInProgress()
|
||||||
|
{
|
||||||
|
FakeAlarmCommandHandler handler = new() { BlockPoll = true };
|
||||||
|
FakeMxAccessComObjectFactory factory = new();
|
||||||
|
FakeMxAccessEventSink eventSink = new();
|
||||||
|
using StaRuntime runtime = CreateRuntime();
|
||||||
|
using MxAccessStaSession session = new(
|
||||||
|
runtime,
|
||||||
|
factory,
|
||||||
|
eventSink,
|
||||||
|
new MxAccessEventQueue(),
|
||||||
|
(_eq, _affinity, _comFactory) => handler);
|
||||||
|
|
||||||
|
await session.StartAsync("session-1", workerProcessId: 1);
|
||||||
|
|
||||||
|
// Wait until PollOnce is blocked mid-call on the STA thread.
|
||||||
|
Assert.True(
|
||||||
|
handler.WaitForPollEntered(TimeSpan.FromSeconds(5)),
|
||||||
|
"Expected the alarm poll to start within 5 seconds.");
|
||||||
|
|
||||||
|
// Captured mid-poll, the heartbeat advertises the in-progress STA call.
|
||||||
|
Assert.True(session.CaptureHeartbeat().StaCallInProgress);
|
||||||
|
|
||||||
|
// Release the poll and stop blocking; the flag clears once the poll returns.
|
||||||
|
handler.BlockPoll = false;
|
||||||
|
handler.ReleasePoll();
|
||||||
|
|
||||||
|
using CancellationTokenSource timeout = new(TimeSpan.FromSeconds(5));
|
||||||
|
while (session.CaptureHeartbeat().StaCallInProgress && !timeout.IsCancellationRequested)
|
||||||
|
{
|
||||||
|
await Task.Delay(25, CancellationToken.None);
|
||||||
|
}
|
||||||
|
|
||||||
|
Assert.False(
|
||||||
|
session.CaptureHeartbeat().StaCallInProgress,
|
||||||
|
"Expected StaCallInProgress to clear once the alarm poll returned.");
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// The STA-affinity guard throws when an
|
/// The STA-affinity guard throws when an
|
||||||
/// IMxAccessAlarmConsumer call is attempted off the thread that created
|
/// IMxAccessAlarmConsumer call is attempted off the thread that created
|
||||||
@@ -472,6 +519,8 @@ public sealed class MxAccessStaSessionTests
|
|||||||
private sealed class FakeAlarmCommandHandler : IAlarmCommandHandler
|
private sealed class FakeAlarmCommandHandler : IAlarmCommandHandler
|
||||||
{
|
{
|
||||||
private readonly object gate = new object();
|
private readonly object gate = new object();
|
||||||
|
private readonly ManualResetEventSlim pollEntered = new(false);
|
||||||
|
private readonly ManualResetEventSlim releasePoll = new(false);
|
||||||
private int pollCount;
|
private int pollCount;
|
||||||
private int? lastPollThreadId;
|
private int? lastPollThreadId;
|
||||||
|
|
||||||
@@ -484,6 +533,17 @@ public sealed class MxAccessStaSessionTests
|
|||||||
/// <summary>Exception thrown by PollOnce; null to succeed.</summary>
|
/// <summary>Exception thrown by PollOnce; null to succeed.</summary>
|
||||||
public Exception? PollException { get; set; }
|
public Exception? PollException { get; set; }
|
||||||
|
|
||||||
|
/// <summary>When set, <see cref="PollOnce"/> blocks until <see cref="ReleasePoll"/> is called.</summary>
|
||||||
|
public bool BlockPoll { get; set; }
|
||||||
|
|
||||||
|
/// <summary>Waits until a blocking <see cref="PollOnce"/> has entered and is blocked.</summary>
|
||||||
|
/// <param name="timeout">Maximum time to wait.</param>
|
||||||
|
/// <returns>True if a poll entered within the timeout.</returns>
|
||||||
|
public bool WaitForPollEntered(TimeSpan timeout) => pollEntered.Wait(timeout);
|
||||||
|
|
||||||
|
/// <summary>Releases a <see cref="PollOnce"/> blocked on the gate.</summary>
|
||||||
|
public void ReleasePoll() => releasePoll.Set();
|
||||||
|
|
||||||
/// <summary>Gets the count of PollOnce calls.</summary>
|
/// <summary>Gets the count of PollOnce calls.</summary>
|
||||||
public int PollCount
|
public int PollCount
|
||||||
{
|
{
|
||||||
@@ -533,6 +593,12 @@ public sealed class MxAccessStaSessionTests
|
|||||||
lastPollThreadId = Thread.CurrentThread.ManagedThreadId;
|
lastPollThreadId = Thread.CurrentThread.ManagedThreadId;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (BlockPoll)
|
||||||
|
{
|
||||||
|
pollEntered.Set();
|
||||||
|
releasePoll.Wait(TimeSpan.FromSeconds(10));
|
||||||
|
}
|
||||||
|
|
||||||
if (PollException is not null)
|
if (PollException is not null)
|
||||||
{
|
{
|
||||||
throw PollException;
|
throw PollException;
|
||||||
@@ -540,6 +606,10 @@ public sealed class MxAccessStaSessionTests
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public void Dispose() { }
|
public void Dispose()
|
||||||
|
{
|
||||||
|
pollEntered.Dispose();
|
||||||
|
releasePoll.Dispose();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -372,6 +372,23 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Enqueues a batch of worker events atomically under one lock so the drain loop cannot
|
||||||
|
/// observe a partial batch. Lets a test assert the drain loop coalesces a whole batch into one
|
||||||
|
/// flush (WRK-25) without racing a mid-enqueue drain that would split the batch.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="workerEvents">The events to enqueue in order.</param>
|
||||||
|
public void EnqueueEvents(IEnumerable<WorkerEvent> workerEvents)
|
||||||
|
{
|
||||||
|
lock (gate)
|
||||||
|
{
|
||||||
|
foreach (WorkerEvent workerEvent in workerEvents)
|
||||||
|
{
|
||||||
|
events.Enqueue(workerEvent);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public void Dispose()
|
public void Dispose()
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -18,6 +18,17 @@ public sealed class WorkerFrameProtocolOptions
|
|||||||
/// </summary>
|
/// </summary>
|
||||||
public const int MaxNegotiableFrameBytes = 256 * 1024 * 1024;
|
public const int MaxNegotiableFrameBytes = 256 * 1024 * 1024;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Lower floor the worker will accept for a gateway-negotiated frame maximum
|
||||||
|
/// (<c>GatewayHello.max_frame_bytes</c>). Matches the gateway's own
|
||||||
|
/// <c>GatewayOptionsValidator.MinimumMaxMessageBytes</c> validation floor so the worker never
|
||||||
|
/// rejects a value the gateway's own validator accepts as legal configuration, yet a nonsensical
|
||||||
|
/// tiny value (a gateway bug or a foreign/old peer) is rejected at the handshake rather than
|
||||||
|
/// leaving a session that handshakes cleanly and then fails every subsequent frame with
|
||||||
|
/// per-frame size errors. 1024 bytes still guarantees hellos, heartbeats, acks, and faults fit.
|
||||||
|
/// </summary>
|
||||||
|
public const int MinNegotiableFrameBytes = 1024;
|
||||||
|
|
||||||
/// <summary>Initializes a new instance of the WorkerFrameProtocolOptions class from WorkerOptions.</summary>
|
/// <summary>Initializes a new instance of the WorkerFrameProtocolOptions class from WorkerOptions.</summary>
|
||||||
/// <param name="options">Worker initialization options.</param>
|
/// <param name="options">Worker initialization options.</param>
|
||||||
public WorkerFrameProtocolOptions(WorkerOptions options)
|
public WorkerFrameProtocolOptions(WorkerOptions options)
|
||||||
@@ -118,7 +129,9 @@ public sealed class WorkerFrameProtocolOptions
|
|||||||
/// <summary>
|
/// <summary>
|
||||||
/// Adopts the gateway-negotiated frame maximum conveyed in <c>GatewayHello.max_frame_bytes</c>.
|
/// Adopts the gateway-negotiated frame maximum conveyed in <c>GatewayHello.max_frame_bytes</c>.
|
||||||
/// A value of 0 (an older gateway that never set the field) is ignored and the
|
/// A value of 0 (an older gateway that never set the field) is ignored and the
|
||||||
/// constructor default is kept. A value above <see cref="MaxNegotiableFrameBytes"/> is rejected.
|
/// constructor default is kept. A value outside the accepted range
|
||||||
|
/// [<see cref="MinNegotiableFrameBytes"/>, <see cref="MaxNegotiableFrameBytes"/>] is rejected so a
|
||||||
|
/// nonsensical negotiated value faults at the handshake rather than mid-session.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="negotiatedMaxFrameBytes">The gateway-negotiated maximum, or 0 for "keep default".</param>
|
/// <param name="negotiatedMaxFrameBytes">The gateway-negotiated maximum, or 0 for "keep default".</param>
|
||||||
internal void AdoptNegotiatedMaxMessageBytes(uint negotiatedMaxFrameBytes)
|
internal void AdoptNegotiatedMaxMessageBytes(uint negotiatedMaxFrameBytes)
|
||||||
@@ -128,6 +141,14 @@ public sealed class WorkerFrameProtocolOptions
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (negotiatedMaxFrameBytes < MinNegotiableFrameBytes)
|
||||||
|
{
|
||||||
|
throw new WorkerFrameProtocolException(
|
||||||
|
WorkerFrameProtocolErrorCode.InvalidConfiguration,
|
||||||
|
$"GatewayHello negotiated frame maximum {negotiatedMaxFrameBytes} is below the worker floor "
|
||||||
|
+ $"of {MinNegotiableFrameBytes} bytes.");
|
||||||
|
}
|
||||||
|
|
||||||
if (negotiatedMaxFrameBytes > MaxNegotiableFrameBytes)
|
if (negotiatedMaxFrameBytes > MaxNegotiableFrameBytes)
|
||||||
{
|
{
|
||||||
throw new WorkerFrameProtocolException(
|
throw new WorkerFrameProtocolException(
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
using System;
|
using System;
|
||||||
using System.Collections.Generic;
|
using System.Collections.Generic;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
|
using System.Runtime.ExceptionServices;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Google.Protobuf;
|
using Google.Protobuf;
|
||||||
@@ -33,6 +34,16 @@ public sealed class WorkerFrameWriter
|
|||||||
|
|
||||||
/// <summary>Gets the completion source signaled once the frame has been written or has failed.</summary>
|
/// <summary>Gets the completion source signaled once the frame has been written or has failed.</summary>
|
||||||
public TaskCompletionSource<bool> Completion { get; }
|
public TaskCompletionSource<bool> Completion { get; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Set to <c>true</c> by <see cref="DequeueNext"/> — under <c>_gate</c> — at the instant the
|
||||||
|
/// draining lock-holder takes ownership of this frame to write it. A cancelled caller
|
||||||
|
/// tombstones its frame only while it is still unclaimed, so a claim and a cancel can never
|
||||||
|
/// both win: the flag is the interlock between the two. A frame already claimed is mid-write
|
||||||
|
/// and can no longer be recalled (see <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>).
|
||||||
|
/// Mutated only under <c>_gate</c>.
|
||||||
|
/// </summary>
|
||||||
|
public bool Claimed;
|
||||||
}
|
}
|
||||||
|
|
||||||
private readonly WorkerFrameProtocolOptions _options;
|
private readonly WorkerFrameProtocolOptions _options;
|
||||||
@@ -75,6 +86,13 @@ public sealed class WorkerFrameWriter
|
|||||||
/// <param name="priority">Scheduling priority; control frames are written ahead of event frames.</param>
|
/// <param name="priority">Scheduling priority; control frames are written ahead of event frames.</param>
|
||||||
/// <param name="cancellationToken">Token to cancel waiting for the write lock.</param>
|
/// <param name="cancellationToken">Token to cancel waiting for the write lock.</param>
|
||||||
/// <returns>A task that completes when the frame has been written and flushed.</returns>
|
/// <returns>A task that completes when the frame has been written and flushed.</returns>
|
||||||
|
/// <remarks>
|
||||||
|
/// Cancellation contract (WRK-22): if the token fires while this call is waiting for the write
|
||||||
|
/// lock, the frame is tombstoned so it is never written — unless a draining lock-holder has
|
||||||
|
/// already claimed it, in which case the frame may still reach the wire even though this call
|
||||||
|
/// observes <see cref="OperationCanceledException"/>. That residual window is by design: blocking
|
||||||
|
/// the canceller behind the very write it is abandoning would defeat the point of cancellation.
|
||||||
|
/// </remarks>
|
||||||
public async Task WriteAsync(
|
public async Task WriteAsync(
|
||||||
WorkerEnvelope envelope,
|
WorkerEnvelope envelope,
|
||||||
WorkerFrameWritePriority priority,
|
WorkerFrameWritePriority priority,
|
||||||
@@ -101,7 +119,19 @@ public sealed class WorkerFrameWriter
|
|||||||
// Contend for the single writer: whoever wins drains every currently-queued frame in priority
|
// Contend for the single writer: whoever wins drains every currently-queued frame in priority
|
||||||
// order, so this frame is written by this call or by a concurrent caller that got the lock
|
// order, so this frame is written by this call or by a concurrent caller that got the lock
|
||||||
// first. Either way it completes via its own TaskCompletionSource.
|
// first. Either way it completes via its own TaskCompletionSource.
|
||||||
|
try
|
||||||
|
{
|
||||||
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
|
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
// Tombstone the queued frame so DequeueNext skips it — but only if a draining lock-holder
|
||||||
|
// has not already claimed it. If it is claimed it is mid-write and cannot be recalled; the
|
||||||
|
// caller still observes cancellation while the frame reaches the wire (documented above).
|
||||||
|
TombstoneIfUnclaimed(frame, cancellationToken);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await DrainQueuedFramesAsync().ConfigureAwait(false);
|
await DrainQueuedFramesAsync().ConfigureAwait(false);
|
||||||
@@ -114,6 +144,123 @@ public sealed class WorkerFrameWriter
|
|||||||
await frame.Completion.Task.ConfigureAwait(false);
|
await frame.Completion.Task.ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Queues a whole batch of envelopes at one priority under a single lock acquisition and drains
|
||||||
|
/// it, so a burst of frames — the event drain loop's hot path — pays one flush for the batch
|
||||||
|
/// rather than one per frame (WRK-25, realizing the WRK-12 coalescing on the path it was built
|
||||||
|
/// for). Intra-batch order is preserved because the enqueue is atomic under <c>_gate</c> and each
|
||||||
|
/// class queue is FIFO; the control-before-event guarantee still holds because any concurrently
|
||||||
|
/// queued control frame is drained ahead of this batch by <see cref="DequeueNext"/>. Every frame's
|
||||||
|
/// "written and flushed before completion" contract is unchanged.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="envelopes">Envelopes to write, in order.</param>
|
||||||
|
/// <param name="priority">Scheduling priority for the whole batch.</param>
|
||||||
|
/// <param name="cancellationToken">Token to cancel waiting for the write lock.</param>
|
||||||
|
/// <returns>A task that completes when every frame in the batch has been written and flushed.</returns>
|
||||||
|
/// <remarks>
|
||||||
|
/// A per-frame rejection inside the batch (for example one oversized event) surfaces from the
|
||||||
|
/// awaited completions as its <see cref="WorkerFrameProtocolException"/>; the remaining frames are
|
||||||
|
/// still observed so none faults unobserved. Cancellation while waiting for the lock tombstones
|
||||||
|
/// every still-unclaimed frame in the batch, per the WRK-22 contract on
|
||||||
|
/// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>.
|
||||||
|
/// </remarks>
|
||||||
|
public async Task WriteBatchAsync(
|
||||||
|
IReadOnlyList<WorkerEnvelope> envelopes,
|
||||||
|
WorkerFrameWritePriority priority,
|
||||||
|
CancellationToken cancellationToken = default)
|
||||||
|
{
|
||||||
|
if (envelopes is null)
|
||||||
|
{
|
||||||
|
throw new ArgumentNullException(nameof(envelopes));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (envelopes.Count == 0)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
PendingFrame[] frames = new PendingFrame[envelopes.Count];
|
||||||
|
for (int index = 0; index < envelopes.Count; index++)
|
||||||
|
{
|
||||||
|
WorkerEnvelope envelope = envelopes[index]
|
||||||
|
?? throw new ArgumentException("Batch envelopes must not contain null.", nameof(envelopes));
|
||||||
|
frames[index] = new PendingFrame(envelope);
|
||||||
|
}
|
||||||
|
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
Queue<PendingFrame> queue = priority == WorkerFrameWritePriority.Event ? _eventFrames : _controlFrames;
|
||||||
|
foreach (PendingFrame frame in frames)
|
||||||
|
{
|
||||||
|
queue.Enqueue(frame);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (OperationCanceledException)
|
||||||
|
{
|
||||||
|
TombstoneUnclaimed(frames, cancellationToken);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await DrainQueuedFramesAsync().ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_writeLock.Release();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Await every completion so no per-frame rejection faults unobserved, but surface the first
|
||||||
|
// failure (in batch order) to the caller — the drain loop maps it back to the offending event.
|
||||||
|
Exception? firstFailure = null;
|
||||||
|
foreach (PendingFrame frame in frames)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await frame.Completion.Task.ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception exception)
|
||||||
|
{
|
||||||
|
firstFailure ??= exception;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (firstFailure is not null)
|
||||||
|
{
|
||||||
|
ExceptionDispatchInfo.Capture(firstFailure).Throw();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void TombstoneIfUnclaimed(PendingFrame frame, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
if (!frame.Claimed)
|
||||||
|
{
|
||||||
|
frame.Completion.TrySetCanceled(cancellationToken);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void TombstoneUnclaimed(PendingFrame[] frames, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
lock (_gate)
|
||||||
|
{
|
||||||
|
foreach (PendingFrame frame in frames)
|
||||||
|
{
|
||||||
|
if (!frame.Claimed)
|
||||||
|
{
|
||||||
|
frame.Completion.TrySetCanceled(cancellationToken);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Runs only under _writeLock. Drains control frames before event frames, stamping and writing each.
|
// Runs only under _writeLock. Drains control frames before event frames, stamping and writing each.
|
||||||
// The stream write itself is not cancellable: a frame is written atomically or fails, never left
|
// The stream write itself is not cancellable: a frame is written atomically or fails, never left
|
||||||
// half-written on the pipe because a caller gave up waiting.
|
// half-written on the pipe because a caller gave up waiting.
|
||||||
@@ -198,18 +345,36 @@ public sealed class WorkerFrameWriter
|
|||||||
or WorkerFrameProtocolErrorCode.SessionMismatch;
|
or WorkerFrameProtocolErrorCode.SessionMismatch;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Returns the next frame to write, control frames first, skipping any frame a cancelled caller
|
||||||
|
// tombstoned while it waited for the lock (WRK-22). The frame actually returned is marked Claimed
|
||||||
|
// under _gate in the same critical section that checks the tombstone, so a claim and a concurrent
|
||||||
|
// cancel are mutually exclusive: whichever acquires _gate first wins.
|
||||||
private PendingFrame? DequeueNext()
|
private PendingFrame? DequeueNext()
|
||||||
{
|
{
|
||||||
lock (_gate)
|
lock (_gate)
|
||||||
{
|
{
|
||||||
if (_controlFrames.Count > 0)
|
while (_controlFrames.Count > 0)
|
||||||
{
|
{
|
||||||
return _controlFrames.Dequeue();
|
PendingFrame frame = _controlFrames.Dequeue();
|
||||||
|
if (frame.Completion.Task.IsCanceled)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_eventFrames.Count > 0)
|
frame.Claimed = true;
|
||||||
|
return frame;
|
||||||
|
}
|
||||||
|
|
||||||
|
while (_eventFrames.Count > 0)
|
||||||
{
|
{
|
||||||
return _eventFrames.Dequeue();
|
PendingFrame frame = _eventFrames.Dequeue();
|
||||||
|
if (frame.Completion.Task.IsCanceled)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
frame.Claimed = true;
|
||||||
|
return frame;
|
||||||
}
|
}
|
||||||
|
|
||||||
return null;
|
return null;
|
||||||
|
|||||||
@@ -371,25 +371,55 @@ public sealed class WorkerPipeSession
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
foreach (WorkerEvent workerEvent in events)
|
// Submit the whole drained batch through the writer's batch entry point under one lock
|
||||||
|
// acquisition so the burst pays a single flush instead of one per event (WRK-25). Events
|
||||||
|
// are the low-priority frame class: the writer holds them behind any pending control frame
|
||||||
|
// (reply, fault, heartbeat, shutdown ack) so those are not delayed behind an event backlog,
|
||||||
|
// and intra-batch order is preserved.
|
||||||
|
WorkerEnvelope[] envelopes = new WorkerEnvelope[events.Count];
|
||||||
|
for (int index = 0; index < events.Count; index++)
|
||||||
{
|
{
|
||||||
// Events are the low-priority frame class: the writer holds them behind any pending
|
envelopes[index] = CreateEnvelope(events[index]);
|
||||||
// control frame (reply, fault, heartbeat, shutdown ack) so those are not delayed
|
}
|
||||||
// behind an event backlog.
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await _writer
|
await _writer
|
||||||
.WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
|
.WriteBatchAsync(envelopes, WorkerFrameWritePriority.Event, cancellationToken)
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
catch (WorkerFrameProtocolException exception)
|
catch (WorkerFrameProtocolException exception)
|
||||||
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
|
||||||
{
|
{
|
||||||
await FaultOnOversizedEventAsync(workerEvent, exception, cancellationToken)
|
// A single oversized event surfaces from the batch's awaited completions; the death is
|
||||||
|
// still IPC-30's structured, event-naming fault. Map the rejection back to the first
|
||||||
|
// event in batch order whose envelope overshoots the negotiated maximum — the same
|
||||||
|
// frame the writer rejected first.
|
||||||
|
await FaultOnOversizedEventAsync(
|
||||||
|
FindOversizedEvent(events, envelopes),
|
||||||
|
exception,
|
||||||
|
cancellationToken)
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private WorkerEvent FindOversizedEvent(
|
||||||
|
IReadOnlyList<WorkerEvent> events,
|
||||||
|
WorkerEnvelope[] envelopes)
|
||||||
|
{
|
||||||
|
for (int index = 0; index < envelopes.Length; index++)
|
||||||
|
{
|
||||||
|
if (envelopes[index].CalculateSize() > _options.MaxMessageBytes)
|
||||||
|
{
|
||||||
|
return events[index];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Unreachable in practice: WriteBatchAsync surfaced MessageTooLarge, so at least one envelope
|
||||||
|
// exceeded the negotiated maximum. Fall back to the first event so the fault still names a
|
||||||
|
// concrete event rather than throwing a second, less useful exception from the fault path.
|
||||||
|
return events[0];
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
@@ -1085,16 +1115,16 @@ public sealed class WorkerPipeSession
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (!string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId)
|
if ((!string.IsNullOrEmpty(snapshot.CurrentCommandCorrelationId) || snapshot.StaCallInProgress)
|
||||||
&& staleFor <= _sessionOptions.HeartbeatStuckCeiling)
|
&& staleFor <= _sessionOptions.HeartbeatStuckCeiling)
|
||||||
{
|
{
|
||||||
// A command is in flight and we are still within the defensive
|
// A command is in flight, or an STA call outside the dispatcher (the alarm poll, WRK-27) is
|
||||||
// suppression ceiling — the STA is busy executing it, not
|
// executing, and we are still within the defensive suppression ceiling — the STA is busy
|
||||||
// hung. The next MarkActivity() in StaRuntime.ProcessQueuedCommands
|
// doing that work, not hung. The next MarkActivity() in StaRuntime.ProcessQueuedCommands
|
||||||
// will refresh LastActivityUtc once the command returns, at which
|
// will refresh LastActivityUtc once the work returns, at which point this branch stops
|
||||||
// point this branch stops being taken. The heartbeat already
|
// being taken. The heartbeat already surfaces the in-flight correlation id so the gateway
|
||||||
// surfaces the in-flight correlation id so the gateway can apply
|
// can apply its own per-command timeout if it considers the command too slow; a poll that
|
||||||
// its own per-command timeout if it considers the command too slow.
|
// blocks the STA past the ceiling still faults, which is the ceiling's contract.
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -24,6 +24,14 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
private CancellationTokenSource? alarmPollCts;
|
private CancellationTokenSource? alarmPollCts;
|
||||||
private Task? alarmPollTask;
|
private Task? alarmPollTask;
|
||||||
private int? alarmConsumerThreadId;
|
private int? alarmConsumerThreadId;
|
||||||
|
|
||||||
|
// True on the STA thread exactly around the alarm PollOnce COM call. The alarm poll runs outside
|
||||||
|
// the StaCommandDispatcher (so it does not inflate PendingCommandCount or perturb command dispatch
|
||||||
|
// ordering), which means CaptureHeartbeat would otherwise see no in-flight activity during a long
|
||||||
|
// poll and the watchdog would fault the session at the 15 s grace instead of the 75 s ceiling
|
||||||
|
// granted to dispatched commands. Surfacing the poll on the heartbeat closes that asymmetry
|
||||||
|
// (WRK-27). Volatile: written on the STA thread, read on the heartbeat thread.
|
||||||
|
private volatile bool staAlarmPollInProgress;
|
||||||
private bool disposed;
|
private bool disposed;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
@@ -246,9 +254,21 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
{
|
{
|
||||||
await staRuntime.InvokeAsync(
|
await staRuntime.InvokeAsync(
|
||||||
() =>
|
() =>
|
||||||
|
{
|
||||||
|
// Advertise the poll to the watchdog for exactly the span of the COM call
|
||||||
|
// (WRK-27): set on the STA thread immediately before the affinity check and
|
||||||
|
// PollOnce, clear in the finally so a heartbeat captured mid-poll reports
|
||||||
|
// StaCallInProgress and one captured after does not.
|
||||||
|
staAlarmPollInProgress = true;
|
||||||
|
try
|
||||||
{
|
{
|
||||||
EnsureOnAlarmConsumerThread();
|
EnsureOnAlarmConsumerThread();
|
||||||
handler.PollOnce();
|
handler.PollOnce();
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
staAlarmPollInProgress = false;
|
||||||
|
}
|
||||||
},
|
},
|
||||||
cancellationToken).ConfigureAwait(false);
|
cancellationToken).ConfigureAwait(false);
|
||||||
}
|
}
|
||||||
@@ -377,7 +397,8 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
|
|||||||
pendingCommandCount,
|
pendingCommandCount,
|
||||||
(uint)eventQueue.Count,
|
(uint)eventQueue.Count,
|
||||||
eventQueue.LastEventSequence,
|
eventQueue.LastEventSequence,
|
||||||
currentCommandCorrelationId);
|
currentCommandCorrelationId,
|
||||||
|
staAlarmPollInProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
|
|||||||
@@ -10,18 +10,26 @@ public sealed class WorkerRuntimeHeartbeatSnapshot
|
|||||||
/// <param name="outboundEventQueueDepth">Current depth of the worker event queue.</param>
|
/// <param name="outboundEventQueueDepth">Current depth of the worker event queue.</param>
|
||||||
/// <param name="lastEventSequence">Sequence number of the most recent event.</param>
|
/// <param name="lastEventSequence">Sequence number of the most recent event.</param>
|
||||||
/// <param name="currentCommandCorrelationId">Correlation ID of the in-flight command.</param>
|
/// <param name="currentCommandCorrelationId">Correlation ID of the in-flight command.</param>
|
||||||
|
/// <param name="staCallInProgress">
|
||||||
|
/// True while an STA call outside the command dispatcher is executing (currently the alarm poll,
|
||||||
|
/// WRK-27). The watchdog treats this like an in-flight command: it suppresses the stale-STA fault
|
||||||
|
/// up to the stuck ceiling instead of the shorter grace, so a healthy-but-slow poll does not fault
|
||||||
|
/// a healthy session. Named generically so any future non-dispatcher STA work reuses it.
|
||||||
|
/// </param>
|
||||||
public WorkerRuntimeHeartbeatSnapshot(
|
public WorkerRuntimeHeartbeatSnapshot(
|
||||||
DateTimeOffset lastStaActivityUtc,
|
DateTimeOffset lastStaActivityUtc,
|
||||||
uint pendingCommandCount,
|
uint pendingCommandCount,
|
||||||
uint outboundEventQueueDepth,
|
uint outboundEventQueueDepth,
|
||||||
ulong lastEventSequence,
|
ulong lastEventSequence,
|
||||||
string currentCommandCorrelationId)
|
string currentCommandCorrelationId,
|
||||||
|
bool staCallInProgress = false)
|
||||||
{
|
{
|
||||||
LastStaActivityUtc = lastStaActivityUtc;
|
LastStaActivityUtc = lastStaActivityUtc;
|
||||||
PendingCommandCount = pendingCommandCount;
|
PendingCommandCount = pendingCommandCount;
|
||||||
OutboundEventQueueDepth = outboundEventQueueDepth;
|
OutboundEventQueueDepth = outboundEventQueueDepth;
|
||||||
LastEventSequence = lastEventSequence;
|
LastEventSequence = lastEventSequence;
|
||||||
CurrentCommandCorrelationId = currentCommandCorrelationId ?? string.Empty;
|
CurrentCommandCorrelationId = currentCommandCorrelationId ?? string.Empty;
|
||||||
|
StaCallInProgress = staCallInProgress;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Gets the last STA activity timestamp in UTC.</summary>
|
/// <summary>Gets the last STA activity timestamp in UTC.</summary>
|
||||||
@@ -38,4 +46,11 @@ public sealed class WorkerRuntimeHeartbeatSnapshot
|
|||||||
|
|
||||||
/// <summary>Gets the correlation ID of the in-flight command.</summary>
|
/// <summary>Gets the correlation ID of the in-flight command.</summary>
|
||||||
public string CurrentCommandCorrelationId { get; }
|
public string CurrentCommandCorrelationId { get; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets a value indicating whether an STA call outside the command dispatcher (the alarm poll) is
|
||||||
|
/// executing. When true the watchdog grants the poll the same grace-to-ceiling suppression as a
|
||||||
|
/// dispatched command (WRK-27).
|
||||||
|
/// </summary>
|
||||||
|
public bool StaCallInProgress { get; }
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user