fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
This commit is contained in:
Joseph Doherty
2026-08-07 05:38:23 -04:00
parent ead921cace
commit 33ba612ddd
18 changed files with 1118 additions and 57 deletions
@@ -43,9 +43,9 @@ Sequenced by cluster; a cluster is one change set.
| GWC-25 | Medium | S | CLI-35/36 (coord) | Not started | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
| CLI-35 | Medium | S | GWC-25 (coord) | Not started | Python CLI `stream-events` crashes on a ReplayGap |
| CLI-36 | Medium | S | GWC-25 (coord) | Not started | Go CLI `stream-events` silently destroys the ReplayGap signal |
| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Not started | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
| IPC-23 | Medium | S | WRK-21 | Not started | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
| IPC-30 | Low | M | WRK-21 (same batch) | Not started | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Done | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
| IPC-23 | Medium | S | WRK-21 | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
| IPC-30 | Low | M | WRK-21 (same batch) | Done | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
| SEC-31 | Medium | M | — | Not started | Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS) |
| SEC-32 | Low | S | SEC-31 | Not started | Failure-limiter LRU flushable by junk-token spray; token prefix never validated |
| IPC-24 | Medium | S | — | Not started | CI's unconditional Java churn-revert masks real drift |
@@ -71,27 +71,27 @@ Full design + implementation for each row lives in the linked domain doc under i
| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Not started | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Done | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
| WRK-22 | Low | — | S | IPC-26 (fix owned here) | Not started | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
| WRK-23 | Low | — | S | WRK-21 | Not started | Rejected frames consume sequence numbers, producing wire gaps |
| WRK-23 | Low | — | S | WRK-21 | Done | Rejected frames consume sequence numbers, producing wire gaps |
| WRK-24 | Low | — | S | — | Not started | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
| WRK-25 | Low | P2 | S | WRK-22 (shared seam) | Not started | WRK-12 flush coalescing never engages on the event hot path |
| WRK-26 | Low | P1 | S | WRK-23 (soft); discharges IPC-29 | Not started | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Not started | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (same batch) | Not started | 10,000 drain cap is a duplicated magic constant |
| WRK-28 | Low | — | S | WRK-21 (same batch) | Done | 10,000 drain cap is a duplicated magic constant |
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | Not started | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave |
| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | In progress — mechanics landed with WRK-21; proto-comment/doc wave pending | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave |
| IPC-24 | Medium | P0 | S | — | Not started | CI's unconditional Java churn-revert masks real drift |
| IPC-25 | Medium | P0 | M | — | Not started | Stale Go/Python worker bindings: regenerate (pinned toolchains) + check-codegen Check 4 |
| IPC-26 | Low | P2 | S | WRK-22 (mechanics) | Not started | Cancelled write leaves ghost frame — cancelled means never written |
| IPC-27 | Low | P2 | S | — | Not started | Descriptor-freshness test blind to enums/services/galaxy descriptor |
| IPC-28 | Low | — | S | — | Not started | docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping |
| IPC-29 | Low | — | S | WRK-26 (discharged by) | Not started | WorkerFrameProtocol.md missing write-scheduling/sequencing section |
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Not started | Oversized event frame: keep session-fatal, make the death structured |
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Done | Oversized event frame: keep session-fatal, make the death structured |
| IPC-31 | Info | — | — | — | N/A | Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded |
| IPC-32 | Info | — | S | IPC-25 (folded in) | Not started | check-codegen banner relabel 1/4…4/4 |