A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder
claimed its frame unwinds without awaiting that frame's completion; the same
holds for a frame already faulted by a concurrent FailAllQueued, where
TrySetCanceled loses. A later wire-write failure then lands TrySetException on
a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException.
The tombstone helpers now attach a fault-observing continuation to every frame
in the cancelled call (a cancelled task never fires OnlyOnFaulted, so
unconditional attach is safe), outside _gate because an already-faulted task
runs the continuation inline.
NEXT-05 is resolved as a documented decision, not a code change: tombstoned
entries keep their lazy DequeueNext purge — any subsequent write drains both
queues to empty and the heartbeat loop bounds residency to one interval, while
eager Queue<T> rebuilds under _gate would add ordering-invariant surface for
no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the
WRK-22 residual-window contract.
New regression test drives the exact abandonment: gated stream holds writer A
mid-write, the queued event frame is claimed and blocked mid-write, its caller
is cancelled, the write then faults with a marker exception, and the test
asserts the marker never reaches UnobservedTaskException after a forced GC.
net48 x86 build/test runs on windev with the rest of this batch.
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).
WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.
WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).
WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.
Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.
WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.
WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.
IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.
Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
WRK-06: MxStatusProxyConverter caches the four resolved FieldInfo per status
type in a static ConcurrentDictionary (the GetField metadata scan ran 4x per
status per event on the STA path). GetValue+Convert.ToInt32 still run per event
(late-bound RCW). Exceptions byte-identical: missing-field message unchanged
(ResolveField, not cached on throw via GetOrAdd); null-value message unchanged.
WRK-11: MxAccessEventQueue.Enqueue takes ownership of the passed MxEvent -
stamps WorkerSequence/WorkerTimestamp on it in place and enqueues it, no
Clone(). Audited all 3 callers (base/alarm event sinks, provider-mode handler):
each builds a fresh event per Enqueue, none reuse it. MxAccessValueCache.Set now
deep-copies its retained Value/SourceTimestamp/Statuses so the cache snapshot
never aliases the queue-owned (later serialized) event. Net: alarm/other events
clone nothing (was full clone); data-change clones payload-only.
WRK-12: WorkerFrameWriter coalesces the flush across a drained batch - each
frame is written but not flushed individually; one FlushAsync after the batch,
then all written frames complete. Preserves the written+flushed completion
contract; a burst of N events costs 1 flush, not N. On write failure the whole
in-flight batch + queue fail so no caller hangs.
IPC-15 (doc): the multi-event WorkerEnvelope body remains unimplemented (wire
still carries one event per worker_event frame); gateway.md Performance section
now distinguishes the shipped flush-coalescing from that deferred proto change.
net48-safe (no init/records; readonly struct cache entry). Worker builds x86
only - verification on windev. Tests added: converter cache-reuse, queue
ownership-transfer, value-cache snapshot independence, writer batch-flush-once.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Worker half of the Wave 3 size/backpressure + write-ordering pass:
- IPC-02: the worker adopts GatewayHello.max_frame_bytes during the handshake
(WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes) instead of a
hard-coded default; 0 keeps the default, a value above a 256 MiB ceiling is
rejected. Reader and writer share the options instance, applied before the
message loop.
- IPC-04: DrainEvents caps each reply at MaxDrainEventsPerReply (10_000) and
treats max_events = 0 as that cap rather than 'drain the entire queue', so one
diagnostic drain cannot pack a session-killing reply frame.
- WRK-04: WorkerFrameWriter stamps the envelope Sequence at the actual point of
writing (under the write lock) instead of at envelope creation, so the on-wire
order and the stamped sequence always agree under concurrent producers.
- WRK-07: the writer is now a cooperative priority scheduler — callers enqueue at
Control or Event priority and the draining lock-holder writes all control
frames before any event frame, so replies/faults/heartbeats jump ahead of an
event backlog. Per-frame validation/size rejections fail only that frame; a
stream write failure fails all queued frames.
Tests: monotonic gap-free sequence under concurrency, control-before-event
priority (gated stream), negotiated-max adoption, DrainEvents zero-bound.
Worker builds x86 only — verified on windev.
Apply the ZB.MOM.WW. prefix to all gateway-side projects, folders,
.csproj/.sln contents, C# namespaces, using directives, generated proto
C# (csharp_namespace + checked-in generated files), InternalsVisibleTo
attributes, project-name string literals (LoadProject, .sln lookups,
worker exe paths, staticwebassets manifest), and the install/script/doc
references that point at any of the above. Migrate the solution from
.sln to .slnx via `dotnet sln migrate` and delete the old file.
External-runtime identifiers are intentionally NOT prefixed so external
configuration keeps working:
- GatewayMetrics.cs MeterName ("MxGateway.Server")
- DashboardAuthenticationDefaults Scheme/Policy ("MxGateway.Dashboard")
- GatewayRequestLoggingMiddleware logger category ("MxGateway.Request")
- StaRuntime thread name ("MxGateway.Worker.STA")
- appsettings.json root section "MxGateway" + env-var prefix
MxGateway__... and secret-name MxGateway:ApiKeyPepper
- C:\ProgramData\MxGateway\ data dir paths
Also fixes two tests that were not rename-related but became visible
while validating the rename:
- WorkerLiveMxAccessSmokeTests.ShutDownAsync: cancellation that the
gateway service correctly maps to RpcException(Cancelled) per gRPC
convention was being misclassified as a stream fault. Added a sibling
catch on RpcException with StatusCode.Cancelled.
- IntegrationTestEnvironment.ResolveRepositoryRoot: extracted IsRepositoryRoot
and made it accept either a .git marker OR a .sln/.slnx next to src/
so the worker-exe walker works in non-git working copies.
clients/proto/proto-inputs.json's protoRoot updated to point at
src/ZB.MOM.WW.MxGateway.Contracts/Protos.
Verified by `dotnet build` and a full `dotnet test` of the .slnx with
MXGATEWAY_RUN_LIVE_{MXACCESS,LDAP,GALAXY}_TESTS=1:
Tests: 472/472 pass
Worker.Tests: 280/280 pass (4 dev-rig [Fact(Skip=...)] skipped)
IntegrationTests: 18/18 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty