The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.
Option (a) from the finding: close the coupling at both ends.
- GatewayOptionsValidator adds the upper half of the headroom rule
(Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
usable.
The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.
Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.
The worker half cannot be built on macOS; verified on windev separately.
Sweep code/doc comments citing unregistered GWC-30/GWC-31 (2026-08-15 perf
pass) to PERF-02/PERF-07/PERF-22, keyed to task numbers in
docs/plans/2026-08-15-perf-review-remediation.md; add a PERF-ID register
table to that plan. Genuine cycle-2 GWC-30 citations (gateway-side frame-
reader 4-byte prefix fix in Server/Workers/WorkerFrameReader.cs and its
test) are left untouched.
Add a line to REVIEW-PROCESS.md reserving the GWC/WRK/IPC/SEC/CLI/TST
prefixes for the archreview domain registers.
Tracker housekeeping per archreview/2026-08-19/00-overall.md:
- GWC-16 -> Won't fix (rationale: PERF(followup) comment documents the
double session lookup as measured-negligible, deliberately not collapsed)
- IPC-08 -> Done (closed by 7c1ea12/a1a38b5; residual tracked as IPC-34)
- IPC-31 -> Done (superseded by GWC-28's write-time sequence stamping)
- TST-06, TST-17 rows refreshed per cycle-3 60-testing-docs-gaps.md
- WRK-31 (and defensively WRK-29/30) registered as reserved/do-not-reuse
interim IDs burned by e913dab, in both the cycle-2 tracker and the
cycle-2 worker remediation doc
Flip Batch C to Done in archreview/2026-08-19/remediation/00-tracking.md.
Worker-side files (Worker/Ipc/WorkerFrameReader.cs, WorkerFrameWriter.cs)
are comment-only edits, unverified here (net48 x86, windev-only build).
ReportWatchdogFaultIfNeededAsync built the StaHung fault with
CommandMethod = snapshot.CurrentCommandCorrelationId, so the wire
fault's command_method field carried an opaque correlation id (or
empty) instead of a method name -- a consumer reading CommandMethod as
a method name got a confusing value. Use a stable "Watchdog" label and
fold the stuck command's correlation id (when present) into the
diagnostic message instead. No test asserts CommandMethod for the
StaHung fault; the existing assertions only check Category and that
DiagnosticMessage contains "STA activity is stale", both unchanged.
TryStartCommandTask discarded ObserveCommandTaskAsync's task without a
fault-observing continuation -- the one gap in the NEXT-04 discipline
applied everywhere else in this file. ObserveCommandTaskAsync only
swallows OperationCanceledException, so any other exception escaping
ProcessCommandAsync's own catch path would fault the discarded wrapper
and surface only as a finalizer-time unobserved-task exception. Wrap it
with the existing ObserveFaultWhenever helper, same as every other
discarded task in this file.
Targeted re-read of the 203-file fca978d sweep (docs(src): add missing
XML docs and strip tracking-ID comments): a mechanical pre-pass narrowed
1,383 deletions to 68 files / 816 residual prose lines, and a judged
review of every one found 17 collateral deletions across 10 files —
rationale prose deleted alongside resolved markers with no equivalent
surviving anywhere in the tree. Restored (markers stay stripped, per the
sweep's intent):
- SessionManager: the three metrics-accounting invariants (kill-path
gauge decrement safety, shutdown kill-fallback registry guard vs
double bookkeeping, SessionClosed-not-SessionRemoved on failed close)
- SessionManagerTests: the matching accounting expectation note and the
reason-string propagation pins (test summary + FakeWorkerClient.LastKillReason)
- MxAccessGatewayService.AcknowledgeAlarm: the routing remarks (GUID vs
Provider!Group.Tag vs InvalidRequest; session-less via IGatewayAlarmService)
— inheritdoc resolves to nothing (proto-generated base is undocumented)
- HubTokenService.Validate: why the hollow-token guard exists
(non-empty AuthenticationType alone satisfies IsAuthenticated)
- DashboardSessionAdminService: why both broad catches exist (keep raw
teardown exceptions out of Blazor's error boundary), Close + Kill paths
- WorkerPipeSession.RunAsync: why the factory result throws instead of
NREing (unambiguous failure; finally-block Dispose can't no-op)
- LmxSubtagAlarmSource: Advise idempotency; Write is always unsecured
(user id 0), never WriteSecured semantics
- WnWrapAlarmConsumer: the v1-prefix path is what WIN-911-style code uses
- DashboardSnapshotPublisherTests: what the 10ms slack absorbs
(Task.Delay's coarse Windows timer quantum)
- DashboardBrowseAndAlarmModelTests: why the label text is pinned, not
just the CSS class
Everything else flagged verified benign: inheritdoc replacements resolve
to equal-or-richer interface docs, or the substance survives relocated.
NonWindows slnx 0W/0E; touched gateway test classes 65/65.
Both tasks the detached-lock-wait path starts and discards now carry the
file's fault-observing continuation, factored out of ObserveAbandonedFault as
ObserveFault so the idiom has one definition. DrainDetachedAsync swallows the
drain but its _writeLock.Release() sits in a finally outside that catch, so a
Release that ever throws — a SemaphoreFullException from some future
double-release regression — had no awaiter and would have surfaced on net48 as
TaskScheduler.UnobservedTaskException at finalization instead of an
attributable failure. Same for a throw out of OnDetachedLockWaitSettled.
Review's Minor (distinguishing a cancelled from a faulted lock wait before
tombstoning) is deliberately not taken: a faulted WaitAsync is unreachable
here — nothing disposes _writeLock — so the branch would be untestable new
logic whose only effect is internal state, the caller already receiving the
fault itself from the rethrow. Recorded as a comment at the site instead.
edited on macOS, windev verification pending (plan Task 11). Re-ran the net10
scratch harness over WorkerFrameWriter and the writer suite: 0 warnings,
31/31 pass.
WriteAsync enqueued its frame and then contended unconditionally for the write
lock, so a caller that lost the race stayed in WaitAsync until the winning
drainer released — even though that winner writes, flushes, and completes the
loser's control frame at the control-to-event class boundary, part-way through
its pass. The boundary flush made the delivery point honest; the awaited task
was still charged for the whole event backlog it had just been flushed ahead of.
WriteAsync now awaits its own frame's completion racing the lock acquisition.
Completion first: the caller returns at its frame's delivery point and the
outstanding acquisition is detached, not dropped — a continuation drains
whatever is queued and releases, so the lock is never acquired and silently
held and a frame enqueued between the previous drainer's last dequeue and its
release is still written. Lock first: drain as before. Cancellation keeps the
WRK-22 tombstone semantics exactly, and a wait cancelled after the caller has
already detached releases nothing (SemaphoreSlim hands no count to a wait it
cancels), so no count leaks and no queued frame is stranded. A token that fires
after the frame's completion won the race changes nothing — the frame was
delivered. WriteBatchAsync deliberately keeps the plain wait-then-drain shape:
its last completion resolves at the end-of-pass flush anyway.
Three tests: the latency win (a control caller returning while the winning
WriteBatchAsync event burst is demonstrably still blocked mid-pass), a
mixed-priority concurrency soak pinning exactly-once writes and a single
drainer, and the cancel-after-detach corner (a wrongly released count would
surface as the drainer's own Release throwing SemaphoreFullException).
edited on macOS, windev verification pending (plan Task 11). Verified here by
compiling and running WorkerFrameWriter plus the writer suite against net10.0
in a scratch harness: 31/31 pass, and the two behaviour-pinning tests fail
against the pre-change parked implementation.
The two-class writer already got control bytes out ahead of a queued event
backlog, but a frame counts as delivered only once flushed, and the drain
deferred its single FlushAsync — and every TrySetResult — to the end of the
pass. A heartbeat, command reply, fault, or shutdown ack was therefore written
first and completed last, behind up to a full 128-frame event batch.
The drain now records each frame's priority class on PendingFrame and flushes
at every control-to-event boundary, completing and clearing the written set
there. Cost stays bounded: a pure-event pass still pays exactly one flush, a
run of control frames still pays one for the run, and only a pass that mixes
both classes pays a second — never one flush per control frame, the
syscall-per-heartbeat cost WRK-12 removed.
A boundary flush that itself fails is a new failure window and is handled like
the end-of-pass flush failure, additionally failing the event frame the drain
had already claimed off its queue and every frame still queued. Frames a
boundary flush completed leave the written set, so a later failure in the same
pass can no longer reach back and fail an already-delivered control frame.
The awaited task of a caller that lost the write-lock race is still bounded by
the winning drainer's pass — that enqueue-then-contend parking is unchanged and
now documented on WriteAsync and in docs/WorkerFrameProtocol.md.
A WriteAsync/WriteBatchAsync caller cancelled after the draining lock-holder
claimed its frame unwinds without awaiting that frame's completion; the same
holds for a frame already faulted by a concurrent FailAllQueued, where
TrySetCanceled loses. A later wire-write failure then lands TrySetException on
a task with no awaiter and surfaces as TaskScheduler.UnobservedTaskException.
The tombstone helpers now attach a fault-observing continuation to every frame
in the cancelled call (a cancelled task never fires OnlyOnFaulted, so
unconditional attach is safe), outside _gate because an already-faulted task
runs the continuation inline.
NEXT-05 is resolved as a documented decision, not a code change: tombstoned
entries keep their lazy DequeueNext purge — any subsequent write drains both
queues to empty and the heartbeat loop bounds residency to one interval, while
eager Queue<T> rebuilds under _gate would add ordering-invariant surface for
no gain. Rationale recorded in docs/WorkerFrameProtocol.md alongside the
WRK-22 residual-window contract.
New regression test drives the exact abandonment: gated stream holds writer A
mid-write, the queued event frame is claimed and blocked mid-write, its caller
is cancelled, the write then faults with a marker exception, and the test
asserts the marker never reaches UnobservedTaskException after a forced GC.
net48 x86 build/test runs on windev with the rest of this batch.
WRK-22/IPC-26: tombstone a WriteAsync/WriteBatchAsync cancelled while
waiting for the write lock (PendingFrame.Claimed under _gate; DequeueNext
skips cancelled, claims the frame it returns) so a cancelled write never
reaches the wire unless already claimed mid-write (documented residual).
WRK-25: add WriteBatchAsync; RunEventDrainLoopAsync submits the drained
event batch through it, so a burst of N events costs one flush not N.
IPC-30 oversized-event structured fault preserved via FindOversizedEvent.
WRK-24: reject a below-1024 negotiated frame maximum at the handshake
(MinNegotiableFrameBytes, matching GatewayOptionsValidator floor).
WRK-27: alarm poll advertises StaCallInProgress on the heartbeat snapshot
so the watchdog suppresses to the ceiling, not the grace.
Docs (WorkerFrameProtocol.md, MxAccessWorkerInstanceDesign.md) and the
2026-07-12 remediation registers/change-log updated in the same commit.
Review follow-ups on the WRK-21 cluster.
1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
byte-heavy event, so DrainEvents truncated on every call and the drain-until-
empty loop never terminated. It now takes the max of (frameMax - reserve) and
frameMax/2, so the budget is monotonic and never below half the frame max. New
test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
byte-heavy queue at the exact validator floor and asserts it drains to empty
with no head ever reported oversized.
2. The reply-too-large fallback write is now itself size-guarded
(WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
at a pathologically tiny negotiated max below the gateway's floor the fallback
could also throw MessageTooLarge and — uncaught — kill the session, defeating the
"no diagnostics command is session-fatal" invariant. It now log-and-swallows;
comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.
3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
already includes the event's tag and length prefix (the same shape the reply's
repeated events field packs), so the 8 bytes is pure slack over an already-
conservative estimate, not compensation for a missing wrapper.
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.
WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.
WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.
IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.
Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
Resolve all CommentChecker findings across the gateway server, worker, tests,
and .NET client (314 -> 0 real issues): add missing <returns>/<summary>/<param>
on public and test members, convert Stream/interface overrides to <inheritdoc/>,
and remove internal task/issue tracking IDs (SEC-*, IPC-*, WRK-*, GWC-*, TST-*,
Client.Dotnet-*) from shipped code documentation while preserving the design
rationale prose. Shipped comments should not carry internal bookkeeping, and
complete XML docs keep the analyzer/TreatWarningsAsErrors gate and generated API
docs clean. The 6 remaining flags are heuristic false positives (MD5, UTC-4,
capacity-1, near-1601) left intact so real documentation is not corrupted.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
WRK-06: MxStatusProxyConverter caches the four resolved FieldInfo per status
type in a static ConcurrentDictionary (the GetField metadata scan ran 4x per
status per event on the STA path). GetValue+Convert.ToInt32 still run per event
(late-bound RCW). Exceptions byte-identical: missing-field message unchanged
(ResolveField, not cached on throw via GetOrAdd); null-value message unchanged.
WRK-11: MxAccessEventQueue.Enqueue takes ownership of the passed MxEvent -
stamps WorkerSequence/WorkerTimestamp on it in place and enqueues it, no
Clone(). Audited all 3 callers (base/alarm event sinks, provider-mode handler):
each builds a fresh event per Enqueue, none reuse it. MxAccessValueCache.Set now
deep-copies its retained Value/SourceTimestamp/Statuses so the cache snapshot
never aliases the queue-owned (later serialized) event. Net: alarm/other events
clone nothing (was full clone); data-change clones payload-only.
WRK-12: WorkerFrameWriter coalesces the flush across a drained batch - each
frame is written but not flushed individually; one FlushAsync after the batch,
then all written frames complete. Preserves the written+flushed completion
contract; a burst of N events costs 1 flush, not N. On write failure the whole
in-flight batch + queue fail so no caller hangs.
IPC-15 (doc): the multi-event WorkerEnvelope body remains unimplemented (wire
still carries one event per worker_event frame); gateway.md Performance section
now distinguishes the shipped flush-coalescing from that deferred proto change.
net48-safe (no init/records; readonly struct cache entry). Worker builds x86
only - verification on windev. Tests added: converter cache-reuse, queue
ownership-transfer, value-cache snapshot independence, writer batch-flush-once.
Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
Worker half of the Wave 3 size/backpressure + write-ordering pass:
- IPC-02: the worker adopts GatewayHello.max_frame_bytes during the handshake
(WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes) instead of a
hard-coded default; 0 keeps the default, a value above a 256 MiB ceiling is
rejected. Reader and writer share the options instance, applied before the
message loop.
- IPC-04: DrainEvents caps each reply at MaxDrainEventsPerReply (10_000) and
treats max_events = 0 as that cap rather than 'drain the entire queue', so one
diagnostic drain cannot pack a session-killing reply frame.
- WRK-04: WorkerFrameWriter stamps the envelope Sequence at the actual point of
writing (under the write lock) instead of at envelope creation, so the on-wire
order and the stamped sequence always agree under concurrent producers.
- WRK-07: the writer is now a cooperative priority scheduler — callers enqueue at
Control or Event priority and the draining lock-holder writes all control
frames before any event frame, so replies/faults/heartbeats jump ahead of an
event backlog. Per-frame validation/size rejections fail only that frame; a
stream write failure fails all queued frames.
Tests: monotonic gap-free sequence under concurrency, control-before-event
priority (gated stream), negotiated-max adoption, DrainEvents zero-bound.
Worker builds x86 only — verified on windev.
Answer the five worker control/lifecycle commands at the WorkerPipeSession
message-loop layer instead of the STA-bound MxAccessCommandExecutor. These
replies are built from process-level state (worker pid, assembly version,
worker lifecycle, the runtime session's event queue) the executor cannot see,
and ShutdownWorker must emit its OK reply before the graceful shutdown joins
the STA thread - dispatching it onto the STA would deadlock.
- Ping: OK reply, echoes message into diagnostic_message.
- GetSessionState: maps WorkerState to proto SessionState.
- GetWorkerInfo: pid, worker version, MXAccess ProgID/CLSID.
- DrainEvents: drains the runtime event queue into DrainEventsReply.
- ShutdownWorker: OK reply, then graceful shutdown, then stops the loop.
Tests added in WorkerPipeSessionTests; FakeRuntimeSession gains a
batch-size drain suppressor so DrainEvents does not race the background
drain loop.
Apply the ZB.MOM.WW. prefix to all gateway-side projects, folders,
.csproj/.sln contents, C# namespaces, using directives, generated proto
C# (csharp_namespace + checked-in generated files), InternalsVisibleTo
attributes, project-name string literals (LoadProject, .sln lookups,
worker exe paths, staticwebassets manifest), and the install/script/doc
references that point at any of the above. Migrate the solution from
.sln to .slnx via `dotnet sln migrate` and delete the old file.
External-runtime identifiers are intentionally NOT prefixed so external
configuration keeps working:
- GatewayMetrics.cs MeterName ("MxGateway.Server")
- DashboardAuthenticationDefaults Scheme/Policy ("MxGateway.Dashboard")
- GatewayRequestLoggingMiddleware logger category ("MxGateway.Request")
- StaRuntime thread name ("MxGateway.Worker.STA")
- appsettings.json root section "MxGateway" + env-var prefix
MxGateway__... and secret-name MxGateway:ApiKeyPepper
- C:\ProgramData\MxGateway\ data dir paths
Also fixes two tests that were not rename-related but became visible
while validating the rename:
- WorkerLiveMxAccessSmokeTests.ShutDownAsync: cancellation that the
gateway service correctly maps to RpcException(Cancelled) per gRPC
convention was being misclassified as a stream fault. Added a sibling
catch on RpcException with StatusCode.Cancelled.
- IntegrationTestEnvironment.ResolveRepositoryRoot: extracted IsRepositoryRoot
and made it accept either a .git marker OR a .sln/.slnx next to src/
so the worker-exe walker works in non-git working copies.
clients/proto/proto-inputs.json's protoRoot updated to point at
src/ZB.MOM.WW.MxGateway.Contracts/Protos.
Verified by `dotnet build` and a full `dotnet test` of the .slnx with
MXGATEWAY_RUN_LIVE_{MXACCESS,LDAP,GALAXY}_TESTS=1:
Tests: 472/472 pass
Worker.Tests: 280/280 pass (4 dev-rig [Fact(Skip=...)] skipped)
IntegrationTests: 18/18 pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty