mxaccessgw/archreview/2026-07-12/10-gateway-core.md

Gateway Server Core — Post-Remediation Re-Review

• Date: 2026-07-12
• Commit: 4f5371f (main — merge of P0/P1/P2 remediation tiers + CI)
• Baseline: 59856b8 (pre-remediation)
• Method: static review on macOS; every cited file read in full at HEAD. No build, no test, no source change.
• Scope: src/ZB.MOM.WW.MxGateway.Server — Sessions/, Workers/, Grpc/ (command invoke + EventStreamService), Alarms/. Excludes Security/authn/authz, dashboard, metrics endpoints (other agent) except where session/stream behavior touches them.
• Prior report: archreview/10-gateway-core.md (2026-07-08); remediation design archreview/remediation/10-gateway-core.md; status source archreview/remediation/00-tracking.md.

Prior-finding verification

Every Done claim was verified against code read at HEAD; every open finding was re-checked for incidental fixes. All 10 Done claims hold and are sound. All 12 open findings are still open (none incidentally fixed); GWC-13 was partially improved. No Done claim failed verification.

ID	Claimed	Verified?	Evidence (HEAD)	Notes
GWC-01	Done	✅ holds	Workers/WorkerClient.cs:79-92 (_events now SingleReader = true with invariant comment), :267-282 (Interlocked.CompareExchange claimed-once guard in ReadEventsAsync, throws on second consumer); Sessions/GatewaySession.cs:553-562 (AttachInternalEventSubscriber, isInternal: true); Sessions/SessionManager.cs:195-208 (ReadAlarmEventsAsync lease-based); Alarms/GatewayAlarmMonitor.cs:232-234 (monitor consumes mapped MxEvents through the distributor, no raw drain)	Sound. The dual-drain race is structurally closed and future regressions fail loudly. Residual one-time gap at monitor startup → new GWC-26.
GWC-02	Done	✅ holds	Sessions/GatewaySession.cs:755-772 (MarkFaulted stamps _faultedAtUtc once), :847-853 + :1663-1667 (IsFaultedReapable(Core) with FaultedGraceSeconds), :1637 (faulted arm inside the atomic TryBeginCloseIfExpired re-check); Sessions/SessionManager.cs:22 (FaultedReason), :287-293 (sweep ladder lease → faulted → detach-grace); Configuration/SessionOptions.cs:58 (FaultedGraceSeconds, default 0 = next sweep); Configuration/GatewayOptionsValidator.cs:265-266	Sound; uses the existing TOCTOU-safe close gate as designed. Clock consistency verified: fault stamp and sweeper share the same TimeProvider (SessionManager.cs:450-454, :474-477).
GWC-03	Done	✅ holds	Sessions/SparseArrayExpander.cs:72-76 (configured cap checked before allocation, message names MxGateway:Events:MaxSparseArrayLength), :78-82 (Array.MaxLength backstop kept); Configuration/EventOptions.cs:39 (default 1,000,000); Sessions/GatewaySession.cs:1120-1126 (wired from EventOptions at the choke point); Configuration/GatewayOptionsValidator.cs:303-304 (range-validated at startup)	Sound. Note the rejection still throws RpcException — the new check inherits open GWC-17's layering leak.
GWC-04	Done	✅ holds	Workers/WorkerClient.cs:28-33/:93-100 (unbounded _eventStaging, single-reader/single-writer), :518-554 (DispatchEnvelope fully synchronous; replies/heartbeats/faults/acks never queue behind events), :565-573 (StageWorkerEvent non-blocking TryWrite), :580-593 (EventWriteLoopAsync owns the timed write + sustained-overflow ProtocolViolation fault, :610-647); loop registered in WaitForBackgroundTasksAsync (:1069) and completed on close/fault/dispose (:363-365, :803-805, :837-839)	Fix works as designed for the reported defect (reply-behind-event stall). However the unbounded staging channel introduces a backpressure regression → new GWC-24.
GWC-05	Not started	✅ still open	Sessions/SessionWorkerClientFactory.cs:158-166 — plain NamedPipeServerStream(..., PipeOptions.Asynchronous), no ACL, no CurrentUserOnly	Local pipe-squatting DoS unchanged; gateway.md pipe-security contract still unmet.
GWC-06	Done	✅ holds	Grpc/MxAccessGatewayService.cs:155-163 — Stopwatch.GetTimestamp() / Stopwatch.GetElapsedTime(ts), no per-event allocation
GWC-07	Done	✅ holds	Grpc/MxAccessGrpcMapper.cs:64-82 — MapEvent transfers ownership of workerEvent.Event (no .Clone()), with the ownership-transfer invariant documented and tied to GWC-01's single-reader guard	Downstream sharing audited: subscribers/replay ring are read-only; the dashboard broadcaster redacts on a deep clone (per SEC-25 change log), so the shared instance is never mutated. Safe.
GWC-08	Done	✅ holds	Workers/WorkerFrameWriter.cs:57-76 — single ArrayPool rent, LE prefix + WriteTo(Span), one WriteAsync, finally return; Workers/WorkerFrameReader.cs:51-79 — pooled payload buffer, length-bounded read/parse, returned in finally; validation order preserved (:37-49 before rent)	Sound. Residual nit: the reader still allocates a fresh 4-byte prefix array per frame (WorkerFrameReader.cs:33) → GWC-30 (Info).
GWC-09	Not started	✅ still open	Workers/WorkerProcessStartedProbe.cs:5-19 (no-op HasExited check throwing WorkerProcessLaunchException); Workers/WorkerProcessLauncher.cs:290-298 (ShouldRetryStartupProbe still excludes WorkerProcessLaunchException); Configuration/WorkerOptions.cs:19-22 (both dead options remain)	Retry pipeline still can never retry.
GWC-10	Not started	✅ still open	Workers/WorkerEnvelopeValidator.cs:15-39 — validates version/session/body only; no sequence tracking anywhere in WorkerClient	Related new gateway-side emission defect → GWC-28.
GWC-11	Not started	✅ still open	Sessions/GatewaySession.cs:18 (_workerClient not volatile); lock-free reads at :280 (WorkerProcessId), :285 (new public WorkerClient property — one more unguarded read than at baseline), :1533 (CloseAsync), :1684, :1719, :1831 (DisposeAsync)
GWC-12	Not started	✅ still open	Workers/WorkerClient.cs:353-360 — plain if (_disposed) return; _disposed = true;, no interlock
GWC-13	Not started	✅ still open (partially improved)	Sessions/GatewaySession.cs:1910-1978 — still a 25 ms poll loop, but the delay now routes through the injected TimeProvider (:1940-1944), improving testability	Design doc recommended defer-until-used; acceptable.
GWC-14	Done	✅ holds	Sessions/SessionEventDistributor.cs:100-118 (preallocated circular ReplayEntry[], head+count, invariants documented), :776-810 (allocation-free append with capacity overwrite), :814 (ReplayEntryAt modular indexing), :818-831 (age eviction advances head)	Wraparound math verified (_replayBuffer.Length == _replayBufferCapacity; capacity-0 guarded before any modulo). Query paths preserve ascending order. Correct.
GWC-15	Done	✅ holds	Grpc/EventStreamService.cs:123-124 (per-stream backlog source registration; gauge reads Reader.Count only at scrape), :197 (disposed before the lease in finally); Metrics/GatewayMetrics.cs:314-320, :560-573 (idempotent registration handle)
GWC-16	Not started	✅ still open	Grpc/MxAccessGatewayService.cs:104 (ResolveSession) + :122-123 (sessionManager.InvokeAsync(request.SessionId, …)) → Sessions/SessionManager.cs:163 (GetRequiredSession) — still two registry lookups per command
GWC-17	Not started	✅ still open	Sessions/SparseArrayExpander.cs:296-297 (RpcException(InvalidArgument) below the gRPC layer); invoked from Sessions/GatewaySession.cs:1043-1045 (comment explicitly documents the leak)	GWC-03's new cap rejection flows through the same leak.
GWC-18	Not started	✅ still open	Sessions/GatewaySession.cs:13 — public sealed class GatewaySession declares no IAsyncDisposable; DisposeAsync at :1741
GWC-19	Not started	✅ still open	Sessions/GatewaySession.cs:1684-1688 — KillWorker(string) present; grep across src/ + clients/ finds zero callers (only KillWorkerWithCloseGateAsync is used)
GWC-20	Not started	✅ still open	Configuration/WorkerOptions.cs:30-31 (HeartbeatIntervalSeconds doc/name unchanged) bound to the check interval at Sessions/SessionWorkerClientFactory.cs:86; Workers/WorkerClient.cs:458 heartbeat loop still uses raw Task.Delay without TimeProvider
GWC-21	Not started	✅ still open	Sessions/SessionWorkerClientFactory.cs:83-89 populates 4 of 6 WorkerClientOptions; EventChannelFullModeTimeout / HeartbeatStuckCeiling fixed at defaults (Workers/WorkerClientOptions.cs:13, :24)	Post-GWC-04 the full-mode timeout is now the sole structural backpressure bound (see GWC-24), which raises the value of exposing it.
GWC-22	Not started	✅ still open	Grpc/EventStreamService.cs:200 — metrics.StreamDisconnected("Detached") hardcoded in the finally; fault/overflow paths (:164-174) do not differentiate
GWC-23	N/A	✅ unchanged	Validator comment still documents the knowingly-dead knob	No action, as agreed.

Also verified in passing (other-domain fixes landing in gateway-core files, all sound): IPC-03 gateway half — pre-checked command envelope size failing only the offending correlation (Workers/WorkerClient.cs:209-221, CommandTooLarge → ResourceExhausted at Grpc/MxAccessGatewayService.cs:955); IPC-02 — GatewayHello.MaxFrameBytes conveyed (WorkerClient.cs:986-988) with startup cross-validation Worker.MaxMessageBytes >= gRPC cap + 64 KiB reserve (Configuration/GatewayOptionsValidator.cs:481-495); IPC-04 — DrainEvents max_events bounded at 10,000 (Grpc/MxAccessGrpcRequestValidator.cs:13, :81-84); TST-02 — owner-scoped StreamEvents attach (Grpc/EventStreamService.cs:57-62).

New findings

IDs continue from the prior register.

GWC-24 — The GWC-04 staging channel is unbounded: sustained slow consumption now grows memory instead of faulting, and the backlog is invisible to metrics

• Severity: Medium
• Category: Stability / backpressure regression
• Location: Workers/WorkerClient.cs:93-100 (unbounded _eventStaging), :565-573 (StageWorkerEvent TryWrite always succeeds), :610-647 (fault fires only when a single timed write exceeds EventChannelFullModeTimeout), :616/:627 (_eventQueueDepth gauge counts _events only)
• Description: Before GWC-04, a full _events channel blocked the read loop, which propagated backpressure through the pipe to the worker's own bounded queue — end-to-end, structurally bounded. Now the read loop stages events into an unbounded channel and only EventWriteLoopAsync faces the bounded _events. The sustained-overflow fault fires only when one WriteAsync waits longer than 5 s; if the consumer (the distributor pump) drains slower than the worker produces but each individual write completes within the window, _eventStaging grows without bound and without any fault — and the growth is invisible, because SetWorkerEventQueueDepth reflects only _events, never the staging backlog. The code comment (:28-33) asserts staging "only fills during the bounded EventChannelFullModeTimeout window", which is true only for a full stall, not for sustained slow drain. Mitigating factor: the pump is non-blocking (per-subscriber TryWrite fan-out, SessionEventDistributor.cs:582-594), so a persistently slow-but-live consumer requires the pump itself to be starved — unlikely but no longer structurally impossible. The repo's fail-fast backpressure invariant now rests on pump liveness rather than on channel bounds.
• Recommendation: Bound the staging channel (e.g. 2 × EventChannelCapacity) and treat a TryWrite miss there as an immediate ProtocolViolation fault (a full staging channel means the event writer is already saturated); alternatively track total staged-but-undelivered depth and fault past a ceiling. Include the staging depth in the worker-event queue-depth gauge either way.

GWC-25 — ReplayGap sentinel carries oldest_available_sequence = 0 when the ring is empty, violating the proto resume contract and dead-streaming a compliant client

• Severity: Medium
• Category: Correctness / contract
• Location: Sessions/SessionEventDistributor.cs:463-467 (RegisterWithReplay, _replayCount == 0 branch: oldestAvailableSequence = 0); emitted verbatim by Grpc/EventStreamService.cs:133-139 + :210-222; contract at src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto:755-763
• Description: The proto contract states "oldest_available_sequence itself IS still retained: a client that wishes to resume without incurring another gap MUST set after_worker_sequence = oldest_available_sequence - 1". When a resume finds a gap with an empty ring (all entries age-evicted — default ReplayRetentionSeconds = 300 and EvictAged runs inside RegisterWithReplay at :458 — or ReplayBufferCapacity = 0 with the cursor behind _highestSequenceSeen), the sentinel carries oldest_available_sequence = 0. A contract-compliant client computes 0 − 1 on a uint64 → 2^64−1, and its next resume passes after_worker_sequence = ulong.MaxValue: RegisterWithReplay replays nothing, reports no gap, and the live filter mxEvent.WorkerSequence <= afterWorkerSequence (Grpc/EventStreamService.cs:179) then drops every live event — a silently dead stream with no error. All five clients shipped the oldest − 1 resume formula in CLI-15 (per the tracking change log), so the failure mode is reachable end-to-end.
• Recommendation: In the empty-ring-with-gap branch, emit oldest_available_sequence = _highestSequenceSeen + 1 (the next sequence that can possibly be delivered), so oldest − 1 = highestSeen resumes cleanly; keep 0 only for the never-populated case where no gap is reported anyway. Alternatively define the 0 special case in the proto and fix all five clients — the server-side fix is one line and needs no client changes, so prefer it. Add a distributor test for gap-with-empty-ring and an e2e resume after full age eviction.

GWC-26 — Alarm monitor attaches its event subscriber only after SubscribeAlarms + first reconcile; transitions in that window bypass the feed, and a missed Acknowledge is never broadcast

• Severity: Low
• Category: Stability / alarm-feed completeness (residual of GWC-01)
• Location: Alarms/GatewayAlarmMonitor.cs:213-214 (subscribe + reconcile) precede the attach at :232-234; late-subscriber semantics at Sessions/SessionEventDistributor.cs:579-582; ApplyReconcile presence-delta-only broadcast at Alarms/GatewayAlarmMonitor.cs:514-553
• Description: The distributor pump has been running since MarkReady (dashboard mirror registers first, Sessions/GatewaySession.cs:445-449, :621), and a subscriber registered mid-stream misses previously fanned events by design. Alarm transitions arriving between the worker's SubscribeAlarms activation and the monitor's internal-subscriber registration (two full worker command round-trips later) are fanned only to the dashboard mirror. Raise/Clear are repaired by the next reconcile, but an Acknowledge in the window updates _alarms silently on the next reconcile snapshot replace (:547-551) without any Broadcast — live StreamAlarms subscribers keep showing the alarm unacked until it clears. This is the same defect class GWC-01 fixed, shrunk from "random half of all events, forever" to "a one-shot window of a few hundred ms per monitor lifecycle".
• Recommendation: Attach the internal subscriber before invoking SubscribeAlarmsAsync (take the lease from AttachInternalEventSubscriber explicitly, then subscribe, then drain), or buffer the lease's channel during subscribe/reconcile. Also consider broadcasting an acked-state delta from ApplyReconcile when an existing reference's CurrentState changes.

GWC-27 — AttachInternalEventSubscriber / ReadAlarmEventsAsync bypass the readiness gate; a premature attach permanently poisons the session's event distributor

• Severity: Low
• Category: Latent lifecycle hazard
• Location: Sessions/GatewaySession.cs:553-562 (no _state == Ready check, contrast AttachEventSubscriber :889-916); Sessions/SessionManager.cs:195-208 (public ReadAlarmEventsAsync, also unchecked)
• Description: The internal-attach path starts the distributor pump unconditionally. If invoked before the session is Ready, the pump's source (MapWorkerEventsAsync → GetReadyWorkerClientAsync, :740-749/:1910) throws SessionNotReady; PumpAsync completes all subscribers with the error and latches _completed (Sessions/SessionEventDistributor.cs:603-612, :662-680). Because _eventDistributorStarted is never reset (Sessions/GatewaySession.cs:527-531) and the distributor is replaced only in DisposeAsync, every later registration — including gRPC StreamEvents attaches — receives an immediately-completed channel carrying the stale error: a Ready session with permanently dead event streaming. Not reachable today (the alarm monitor, the only caller, attaches after OpenSessionAsync returns a Ready session), but the hazard is one future call site away and fails in the worst way (silent, permanent, per-session).
• Recommendation: Mirror AttachEventSubscriber's readiness check at the top of AttachInternalEventSubscriber (throw SessionManagerErrorCode.SessionNotReady before EnsureDistributorCreated).

GWC-28 — Gateway→worker envelope sequence is stamped at envelope creation, not at write, so concurrent invokes can emit non-monotonic sequences on the wire

• Severity: Low
• Category: IPC contract / conventions
• Location: Workers/WorkerClient.cs:1031-1044 (CreateEnvelope stamps Interlocked.Increment(ref _nextSequence)); enqueue at :957-972; single write loop :390-409; the gateway WorkerFrameWriter does not re-stamp (Workers/WorkerFrameWriter.cs:35-77)
• Description: Two concurrent InvokeAsync callers can stamp sequences 1 and 2 but win the channel WriteAsync race in the order 2, 1, putting out-of-order sequences on the pipe. This is exactly the defect WRK-04 fixed on the worker side (its writer now stamps at the point of writing under the write lock); the gateway half was not given the same treatment. Benign today — nothing validates inbound sequence on either side (GWC-10/IPC-10 both open) — but it violates gateway.md's "monotonic per sender" contract and would start faulting sessions the moment worker-side validation is added.
• Recommendation: Move the stamp into WriteLoopAsync immediately before _writer.WriteAsync (single consumer, naturally serialized), matching the worker's WRK-04 fix; coordinate with GWC-10 if enforcement is added.

GWC-29 — Invoke deep-clones the entire request only to discard the cloned command

• Severity: Low
• Category: Performance (hot path)
• Location: Grpc/MxAccessGatewayService.cs:119-121; Grpc/MxAccessGrpcMapper.cs:27-37
• Description: MxCommandRequest invokeRequest = request.Clone() deep-clones the request including its command payload (potentially a large bulk-write graph), then immediately overwrites invokeRequest.Command with commandToInvoke, discarding the cloned command. MapCommand then performs the one clone that is actually needed (request.Command.Clone()). Net: one full wasted command deep-clone on every Invoke — the same cost class GWC-07/IPC-05 eliminated on the event and envelope paths, still present on the command path.
• Recommendation: Skip the request clone: add a MapCommand(MxCommand command) overload (or construct a minimal request) since MapCommand reads only the command, and pass commandToInvoke directly.

GWC-30 — Frame reader still allocates a fresh 4-byte prefix array per frame

• Severity: Info
• Category: Performance (residual of GWC-08)
• Location: Workers/WorkerFrameReader.cs:33
• Description: The GWC-08 design suggested a per-reader scratch buffer for the length prefix; the payload was pooled but the new byte[sizeof(uint)] per frame remains. Gen-0, 4 bytes — negligible; recorded only so the next hot-path pass knows it was seen, not missed.
• Recommendation: Optional: a reusable per-reader prefix field (the reader is single-consumer by construction).

Severity roll-up (new findings)

Severity	Count	IDs
Critical	0	—
High	0	—
Medium	2	GWC-24, GWC-25
Low	4	GWC-26, GWC-27, GWC-28, GWC-29
Info	1	GWC-30

Still good / preserved

The qualities the prior review praised survived the remediation churn intact. Session state writes remain single-lock disciplined (_syncRoot everywhere, including the new fault-timestamp and faulted-reap paths); the close gate still serializes Close/Kill/sweep with atomic TOCTOU re-checks (TryBeginCloseIfExpired gained the faulted arm without weakening the reattach-wins race resolution). The replay→live handoff argument still holds in the circular-array rewrite — the _replayLock-atomic snapshot+register in RegisterWithReplay is unchanged in structure and its no-gap/no-duplicate documentation is exemplary. The rewritten ring's eviction, ordering, and wraparound logic are correct. The GWC-01 fix is a model remediation: it doesn't just fix the bug, it converts the whole bug class into a loud InvalidOperationException via SingleReader = true plus the claimed-once guard. Repo invariants were respected throughout: no synthesized events (the ReplayGap sentinel is a documented control signal, correctly never fanned or drained elsewhere), one worker per session, and the frame wire format is byte-identical after the pooled-I/O rewrite. Convention adherence (file-scoped namespaces, sealed, Async suffixes, MXAccess-aligned naming) remains strong in all new code; GatewaySession has grown to 2,128 lines and the split-out recommendation from the prior report stands.