mxaccessgw/archreview/2026-07-12/10-gateway-core.md

# Gateway Server Core — Post-Remediation Re-Review

- **Date:** 2026-07-12
- **Commit:** `4f5371f` (`main` — merge of P0/P1/P2 remediation tiers + CI)
- **Baseline:** `59856b8` (pre-remediation)
- **Method:** static review on macOS; every cited file read in full at HEAD. No build, no test, no source change.
- **Scope:** `src/ZB.MOM.WW.MxGateway.Server` — `Sessions/`, `Workers/`, `Grpc/` (command invoke + `EventStreamService`), `Alarms/`. Excludes Security/authn/authz, dashboard, metrics endpoints (other agent) except where session/stream behavior touches them.
- **Prior report:** `archreview/10-gateway-core.md` (2026-07-08); remediation design `archreview/remediation/10-gateway-core.md`; status source `archreview/remediation/00-tracking.md`.

## Prior-finding verification

Every `Done` claim was verified against code read at HEAD; every open finding was re-checked for incidental fixes. **All 10 Done claims hold and are sound. All 12 open findings are still open (none incidentally fixed); GWC-13 was partially improved.** No Done claim failed verification.

| ID | Claimed | Verified? | Evidence (HEAD) | Notes |
|----|---------|-----------|-----------------|-------|
| GWC-01 | Done | ✅ holds | `Workers/WorkerClient.cs:79-92` (`_events` now `SingleReader = true` with invariant comment), `:267-282` (`Interlocked.CompareExchange` claimed-once guard in `ReadEventsAsync`, throws on second consumer); `Sessions/GatewaySession.cs:553-562` (`AttachInternalEventSubscriber`, `isInternal: true`); `Sessions/SessionManager.cs:195-208` (`ReadAlarmEventsAsync` lease-based); `Alarms/GatewayAlarmMonitor.cs:232-234` (monitor consumes mapped `MxEvent`s through the distributor, no raw drain) | Sound. The dual-drain race is structurally closed and future regressions fail loudly. Residual one-time gap at monitor startup → new GWC-26. |
| GWC-02 | Done | ✅ holds | `Sessions/GatewaySession.cs:755-772` (`MarkFaulted` stamps `_faultedAtUtc` once), `:847-853` + `:1663-1667` (`IsFaultedReapable(Core)` with `FaultedGraceSeconds`), `:1637` (faulted arm inside the atomic `TryBeginCloseIfExpired` re-check); `Sessions/SessionManager.cs:22` (`FaultedReason`), `:287-293` (sweep ladder lease → faulted → detach-grace); `Configuration/SessionOptions.cs:58` (`FaultedGraceSeconds`, default 0 = next sweep); `Configuration/GatewayOptionsValidator.cs:265-266` | Sound; uses the existing TOCTOU-safe close gate as designed. Clock consistency verified: fault stamp and sweeper share the same `TimeProvider` (`SessionManager.cs:450-454`, `:474-477`). |
| GWC-03 | Done | ✅ holds | `Sessions/SparseArrayExpander.cs:72-76` (configured cap checked **before** allocation, message names `MxGateway:Events:MaxSparseArrayLength`), `:78-82` (`Array.MaxLength` backstop kept); `Configuration/EventOptions.cs:39` (default 1,000,000); `Sessions/GatewaySession.cs:1120-1126` (wired from `EventOptions` at the choke point); `Configuration/GatewayOptionsValidator.cs:303-304` (range-validated at startup) | Sound. Note the rejection still throws `RpcException` — the new check inherits open GWC-17's layering leak. |
| GWC-04 | Done | ✅ holds | `Workers/WorkerClient.cs:28-33`/`:93-100` (unbounded `_eventStaging`, single-reader/single-writer), `:518-554` (`DispatchEnvelope` fully synchronous; replies/heartbeats/faults/acks never queue behind events), `:565-573` (`StageWorkerEvent` non-blocking `TryWrite`), `:580-593` (`EventWriteLoopAsync` owns the timed write + sustained-overflow `ProtocolViolation` fault, `:610-647`); loop registered in `WaitForBackgroundTasksAsync` (`:1069`) and completed on close/fault/dispose (`:363-365`, `:803-805`, `:837-839`) | Fix works as designed for the reported defect (reply-behind-event stall). However the *unbounded* staging channel introduces a backpressure regression → new GWC-24. |
| GWC-05 | Not started | ✅ still open | `Sessions/SessionWorkerClientFactory.cs:158-166` — plain `NamedPipeServerStream(..., PipeOptions.Asynchronous)`, no ACL, no `CurrentUserOnly` | Local pipe-squatting DoS unchanged; `gateway.md` pipe-security contract still unmet. |
| GWC-06 | Done | ✅ holds | `Grpc/MxAccessGatewayService.cs:155-163` — `Stopwatch.GetTimestamp()` / `Stopwatch.GetElapsedTime(ts)`, no per-event allocation | |
| GWC-07 | Done | ✅ holds | `Grpc/MxAccessGrpcMapper.cs:64-82` — `MapEvent` transfers ownership of `workerEvent.Event` (no `.Clone()`), with the ownership-transfer invariant documented and tied to GWC-01's single-reader guard | Downstream sharing audited: subscribers/replay ring are read-only; the dashboard broadcaster redacts on a deep clone (per SEC-25 change log), so the shared instance is never mutated. Safe. |
| GWC-08 | Done | ✅ holds | `Workers/WorkerFrameWriter.cs:57-76` — single `ArrayPool` rent, LE prefix + `WriteTo(Span)`, one `WriteAsync`, `finally` return; `Workers/WorkerFrameReader.cs:51-79` — pooled payload buffer, length-bounded read/parse, returned in `finally`; validation order preserved (`:37-49` before rent) | Sound. Residual nit: the reader still allocates a fresh 4-byte prefix array per frame (`WorkerFrameReader.cs:33`) → GWC-30 (Info). |
| GWC-09 | Not started | ✅ still open | `Workers/WorkerProcessStartedProbe.cs:5-19` (no-op `HasExited` check throwing `WorkerProcessLaunchException`); `Workers/WorkerProcessLauncher.cs:290-298` (`ShouldRetryStartupProbe` still excludes `WorkerProcessLaunchException`); `Configuration/WorkerOptions.cs:19-22` (both dead options remain) | Retry pipeline still can never retry. |
| GWC-10 | Not started | ✅ still open | `Workers/WorkerEnvelopeValidator.cs:15-39` — validates version/session/body only; no sequence tracking anywhere in `WorkerClient` | Related new gateway-side emission defect → GWC-28. |
| GWC-11 | Not started | ✅ still open | `Sessions/GatewaySession.cs:18` (`_workerClient` not volatile); lock-free reads at `:280` (`WorkerProcessId`), `:285` (new public `WorkerClient` property — one **more** unguarded read than at baseline), `:1533` (`CloseAsync`), `:1684`, `:1719`, `:1831` (`DisposeAsync`) | |
| GWC-12 | Not started | ✅ still open | `Workers/WorkerClient.cs:353-360` — plain `if (_disposed) return; _disposed = true;`, no interlock | |
| GWC-13 | Not started | ✅ still open (partially improved) | `Sessions/GatewaySession.cs:1910-1978` — still a 25 ms poll loop, but the delay now routes through the injected `TimeProvider` (`:1940-1944`), improving testability | Design doc recommended defer-until-used; acceptable. |
| GWC-14 | Done | ✅ holds | `Sessions/SessionEventDistributor.cs:100-118` (preallocated circular `ReplayEntry[]`, head+count, invariants documented), `:776-810` (allocation-free append with capacity overwrite), `:814` (`ReplayEntryAt` modular indexing), `:818-831` (age eviction advances head) | Wraparound math verified (`_replayBuffer.Length == _replayBufferCapacity`; capacity-0 guarded before any modulo). Query paths preserve ascending order. Correct. |
| GWC-15 | Done | ✅ holds | `Grpc/EventStreamService.cs:123-124` (per-stream backlog source registration; gauge reads `Reader.Count` only at scrape), `:197` (disposed before the lease in `finally`); `Metrics/GatewayMetrics.cs:314-320`, `:560-573` (idempotent registration handle) | |
| GWC-16 | Not started | ✅ still open | `Grpc/MxAccessGatewayService.cs:104` (`ResolveSession`) + `:122-123` (`sessionManager.InvokeAsync(request.SessionId, …)`) → `Sessions/SessionManager.cs:163` (`GetRequiredSession`) — still two registry lookups per command | |
| GWC-17 | Not started | ✅ still open | `Sessions/SparseArrayExpander.cs:296-297` (`RpcException(InvalidArgument)` below the gRPC layer); invoked from `Sessions/GatewaySession.cs:1043-1045` (comment explicitly documents the leak) | GWC-03's new cap rejection flows through the same leak. |
| GWC-18 | Not started | ✅ still open | `Sessions/GatewaySession.cs:13` — `public sealed class GatewaySession` declares no `IAsyncDisposable`; `DisposeAsync` at `:1741` | |
| GWC-19 | Not started | ✅ still open | `Sessions/GatewaySession.cs:1684-1688` — `KillWorker(string)` present; grep across `src/` + `clients/` finds zero callers (only `KillWorkerWithCloseGateAsync` is used) | |
| GWC-20 | Not started | ✅ still open | `Configuration/WorkerOptions.cs:30-31` (`HeartbeatIntervalSeconds` doc/name unchanged) bound to the check interval at `Sessions/SessionWorkerClientFactory.cs:86`; `Workers/WorkerClient.cs:458` heartbeat loop still uses raw `Task.Delay` without `TimeProvider` | |
| GWC-21 | Not started | ✅ still open | `Sessions/SessionWorkerClientFactory.cs:83-89` populates 4 of 6 `WorkerClientOptions`; `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` fixed at defaults (`Workers/WorkerClientOptions.cs:13`, `:24`) | Post-GWC-04 the full-mode timeout is now the *sole* structural backpressure bound (see GWC-24), which raises the value of exposing it. |
| GWC-22 | Not started | ✅ still open | `Grpc/EventStreamService.cs:200` — `metrics.StreamDisconnected("Detached")` hardcoded in the `finally`; fault/overflow paths (`:164-174`) do not differentiate | |
| GWC-23 | N/A | ✅ unchanged | Validator comment still documents the knowingly-dead knob | No action, as agreed. |

Also verified in passing (other-domain fixes landing in gateway-core files, all sound): IPC-03 gateway half — pre-checked command envelope size failing only the offending correlation (`Workers/WorkerClient.cs:209-221`, `CommandTooLarge` → `ResourceExhausted` at `Grpc/MxAccessGatewayService.cs:955`); IPC-02 — `GatewayHello.MaxFrameBytes` conveyed (`WorkerClient.cs:986-988`) with startup cross-validation `Worker.MaxMessageBytes >= gRPC cap + 64 KiB reserve` (`Configuration/GatewayOptionsValidator.cs:481-495`); IPC-04 — `DrainEvents max_events` bounded at 10,000 (`Grpc/MxAccessGrpcRequestValidator.cs:13`, `:81-84`); TST-02 — owner-scoped `StreamEvents` attach (`Grpc/EventStreamService.cs:57-62`).

## New findings

IDs continue from the prior register.

### GWC-24 — The GWC-04 staging channel is unbounded: sustained slow consumption now grows memory instead of faulting, and the backlog is invisible to metrics

- **Severity:** Medium
- **Category:** Stability / backpressure regression
- **Location:** `Workers/WorkerClient.cs:93-100` (unbounded `_eventStaging`), `:565-573` (`StageWorkerEvent` `TryWrite` always succeeds), `:610-647` (fault fires only when a *single* timed write exceeds `EventChannelFullModeTimeout`), `:616`/`:627` (`_eventQueueDepth` gauge counts `_events` only)
- **Description:** Before GWC-04, a full `_events` channel blocked the read loop, which propagated backpressure through the pipe to the worker's own bounded queue — end-to-end, structurally bounded. Now the read loop stages events into an unbounded channel and only `EventWriteLoopAsync` faces the bounded `_events`. The sustained-overflow fault fires only when one `WriteAsync` waits longer than 5 s; if the consumer (the distributor pump) drains slower than the worker produces but each individual write completes within the window, `_eventStaging` grows without bound and without any fault — and the growth is invisible, because `SetWorkerEventQueueDepth` reflects only `_events`, never the staging backlog. The code comment (`:28-33`) asserts staging "only fills during the bounded EventChannelFullModeTimeout window", which is true only for a full stall, not for sustained slow drain. Mitigating factor: the pump is non-blocking (per-subscriber `TryWrite` fan-out, `SessionEventDistributor.cs:582-594`), so a persistently slow-but-live consumer requires the pump itself to be starved — unlikely but no longer structurally impossible. The repo's fail-fast backpressure invariant now rests on pump liveness rather than on channel bounds.
- **Recommendation:** Bound the staging channel (e.g. `2 × EventChannelCapacity`) and treat a `TryWrite` miss there as an immediate `ProtocolViolation` fault (a full staging channel means the event writer is already saturated); alternatively track total staged-but-undelivered depth and fault past a ceiling. Include the staging depth in the worker-event queue-depth gauge either way.

### GWC-25 — `ReplayGap` sentinel carries `oldest_available_sequence = 0` when the ring is empty, violating the proto resume contract and dead-streaming a compliant client

- **Severity:** Medium
- **Category:** Correctness / contract
- **Location:** `Sessions/SessionEventDistributor.cs:463-467` (`RegisterWithReplay`, `_replayCount == 0` branch: `oldestAvailableSequence = 0`); emitted verbatim by `Grpc/EventStreamService.cs:133-139` + `:210-222`; contract at `src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto:755-763`
- **Description:** The proto contract states "`oldest_available_sequence` itself IS still retained: a client that wishes to resume without incurring another gap MUST set `after_worker_sequence = oldest_available_sequence - 1`". When a resume finds a gap with an *empty* ring (all entries age-evicted — default `ReplayRetentionSeconds = 300` and `EvictAged` runs inside `RegisterWithReplay` at `:458` — or `ReplayBufferCapacity = 0` with the cursor behind `_highestSequenceSeen`), the sentinel carries `oldest_available_sequence = 0`. A contract-compliant client computes `0 − 1` on a uint64 → `2^64−1`, and its next resume passes `after_worker_sequence = ulong.MaxValue`: `RegisterWithReplay` replays nothing, reports no gap, and the live filter `mxEvent.WorkerSequence <= afterWorkerSequence` (`Grpc/EventStreamService.cs:179`) then drops **every** live event — a silently dead stream with no error. All five clients shipped the `oldest − 1` resume formula in CLI-15 (per the tracking change log), so the failure mode is reachable end-to-end.
- **Recommendation:** In the empty-ring-with-gap branch, emit `oldest_available_sequence = _highestSequenceSeen + 1` (the next sequence that can possibly be delivered), so `oldest − 1 = highestSeen` resumes cleanly; keep 0 only for the never-populated case where no gap is reported anyway. Alternatively define the 0 special case in the proto and fix all five clients — the server-side fix is one line and needs no client changes, so prefer it. Add a distributor test for gap-with-empty-ring and an e2e resume after full age eviction.

### GWC-26 — Alarm monitor attaches its event subscriber only *after* SubscribeAlarms + first reconcile; transitions in that window bypass the feed, and a missed Acknowledge is never broadcast

- **Severity:** Low
- **Category:** Stability / alarm-feed completeness (residual of GWC-01)
- **Location:** `Alarms/GatewayAlarmMonitor.cs:213-214` (subscribe + reconcile) precede the attach at `:232-234`; late-subscriber semantics at `Sessions/SessionEventDistributor.cs:579-582`; `ApplyReconcile` presence-delta-only broadcast at `Alarms/GatewayAlarmMonitor.cs:514-553`
- **Description:** The distributor pump has been running since `MarkReady` (dashboard mirror registers first, `Sessions/GatewaySession.cs:445-449`, `:621`), and a subscriber registered mid-stream misses previously fanned events by design. Alarm transitions arriving between the worker's `SubscribeAlarms` activation and the monitor's internal-subscriber registration (two full worker command round-trips later) are fanned only to the dashboard mirror. Raise/Clear are repaired by the next reconcile, but an Acknowledge in the window updates `_alarms` silently on the next reconcile snapshot replace (`:547-551`) without any `Broadcast` — live `StreamAlarms` subscribers keep showing the alarm unacked until it clears. This is the same defect class GWC-01 fixed, shrunk from "random half of all events, forever" to "a one-shot window of a few hundred ms per monitor lifecycle".
- **Recommendation:** Attach the internal subscriber before invoking `SubscribeAlarmsAsync` (take the lease from `AttachInternalEventSubscriber` explicitly, then subscribe, then drain), or buffer the lease's channel during subscribe/reconcile. Also consider broadcasting an acked-state delta from `ApplyReconcile` when an existing reference's `CurrentState` changes.

### GWC-27 — `AttachInternalEventSubscriber` / `ReadAlarmEventsAsync` bypass the readiness gate; a premature attach permanently poisons the session's event distributor

- **Severity:** Low
- **Category:** Latent lifecycle hazard
- **Location:** `Sessions/GatewaySession.cs:553-562` (no `_state == Ready` check, contrast `AttachEventSubscriber` `:889-916`); `Sessions/SessionManager.cs:195-208` (public `ReadAlarmEventsAsync`, also unchecked)
- **Description:** The internal-attach path starts the distributor pump unconditionally. If invoked before the session is Ready, the pump's source (`MapWorkerEventsAsync` → `GetReadyWorkerClientAsync`, `:740-749`/`:1910`) throws `SessionNotReady`; `PumpAsync` completes all subscribers with the error and latches `_completed` (`Sessions/SessionEventDistributor.cs:603-612`, `:662-680`). Because `_eventDistributorStarted` is never reset (`Sessions/GatewaySession.cs:527-531`) and the distributor is replaced only in `DisposeAsync`, every later registration — including gRPC `StreamEvents` attaches — receives an immediately-completed channel carrying the stale error: a Ready session with permanently dead event streaming. Not reachable today (the alarm monitor, the only caller, attaches after `OpenSessionAsync` returns a Ready session), but the hazard is one future call site away and fails in the worst way (silent, permanent, per-session).
- **Recommendation:** Mirror `AttachEventSubscriber`'s readiness check at the top of `AttachInternalEventSubscriber` (throw `SessionManagerErrorCode.SessionNotReady` before `EnsureDistributorCreated`).

### GWC-28 — Gateway→worker envelope `sequence` is stamped at envelope creation, not at write, so concurrent invokes can emit non-monotonic sequences on the wire

- **Severity:** Low
- **Category:** IPC contract / conventions
- **Location:** `Workers/WorkerClient.cs:1031-1044` (`CreateEnvelope` stamps `Interlocked.Increment(ref _nextSequence)`); enqueue at `:957-972`; single write loop `:390-409`; the gateway `WorkerFrameWriter` does not re-stamp (`Workers/WorkerFrameWriter.cs:35-77`)
- **Description:** Two concurrent `InvokeAsync` callers can stamp sequences 1 and 2 but win the channel `WriteAsync` race in the order 2, 1, putting out-of-order sequences on the pipe. This is exactly the defect WRK-04 fixed on the worker side (its writer now stamps at the point of writing under the write lock); the gateway half was not given the same treatment. Benign today — nothing validates inbound sequence on either side (GWC-10/IPC-10 both open) — but it violates gateway.md's "monotonic per sender" contract and would start faulting sessions the moment worker-side validation is added.
- **Recommendation:** Move the stamp into `WriteLoopAsync` immediately before `_writer.WriteAsync` (single consumer, naturally serialized), matching the worker's WRK-04 fix; coordinate with GWC-10 if enforcement is added.

### GWC-29 — `Invoke` deep-clones the entire request only to discard the cloned command

- **Severity:** Low
- **Category:** Performance (hot path)
- **Location:** `Grpc/MxAccessGatewayService.cs:119-121`; `Grpc/MxAccessGrpcMapper.cs:27-37`
- **Description:** `MxCommandRequest invokeRequest = request.Clone()` deep-clones the request *including its command payload* (potentially a large bulk-write graph), then immediately overwrites `invokeRequest.Command` with `commandToInvoke`, discarding the cloned command. `MapCommand` then performs the one clone that is actually needed (`request.Command.Clone()`). Net: one full wasted command deep-clone on every `Invoke` — the same cost class GWC-07/IPC-05 eliminated on the event and envelope paths, still present on the command path.
- **Recommendation:** Skip the request clone: add a `MapCommand(MxCommand command)` overload (or construct a minimal request) since `MapCommand` reads only the command, and pass `commandToInvoke` directly.

### GWC-30 — Frame reader still allocates a fresh 4-byte prefix array per frame

- **Severity:** Info
- **Category:** Performance (residual of GWC-08)
- **Location:** `Workers/WorkerFrameReader.cs:33`
- **Description:** The GWC-08 design suggested a per-reader scratch buffer for the length prefix; the payload was pooled but the `new byte[sizeof(uint)]` per frame remains. Gen-0, 4 bytes — negligible; recorded only so the next hot-path pass knows it was seen, not missed.
- **Recommendation:** Optional: a reusable per-reader prefix field (the reader is single-consumer by construction).

## Severity roll-up (new findings)

| Severity | Count | IDs |
|----------|:-:|-----|
| Critical | 0 | — |
| High | 0 | — |
| Medium | 2 | GWC-24, GWC-25 |
| Low | 4 | GWC-26, GWC-27, GWC-28, GWC-29 |
| Info | 1 | GWC-30 |

## Still good / preserved

The qualities the prior review praised survived the remediation churn intact. Session state writes remain single-lock disciplined (`_syncRoot` everywhere, including the new fault-timestamp and faulted-reap paths); the close gate still serializes Close/Kill/sweep with atomic TOCTOU re-checks (`TryBeginCloseIfExpired` gained the faulted arm without weakening the reattach-wins race resolution). The replay→live handoff argument still holds in the circular-array rewrite — the `_replayLock`-atomic snapshot+register in `RegisterWithReplay` is unchanged in structure and its no-gap/no-duplicate documentation is exemplary. The rewritten ring's eviction, ordering, and wraparound logic are correct. The GWC-01 fix is a model remediation: it doesn't just fix the bug, it converts the whole bug class into a loud `InvalidOperationException` via `SingleReader = true` plus the claimed-once guard. Repo invariants were respected throughout: no synthesized events (the `ReplayGap` sentinel is a documented control signal, correctly never fanned or drained elsewhere), one worker per session, and the frame wire format is byte-identical after the pooled-I/O rewrite. Convention adherence (file-scoped namespaces, `sealed`, `Async` suffixes, MXAccess-aligned naming) remains strong in all new code; `GatewaySession` has grown to 2,128 lines and the split-out recommendation from the prior report stands.