# MxAccessGateway — Overall Architecture Review

Date: 2026-07-08. Branch: `feat/jdk17-client-retarget`. Working tree: macOS (static review only; no builds or test runs).

## Scope and method

Six parallel review agents each examined one domain, reading source directly and citing verified `path:line` evidence. This document synthesizes their findings; the domain reports carry the full evidence and recommendations:

| Report | Domain |
|---|---|
| [10-gateway-core.md](10-gateway-core.md) | Gateway server: sessions, worker lifecycle, IPC client, gRPC streaming, alarms |
| [20-worker.md](20-worker.md) | Worker process: STA pump, COM lifetime, pipe session, event queue |
| [30-contracts-ipc.md](30-contracts-ipc.md) | Protos, frame protocol (both sides), codegen pipeline |
| [40-security-dashboard.md](40-security-dashboard.md) | API-key auth, LDAP dashboard, SignalR hubs, metrics, diagnostics |
| [50-clients.md](50-clients.md) | .NET, Go, Java, Python, Rust clients; cross-client parity matrix |
| [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | Test architecture, documentation currency, CI/ops, repo-wide gaps |

Review lenses: stability, performance, conventions, underdeveloped areas.

## Overall verdict

The core architecture is sound and unusually well executed for its stage. The hard problems the two-process design exists to solve are solved correctly: the worker runs a real `MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage` STA pump with strict COM confinement and disciplined `Marshal.FinalReleaseComObject` teardown; the gateway's session lifecycle machinery (locking, close/kill gating, TOCTOU re-checks, replay→live handoff) is carefully proven and documented inline; the frame protocol validates lengths before allocating, reads exactly, and follows real proto-evolution hygiene; auth is a single fail-closed interceptor over peppered-HMAC keys with constant-time comparison; and the five language clients are more uniform and complete than most multi-language SDK efforts.

The risk is concentrated in four places, none of them structural rewrites:

1. **One confirmed critical defect** — the alarm monitor and the session event distributor both drain the same worker event channel, silently splitting the alarm feed.
2. **Silent failure modes at the edges** — sessions and streams that die or drop work without an error reaching anyone (faulted sessions never reaped, Go stream closes silently on overflow, worker drops post-shutdown commands without reply, Rust never checks MXSTATUS).
3. **Nothing is guarded by automation** — zero CI across a five-language, dual-runtime matrix; every known codegen fragility, the stale client descriptor set, and the Generated/-commit rule are protected only by operator memory.
4. **A half-shipped session-resilience epic** — reconnect/replay shipped server-side and is on by default, but owner re-validation (a security control), client-side `ReplayGap` handling, and the end-to-end test remain undone.

## Severity roll-up

Counts from the domain digests; the sub-reports are authoritative.

| Domain | Critical | High | Medium | Low |
|---|---|---|---|---|
| Gateway core | 1 | 2 | 6 | 11 |
| Worker | — | 1 | 6 | 11 |
| Contracts & IPC | — | 1 | 8 | 8 |
| Security & dashboard | — | — | 10 | 5 |
| Clients | — | 4 | 11 | 2 |
| Testing, docs & gaps | — | 4 | 13 | 2 |
| **Total** | **1** | **12** | **54** | **39** |

## Critical and high findings

### Critical

- **Alarm feed split between two consumers.** `GatewayAlarmMonitor` and the per-session distributor pump both drain the same worker event channel, so alarm transitions are randomly stolen by whichever consumer wins; missed acks are never repaired by reconcile. `Alarms/GatewayAlarmMonitor.cs:229` vs `Sessions/GatewaySession.cs:554-583`. ([10](10-gateway-core.md))

### High — stability

- **Faulted sessions are never reaped.** `MarkFaulted` neither kills the worker nor schedules teardown; the sweeper only checks lease/detach-grace, pinning the worker process and session slot for up to 30 minutes. `Sessions/SessionManager.cs:273`. ([10](10-gateway-core.md))
- **Sparse-array bound unimplemented.** The doc-promised configurable max length is missing; only `Array.MaxLength` gates expansion, so one Write can force a multi-GB allocation before the frame-size check. `Sessions/SparseArrayExpander.cs:65`. ([10](10-gateway-core.md))
- **ReadBulk trips the stuck-STA watchdog.** Bulk reads over uncached tags freeze `LastActivityUtc` past the 75 s ceiling, so a healthy session is declared hung and its replies dropped. `MxAccessSession.cs:919-931` + `StaRuntime.cs:90`. ([20](20-worker.md))
- **Go event stream fails silently.** On a 16-slot buffer overflow the client cancels and closes the channel with no terminal error — indistinguishable from graceful end. `clients/go/mxgateway/session.go:751`. ([50](50-clients.md))
- **Rust never validates MXSTATUS.** `invoke` ignores hresult/status arrays, so failed writes can read as success. `clients/rust/src/error.rs:214`. ([50](50-clients.md))

### High — release and process integrity

- **No CI anywhere.** No pipeline files exist; the entire five-language, dual-runtime matrix is verified by hand. ([60](60-testing-docs-gaps.md))
- **Client descriptor set seven weeks stale.** The published protoset lacks `MxSparseArray`/`ReplayGap`/provider-status; the `-Check` script exists but nothing runs it. `clients/proto/descriptors/`. ([30](30-contracts-ipc.md))
- **Rust crate unbuildable outside the repo.** `build.rs` reads `../../src/.../Protos` and packaging hides it with `--no-verify`. `clients/rust/build.rs:8`. ([50](50-clients.md))
- **Session-resilience epic half-shipped.** 16 of 28 tasks open, including reconnect owner re-validation (security), client `ReplayGap` handling (no client implements it), and the replay end-to-end test — for behavior on by default (`DetachGrace=30`, `ReplayBuffer=1024`; CLAUDE.md wrongly claims no-retention defaults). ([60](60-testing-docs-gaps.md), [50](50-clients.md))
- **Typed-command parity gap across all clients.** No client exposes typed `WriteSecured`/`AuthenticateUser`/`AdviseSupervisory`/buffered helpers; the secured-write path is reachable only via raw Invoke. ([50](50-clients.md))

## Cross-cutting themes

**1. Silent failure modes.** The system's fail-fast philosophy is right, but several edges fail *silently* instead: faulted-session limbo (gateway), post-shutdown command drops with no reply and silent STA-thread death (worker), Go stream close, Rust status blindness, and the client stream-buffer overflow behavior diverging per language (Java errors, Go doesn't). The fix pattern is uniform — every death or drop must produce an observable error or fault frame.

**2. Backpressure and size-limit topology.** Related limits are set independently and interact badly: the worker hard-pins frames at 16 MiB while the gateway's `MaxMessageBytes` config (up to 256 MiB) is never conveyed to it; gRPC and pipe limits are equal with no envelope headroom; `DrainEvents max_events=0` builds one unbounded frame; the gateway read loop blocks up to 5 s per event behind a full channel, stalling command replies and heartbeats; clients hard-code 16-slot stream buffers. A legitimate large payload or event burst converts into whole-session death rather than a per-command error. This cluster deserves one coordinated design pass ([30](30-contracts-ipc.md), [10](10-gateway-core.md), [20](20-worker.md), [50](50-clients.md)).

**3. Everything is guarded by operator memory, not automation.** No CI, stale descriptor set, unpinned Python grpcio-tools range, Java's tracked-generated-file churn with no check, the Generated/-commit-after-regen rule undocumented, `--no-verify` Rust packaging. A minimal CI (build + targeted tests + descriptor/codegen freshness checks) removes an entire failure class at once.

**4. Documentation drift in load-bearing places.** gateway.md's `WorkerEnvelope` sketch has wrong types and field numbers; CLAUDE.md misstates retention defaults; the dashboard cookie is `MxGatewayDashboard`, not the documented `__Host-MxGatewayDashboard`; docs' `GroupToRole` sample value now fails startup validation; Java docs still say JDK 21; `docs/Grpc.md` omits `QueryActiveAlarms`. The repo's own rule — docs change in the same commit as behavior — has not been enforced retroactively.

**5. Cross-platform path defaults.** Windows-literal defaults (`AuthenticationOptions.SqlitePath`, `TlsOptions` PFX path) become CWD-relative filenames on non-Windows hosts — confirmed by a stray `C:\ProgramData\MxGateway\gateway-auth.db` file created inside the source tree (empty and gitignored this time; the same defect would write a private-key PFX into the tree). ([40](40-security-dashboard.md))

**6. Policy-layer security tightening.** The crypto and interceptor foundations are solid; the gaps are policy: `AllowAnonymousLocalhost` satisfies AdminOnly (loopback processes reach API-key inventory and any session's events), `DisableLogin` auto-admins remote requests with no production guard, `QueryActiveAlarms` silently requires admin because it is missing from the scope resolver (masked by mislabeled tests), hub bearer tokens are irrevocable for 30 minutes and accepted via query string, plaintext LDAP with a committed service-account password, no key expiry, and no rate limiting or lockout. ([40](40-security-dashboard.md))

**7. Event hot-path performance headroom.** Not urgent, but consistent across processes: every event is deep-cloned at least twice (worker queue clone, gateway `MapEvent` clone), the worker converts `MXSTATUS_PROXY` via reflection per status, events are written one frame + flush at a time on a 25 ms poll with no batching, the gateway frame reader/writer allocate per frame (worker side already uses `ArrayPool`), and a `Stopwatch` is allocated per streamed event. These compound under alarm bursts — the exact load the system exists to handle.

## Prioritized roadmap

**P0 — correctness and safety (do before wider rollout)**
1. Fix the alarm-monitor/distributor channel split (the one critical).
2. Reap faulted sessions: `MarkFaulted` must terminate the worker and schedule teardown.
3. Implement the sparse-array max-length bound.
4. Fix the ReadBulk/watchdog interaction (activity heartbeat during long COM calls).
5. Close the reconnect owner re-validation gap or flip retention defaults off until it lands; correct the CLAUDE.md claim.
6. Fix Go silent stream close and Rust MXSTATUS validation.

**P1 — process and hardening (next few weeks)**
7. Stand up minimal CI: NonWindows build + gateway fake-worker tests + client builds/tests + descriptor freshness check; add a Windows job for the x86 worker when runner access allows.
8. One coordinated pass on the size/backpressure topology (convey `MaxMessageBytes` to the worker, bound `DrainEvents`, decouple event backlog from command replies, make client buffer-overflow behavior uniform and loud).
9. Security policy pass: scope-resolver entry for `QueryActiveAlarms`, constrain `AllowAnonymousLocalhost`/`DisableLogin` (production guard), LDAPS or StartTLS, remove the committed service password, revocable hub tokens.
10. Cross-platform default paths (SQLite, PFX) via `Environment.SpecialFolder`/config requirement on non-Windows.
11. Rust crate packaging fix (vendor protos into the crate; drop `--no-verify`).

**P2 — completeness and polish**
12. Finish the session-resilience epic: client `ReplayGap` handling in all five clients, replay e2e test, per-session event ACL (`EventsHub.cs:39` TODO).
13. Typed `WriteSecured`/`AuthenticateUser`/buffered-read helpers across clients.
14. Event hot-path performance pass (kill double clones, cache the status converter, batch event frames, pool gateway frame buffers).
15. Documentation-drift sweep (gateway.md envelope sketch, cookie name, role sample, Java 17, Grpc.md RPC table, ClientPackaging.md) and one-time repo-root triage (oldtasks.md, stillpending.md, *-docs-issues.md, code-reviews/).
16. Version alignment across server assemblies and the five clients.

## What is demonstrably good

Worth stating so it is preserved under change: the STA pump and COM teardown discipline; the gateway's inline-proven locking and lifecycle reasoning; frame-protocol length validation and typed errors; proto evolution hygiene (reserved fields, UNSPECIFIED-zero enums, byte-identical galaxy proto vs the shared package); fail-closed auth with constant-time comparison and a real redaction seam (no secret-logging call sites found); the layered test architecture with a high-fidelity `FakeWorkerHarness` that reuses production frame code; config docs whose tables match code defaults; and five clients with an unusually consistent surface, fixture-driven tests, and a clean, verified JDK 17 retarget.
