mxaccessgw/archreview/00-overall.md

MxAccessGateway — Overall Architecture Review

Date: 2026-07-08. Branch: feat/jdk17-client-retarget. Working tree: macOS (static review only; no builds or test runs).

Scope and method

Six parallel review agents each examined one domain, reading source directly and citing verified path:line evidence. This document synthesizes their findings; the domain reports carry the full evidence and recommendations:

Report	Domain
10-gateway-core.md	Gateway server: sessions, worker lifecycle, IPC client, gRPC streaming, alarms
20-worker.md	Worker process: STA pump, COM lifetime, pipe session, event queue
30-contracts-ipc.md	Protos, frame protocol (both sides), codegen pipeline
40-security-dashboard.md	API-key auth, LDAP dashboard, SignalR hubs, metrics, diagnostics
50-clients.md	.NET, Go, Java, Python, Rust clients; cross-client parity matrix
60-testing-docs-gaps.md	Test architecture, documentation currency, CI/ops, repo-wide gaps

Review lenses: stability, performance, conventions, underdeveloped areas.

Overall verdict

The core architecture is sound and unusually well executed for its stage. The hard problems the two-process design exists to solve are solved correctly: the worker runs a real MsgWaitForMultipleObjectsEx + PeekMessage/DispatchMessage STA pump with strict COM confinement and disciplined Marshal.FinalReleaseComObject teardown; the gateway's session lifecycle machinery (locking, close/kill gating, TOCTOU re-checks, replay→live handoff) is carefully proven and documented inline; the frame protocol validates lengths before allocating, reads exactly, and follows real proto-evolution hygiene; auth is a single fail-closed interceptor over peppered-HMAC keys with constant-time comparison; and the five language clients are more uniform and complete than most multi-language SDK efforts.

The risk is concentrated in four places, none of them structural rewrites:

1. One confirmed critical defect — the alarm monitor and the session event distributor both drain the same worker event channel, silently splitting the alarm feed.
2. Silent failure modes at the edges — sessions and streams that die or drop work without an error reaching anyone (faulted sessions never reaped, Go stream closes silently on overflow, worker drops post-shutdown commands without reply, Rust never checks MXSTATUS).
3. Nothing is guarded by automation — zero CI across a five-language, dual-runtime matrix; every known codegen fragility, the stale client descriptor set, and the Generated/-commit rule are protected only by operator memory.
4. A half-shipped session-resilience epic — reconnect/replay shipped server-side and is on by default, but owner re-validation (a security control), client-side ReplayGap handling, and the end-to-end test remain undone.

Severity roll-up

Counts from the domain digests; the sub-reports are authoritative.

Domain	Critical	High	Medium	Low
Gateway core	1	2	6	11
Worker	—	1	6	11
Contracts & IPC	—	1	8	8
Security & dashboard	—	—	10	5
Clients	—	4	11	2
Testing, docs & gaps	—	4	13	2
Total	1	12	54	39

Critical and high findings

Critical

• Alarm feed split between two consumers. GatewayAlarmMonitor and the per-session distributor pump both drain the same worker event channel, so alarm transitions are randomly stolen by whichever consumer wins; missed acks are never repaired by reconcile. Alarms/GatewayAlarmMonitor.cs:229 vs Sessions/GatewaySession.cs:554-583. (10)

High — stability

• Faulted sessions are never reaped. MarkFaulted neither kills the worker nor schedules teardown; the sweeper only checks lease/detach-grace, pinning the worker process and session slot for up to 30 minutes. Sessions/SessionManager.cs:273. (10)
• Sparse-array bound unimplemented. The doc-promised configurable max length is missing; only Array.MaxLength gates expansion, so one Write can force a multi-GB allocation before the frame-size check. Sessions/SparseArrayExpander.cs:65. (10)
• ReadBulk trips the stuck-STA watchdog. Bulk reads over uncached tags freeze LastActivityUtc past the 75 s ceiling, so a healthy session is declared hung and its replies dropped. MxAccessSession.cs:919-931 + StaRuntime.cs:90. (20)
• Go event stream fails silently. On a 16-slot buffer overflow the client cancels and closes the channel with no terminal error — indistinguishable from graceful end. clients/go/mxgateway/session.go:751. (50)
• Rust never validates MXSTATUS. invoke ignores hresult/status arrays, so failed writes can read as success. clients/rust/src/error.rs:214. (50)

High — release and process integrity

• No CI anywhere. No pipeline files exist; the entire five-language, dual-runtime matrix is verified by hand. (60)
• Client descriptor set seven weeks stale. The published protoset lacks MxSparseArray/ReplayGap/provider-status; the -Check script exists but nothing runs it. clients/proto/descriptors/. (30)
• Rust crate unbuildable outside the repo. build.rs reads ../../src/.../Protos and packaging hides it with --no-verify. clients/rust/build.rs:8. (50)
• Session-resilience epic half-shipped. 16 of 28 tasks open, including reconnect owner re-validation (security), client ReplayGap handling (no client implements it), and the replay end-to-end test — for behavior on by default (DetachGrace=30, ReplayBuffer=1024; CLAUDE.md wrongly claims no-retention defaults). (60, 50)
• Typed-command parity gap across all clients. No client exposes typed WriteSecured/AuthenticateUser/AdviseSupervisory/buffered helpers; the secured-write path is reachable only via raw Invoke. (50)

Cross-cutting themes

1. Silent failure modes. The system's fail-fast philosophy is right, but several edges fail silently instead: faulted-session limbo (gateway), post-shutdown command drops with no reply and silent STA-thread death (worker), Go stream close, Rust status blindness, and the client stream-buffer overflow behavior diverging per language (Java errors, Go doesn't). The fix pattern is uniform — every death or drop must produce an observable error or fault frame.

2. Backpressure and size-limit topology. Related limits are set independently and interact badly: the worker hard-pins frames at 16 MiB while the gateway's MaxMessageBytes config (up to 256 MiB) is never conveyed to it; gRPC and pipe limits are equal with no envelope headroom; DrainEvents max_events=0 builds one unbounded frame; the gateway read loop blocks up to 5 s per event behind a full channel, stalling command replies and heartbeats; clients hard-code 16-slot stream buffers. A legitimate large payload or event burst converts into whole-session death rather than a per-command error. This cluster deserves one coordinated design pass (30, 10, 20, 50).

3. Everything is guarded by operator memory, not automation. No CI, stale descriptor set, unpinned Python grpcio-tools range, Java's tracked-generated-file churn with no check, the Generated/-commit-after-regen rule undocumented, --no-verify Rust packaging. A minimal CI (build + targeted tests + descriptor/codegen freshness checks) removes an entire failure class at once.

4. Documentation drift in load-bearing places. gateway.md's WorkerEnvelope sketch has wrong types and field numbers; CLAUDE.md misstates retention defaults; the dashboard cookie is MxGatewayDashboard, not the documented __Host-MxGatewayDashboard; docs' GroupToRole sample value now fails startup validation; Java docs still say JDK 21; docs/Grpc.md omits QueryActiveAlarms. The repo's own rule — docs change in the same commit as behavior — has not been enforced retroactively.

5. Cross-platform path defaults. Windows-literal defaults (AuthenticationOptions.SqlitePath, TlsOptions PFX path) become CWD-relative filenames on non-Windows hosts — confirmed by a stray C:\ProgramData\MxGateway\gateway-auth.db file created inside the source tree (empty and gitignored this time; the same defect would write a private-key PFX into the tree). (40)

6. Policy-layer security tightening. The crypto and interceptor foundations are solid; the gaps are policy: AllowAnonymousLocalhost satisfies AdminOnly (loopback processes reach API-key inventory and any session's events), DisableLogin auto-admins remote requests with no production guard, QueryActiveAlarms silently requires admin because it is missing from the scope resolver (masked by mislabeled tests), hub bearer tokens are irrevocable for 30 minutes and accepted via query string, plaintext LDAP with a committed service-account password, no key expiry, and no rate limiting or lockout. (40)

7. Event hot-path performance headroom. Not urgent, but consistent across processes: every event is deep-cloned at least twice (worker queue clone, gateway MapEvent clone), the worker converts MXSTATUS_PROXY via reflection per status, events are written one frame + flush at a time on a 25 ms poll with no batching, the gateway frame reader/writer allocate per frame (worker side already uses ArrayPool), and a Stopwatch is allocated per streamed event. These compound under alarm bursts — the exact load the system exists to handle.

Prioritized roadmap

P0 — correctness and safety (do before wider rollout)

1. Fix the alarm-monitor/distributor channel split (the one critical).
2. Reap faulted sessions: MarkFaulted must terminate the worker and schedule teardown.
3. Implement the sparse-array max-length bound.
4. Fix the ReadBulk/watchdog interaction (activity heartbeat during long COM calls).
5. Close the reconnect owner re-validation gap or flip retention defaults off until it lands; correct the CLAUDE.md claim.
6. Fix Go silent stream close and Rust MXSTATUS validation.

P1 — process and hardening (next few weeks) 7. Stand up minimal CI: NonWindows build + gateway fake-worker tests + client builds/tests + descriptor freshness check; add a Windows job for the x86 worker when runner access allows. 8. One coordinated pass on the size/backpressure topology (convey MaxMessageBytes to the worker, bound DrainEvents, decouple event backlog from command replies, make client buffer-overflow behavior uniform and loud). 9. Security policy pass: scope-resolver entry for QueryActiveAlarms, constrain AllowAnonymousLocalhost/DisableLogin (production guard), LDAPS or StartTLS, remove the committed service password, revocable hub tokens. 10. Cross-platform default paths (SQLite, PFX) via Environment.SpecialFolder/config requirement on non-Windows. 11. Rust crate packaging fix (vendor protos into the crate; drop --no-verify).

P2 — completeness and polish 12. Finish the session-resilience epic: client ReplayGap handling in all five clients, replay e2e test, per-session event ACL (EventsHub.cs:39 TODO). 13. Typed WriteSecured/AuthenticateUser/buffered-read helpers across clients. 14. Event hot-path performance pass (kill double clones, cache the status converter, batch event frames, pool gateway frame buffers). 15. Documentation-drift sweep (gateway.md envelope sketch, cookie name, role sample, Java 17, Grpc.md RPC table, ClientPackaging.md) and one-time repo-root triage (oldtasks.md, stillpending.md, *-docs-issues.md, code-reviews/). 16. Version alignment across server assemblies and the five clients.

What is demonstrably good

Worth stating so it is preserved under change: the STA pump and COM teardown discipline; the gateway's inline-proven locking and lifecycle reasoning; frame-protocol length validation and typed errors; proto evolution hygiene (reserved fields, UNSPECIFIED-zero enums, byte-identical galaxy proto vs the shared package); fail-closed auth with constant-time comparison and a real redaction seam (no secret-logging call sites found); the layered test architecture with a high-fidelity FakeWorkerHarness that reuses production frame code; config docs whose tables match code defaults; and five clients with an unusually consistent surface, fixture-driven tests, and a clean, verified JDK 17 retarget.