mxaccessgw/archreview/2026-07-12/00-overall.md

# MxAccessGateway — Architecture Review Update (post-remediation)

Date: 2026-07-12. Branch: `main` at `4f5371f` (merge of `fix/archreview-p2` into main, performed as part of this review so the target includes P0 + P1 + P2 + CI). Working tree: macOS (static review only; NonWindows build + .NET client tests were run to validate the merge, not the findings).

## Scope and method

This is a re-run of the 2026-07-08 review ([`../00-overall.md`](../00-overall.md)), executed the same way: six parallel review agents, one per domain, each with three passes — (1) verify every prior finding's tracked status against actual code at HEAD, (2) deep-review the remediation code itself for new defects, (3) a fresh sweep for anything the first review missed. The domain reports carry the full evidence:

| Report | Domain |
|---|---|
| [10-gateway-core.md](10-gateway-core.md) | Gateway server: sessions, worker lifecycle, IPC client, gRPC streaming, alarms |
| [20-worker.md](20-worker.md) | Worker process: STA pump, COM lifetime, pipe session, event queue |
| [30-contracts-ipc.md](30-contracts-ipc.md) | Protos, frame protocol (both sides), codegen pipeline |
| [40-security-dashboard.md](40-security-dashboard.md) | API-key auth, LDAP dashboard, SignalR hubs, metrics, diagnostics |
| [50-clients.md](50-clients.md) | .NET, Go, Java, Python, Rust clients; cross-client parity |
| [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | Test architecture, documentation currency, CI/ops, repo-wide gaps |

## Overall verdict

The remediation was real. Every one of the prior review's 1 Critical and 12 High findings is verifiably fixed at HEAD, and **not a single finding marked `Done` in the tracker turned out to be false** — the worst cases are partial (ten `Done` claims carry material caveats, detailed below). The new top line is **0 Critical, 1 High** across 47 new findings, and the one High is an automation gap, not a product defect. The risk profile has genuinely shifted from "correctness defects in the core" to "sharp edges in the newly added machinery plus guard gaps around the Windows tier".

Three patterns dominate the new findings:

1. **The remediation's own edges.** Several fixes solve the tracked defect but leave a residual failure mode one layer out: the DrainEvents bound is count-only, so a byte-heavy drain still builds a session-killing oversized frame (WRK-21/IPC-23 — the exact failure the bound exists to prevent); the event-staging channel that decoupled the read loop is unbounded and invisible to the queue-depth gauge (GWC-24); the new gRPC failure limiter can be turned into a lockout-DoS against a legitimate key holder because it partitions on the attacker-supplied key id *before* verification (SEC-31).
2. **ReplayGap shipped with broken ends.** The server emits `oldest_available_sequence = 0` when the ring is empty — a spec-compliant client resume then underflows to `2^64−1` and silently drops every event forever (GWC-25, one-line fix) — and while all five client *libraries* surface the sentinel, the Python CLI crashes on it and the Go CLI destroys it (CLI-35/36).
3. **Guard gaps the CI can't see.** With the Windows jobs removed, the x86 worker build, all of Worker.Tests (including the new priority-scheduler tests), and the live-MXAccess smoke have zero automation (TST-25); the committed Go and Python *worker* bindings are stale on main right now (missing `max_frame_bytes`; IPC-25); and CI's unconditional `git checkout` of the two Java aggregate files masks real Java drift for any message-level proto change (IPC-24).

## Prior-finding verification

- **All Critical/High fixes verified sound at HEAD**, including the alarm-feed split (GWC-01), faulted-session reaping (GWC-02), sparse-array bound (GWC-03), ReadBulk watchdog heartbeat (WRK-01), Go loud overflow (CLI-01), Rust MXSTATUS validation (CLI-03), owner re-validation (TST-02), and CI existence (TST-03, with the Windows-job caveat above).
- **`Done` claims with material caveats (10):**
  - **IPC-04 / WRK-21** — DrainEvents cap is count-based only; oversized reply still faults the whole session and loses drained events.
  - **IPC-09** — codegen guards have two holes: Java drift masking in CI (IPC-24) and no guard at all over Go/Python worker bindings, which are stale at HEAD (IPC-25).
  - **WRK-12** — flush coalescing is correct in the writer but never engages on the event hot path: the drain loop awaits each write+flush serially (`WorkerPipeSession.cs:374-382`), so events still cost one flush each.
  - **WRK-07** — the two-class priority scheduler landed but `docs/MxAccessWorkerInstanceDesign.md` still specifies a five-level order (WRK-26).
  - **SEC-01** — path-rooting shipped, but `IsRootedForAnyPlatform` accepts Windows paths on Unix, so the shipped appsettings still materializes a literal `C:\ProgramData\...` auth DB on macOS at runtime (SEC-33); Galaxy `SnapshotCachePath` has no rooting check.
  - **SEC-06** — production plaintext-LDAP hard-stop is enforced, but the committed dev service-account password is still in `appsettings.json`, unrotated (SEC-36).
  - **CLI-15** — library-complete in all five clients, but the Python CLI crashes on ReplayGap and the Go CLI drops it (CLI-35/36).
  - **CLI-02** — Rust packaging fix is sound; the required README/ClientPackaging doc updates were skipped (CLI-42).
  - **CLI-18/21/26/29** — versions aligned to the already-published 0.1.2 despite breaking API changes since; next publish collides or ships a different API under the same version (CLI-39).
  - **Docs claim** (`ClientLibrariesDesign.md`) that typed helpers use `hresult < 0` — false for .NET/Go/Java, which still use `!= 0` (CLI-08 remains open inside the new surfaces; CLI-38).
- **Incidentally fixed open items:** CLI-24 (Java single-consumer javadoc) and CLI-34 (gitignore coverage) can be closed in the tracker.
- All other open findings were confirmed still present at HEAD; none silently regressed.

## Severity roll-up (new findings only)

| Domain | Critical | High | Medium | Low | Info | IDs |
|---|:-:|:-:|:-:|:-:|:-:|---|
| Gateway core | — | — | 2 | 4 | 1 | GWC-24..30 |
| Worker | — | — | 1 | 7 | — | WRK-21..28 |
| Contracts & IPC | — | — | 3 | 5 | 2 | IPC-23..32 |
| Security & dashboard | — | — | 1 | 4 | 1 | SEC-31..36 |
| Clients | — | — | 5 | 6 | — | CLI-35..45 |
| Testing, docs & gaps | — | 1 | 2 | 2 | — | TST-25..29 |
| **Total** | **0** | **1** | **14** | **28** | **4** | **47** |

(Prior review: 1 Critical, 12 High, 54 Medium, 39 Low.)

## High and headline findings

### High

- **TST-25 — the entire Windows/x86 test tier has zero automation.** With the `windows`/`live-mxaccess` CI jobs removed (`abb0930`), the x86 worker build, all of Worker.Tests — including the tests for the *new* priority write scheduler, negotiated frame max, and drain bound — and the live-MXAccess smoke run only when someone remembers the manual windev worktree process, while `docs/GatewayTesting.md` and `check-codegen.ps1` still describe the removed job as "the definitive guard" (TST-26). Cheapest fix: an SSH-driven scheduled step against windev rather than waiting on a Windows act_runner. ([60](60-testing-docs-gaps.md))

### Medium — headline items

- **WRK-21 / IPC-23 — DrainEvents byte-blindness is still session-fatal.** The 10,000-count cap doesn't bound bytes; array-heavy events averaging >~1.7 KiB build a reply over `MaxMessageBytes`, the writer's rejection is rethrown out of `WorkerPipeSession.RunAsync`, the worker exits, and the already-drained events are lost. Cap the reply by serialized size as well as count. ([20](20-worker.md), [30](30-contracts-ipc.md))
- **GWC-25 — ReplayGap `oldest_available_sequence = 0` on an empty ring.** Violates the proto contract; a compliant client resumes at `0 − 1`, underflows, and the live filter silently drops every event forever. Reachable via the default 300 s age eviction. One-line fix (`_highestSequenceSeen + 1`). ([10](10-gateway-core.md))
- **GWC-24 — the event staging channel is unbounded and invisible.** The GWC-04 decoupling removed end-to-end pipe backpressure; a slow-but-live consumer grows gateway memory without bound, without fault, and the queue-depth gauge doesn't count it. ([10](10-gateway-core.md))
- **SEC-31 — failure-limiter lockout DoS.** The gRPC limiter partitions on the key id parsed from the *unauthenticated* token and blocks before verification: any peer who knows a key id (non-secret) can send 10 garbage requests/min and lock out the legitimate holder indefinitely. Partition by peer, or count failures only after verification. ([40](40-security-dashboard.md))
- **IPC-25 / IPC-24 — codegen guard holes, one already live.** Go and Python worker bindings on main lack `max_frame_bytes` (stale since the 2026-07-09 proto change; nothing checks them); CI's unconditional checkout of `MxaccessGateway.java`/`MxaccessWorker.java` reverts exactly the files where real message-level Java drift would appear. ([30](30-contracts-ipc.md))
- **CLI-35/36/37/38 — the new client surfaces re-diverge.** Python CLI crashes on ReplayGap; Go CLI prints a nil event instead of the gap; four clients branch status checks on `success` while the proto mandates `category` (only .NET complies — identical replies pass four clients and throw in one); the `< 0` HRESULT rule is documented but implemented only in Rust/Python. ([50](50-clients.md))
- **CLI-39 — version constants aligned to an already-published version.** 0.1.2 now labels a breaking-changed API (Rust stream item type, Python stream union); bump before any publish. ([50](50-clients.md))
- **TST-27 — `ShowTagValues` documented as "Reserved" but now live.** SEC-25 wired it into the SignalR mirror redaction; a security-relevant flag is documented as a no-op. ([60](60-testing-docs-gaps.md))

## Cross-cutting themes

1. **Residual failure modes one layer out from each fix.** The remediation pattern to watch: the tracked defect is fixed, but the same failure re-emerges at the next boundary (drain bound vs frame max; read-loop decoupling vs unbounded staging; per-frame rejection vs event frames still session-fatal one layer up, IPC-30). Fixes here should be specified against the *invariant* ("no diagnostics command may kill a session", "gateway memory per session is bounded") rather than the symptom.
2. **The reconnect/replay story needs an end-to-end walk.** Server sentinel arithmetic (GWC-25), two broken CLIs (CLI-35/36), and no .NET session re-attach (CLI-05) mean the shipped, on-by-default resilience feature still can't be exercised cleanly end-to-end by a user in every language.
3. **Cross-client semantic parity regressed inside the new code.** The typed-command epic achieved surface parity but re-introduced behavioral divergence: HRESULT sign semantics (3v2 split), success-vs-category (4v1), credential-scrub strength, malformed-AuthenticateUser handling, and even credential env-var names (CLI-45). A single conformance checklist per behavior, tested per client, is the fix shape.
4. **Automation guards still trail the code.** CI exists and is green, but the Windows tier is unguarded (TST-25), two vendored binding sets drifted already (IPC-25), Java drift is masked by the pipeline itself (IPC-24), and the descriptor-freshness *test* is blind to enums/services (IPC-27).
5. **The same-commit docs rule was violated by the remediation itself.** TST-26 (removed CI jobs still documented), TST-27 (`ShowTagValues`), WRK-26 (five-level priority spec vs two-class implementation), CLI-42 (Rust packaging docs) — worth one batch pass, since the repo rule exists precisely to prevent this accumulation.
6. **Security posture: structural controls hold; new machinery needs a second pass.** Fail-closed interceptor, constant-time compare, redaction seam, owner-scoped streams all verified intact. The new limiter (SEC-31/32), the 15 s verification-cache window vs revocation/expiry (SEC-34), the Unix path-rooting residual (SEC-33), and the still-committed dev LDAP password (SEC-36) are the follow-ups.

## Prioritized roadmap

**P0 — correctness/safety, all small**
1. GWC-25: emit `_highestSequenceSeen + 1` as `oldest_available_sequence` on an empty ring (one line) + CLI-35/36 CLI ReplayGap handling.
2. WRK-21/IPC-23 (+ IPC-30): byte-aware DrainEvents reply capping; no diagnostics command may be session-fatal.
3. SEC-31: re-partition the failure limiter (per-peer, or post-verification counting); SEC-32 while in there.
4. IPC-25: regenerate Go/Python worker bindings and add them to check-codegen; IPC-24: replace CI's unconditional Java checkout with a real churn-vs-drift discriminator.

**P1 — process and hardening**
5. TST-25: automated Windows tier (SSH-driven windev job for x86 build + Worker.Tests; scheduled live smoke) and fix the docs that claim it exists (TST-26).
6. GWC-24: bound the staging channel (or fault on sustained growth) and include it in the queue-depth gauge.
7. Client conformance pass: `< 0` HRESULT + `category` branching everywhere (CLI-37/38, closes CLI-08), env-var name alignment (CLI-45), version bump before publish (CLI-39).
8. Doc-drift batch: TST-27, WRK-26, CLI-42, plus SEC-33/36 residuals.

**P2 — the rest**
9. New Lows (WRK-22 cancelled-write ghost frames, WRK-24 tiny negotiated max, WRK-27 alarm-poll watchdog bypass, IPC-26/27, SEC-34 cache/revocation window, GWC-26/27/28) alongside the pre-existing open backlog (GWC-05/09/10…, WRK-02/03/05…, SEC-13…27, CLI-05…33, TST-05…24), which this review confirmed unchanged.

## What is demonstrably good

The prior review's "preserve under change" list survived the remediation intact: STA pump and COM teardown discipline, inline-proven session locking, frame-protocol validation (now with negotiated maxima and headroom cross-validation at startup), additive-only proto evolution with byte-identical vendored copies, fail-closed auth with constant-time comparison, and the layered fake-worker test architecture. New entries worth preserving: the owner-scoped `StreamEvents` trust boundary, the per-correlation `CommandTooLarge` → `ResourceExhausted` path (fault isolation done right — the model the DrainEvents fix should copy), the pooled single-write gateway framing, and the two-class priority write scheduler's control-before-event guarantee.