mxaccessgw/archreview/60-testing-docs-gaps.md

# Testing, Documentation & Underdeveloped Areas — Architecture Review

## Scope & method

This review covers test architecture (`src/ZB.MOM.WW.MxGateway.Tests`, `src/ZB.MOM.WW.MxGateway.Worker.Tests`, `src/ZB.MOM.WW.MxGateway.IntegrationTests`, `scripts/`), documentation currency (`gateway.md`, `docs/Sessions.md`, `docs/GatewayConfiguration.md`, `docs/DesignDecisions.md`, `docs/GatewayTesting.md`, CLAUDE.md, repo-root working artifacts), and repo-wide underdeveloped areas (TODO sweep, half-shipped features, operational gaps). Method is static reading of code and docs on the macOS tree; no test suites were run. Generated code, `bin/`, and `obj/` are excluded. All paths are relative to `/Users/dohertj2/Desktop/MxAccessGateway`.

## Executive summary

- Test coverage of the gateway core is strong and unusually well-layered: 46 gateway test classes, 34 worker test classes, and an 8-scenario opt-in live MXAccess suite map cleanly onto session lifecycle, pipe framing, event fan-out/replay, auth, and dashboard. The `FakeWorkerHarness` reuses the production `WorkerFrameReader`/`WorkerFrameWriter`, so fake tests exercise real frame validation rather than a mock transport.
- There is **no CI configuration anywhere in the repository** — no `.github/`, `.gitea/`, or pipeline files. Every documented verification step (per-task filtered `dotnet test`, five client toolchains, opt-in live matrices) is manual operator discipline; nothing prevents a cross-component regression from landing on `main` untested.
- The session-resilience epic is half-shipped: 12 of 28 tasks are merged (`oldtasks.md`), leaving reconnect owner re-validation (a security gap), client-side `ReplayGap` handling in all five clients, per-session dashboard ACL, and orphan-worker reattach pending. The server emits a reconnect protocol no client yet understands.
- Fake-vs-real fidelity has one structural blind spot by design: `FakeWorkerHarness` cannot simulate STA behavior, COM faults, process crash, or MXAccess semantics — those live only in the opt-in `WorkerLiveMxAccessSmokeTests` and dev-rig probes, which never run automatically.
- Timing-based tests are mostly disciplined (`ManualTimeProvider`, cancellation-token-bounded polls), but a handful of real-clock sleeps followed by negative assertions remain (`WorkerClientTests.cs:433`, `SessionManagerTests.cs:402`) and are latent flakes under load. The known macOS `SessionWorkerClientFactoryFakeWorkerTests` timeout failures are pre-existing and environmental.
- Documentation currency is better than typical: `docs/GatewayConfiguration.md` defaults match `Configuration/*.cs` exactly, and `docs/Sessions.md`/`docs/DesignDecisions.md` reflect the shipped reconnect/fan-out behavior. The stale spots are concentrated in `gateway.md` (design-era `WorkerEnvelope` sketch, single-subscriber policy line, never-built `Session` bidi RPC) and one materially wrong sentence in CLAUDE.md about default retention.
- Repo-root hygiene is mixed: `stillpending.md` and `oldtasks.md` are tracked, annotated snapshots that still serve as the de-facto backlog (there is no issue tracker in evidence); the five untracked `*-docs-{issues,fixed,final}.md` artifacts (15k+ lines) are dead local working files; `code-reviews/` + `REVIEW-PROCESS.md` are a completed, self-consistent review system with zero open findings.
- Operationally the gateway is under-instrumented for a service: one health check (auth-store) with no Galaxy SQL, LDAP, alarm-monitor, or worker-launchability checks; hand-rolled NSSM deployment with config living in service env vars outside the repo; and no version stamping on the server/worker assemblies while client packages drift (Java 0.2.0 vs 0.1.2 everywhere else).

## Findings

### Coverage map

| Component | Unit | Fake-worker integration | Live integration | Gap notes |
|---|---|---|---|---|
| Session lifecycle (open/close/lease/detach-grace) | `SessionManagerTests`, `GatewaySessionTests` | `GatewayEndToEndFakeWorkerSmokeTests`, `SessionWorkerClientFactoryFakeWorkerTests` | `WorkerLiveMxAccessSmokeTests` | `SessionLeaseMonitorHostedService` timer loop itself untested (sweep logic covered via `CloseExpiredLeasesAsync` tests at `SessionManagerTests.cs:859-930`) |
| Worker launch / validation / orphan cleanup | `WorkerProcessLauncherTests`, `WorkerExecutableValidatorTests`, `OrphanWorkerTerminatorTests` | scripted fake launcher | live x86 launch | `OrphanWorkerCleanupHostedService` wrapper untested; reattach not implemented (epic Phase 5) |
| Pipe framing / IPC envelope | `WorkerFrameProtocolTests` (both projects), `WorkerClientTests`, `WorkerPipeClientTests`, `WorkerPipeSessionTests` | `FakeWorkerHarnessTests` incl. malformed/oversized frames | live | good — malformed payloads and oversized headers explicitly scripted |
| Worker crash / heartbeat / fault | `WorkerClientTests` with `ManualTimeProvider` | fault frames via harness | abnormal-exit kill test | real process crash covered only opt-in |
| Event streaming / fan-out / replay | `EventStreamServiceTests` (15 tests), `SessionEventDistributorTests` (22) | `GatewayEndToEndMultiSubscriberTests` | stream phase of live smoke | no end-to-end reconnect/replay fake-worker test (epic Task 15 pending); no client handles `ReplayGap` (Task 14) |
| Backpressure | distributor/stream tests | overflow via harness | e2e script drain-loop workaround | policy interplay (`FailFast` vs multi-subscriber degrade) unit-covered only |
| Auth (API keys, scopes, audit) | `ApiKeyVerifierTests`, `SqliteAuthStoreTests`, `GatewayGrpcAuthorizationInterceptorTests`, `ConstraintEnforcerTests`, audit tests | — | e2e script auth-rejection phase (opt-in) | good |
| Dashboard | ~17 test classes under `Gateway/Dashboard/` | — | `DashboardLdapLiveTests` (opt-in) | `DashboardLiveDataService`, `EventsHub` hub methods, `AlarmsHubPublisher`, `DashboardHubConnectionFactory` have no direct tests |
| Alarms (consumer, failover, monitor) | `FailoverAlarmConsumerTests`, `SubtagAlarmStateMachineTests`, `GatewayAlarmMonitorProviderModeTests`, `AlarmFailoverEndToEndTests` | fake alarm service | dev-rig probes (`[Fact(Skip=...)]`) | live failover undrivable on rig; `provider_switches` metric never live-exercised; subtag Clear unvalidated |
| Worker STA / COM / conversion | `StaMessagePumpTests`, `StaCommandDispatcherTests`, `MxAccessCommandExecutorTests`, `VariantConverterTests`, etc. | — | `MxAccessLiveComCreationTests` (opt-in) | multi-sample `OnBufferedDataChange` conversion never observed live (`stillpending.md` §3.2) |
| Galaxy browse | `GalaxyFilterInputSafetyTests`, `GalaxyRepositoryHostWiringTests` | — | `GalaxyRepositoryLiveTests` (opt-in) | core browse logic now lives in the external `ZB.MOM.WW.GalaxyRepository` package |
| Clients (×5) | dotnet 13 / go 8 / rust 3 / python 13 / java 7 test files | — | `scripts/run-client-e2e-tests.ps1` only | no per-client wire-level integration tests; all client↔gateway behavior unverified in default runs |

### Test Coverage

- **High — Reconnect/replay has no end-to-end test and no client-side consumer.** The server emits the `ReplayGap` sentinel and replays the ring (`docs/Sessions.md` "Reconnect and replay"), unit-covered in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/EventStreamServiceTests.cs` and `MxAccessGatewayServiceTests.cs`, but the epic's fake-worker reconnect integration test (Task 15) and client `ReplayGap` handling in all five clients (Task 14) are pending per `oldtasks.md:40-44`. Impact: the shipped, default-on reconnect protocol (DetachGraceSeconds=30, ReplayBufferCapacity=1024) is unproven end-to-end and unusable by every official client. Recommendation: land Tasks 14–15 before advertising reconnect; until then document it as server-only.
- **High — Reconnect owner re-validation is not implemented.** Epic Task 13 ("Owner re-validation on reconnect", `oldtasks.md:40`) is pending, so nothing ties a resuming `StreamEvents` call to the API key that opened the session beyond the `event` scope and knowledge of the session id. Impact: with fan-out or detach-grace enabled, any event-scoped key that learns a session id can attach to another key's session and receive its replayed data. Recommendation: prioritize Task 13; it is a security control, not a resilience feature.
- **Medium — Real-worker control/COM behavior is only verified opt-in.** All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (`stillpending.md` §1.1), but the default suite exercises `Ping`/`GetWorkerInfo`/`DrainEvents`/`ShutdownWorker` only through `FakeWorkerHarness.RespondToControlCommandAsync` (`src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs:417-513`), which returns canned replies. Impact: a worker-side regression in these paths is invisible until someone sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`. Recommendation: schedule the live suite on the windev box on a cadence (see CI finding).
- **Medium — Dashboard live-data path untested.** `DashboardLiveDataService` (owns the shared lazily-opened gateway session backing `/browse` live values, per `gateway.md:128-131`) has no test class; `EventsHub`/`AlarmsHubPublisher` hub methods are likewise untested. Impact: the one dashboard component that holds a real worker session and can fault it has no regression net. Recommendation: add a fake-worker-backed test for session reuse, fault recovery, and disposal.
- **Low — Hosted-service wrappers untested.** `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` delegate to well-tested cores but their timer/startup wiring has no tests. Impact: low; failure mode is obvious at startup.

### Test Quality

- **Medium — Real-clock sleeps with negative assertions are latent flakes.** `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs:433` sleeps 150 ms of wall time ("give the heartbeat monitor a few real check-intervals") then asserts state did *not* change; `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs:402` races a 50 ms delayed state flip against a 500 ms bounded wait. 47 `Thread.Sleep`/`Task.Delay` occurrences exist across 22 test files; most are benign (cancellation-bounded polls, `Timeout.InfiniteTimeSpan` hang simulators, `ManualTimeProvider` used for the actual clock), but the fixed-real-time ones can pass spuriously or fail under load. Impact: intermittent failures erode trust in an already manual test regime. Recommendation: convert the negative-assertion sleeps to `ManualTimeProvider`-driven check-interval pumping or polling-until-stable.
- **Medium — Full-suite orphaned testhost processes remain an unfixed hygiene defect.** CLAUDE.md ("Source Update Workflow") documents that the full gateway suite "leaves orphaned testhost processes" and mandates filtered runs as a workaround. Impact: the workaround is procedural, so any future CI or an unaware contributor inherits zombie processes; it also suggests undisposed pipe servers or hosted services in some fixture. Recommendation: identify the leaking fixture (likely a `NamedPipeServerStream` or hosted-service test without disposal) rather than institutionalizing the workaround. The macOS `SessionWorkerClientFactoryFakeWorkerTests` timeout-message failures are pre-existing platform noise (wrong timeout message on macOS) and are correctly excluded from this finding.
- **Low — `FakeWorkerHarness` fidelity is good within its charter.** It shares `WorkerFrameReader`/`WorkerFrameWriter`/`WorkerEnvelope` with production (`FakeWorkerHarness.cs:36-37`), scripts malformed payloads and oversized headers (`:519-553`), heartbeats, faults, and shutdown acks, and mirrors real control-command reply shapes. What it structurally cannot represent — STA pumping, COM HRESULT semantics, process exit codes, event timing under load — is honestly delegated to the live suite by `docs/GatewayTesting.md`. No action needed beyond keeping its canned replies in lockstep with `WorkerPipeSession` (the §1.1 history shows this drifted once: green fakes masked an unimplemented real worker for months).
- **Low — The e2e script embeds a workaround for a real product sharp edge.** `scripts/run-client-e2e-tests.ps1` (1,715 lines) must interleave `StreamEvents` drains every 15 advised tags because advising without a consumer overflows the worker event channel and faults the session under `FailFast` (`docs/GatewayTesting.md` "Client E2E Scripts" phase 3). Impact: the test harness papering over this documents that real clients doing bulk-advise-then-stream will fault sessions. Recommendation: treat this as product feedback (e.g., subscribe-time channel policy), not just a script detail.

### CI & Operations

- **High — No CI exists.** There is no `.github/`, `.gitea/`, `azure-pipelines.yml`, `Jenkinsfile`, or any pipeline file in the repository. `docs/ImplementationPlanIndex.md` names a Gitea repo and a `packaging-and-ops` milestone, but nothing automated runs the .NET, Go, Rust, Python, or Java suites on push. Impact: the multi-language, multi-target (net10/net48-x86) build matrix is exactly the kind that silently breaks; the repo's own history (net48 CS0246 on unregenerated protos, Java generated-file churn) shows cross-component breakage is routine. Recommendation: a minimal Gitea Actions (or runner-on-windev) pipeline that builds `NonWindows.slnx` + runs the gateway suite on every push, with a Windows job for the x86 worker, would catch the majority class of regressions; add a scheduled opt-in live-MXAccess job on the dev rig.
- **Medium — Health checks cover only the auth store.** `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:71-75` registers a single `AuthStoreHealthCheck`; there is no readiness check for Galaxy SQL reachability, LDAP bind, alarm-monitor session health, or worker-executable presence/launchability, even though each is a documented startup dependency. Impact: `/health` reports ready while the gateway cannot open a session or browse Galaxy. Recommendation: add tagged checks for worker exe validation (cheap, reuses `WorkerExecutableValidator`) and Galaxy/LDAP with caching.
- **Medium — Deployment and upgrade are undocumented in-repo and hand-rolled.** Deployments are NSSM-wrapped services with configuration held in NSSM environment variables (acknowledged in `docs/GatewayConfiguration.md` "Host Endpoints" and the project memory notes); `scripts/` contains packaging and e2e scripts but no deploy/upgrade script, and there is no runbook for the `MxAccessGw`→`OtOpcUa` service-dependency dance. Impact: deploys are tribal knowledge; a second operator cannot reproduce them from the repo. Recommendation: commit a `docs/Deployment.md` plus a publish/deploy script that captures the NSSM env-var config as code.
- **Medium — No version discipline on server artifacts; client versions drift.** The Server and Worker csproj files carry no `Version` property (assemblies stamp 1.0.0), Contracts is `0.1.2` (`src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj:10`), Python and Rust clients are `0.1.2` (`clients/python/pyproject.toml:9`, `clients/rust/Cargo.toml:3`), while Java is `0.2.0` (`clients/java/build.gradle:16`) — and the Java bump required a separately-maintained `CLIENT_VERSION` constant (commit `b6a0d90` fixed a mismatch). Impact: support cannot correlate a deployed gateway or a client wheel to a commit; duplicated version constants have already drifted once. Recommendation: single-source versions (Directory.Build.props for the .NET side; generate client version constants at build) and stamp the server assembly informational version with the git SHA.
- **Low — Log rotation is configured but minimal.** `src/ZB.MOM.WW.MxGateway.Server/appsettings.json:8-11` rolls the file sink daily (`logs/mxgateway-.log`, a path relative to the service working directory) with no explicit `retainedFileCountLimit` or size cap (Serilog defaults to 31 files, unbounded size per day). Impact: acceptable, but a high-rate event day can produce an unbounded single file. Recommendation: set `fileSizeLimitBytes` + `rollOnFileSizeLimit` and an absolute log path for the service deployment.

### Documentation Currency

- **Medium — CLAUDE.md misstates the default retention behavior.** CLAUDE.md ("Repository-Specific Conventions") says "Default config preserves the original single-subscriber, no-retention behavior," but the code defaults enable retention: `DetachGraceSeconds = 30` (`src/ZB.MOM.WW.MxGateway.Server/Configuration/SessionOptions.cs:47`) and `ReplayBufferCapacity = 1024` / `ReplayRetentionSeconds = 300` (`Configuration/EventOptions.cs:11-31`). Only `AllowMultipleEventSubscribers` defaults off. Impact: an agent or operator following CLAUDE.md will assume detached sessions die immediately and no events are buffered. Recommendation: reword to "single-subscriber by default; detach-grace and replay retention are on by default."
- **Medium — gateway.md carries design-era sketches that no longer match the wire contract.** The `WorkerEnvelope` snippet (`gateway.md:291-309`) shows `uint64 correlation_id = 4` and body tags `command = 20 … fault = 26` with no `WorkerShutdownAck`; the actual proto uses `string correlation_id = 4` and `gateway_hello = 10 … worker_fault = 20` including `worker_shutdown_ack = 17` (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:19-38`). Similarly `gateway.md:738` still states the gateway "allow[s] one active client event subscriber per session, reject[s] a second subscriber" as unconditional policy, contradicted by the config-gated fan-out described later in the same file (`:1177-1180`) and by `docs/Sessions.md`. The public-API sketch (`gateway.md:328-334`) includes `rpc Session(stream ClientMessage)` which was never implemented (the real service has no bidi RPC — `mxaccess_gateway.proto:18-37`). Impact: gateway.md is the mandated pre-change reading; internally inconsistent sections cost every reader a reconciliation pass. Recommendation: replace the envelope sketch with a pointer to the proto, mark the single-subscriber paragraph as the default mode, and label `Session` as unbuilt future work.
- **Low — GatewayConfiguration.md, Sessions.md, DesignDecisions.md, and GatewayTesting.md are current.** Spot-checks confirm every table default in `docs/GatewayConfiguration.md` matches `Configuration/{SessionOptions,EventOptions,DashboardOptions,AuthenticationOptions,WorkerOptions}.cs`; `docs/DesignDecisions.md:63-118` correctly records the superseded reconnect/fan-out constraints; `docs/GatewayTesting.md` "all eight tests" matches the 8 `[LiveMxAccessFact]` attributes in `WorkerLiveMxAccessSmokeTests.cs`. The only nit: the Configuration Shape JSON block (`docs/GatewayConfiguration.md:12-83`) omits `DisableLogin`/`AutoLoginUser`/`CookieName`, `Tls`, and the `Alarms:Fallback` block that the tables below it document.
- **Medium — Repo-root working artifacts need triage.** Classification by evidence:
  - `MxAccessGateway-docs-{issues,fixed,final}.md` and `MxGatewayClient-docs-{issues,fixed}.md` — **untracked** (gitignored via `*-docs-issues.md`, `.gitignore:152`; the others simply never committed), 15,416 lines in the largest. These are dead local outputs of a finished docs-review pass; delete or move out of the working tree so greps and agents stop tripping over them.
  - `stillpending.md` (tracked, last touched 2026-06-25) — a 2026-06-15 audit snapshot with resolution annotations that currently serves as the project's only backlog register. Living but structurally a snapshot; either promote it to a maintained register (drop "Generated: … Commit:" framing) or migrate open items (§1.3, §1.4, §3.x, epic remainder) to the Gitea tracker that `docs/ImplementationPlanIndex.md` describes.
  - `oldtasks.md` (tracked, 2026-06-16) — explicitly a "human-readable mirror" of `docs/plans/2026-06-15-session-resilience.md.tasks.json`; keep only until the epic resumes, then delete.
  - `REVIEW-PROCESS.md` + `code-reviews/` — a coherent, generated-index review system, all 351 findings resolved, `Server`/`Tests` re-reviewed 2026-06-25. Living docs; keep.
  - `A2-galaxyrepository-adoption-handoff.md` — completed-migration handoff referenced from CLAUDE.md; archive candidate under `docs/plans/` once its cross-repo follow-ups land.

### Underdeveloped Areas

- **High — Session-resilience epic is 16/28 tasks unfinished with security-relevant remainder.** Pending per `oldtasks.md:39-63`: owner re-validation (Task 13), client ReplayGap handling (14), reconnect integration test (15), per-session dashboard ACL (16–19), orphan-worker reattach (20–28, incl. the `EnableOrphanReattach` flag that does not exist yet). Impact: the repo advertises reconnect and fan-out as shipped while their trust boundary and client support are absent. Recommendation: finish Phase 3 (13–15) as one unit; re-scope Phases 4–5 explicitly if they are no longer planned.
- **Medium — Dashboard EventsHub has no per-session ACL.** `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHub.cs:39` (`TODO(per-session-acl)`): any authenticated dashboard Viewer may join any `session:{id}` group and observe that session's event stream. The remark documents why this is accepted today; it is the only production-source TODO in the repo. Impact: acceptable for single-tenant dashboards, wrong the moment `GroupToRole` admits low-trust viewers. Recommendation: covered by epic Phase 4; keep the TODO until then.
- **Medium — `Dashboard:ShowTagValues` is a dead flag.** It is bound (`Configuration/DashboardOptions.cs:62`), projected into effective configuration, and displayed on the settings page (`Dashboard/Components/Pages/SettingsPage.razor:65`), but gates no value-display behavior anywhere; `docs/GatewayConfiguration.md:174` calls it "Reserved". Impact: operators toggling it see no effect. Recommendation: implement or remove; a config option that renders on the settings page implies function.
- **Medium — Vendor-gated alarm parity residuals are silently lossy.** The 8-arg `AlarmAckByName` operator `domain`/`full_name` fields are accepted on the wire and discarded (`src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs:261-278`, vendor stub returns −55); `AlarmAckByGUID` is `E_NOTIMPL` on this AVEVA build; the `provider_switches` metric's reason tagging has never been exercised live (`stillpending.md` §1.3–1.4, §3.4–3.5). Impact: contract fields that do nothing, invisible to callers. Recommendation: surface the drop in the ack reply diagnostic message until AVEVA implements the stub.
- **Low — The bidirectional `Session` RPC from the original design was never built.** `gateway.md:328-345` presents it as the "best long-term shape"; the service stopped at unary + server-streaming. Impact: none functionally; it is only a docs-currency and roadmap-clarity issue (see Documentation Currency).
- **Low — Client wire behavior has no automated verification.** All five clients have unit tests (13/8/3/13/7 test files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1` against a live rig. Impact: a gateway contract change can pass every default suite and break all five clients (mitigated somewhat by shared-proto codegen and `CrossLanguageSmokeMatrixTests` shape checks). Recommendation: an in-process fake gateway (the Java CLI already has `InProcessGatewayHarness` per `stillpending.md` §8) is the cheapest pattern to replicate per client.

## Top 5 recommendations

1. **Stand up minimal CI.** One pipeline that builds `NonWindows.slnx` and runs the gateway suite per push, a Windows/windev job for the x86 worker + worker tests, and a scheduled opt-in live-MXAccess run. This single change converts most findings above from "process risk" to "caught automatically."
2. **Finish session-resilience Phase 3 (Tasks 13–15).** Owner re-validation is a security control for a default-on retention window; client `ReplayGap` handling and the reconnect integration test make the already-shipped server protocol real.
3. **Fix the two documentation defects that actively mislead:** the CLAUDE.md "no-retention default" sentence and gateway.md's stale `WorkerEnvelope`/single-subscriber/`Session` sections (replace sketches with pointers to the protos and `docs/Sessions.md`).
4. **Triage the repo root:** delete the untracked `*-docs-*.md` artifacts, retire `oldtasks.md` into the plan's tasks.json, and either maintain `stillpending.md` as a living register or migrate its open items into the Gitea tracker the plans already assume.
5. **Harden operations:** add Galaxy/LDAP/worker-executable health checks next to the existing auth-store check, commit a deployment runbook + script capturing the NSSM env-var configuration, single-source versions across server, contracts, and clients, and fix (rather than route around) the orphaned-testhost fixture leak.