mxaccessgw/archreview/60-testing-docs-gaps.md

Testing, Documentation & Underdeveloped Areas — Architecture Review

Scope & method

This review covers test architecture (src/ZB.MOM.WW.MxGateway.Tests, src/ZB.MOM.WW.MxGateway.Worker.Tests, src/ZB.MOM.WW.MxGateway.IntegrationTests, scripts/), documentation currency (gateway.md, docs/Sessions.md, docs/GatewayConfiguration.md, docs/DesignDecisions.md, docs/GatewayTesting.md, CLAUDE.md, repo-root working artifacts), and repo-wide underdeveloped areas (TODO sweep, half-shipped features, operational gaps). Method is static reading of code and docs on the macOS tree; no test suites were run. Generated code, bin/, and obj/ are excluded. All paths are relative to /Users/dohertj2/Desktop/MxAccessGateway.

Executive summary

• Test coverage of the gateway core is strong and unusually well-layered: 46 gateway test classes, 34 worker test classes, and an 8-scenario opt-in live MXAccess suite map cleanly onto session lifecycle, pipe framing, event fan-out/replay, auth, and dashboard. The FakeWorkerHarness reuses the production WorkerFrameReader/WorkerFrameWriter, so fake tests exercise real frame validation rather than a mock transport.
• There is no CI configuration anywhere in the repository — no .github/, .gitea/, or pipeline files. Every documented verification step (per-task filtered dotnet test, five client toolchains, opt-in live matrices) is manual operator discipline; nothing prevents a cross-component regression from landing on main untested.
• The session-resilience epic is half-shipped: 12 of 28 tasks are merged (oldtasks.md), leaving reconnect owner re-validation (a security gap), client-side ReplayGap handling in all five clients, per-session dashboard ACL, and orphan-worker reattach pending. The server emits a reconnect protocol no client yet understands.
• Fake-vs-real fidelity has one structural blind spot by design: FakeWorkerHarness cannot simulate STA behavior, COM faults, process crash, or MXAccess semantics — those live only in the opt-in WorkerLiveMxAccessSmokeTests and dev-rig probes, which never run automatically.
• Timing-based tests are mostly disciplined (ManualTimeProvider, cancellation-token-bounded polls), but a handful of real-clock sleeps followed by negative assertions remain (WorkerClientTests.cs:433, SessionManagerTests.cs:402) and are latent flakes under load. The known macOS SessionWorkerClientFactoryFakeWorkerTests timeout failures are pre-existing and environmental.
• Documentation currency is better than typical: docs/GatewayConfiguration.md defaults match Configuration/*.cs exactly, and docs/Sessions.md/docs/DesignDecisions.md reflect the shipped reconnect/fan-out behavior. The stale spots are concentrated in gateway.md (design-era WorkerEnvelope sketch, single-subscriber policy line, never-built Session bidi RPC) and one materially wrong sentence in CLAUDE.md about default retention.
• Repo-root hygiene is mixed: stillpending.md and oldtasks.md are tracked, annotated snapshots that still serve as the de-facto backlog (there is no issue tracker in evidence); the five untracked *-docs-{issues,fixed,final}.md artifacts (15k+ lines) are dead local working files; code-reviews/ + REVIEW-PROCESS.md are a completed, self-consistent review system with zero open findings.
• Operationally the gateway is under-instrumented for a service: one health check (auth-store) with no Galaxy SQL, LDAP, alarm-monitor, or worker-launchability checks; hand-rolled NSSM deployment with config living in service env vars outside the repo; and no version stamping on the server/worker assemblies while client packages drift (Java 0.2.0 vs 0.1.2 everywhere else).

Findings

Coverage map

Component	Unit	Fake-worker integration	Live integration	Gap notes
Session lifecycle (open/close/lease/detach-grace)	SessionManagerTests, GatewaySessionTests	GatewayEndToEndFakeWorkerSmokeTests, SessionWorkerClientFactoryFakeWorkerTests	WorkerLiveMxAccessSmokeTests	SessionLeaseMonitorHostedService timer loop itself untested (sweep logic covered via CloseExpiredLeasesAsync tests at SessionManagerTests.cs:859-930)
Worker launch / validation / orphan cleanup	WorkerProcessLauncherTests, WorkerExecutableValidatorTests, OrphanWorkerTerminatorTests	scripted fake launcher	live x86 launch	OrphanWorkerCleanupHostedService wrapper untested; reattach not implemented (epic Phase 5)
Pipe framing / IPC envelope	WorkerFrameProtocolTests (both projects), WorkerClientTests, WorkerPipeClientTests, WorkerPipeSessionTests	FakeWorkerHarnessTests incl. malformed/oversized frames	live	good — malformed payloads and oversized headers explicitly scripted
Worker crash / heartbeat / fault	WorkerClientTests with ManualTimeProvider	fault frames via harness	abnormal-exit kill test	real process crash covered only opt-in
Event streaming / fan-out / replay	EventStreamServiceTests (15 tests), SessionEventDistributorTests (22)	GatewayEndToEndMultiSubscriberTests	stream phase of live smoke	no end-to-end reconnect/replay fake-worker test (epic Task 15 pending); no client handles ReplayGap (Task 14)
Backpressure	distributor/stream tests	overflow via harness	e2e script drain-loop workaround	policy interplay (FailFast vs multi-subscriber degrade) unit-covered only
Auth (API keys, scopes, audit)	ApiKeyVerifierTests, SqliteAuthStoreTests, GatewayGrpcAuthorizationInterceptorTests, ConstraintEnforcerTests, audit tests	—	e2e script auth-rejection phase (opt-in)	good
Dashboard	~17 test classes under Gateway/Dashboard/	—	DashboardLdapLiveTests (opt-in)	DashboardLiveDataService, EventsHub hub methods, AlarmsHubPublisher, DashboardHubConnectionFactory have no direct tests
Alarms (consumer, failover, monitor)	FailoverAlarmConsumerTests, SubtagAlarmStateMachineTests, GatewayAlarmMonitorProviderModeTests, AlarmFailoverEndToEndTests	fake alarm service	dev-rig probes ([Fact(Skip=...)])	live failover undrivable on rig; provider_switches metric never live-exercised; subtag Clear unvalidated
Worker STA / COM / conversion	StaMessagePumpTests, StaCommandDispatcherTests, MxAccessCommandExecutorTests, VariantConverterTests, etc.	—	MxAccessLiveComCreationTests (opt-in)	multi-sample OnBufferedDataChange conversion never observed live (stillpending.md §3.2)
Galaxy browse	GalaxyFilterInputSafetyTests, GalaxyRepositoryHostWiringTests	—	GalaxyRepositoryLiveTests (opt-in)	core browse logic now lives in the external ZB.MOM.WW.GalaxyRepository package
Clients (×5)	dotnet 13 / go 8 / rust 3 / python 13 / java 7 test files	—	scripts/run-client-e2e-tests.ps1 only	no per-client wire-level integration tests; all client↔gateway behavior unverified in default runs

Test Coverage

• High — Reconnect/replay has no end-to-end test and no client-side consumer. The server emits the ReplayGap sentinel and replays the ring (docs/Sessions.md "Reconnect and replay"), unit-covered in src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/EventStreamServiceTests.cs and MxAccessGatewayServiceTests.cs, but the epic's fake-worker reconnect integration test (Task 15) and client ReplayGap handling in all five clients (Task 14) are pending per oldtasks.md:40-44. Impact: the shipped, default-on reconnect protocol (DetachGraceSeconds=30, ReplayBufferCapacity=1024) is unproven end-to-end and unusable by every official client. Recommendation: land Tasks 14–15 before advertising reconnect; until then document it as server-only.
• High — Reconnect owner re-validation is not implemented. Epic Task 13 ("Owner re-validation on reconnect", oldtasks.md:40) is pending, so nothing ties a resuming StreamEvents call to the API key that opened the session beyond the event scope and knowledge of the session id. Impact: with fan-out or detach-grace enabled, any event-scoped key that learns a session id can attach to another key's session and receive its replayed data. Recommendation: prioritize Task 13; it is a security control, not a resilience feature.
• Medium — Real-worker control/COM behavior is only verified opt-in. All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (stillpending.md §1.1), but the default suite exercises Ping/GetWorkerInfo/DrainEvents/ShutdownWorker only through FakeWorkerHarness.RespondToControlCommandAsync (src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs:417-513), which returns canned replies. Impact: a worker-side regression in these paths is invisible until someone sets MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1. Recommendation: schedule the live suite on the windev box on a cadence (see CI finding).
• Medium — Dashboard live-data path untested. DashboardLiveDataService (owns the shared lazily-opened gateway session backing /browse live values, per gateway.md:128-131) has no test class; EventsHub/AlarmsHubPublisher hub methods are likewise untested. Impact: the one dashboard component that holds a real worker session and can fault it has no regression net. Recommendation: add a fake-worker-backed test for session reuse, fault recovery, and disposal.
• Low — Hosted-service wrappers untested. SessionLeaseMonitorHostedService and OrphanWorkerCleanupHostedService delegate to well-tested cores but their timer/startup wiring has no tests. Impact: low; failure mode is obvious at startup.

Test Quality

• Medium — Real-clock sleeps with negative assertions are latent flakes. src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs:433 sleeps 150 ms of wall time ("give the heartbeat monitor a few real check-intervals") then asserts state did not change; src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs:402 races a 50 ms delayed state flip against a 500 ms bounded wait. 47 Thread.Sleep/Task.Delay occurrences exist across 22 test files; most are benign (cancellation-bounded polls, Timeout.InfiniteTimeSpan hang simulators, ManualTimeProvider used for the actual clock), but the fixed-real-time ones can pass spuriously or fail under load. Impact: intermittent failures erode trust in an already manual test regime. Recommendation: convert the negative-assertion sleeps to ManualTimeProvider-driven check-interval pumping or polling-until-stable.
• Medium — Full-suite orphaned testhost processes remain an unfixed hygiene defect. CLAUDE.md ("Source Update Workflow") documents that the full gateway suite "leaves orphaned testhost processes" and mandates filtered runs as a workaround. Impact: the workaround is procedural, so any future CI or an unaware contributor inherits zombie processes; it also suggests undisposed pipe servers or hosted services in some fixture. Recommendation: identify the leaking fixture (likely a NamedPipeServerStream or hosted-service test without disposal) rather than institutionalizing the workaround. The macOS SessionWorkerClientFactoryFakeWorkerTests timeout-message failures are pre-existing platform noise (wrong timeout message on macOS) and are correctly excluded from this finding.
• Low — FakeWorkerHarness fidelity is good within its charter. It shares WorkerFrameReader/WorkerFrameWriter/WorkerEnvelope with production (FakeWorkerHarness.cs:36-37), scripts malformed payloads and oversized headers (:519-553), heartbeats, faults, and shutdown acks, and mirrors real control-command reply shapes. What it structurally cannot represent — STA pumping, COM HRESULT semantics, process exit codes, event timing under load — is honestly delegated to the live suite by docs/GatewayTesting.md. No action needed beyond keeping its canned replies in lockstep with WorkerPipeSession (the §1.1 history shows this drifted once: green fakes masked an unimplemented real worker for months).
• Low — The e2e script embeds a workaround for a real product sharp edge. scripts/run-client-e2e-tests.ps1 (1,715 lines) must interleave StreamEvents drains every 15 advised tags because advising without a consumer overflows the worker event channel and faults the session under FailFast (docs/GatewayTesting.md "Client E2E Scripts" phase 3). Impact: the test harness papering over this documents that real clients doing bulk-advise-then-stream will fault sessions. Recommendation: treat this as product feedback (e.g., subscribe-time channel policy), not just a script detail.

CI & Operations

• High — No CI exists. There is no .github/, .gitea/, azure-pipelines.yml, Jenkinsfile, or any pipeline file in the repository. docs/ImplementationPlanIndex.md names a Gitea repo and a packaging-and-ops milestone, but nothing automated runs the .NET, Go, Rust, Python, or Java suites on push. Impact: the multi-language, multi-target (net10/net48-x86) build matrix is exactly the kind that silently breaks; the repo's own history (net48 CS0246 on unregenerated protos, Java generated-file churn) shows cross-component breakage is routine. Recommendation: a minimal Gitea Actions (or runner-on-windev) pipeline that builds NonWindows.slnx + runs the gateway suite on every push, with a Windows job for the x86 worker, would catch the majority class of regressions; add a scheduled opt-in live-MXAccess job on the dev rig.
• Medium — Health checks cover only the auth store. src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:71-75 registers a single AuthStoreHealthCheck; there is no readiness check for Galaxy SQL reachability, LDAP bind, alarm-monitor session health, or worker-executable presence/launchability, even though each is a documented startup dependency. Impact: /health reports ready while the gateway cannot open a session or browse Galaxy. Recommendation: add tagged checks for worker exe validation (cheap, reuses WorkerExecutableValidator) and Galaxy/LDAP with caching.
• Medium — Deployment and upgrade are undocumented in-repo and hand-rolled. Deployments are NSSM-wrapped services with configuration held in NSSM environment variables (acknowledged in docs/GatewayConfiguration.md "Host Endpoints" and the project memory notes); scripts/ contains packaging and e2e scripts but no deploy/upgrade script, and there is no runbook for the MxAccessGw→OtOpcUa service-dependency dance. Impact: deploys are tribal knowledge; a second operator cannot reproduce them from the repo. Recommendation: commit a docs/Deployment.md plus a publish/deploy script that captures the NSSM env-var config as code.
• Medium — No version discipline on server artifacts; client versions drift. The Server and Worker csproj files carry no Version property (assemblies stamp 1.0.0), Contracts is 0.1.2 (src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj:10), Python and Rust clients are 0.1.2 (clients/python/pyproject.toml:9, clients/rust/Cargo.toml:3), while Java is 0.2.0 (clients/java/build.gradle:16) — and the Java bump required a separately-maintained CLIENT_VERSION constant (commit b6a0d90 fixed a mismatch). Impact: support cannot correlate a deployed gateway or a client wheel to a commit; duplicated version constants have already drifted once. Recommendation: single-source versions (Directory.Build.props for the .NET side; generate client version constants at build) and stamp the server assembly informational version with the git SHA.
• Low — Log rotation is configured but minimal. src/ZB.MOM.WW.MxGateway.Server/appsettings.json:8-11 rolls the file sink daily (logs/mxgateway-.log, a path relative to the service working directory) with no explicit retainedFileCountLimit or size cap (Serilog defaults to 31 files, unbounded size per day). Impact: acceptable, but a high-rate event day can produce an unbounded single file. Recommendation: set fileSizeLimitBytes + rollOnFileSizeLimit and an absolute log path for the service deployment.

Documentation Currency

• Medium — CLAUDE.md misstates the default retention behavior. CLAUDE.md ("Repository-Specific Conventions") says "Default config preserves the original single-subscriber, no-retention behavior," but the code defaults enable retention: DetachGraceSeconds = 30 (src/ZB.MOM.WW.MxGateway.Server/Configuration/SessionOptions.cs:47) and ReplayBufferCapacity = 1024 / ReplayRetentionSeconds = 300 (Configuration/EventOptions.cs:11-31). Only AllowMultipleEventSubscribers defaults off. Impact: an agent or operator following CLAUDE.md will assume detached sessions die immediately and no events are buffered. Recommendation: reword to "single-subscriber by default; detach-grace and replay retention are on by default."
• Medium — gateway.md carries design-era sketches that no longer match the wire contract. The WorkerEnvelope snippet (gateway.md:291-309) shows uint64 correlation_id = 4 and body tags command = 20 … fault = 26 with no WorkerShutdownAck; the actual proto uses string correlation_id = 4 and gateway_hello = 10 … worker_fault = 20 including worker_shutdown_ack = 17 (src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:19-38). Similarly gateway.md:738 still states the gateway "allow[s] one active client event subscriber per session, reject[s] a second subscriber" as unconditional policy, contradicted by the config-gated fan-out described later in the same file (:1177-1180) and by docs/Sessions.md. The public-API sketch (gateway.md:328-334) includes rpc Session(stream ClientMessage) which was never implemented (the real service has no bidi RPC — mxaccess_gateway.proto:18-37). Impact: gateway.md is the mandated pre-change reading; internally inconsistent sections cost every reader a reconciliation pass. Recommendation: replace the envelope sketch with a pointer to the proto, mark the single-subscriber paragraph as the default mode, and label Session as unbuilt future work.
• Low — GatewayConfiguration.md, Sessions.md, DesignDecisions.md, and GatewayTesting.md are current. Spot-checks confirm every table default in docs/GatewayConfiguration.md matches Configuration/{SessionOptions,EventOptions,DashboardOptions,AuthenticationOptions,WorkerOptions}.cs; docs/DesignDecisions.md:63-118 correctly records the superseded reconnect/fan-out constraints; docs/GatewayTesting.md "all eight tests" matches the 8 [LiveMxAccessFact] attributes in WorkerLiveMxAccessSmokeTests.cs. The only nit: the Configuration Shape JSON block (docs/GatewayConfiguration.md:12-83) omits DisableLogin/AutoLoginUser/CookieName, Tls, and the Alarms:Fallback block that the tables below it document.
• Medium — Repo-root working artifacts need triage. Classification by evidence:
  • MxAccessGateway-docs-{issues,fixed,final}.md and MxGatewayClient-docs-{issues,fixed}.md — untracked (gitignored via *-docs-issues.md, .gitignore:152; the others simply never committed), 15,416 lines in the largest. These are dead local outputs of a finished docs-review pass; delete or move out of the working tree so greps and agents stop tripping over them.
  • stillpending.md (tracked, last touched 2026-06-25) — a 2026-06-15 audit snapshot with resolution annotations that currently serves as the project's only backlog register. Living but structurally a snapshot; either promote it to a maintained register (drop "Generated: … Commit:" framing) or migrate open items (§1.3, §1.4, §3.x, epic remainder) to the Gitea tracker that docs/ImplementationPlanIndex.md describes.
  • oldtasks.md (tracked, 2026-06-16) — explicitly a "human-readable mirror" of docs/plans/2026-06-15-session-resilience.md.tasks.json; keep only until the epic resumes, then delete.
  • REVIEW-PROCESS.md + code-reviews/ — a coherent, generated-index review system, all 351 findings resolved, Server/Tests re-reviewed 2026-06-25. Living docs; keep.
  • A2-galaxyrepository-adoption-handoff.md — completed-migration handoff referenced from CLAUDE.md; archive candidate under docs/plans/ once its cross-repo follow-ups land.

Underdeveloped Areas

• High — Session-resilience epic is 16/28 tasks unfinished with security-relevant remainder. Pending per oldtasks.md:39-63: owner re-validation (Task 13), client ReplayGap handling (14), reconnect integration test (15), per-session dashboard ACL (16–19), orphan-worker reattach (20–28, incl. the EnableOrphanReattach flag that does not exist yet). Impact: the repo advertises reconnect and fan-out as shipped while their trust boundary and client support are absent. Recommendation: finish Phase 3 (13–15) as one unit; re-scope Phases 4–5 explicitly if they are no longer planned.
• Medium — Dashboard EventsHub has no per-session ACL. src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHub.cs:39 (TODO(per-session-acl)): any authenticated dashboard Viewer may join any session:{id} group and observe that session's event stream. The remark documents why this is accepted today; it is the only production-source TODO in the repo. Impact: acceptable for single-tenant dashboards, wrong the moment GroupToRole admits low-trust viewers. Recommendation: covered by epic Phase 4; keep the TODO until then.
• Medium — Dashboard:ShowTagValues is a dead flag. It is bound (Configuration/DashboardOptions.cs:62), projected into effective configuration, and displayed on the settings page (Dashboard/Components/Pages/SettingsPage.razor:65), but gates no value-display behavior anywhere; docs/GatewayConfiguration.md:174 calls it "Reserved". Impact: operators toggling it see no effect. Recommendation: implement or remove; a config option that renders on the settings page implies function.
• Medium — Vendor-gated alarm parity residuals are silently lossy. The 8-arg AlarmAckByName operator domain/full_name fields are accepted on the wire and discarded (src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs:261-278, vendor stub returns −55); AlarmAckByGUID is E_NOTIMPL on this AVEVA build; the provider_switches metric's reason tagging has never been exercised live (stillpending.md §1.3–1.4, §3.4–3.5). Impact: contract fields that do nothing, invisible to callers. Recommendation: surface the drop in the ack reply diagnostic message until AVEVA implements the stub.
• Low — The bidirectional Session RPC from the original design was never built. gateway.md:328-345 presents it as the "best long-term shape"; the service stopped at unary + server-streaming. Impact: none functionally; it is only a docs-currency and roadmap-clarity issue (see Documentation Currency).
• Low — Client wire behavior has no automated verification. All five clients have unit tests (13/8/3/13/7 test files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run scripts/run-client-e2e-tests.ps1 against a live rig. Impact: a gateway contract change can pass every default suite and break all five clients (mitigated somewhat by shared-proto codegen and CrossLanguageSmokeMatrixTests shape checks). Recommendation: an in-process fake gateway (the Java CLI already has InProcessGatewayHarness per stillpending.md §8) is the cheapest pattern to replicate per client.

Top 5 recommendations

1. Stand up minimal CI. One pipeline that builds NonWindows.slnx and runs the gateway suite per push, a Windows/windev job for the x86 worker + worker tests, and a scheduled opt-in live-MXAccess run. This single change converts most findings above from "process risk" to "caught automatically."
2. Finish session-resilience Phase 3 (Tasks 13–15). Owner re-validation is a security control for a default-on retention window; client ReplayGap handling and the reconnect integration test make the already-shipped server protocol real.
3. Fix the two documentation defects that actively mislead: the CLAUDE.md "no-retention default" sentence and gateway.md's stale WorkerEnvelope/single-subscriber/Session sections (replace sketches with pointers to the protos and docs/Sessions.md).
4. Triage the repo root: delete the untracked *-docs-*.md artifacts, retire oldtasks.md into the plan's tasks.json, and either maintain stillpending.md as a living register or migrate its open items into the Gitea tracker the plans already assume.
5. Harden operations: add Galaxy/LDAP/worker-executable health checks next to the existing auth-store check, commit a deployment runbook + script capturing the NSSM env-var configuration, single-source versions across server, contracts, and clients, and fix (rather than route around) the orphaned-testhost fixture leak.