docs(archreview): add architecture review + per-domain remediation designs and tracker
Adds the 2026-07-08 architecture review (00-overall + six domain reports) and a remediation/ tree: one design+implementation doc per domain covering every finding, plus 00-tracking.md as the master progress tracker. - 153 findings with stable IDs (GWC/WRK/IPC/SEC/CLI/TST), each with design rationale, implementation steps, tests, docs, and verification. - Tracker rolls findings up by severity and P0/P1/P2 roadmap tier, records cross-cutting clusters and per-finding status (all Not started). - Planning docs only; no source changes.
This commit is contained in:
@@ -0,0 +1,109 @@
|
|||||||
|
# MxAccessGateway — Overall Architecture Review
|
||||||
|
|
||||||
|
Date: 2026-07-08. Branch: `feat/jdk17-client-retarget`. Working tree: macOS (static review only; no builds or test runs).
|
||||||
|
|
||||||
|
## Scope and method
|
||||||
|
|
||||||
|
Six parallel review agents each examined one domain, reading source directly and citing verified `path:line` evidence. This document synthesizes their findings; the domain reports carry the full evidence and recommendations:
|
||||||
|
|
||||||
|
| Report | Domain |
|
||||||
|
|---|---|
|
||||||
|
| [10-gateway-core.md](10-gateway-core.md) | Gateway server: sessions, worker lifecycle, IPC client, gRPC streaming, alarms |
|
||||||
|
| [20-worker.md](20-worker.md) | Worker process: STA pump, COM lifetime, pipe session, event queue |
|
||||||
|
| [30-contracts-ipc.md](30-contracts-ipc.md) | Protos, frame protocol (both sides), codegen pipeline |
|
||||||
|
| [40-security-dashboard.md](40-security-dashboard.md) | API-key auth, LDAP dashboard, SignalR hubs, metrics, diagnostics |
|
||||||
|
| [50-clients.md](50-clients.md) | .NET, Go, Java, Python, Rust clients; cross-client parity matrix |
|
||||||
|
| [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | Test architecture, documentation currency, CI/ops, repo-wide gaps |
|
||||||
|
|
||||||
|
Review lenses: stability, performance, conventions, underdeveloped areas.
|
||||||
|
|
||||||
|
## Overall verdict
|
||||||
|
|
||||||
|
The core architecture is sound and unusually well executed for its stage. The hard problems the two-process design exists to solve are solved correctly: the worker runs a real `MsgWaitForMultipleObjectsEx` + `PeekMessage`/`DispatchMessage` STA pump with strict COM confinement and disciplined `Marshal.FinalReleaseComObject` teardown; the gateway's session lifecycle machinery (locking, close/kill gating, TOCTOU re-checks, replay→live handoff) is carefully proven and documented inline; the frame protocol validates lengths before allocating, reads exactly, and follows real proto-evolution hygiene; auth is a single fail-closed interceptor over peppered-HMAC keys with constant-time comparison; and the five language clients are more uniform and complete than most multi-language SDK efforts.
|
||||||
|
|
||||||
|
The risk is concentrated in four places, none of them structural rewrites:
|
||||||
|
|
||||||
|
1. **One confirmed critical defect** — the alarm monitor and the session event distributor both drain the same worker event channel, silently splitting the alarm feed.
|
||||||
|
2. **Silent failure modes at the edges** — sessions and streams that die or drop work without an error reaching anyone (faulted sessions never reaped, Go stream closes silently on overflow, worker drops post-shutdown commands without reply, Rust never checks MXSTATUS).
|
||||||
|
3. **Nothing is guarded by automation** — zero CI across a five-language, dual-runtime matrix; every known codegen fragility, the stale client descriptor set, and the Generated/-commit rule are protected only by operator memory.
|
||||||
|
4. **A half-shipped session-resilience epic** — reconnect/replay shipped server-side and is on by default, but owner re-validation (a security control), client-side `ReplayGap` handling, and the end-to-end test remain undone.
|
||||||
|
|
||||||
|
## Severity roll-up
|
||||||
|
|
||||||
|
Counts from the domain digests; the sub-reports are authoritative.
|
||||||
|
|
||||||
|
| Domain | Critical | High | Medium | Low |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Gateway core | 1 | 2 | 6 | 11 |
|
||||||
|
| Worker | — | 1 | 6 | 11 |
|
||||||
|
| Contracts & IPC | — | 1 | 8 | 8 |
|
||||||
|
| Security & dashboard | — | — | 10 | 5 |
|
||||||
|
| Clients | — | 4 | 11 | 2 |
|
||||||
|
| Testing, docs & gaps | — | 4 | 13 | 2 |
|
||||||
|
| **Total** | **1** | **12** | **54** | **39** |
|
||||||
|
|
||||||
|
## Critical and high findings
|
||||||
|
|
||||||
|
### Critical
|
||||||
|
|
||||||
|
- **Alarm feed split between two consumers.** `GatewayAlarmMonitor` and the per-session distributor pump both drain the same worker event channel, so alarm transitions are randomly stolen by whichever consumer wins; missed acks are never repaired by reconcile. `Alarms/GatewayAlarmMonitor.cs:229` vs `Sessions/GatewaySession.cs:554-583`. ([10](10-gateway-core.md))
|
||||||
|
|
||||||
|
### High — stability
|
||||||
|
|
||||||
|
- **Faulted sessions are never reaped.** `MarkFaulted` neither kills the worker nor schedules teardown; the sweeper only checks lease/detach-grace, pinning the worker process and session slot for up to 30 minutes. `Sessions/SessionManager.cs:273`. ([10](10-gateway-core.md))
|
||||||
|
- **Sparse-array bound unimplemented.** The doc-promised configurable max length is missing; only `Array.MaxLength` gates expansion, so one Write can force a multi-GB allocation before the frame-size check. `Sessions/SparseArrayExpander.cs:65`. ([10](10-gateway-core.md))
|
||||||
|
- **ReadBulk trips the stuck-STA watchdog.** Bulk reads over uncached tags freeze `LastActivityUtc` past the 75 s ceiling, so a healthy session is declared hung and its replies dropped. `MxAccessSession.cs:919-931` + `StaRuntime.cs:90`. ([20](20-worker.md))
|
||||||
|
- **Go event stream fails silently.** On a 16-slot buffer overflow the client cancels and closes the channel with no terminal error — indistinguishable from graceful end. `clients/go/mxgateway/session.go:751`. ([50](50-clients.md))
|
||||||
|
- **Rust never validates MXSTATUS.** `invoke` ignores hresult/status arrays, so failed writes can read as success. `clients/rust/src/error.rs:214`. ([50](50-clients.md))
|
||||||
|
|
||||||
|
### High — release and process integrity
|
||||||
|
|
||||||
|
- **No CI anywhere.** No pipeline files exist; the entire five-language, dual-runtime matrix is verified by hand. ([60](60-testing-docs-gaps.md))
|
||||||
|
- **Client descriptor set seven weeks stale.** The published protoset lacks `MxSparseArray`/`ReplayGap`/provider-status; the `-Check` script exists but nothing runs it. `clients/proto/descriptors/`. ([30](30-contracts-ipc.md))
|
||||||
|
- **Rust crate unbuildable outside the repo.** `build.rs` reads `../../src/.../Protos` and packaging hides it with `--no-verify`. `clients/rust/build.rs:8`. ([50](50-clients.md))
|
||||||
|
- **Session-resilience epic half-shipped.** 16 of 28 tasks open, including reconnect owner re-validation (security), client `ReplayGap` handling (no client implements it), and the replay end-to-end test — for behavior on by default (`DetachGrace=30`, `ReplayBuffer=1024`; CLAUDE.md wrongly claims no-retention defaults). ([60](60-testing-docs-gaps.md), [50](50-clients.md))
|
||||||
|
- **Typed-command parity gap across all clients.** No client exposes typed `WriteSecured`/`AuthenticateUser`/`AdviseSupervisory`/buffered helpers; the secured-write path is reachable only via raw Invoke. ([50](50-clients.md))
|
||||||
|
|
||||||
|
## Cross-cutting themes
|
||||||
|
|
||||||
|
**1. Silent failure modes.** The system's fail-fast philosophy is right, but several edges fail *silently* instead: faulted-session limbo (gateway), post-shutdown command drops with no reply and silent STA-thread death (worker), Go stream close, Rust status blindness, and the client stream-buffer overflow behavior diverging per language (Java errors, Go doesn't). The fix pattern is uniform — every death or drop must produce an observable error or fault frame.
|
||||||
|
|
||||||
|
**2. Backpressure and size-limit topology.** Related limits are set independently and interact badly: the worker hard-pins frames at 16 MiB while the gateway's `MaxMessageBytes` config (up to 256 MiB) is never conveyed to it; gRPC and pipe limits are equal with no envelope headroom; `DrainEvents max_events=0` builds one unbounded frame; the gateway read loop blocks up to 5 s per event behind a full channel, stalling command replies and heartbeats; clients hard-code 16-slot stream buffers. A legitimate large payload or event burst converts into whole-session death rather than a per-command error. This cluster deserves one coordinated design pass ([30](30-contracts-ipc.md), [10](10-gateway-core.md), [20](20-worker.md), [50](50-clients.md)).
|
||||||
|
|
||||||
|
**3. Everything is guarded by operator memory, not automation.** No CI, stale descriptor set, unpinned Python grpcio-tools range, Java's tracked-generated-file churn with no check, the Generated/-commit-after-regen rule undocumented, `--no-verify` Rust packaging. A minimal CI (build + targeted tests + descriptor/codegen freshness checks) removes an entire failure class at once.
|
||||||
|
|
||||||
|
**4. Documentation drift in load-bearing places.** gateway.md's `WorkerEnvelope` sketch has wrong types and field numbers; CLAUDE.md misstates retention defaults; the dashboard cookie is `MxGatewayDashboard`, not the documented `__Host-MxGatewayDashboard`; docs' `GroupToRole` sample value now fails startup validation; Java docs still say JDK 21; `docs/Grpc.md` omits `QueryActiveAlarms`. The repo's own rule — docs change in the same commit as behavior — has not been enforced retroactively.
|
||||||
|
|
||||||
|
**5. Cross-platform path defaults.** Windows-literal defaults (`AuthenticationOptions.SqlitePath`, `TlsOptions` PFX path) become CWD-relative filenames on non-Windows hosts — confirmed by a stray `C:\ProgramData\MxGateway\gateway-auth.db` file created inside the source tree (empty and gitignored this time; the same defect would write a private-key PFX into the tree). ([40](40-security-dashboard.md))
|
||||||
|
|
||||||
|
**6. Policy-layer security tightening.** The crypto and interceptor foundations are solid; the gaps are policy: `AllowAnonymousLocalhost` satisfies AdminOnly (loopback processes reach API-key inventory and any session's events), `DisableLogin` auto-admins remote requests with no production guard, `QueryActiveAlarms` silently requires admin because it is missing from the scope resolver (masked by mislabeled tests), hub bearer tokens are irrevocable for 30 minutes and accepted via query string, plaintext LDAP with a committed service-account password, no key expiry, and no rate limiting or lockout. ([40](40-security-dashboard.md))
|
||||||
|
|
||||||
|
**7. Event hot-path performance headroom.** Not urgent, but consistent across processes: every event is deep-cloned at least twice (worker queue clone, gateway `MapEvent` clone), the worker converts `MXSTATUS_PROXY` via reflection per status, events are written one frame + flush at a time on a 25 ms poll with no batching, the gateway frame reader/writer allocate per frame (worker side already uses `ArrayPool`), and a `Stopwatch` is allocated per streamed event. These compound under alarm bursts — the exact load the system exists to handle.
|
||||||
|
|
||||||
|
## Prioritized roadmap
|
||||||
|
|
||||||
|
**P0 — correctness and safety (do before wider rollout)**
|
||||||
|
1. Fix the alarm-monitor/distributor channel split (the one critical).
|
||||||
|
2. Reap faulted sessions: `MarkFaulted` must terminate the worker and schedule teardown.
|
||||||
|
3. Implement the sparse-array max-length bound.
|
||||||
|
4. Fix the ReadBulk/watchdog interaction (activity heartbeat during long COM calls).
|
||||||
|
5. Close the reconnect owner re-validation gap or flip retention defaults off until it lands; correct the CLAUDE.md claim.
|
||||||
|
6. Fix Go silent stream close and Rust MXSTATUS validation.
|
||||||
|
|
||||||
|
**P1 — process and hardening (next few weeks)**
|
||||||
|
7. Stand up minimal CI: NonWindows build + gateway fake-worker tests + client builds/tests + descriptor freshness check; add a Windows job for the x86 worker when runner access allows.
|
||||||
|
8. One coordinated pass on the size/backpressure topology (convey `MaxMessageBytes` to the worker, bound `DrainEvents`, decouple event backlog from command replies, make client buffer-overflow behavior uniform and loud).
|
||||||
|
9. Security policy pass: scope-resolver entry for `QueryActiveAlarms`, constrain `AllowAnonymousLocalhost`/`DisableLogin` (production guard), LDAPS or StartTLS, remove the committed service password, revocable hub tokens.
|
||||||
|
10. Cross-platform default paths (SQLite, PFX) via `Environment.SpecialFolder`/config requirement on non-Windows.
|
||||||
|
11. Rust crate packaging fix (vendor protos into the crate; drop `--no-verify`).
|
||||||
|
|
||||||
|
**P2 — completeness and polish**
|
||||||
|
12. Finish the session-resilience epic: client `ReplayGap` handling in all five clients, replay e2e test, per-session event ACL (`EventsHub.cs:39` TODO).
|
||||||
|
13. Typed `WriteSecured`/`AuthenticateUser`/buffered-read helpers across clients.
|
||||||
|
14. Event hot-path performance pass (kill double clones, cache the status converter, batch event frames, pool gateway frame buffers).
|
||||||
|
15. Documentation-drift sweep (gateway.md envelope sketch, cookie name, role sample, Java 17, Grpc.md RPC table, ClientPackaging.md) and one-time repo-root triage (oldtasks.md, stillpending.md, *-docs-issues.md, code-reviews/).
|
||||||
|
16. Version alignment across server assemblies and the five clients.
|
||||||
|
|
||||||
|
## What is demonstrably good
|
||||||
|
|
||||||
|
Worth stating so it is preserved under change: the STA pump and COM teardown discipline; the gateway's inline-proven locking and lifecycle reasoning; frame-protocol length validation and typed errors; proto evolution hygiene (reserved fields, UNSPECIFIED-zero enums, byte-identical galaxy proto vs the shared package); fail-closed auth with constant-time comparison and a real redaction seam (no secret-logging call sites found); the layered test architecture with a high-fidelity `FakeWorkerHarness` that reuses production frame code; config docs whose tables match code defaults; and five clients with an unusually consistent surface, fixture-driven tests, and a clean, verified JDK 17 retarget.
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
# Gateway Server Core — Architecture Review
|
||||||
|
|
||||||
|
## Scope & method
|
||||||
|
|
||||||
|
Static review of the Gateway server core in `src/ZB.MOM.WW.MxGateway.Server`, excluding `Security/`, `Dashboard/`, `Metrics/`, and `Diagnostics/` (covered by another agent; DI wiring and metric call sites in scope were followed into those directories only to confirm behavior). Every file in `Program.cs`, `GatewayApplication.cs`, `Configuration/`, `Sessions/`, `Workers/`, `Grpc/`, and `Alarms/` was read in full, including `GatewaySession.cs` (2,058 lines), `WorkerClient.cs`, `SessionEventDistributor.cs`, `SessionManager.cs`, `MxAccessGatewayService.cs`, `EventStreamService.cs`, `GatewayAlarmMonitor.cs`, the frame reader/writer, the process launcher stack, both hosted services, and the full options/validator set. Architecture context comes from `CLAUDE.md`, `gateway.md`, and `docs/Sessions.md`. No source file was modified and no build or test was run (macOS working tree).
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
- The session/worker lifecycle machinery is unusually well hardened: state writes are single-lock disciplined, close/kill paths are serialized through a per-session close gate, TOCTOU races between the lease sweeper and reconnecting subscribers are re-checked atomically, and the distributor's replay→live handoff is provably gap- and duplicate-free. The inline documentation of these invariants is exemplary.
|
||||||
|
- One serious correctness defect exists: the central alarm monitor consumes the worker event channel directly while the session's own event-distributor pump (started for the dashboard mirror on every production session) consumes the same channel concurrently. Events are split between the two consumers, so alarm transitions are randomly lost from the alarm feed; acknowledge transitions are never repaired by the reconcile pass.
|
||||||
|
- Faulted sessions are never reaped. `MarkFaulted` neither kills the worker nor schedules teardown, and the lease sweeper only checks lease/detach-grace expiry, so a faulted session can hold a session slot and a live x86 worker process for up to `DefaultLeaseSeconds` (30 minutes by default).
|
||||||
|
- The worker pipe is created without any ACL or `PipeOptions.CurrentUserOnly`, despite `gateway.md` promising a restricted-ACL, no-anonymous-access pipe. A local process can steal the single pipe instance and fail session startup at will.
|
||||||
|
- The worker read loop blocks on a full gateway event channel for up to 5 seconds per event, stalling command replies and heartbeat processing behind an event burst.
|
||||||
|
- `gateway.md` promises a gateway-configured maximum sparse-array length; the code only caps at `Array.MaxLength`, so one write request can force multi-hundred-megabyte allocations before the frame-size check rejects the result.
|
||||||
|
- The worker startup probe is a no-op whose failure exception is excluded from its own retry pipeline, making `StartupProbeRetryAttempts`/`StartupProbeRetryDelayMilliseconds` dead configuration.
|
||||||
|
- Hot-path efficiency is mostly sound (bounded channels, non-blocking fan-out, `TryWrite` overflow detection) but carries avoidable per-event costs: a `Stopwatch` allocation per streamed event, a full protobuf clone per mapped event, per-frame byte-array allocations with two stream writes, and a `LinkedList`-based replay ring.
|
||||||
|
- Convention adherence to `docs/style-guides/CSharpStyleGuide.md` is strong (file-scoped namespaces, `sealed` by default, `Async` suffix, MXAccess-aligned naming). The main deviations are a gRPC type (`RpcException`) thrown from the Sessions layer, a `DisposeAsync` on `GatewaySession` without declaring `IAsyncDisposable`, and one dead public method with a stale doc reference.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### Stability
|
||||||
|
|
||||||
|
**[Critical] The alarm monitor and the session's event-distributor pump both drain the same single worker event channel, so alarm events are randomly stolen from the alarm feed.**
|
||||||
|
Evidence: `GatewayAlarmMonitor.RunMonitorAsync` consumes worker events directly via `_sessionManager.ReadEventsAsync(session.SessionId, ...)` (`src/ZB.MOM.WW.MxGateway.Server/Alarms/GatewayAlarmMonitor.cs:228-231`), which calls `GatewaySession.ReadEventsAsync` → `WorkerClient.ReadEventsAsync` (`Sessions/GatewaySession.cs:1427-1440`, `Workers/WorkerClient.cs:227-236`). But `GatewaySession.MarkReady` starts the dashboard mirror, which creates and starts the `SessionEventDistributor` pump (`Sessions/GatewaySession.cs:433-437`, `554-583`), and that pump's source is the same `GatewaySession.ReadEventsAsync` (`Sessions/GatewaySession.cs:701-710`). `IDashboardEventBroadcaster` is registered unconditionally (`Dashboard/DashboardServiceCollectionExtensions.cs:50`) and injected into `SessionManager` (`Sessions/SessionManager.cs:65`, passed to every session at `Sessions/SessionManager.cs:440`), so in every production deployment the pump starts at session-ready — including on the alarm monitor's own session. `WorkerClient._events` is a single channel (created with `SingleReader = false`, `Workers/WorkerClient.cs:70-77`), so the two concurrent `ReadAllAsync` enumerators each receive a random subset of events. `docs/Sessions.md:196` states the design intent explicitly: single-subscriber mode exists to prevent "two gRPC streams from racing on the same worker event channel" — the alarm monitor recreates exactly that race internally.
|
||||||
|
Failure scenario: roughly half of `OnAlarmTransition` events land in the dashboard mirror instead of `ApplyTransition`. Raise/Clear are eventually repaired by `ReconcileLoopAsync` (up to `ReconcileIntervalSeconds`, floor 5 s, default 30 s late), but `ApplyReconcile` broadcasts only presence deltas (`Alarms/GatewayAlarmMonitor.cs:511-550`), so a stolen Acknowledge transition is never delivered to `StreamAlarms` subscribers at all — clients show unacked alarms indefinitely. Provider-mode-change events can also be stolen, delaying degraded-state visibility.
|
||||||
|
Recommendation: make the alarm monitor a distributor subscriber (attach via the session's lease API, or an internal `Register` on the distributor) instead of calling `ReadEventsAsync` directly; alternatively assert single consumption of `WorkerClient.ReadEventsAsync` (`SingleReader = true` plus a claimed-once guard) so this class of bug fails loudly instead of silently splitting events.
|
||||||
|
|
||||||
|
**[High] Faulted sessions are never swept, leaving a live worker process and a consumed session slot for up to the full lease duration.**
|
||||||
|
Evidence: `MarkFaulted` only flips state and records the reason (`Sessions/GatewaySession.cs:716-728`); it does not kill the worker, stop the distributor, or notify the registry. The sweeper closes only lease-expired or detach-grace-expired sessions (`Sessions/SessionManager.cs:273-277`); there is no `State == Faulted` branch. Detach-grace deliberately does not stamp faulted sessions (`Sessions/GatewaySession.cs:2022-2028`), so the only path out is lease expiry at `DefaultLeaseSeconds` = 1800 s (`Configuration/SessionOptions.cs:21`). In the FailFast overflow case (`Sessions/GatewaySession.cs:690-694`) the worker is healthy and keeps pumping events into the distributor while the session is permanently unusable (`EvaluateReadyUnderLock` fails every command, `Sessions/GatewaySession.cs:1930-1948`).
|
||||||
|
Failure scenario: a burst-slow client overflows its queue in single-subscriber FailFast mode; the session faults; for the next 30 minutes the gateway holds one of `MaxSessions` (default 64) slots and a running x86 MXAccess worker with live COM subscriptions that no client can use or close (clients rarely call `CloseSession` on a faulted session). A handful of such faults can exhaust session capacity.
|
||||||
|
Recommendation: sweep `Faulted` sessions in `CloseExpiredLeasesAsync` (immediately or after a short grace), or have `MarkFaulted` schedule teardown (kill worker + registry removal) the way `SetFaulted` does on the worker-client side.
|
||||||
|
|
||||||
|
**[Medium] A full gateway event channel stalls the worker read loop — command replies and heartbeats queue behind the 5-second full-mode wait.**
|
||||||
|
Evidence: `ReadLoopAsync` awaits `DispatchEnvelopeAsync` inline (`Workers/WorkerClient.cs:358-362`), and the `WorkerEvent` branch awaits `EnqueueWorkerEventAsync`, which blocks in `WriteAsync` for up to `EventChannelFullModeTimeout` (default 5 s) when `_events` is full (`Workers/WorkerClient.cs:511-553`, `Workers/WorkerClientOptions.cs:13`).
|
||||||
|
Failure scenario: with no event consumer attached (or a stalled distributor), each incoming event costs up to 5 s of read-loop stall before the fault fires; a `WorkerCommandReply` sitting behind the event frame is not dispatched, so an in-flight `InvokeAsync` can hit `CommandTimeout` (`Workers/WorkerClient.cs:187-213`) even though the worker replied in time; heartbeats behind the stall feed the heartbeat watchdog interplay the code specifically tries to compensate for (`Workers/WorkerClient.cs:394-424`).
|
||||||
|
Recommendation: dispatch command replies/heartbeats before (or independently of) event enqueue — e.g. fault-or-drop on event backlog without blocking the loop, or route events through a dedicated writer task so replies never queue behind events.
|
||||||
|
|
||||||
|
**[Medium] The worker named pipe is created with no ACL and without `CurrentUserOnly`, contradicting the documented pipe-security model.**
|
||||||
|
Evidence: `SessionWorkerClientFactory.CreatePipe` uses the plain `NamedPipeServerStream` constructor with `PipeOptions.Asynchronous` only (`Sessions/SessionWorkerClientFactory.cs:158-166`). `gateway.md:279-284` requires "ACL restricted to the gateway identity and the launched worker identity, no anonymous access".
|
||||||
|
Failure scenario: any local process can connect to `mxaccess-gateway-{pid}-{sessionId}` before the real worker does; with `maxNumberOfServerInstances: 1` the legitimate worker can then never connect, so `OpenSession` fails on startup timeout — a trivially repeatable local denial of service. The nonce prevents impersonation (`Workers/WorkerClient.cs:632-637`) but not connection stealing.
|
||||||
|
Recommendation: create the pipe via `NamedPipeServerStreamAcl.Create` with a DACL limited to the service identity (or `PipeOptions.CurrentUserOnly`, since workers run as the gateway identity per `docs/DesignDecisions.md`).
|
||||||
|
|
||||||
|
**[Low] `GatewaySession._workerClient` is written under `_syncRoot` but read without it on several paths.**
|
||||||
|
Evidence: written in `AttachWorkerClient` under lock (`Sessions/GatewaySession.cs:368-376`); read lock-free in `CloseAsync` (`Sessions/GatewaySession.cs:1470`), `DisposeAsync` (`Sessions/GatewaySession.cs:1762`), `KillWorker` (`Sessions/GatewaySession.cs:1617`), and `WorkerProcessId` (`Sessions/GatewaySession.cs:268`). Reference reads are atomic and the manager's call ordering makes a torn interleaving unlikely, but the discipline documented for `_state` is not applied to the field.
|
||||||
|
Recommendation: read `_workerClient` under `_syncRoot` (or mark it `volatile`) for consistency with the class's own locking contract.
|
||||||
|
|
||||||
|
**[Low] `WorkerClient.DisposeAsync` is not safe against concurrent double-dispose.**
|
||||||
|
Evidence: plain `if (_disposed) return; _disposed = true;` with no interlock (`Workers/WorkerClient.cs:296-303`). Two concurrent disposals would both run kill/complete/dispose, and the second `_stopCts.Cancel()` after `_stopCts.Dispose()` would throw `ObjectDisposedException`.
|
||||||
|
Impact: latent only — `SessionManager.RemoveSessionAsync`'s registry `TryRemove` gate makes the session's `DisposeAsync` single-shot in practice.
|
||||||
|
Recommendation: use `Interlocked.Exchange` as the lease classes already do.
|
||||||
|
|
||||||
|
**[Low] Worker-ready wait is a 25 ms poll loop rather than a state-change signal.**
|
||||||
|
Evidence: `GetReadyWorkerClientAsync` polls with `Task.Delay(25ms)` up to `WorkerReadyWaitTimeoutMs` (`Sessions/GatewaySession.cs:1841-1909`). Default-off (`Configuration/SessionOptions.cs:69`), bounded, and testable via `TimeProvider`, so the impact is minor; it is still a poll on the command hot path when enabled.
|
||||||
|
Recommendation: if the option sees real use, replace with a `TaskCompletionSource` pulsed on worker state transitions.
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
**[Medium] A `Stopwatch` instance is allocated for every streamed event.**
|
||||||
|
Evidence: `MxAccessGatewayService.StreamEvents` runs `Stopwatch stopwatch = Stopwatch.StartNew()` inside the per-event loop (`Grpc/MxAccessGatewayService.cs:155-157`).
|
||||||
|
Impact: one heap allocation per event per subscriber on the highest-volume path in the gateway.
|
||||||
|
Recommendation: use `long ts = Stopwatch.GetTimestamp()` + `Stopwatch.GetElapsedTime(ts)` (allocation-free).
|
||||||
|
|
||||||
|
**[Medium] Every mapped event is deep-cloned.**
|
||||||
|
Evidence: `MxAccessGrpcMapper.MapEvent` returns `workerEvent.Event?.Clone()` (`Grpc/MxAccessGrpcMapper.cs:64-73`), invoked once per event by the distributor pump (`Sessions/GatewaySession.cs:701-710`).
|
||||||
|
Impact: a full protobuf deep copy (including value arrays) per event, even though the enclosing `WorkerEvent` is discarded immediately after mapping and nothing else retains the inner message.
|
||||||
|
Recommendation: transfer ownership of `workerEvent.Event` instead of cloning; keep the clone only if a second consumer of the `WorkerEvent` is ever introduced.
|
||||||
|
|
||||||
|
**[Medium] Pipe framing allocates fresh buffers per frame and issues two stream writes per envelope.**
|
||||||
|
Evidence: reader allocates `new byte[4]` and `new byte[payloadLength]` per frame (`Workers/WorkerFrameReader.cs:32`, `50`); writer allocates `new byte[4]` plus `envelope.ToByteArray()` and performs two `WriteAsync` calls (`Workers/WorkerFrameWriter.cs:56-60`).
|
||||||
|
Impact: per-frame GC pressure proportional to event rate, and two pipe syscalls per outbound frame.
|
||||||
|
Recommendation: rent from `ArrayPool<byte>`, serialize length + payload into one buffer, and write once; on the read side reuse a pooled buffer sized to the frame.
|
||||||
|
|
||||||
|
**[Low] The replay ring is a `LinkedList` with a node allocation per retained event.**
|
||||||
|
Evidence: `Sessions/SessionEventDistributor.cs:108`, appended per event under `_replayLock` (`Sessions/SessionEventDistributor.cs:766-793`).
|
||||||
|
Impact: node allocation + poor cache locality on the fan-out hot path; capacity is fixed (`ReplayBufferCapacity`, default 1024), which is exactly the shape a circular array buffer serves allocation-free.
|
||||||
|
Recommendation: replace with a fixed-size ring array; keep the `_replayLock` protocol unchanged.
|
||||||
|
|
||||||
|
**[Low] Per-event gauge reconciliation reads `ChannelReader.Count` on every event.**
|
||||||
|
Evidence: `Grpc/EventStreamService.cs:173-179`. Bounded channel `Count` acquires the channel's internal lock; combined with the metric adjustment this adds measurable per-event overhead at high rates.
|
||||||
|
Recommendation: sample the backlog on an interval (or every N events) rather than per event.
|
||||||
|
|
||||||
|
**[Low] `Invoke` resolves the session twice per command.**
|
||||||
|
Evidence: `ResolveSession(request.SessionId)` (`Grpc/MxAccessGatewayService.cs:104`) followed by `sessionManager.InvokeAsync(request.SessionId, ...)` → `GetRequiredSession` (`Sessions/SessionManager.cs:161`).
|
||||||
|
Impact: a redundant `ConcurrentDictionary` lookup per command — small, but on every command.
|
||||||
|
Recommendation: add an `ISessionManager.InvokeAsync(GatewaySession, ...)` overload or pass the resolved session through.
|
||||||
|
|
||||||
|
### Conventions
|
||||||
|
|
||||||
|
**[Low] The Sessions layer throws `Grpc.Core.RpcException`, leaking a gRPC concern below the service boundary.**
|
||||||
|
Evidence: `SparseArrayExpander.Invalid` creates `RpcException(Status(StatusCode.InvalidArgument, ...))` (`Sessions/SparseArrayExpander.cs:283-284`) and is invoked from `GatewaySession.NormalizeOutboundCommand` (`Sessions/GatewaySession.cs:984-1063`), so `GatewaySession.InvokeAsync` — a transport-agnostic session API also used by the alarm monitor — throws a gRPC exception type. `gateway.md:1063-1065` calls for translation code to live at the gRPC layer.
|
||||||
|
Recommendation: throw a domain exception (e.g. `SessionManagerException` with an `InvalidArgument`-mapping error code) and map it in `MxAccessGatewayService.MapException`.
|
||||||
|
|
||||||
|
**[Low] `GatewaySession` implements `DisposeAsync` without declaring `IAsyncDisposable`.**
|
||||||
|
Evidence: `public sealed class GatewaySession` with no interface list (`Sessions/GatewaySession.cs:13`) but a public `DisposeAsync` (`Sessions/GatewaySession.cs:1672`).
|
||||||
|
Impact: `await using` does not compile against the type; disposal is only discoverable by convention.
|
||||||
|
Recommendation: declare `IAsyncDisposable`.
|
||||||
|
|
||||||
|
**[Low] Dead method with stale documentation: `GatewaySession.KillWorker(string)` has no callers.**
|
||||||
|
Evidence: no `.KillWorker(` call sites exist in server or test sources (grep across `src/`); the gated variant `KillWorkerWithCloseGateAsync` (`Sessions/GatewaySession.cs:1637-1658`) is what `SessionManager.KillWorkerAsync` uses (`Sessions/SessionManager.cs:225`). `docs/Sessions.md:57` still states that `KillWorkerAsync` "calls `GatewaySession.KillWorker` directly".
|
||||||
|
Recommendation: delete `KillWorker` (`Sessions/GatewaySession.cs:1615-1619`) and correct `docs/Sessions.md` in the same change, per the repo's docs-with-source rule.
|
||||||
|
|
||||||
|
**[Low] Heartbeat configuration semantics are conflated between the worker's send interval and the gateway's check interval.**
|
||||||
|
Evidence: `WorkerOptions.HeartbeatIntervalSeconds` is documented as "the interval in seconds for worker heartbeats" (`Configuration/WorkerOptions.cs:31`) but is bound to the gateway-side `HeartbeatCheckInterval` (`Sessions/SessionWorkerClientFactory.cs:86`), whose own default is 1 s (`Workers/WorkerClientOptions.cs:10`). Production therefore checks every 5 s while unit-constructed clients check every 1 s, and the option name does not describe what it controls.
|
||||||
|
Recommendation: either rename the option or add a distinct `HeartbeatCheckIntervalSeconds`; also note `HeartbeatLoopAsync` uses raw `Task.Delay` without the injected `TimeProvider` (`Workers/WorkerClient.cs:400`), unlike the rest of the class.
|
||||||
|
|
||||||
|
**Positive:** the reviewed code otherwise adheres closely to `CSharpStyleGuide.md` — file-scoped namespaces throughout, `sealed` classes by default, `Async` suffixes, MXAccess-aligned naming (`ServerHandle`, `ItemHandle`, `MxStatusProxy` shapes), and no hand-edited generated code. The `GatewaySession` class is, however, 2,058 lines mixing lifecycle, distributor wiring, bulk-command wrappers, and item tracking; splitting the bulk-command facade out would restore single-responsibility without behavior change.
|
||||||
|
|
||||||
|
### Underdeveloped
|
||||||
|
|
||||||
|
**[High] The documented "gateway-configured maximum array length" bound on sparse-array writes is not implemented.**
|
||||||
|
Evidence: `gateway.md:536` lists "`total_length` exceeds the gateway-configured maximum array length" as an `InvalidArgument` rejection; `SparseArrayExpander.Expand` validates only `totalLength > (uint)Array.MaxLength` (`Sessions/SparseArrayExpander.cs:65-69`) — about 2.1 billion elements. No configuration option for the bound exists in `Configuration/`.
|
||||||
|
Failure scenario: a single authorized `Write` carrying `total_length = 500_000_000` forces the gateway to materialize a ~2-4 GB `MxArray` (`Sessions/SparseArrayExpander.cs:98-232`) before the 16 MB `MaxMessageBytes` frame check finally rejects it in `WorkerFrameWriter` (`Workers/WorkerFrameWriter.cs:49-54`) — a memory-exhaustion vector reachable through normal command flow.
|
||||||
|
Recommendation: add the documented configurable cap (validated in `GatewayOptionsValidator`) and enforce it before allocation.
|
||||||
|
|
||||||
|
**[Medium] The worker startup probe is a no-op and its retry pipeline can never retry, making the probe configuration dead.**
|
||||||
|
Evidence: `WorkerProcessStartedProbe.WaitUntilReadyAsync` performs one instantaneous `HasExited` check and throws `WorkerProcessLaunchException` on failure (`Workers/WorkerProcessStartedProbe.cs:6-19`); `ShouldRetryStartupProbe` explicitly excludes `WorkerProcessLaunchException` (and `OperationCanceledException`) from retry (`Workers/WorkerProcessLauncher.cs:291-299`). The Polly pipeline with exponential backoff and jitter (`Workers/WorkerProcessLauncher.cs:264-289`) therefore executes exactly one attempt in all cases, and `StartupProbeRetryAttempts` / `StartupProbeRetryDelayMilliseconds` (`Configuration/WorkerOptions.cs:19-22`, validated at `Configuration/GatewayOptionsValidator.cs:119-126`) have no observable effect.
|
||||||
|
Recommendation: either implement a real readiness probe (retryable transient failures) or delete the pipeline and the two dead options; `docs/WorkerProcessLauncher.md` should be updated in the same change.
|
||||||
|
|
||||||
|
**[Medium] The envelope `sequence` monotonicity rule is specified but never enforced.**
|
||||||
|
Evidence: `gateway.md:313` states "`sequence` is monotonic per sender"; `WorkerEnvelopeValidator.Validate` checks only protocol version, session id, and body presence (`Workers/WorkerEnvelopeValidator.cs:15-39`). Nothing on the gateway side detects out-of-order, duplicated, or replayed frames from a misbehaving worker.
|
||||||
|
Impact: a worker bug that reorders or repeats frames is invisible; event ordering guarantees ("keep event order stable per worker") rest solely on the worker's writer discipline.
|
||||||
|
Recommendation: track last-received sequence per connection and fault the worker client on regression (`ProtocolViolation`), which matches the existing fault model.
|
||||||
|
|
||||||
|
**[Low] `EventChannelFullModeTimeout` and `HeartbeatStuckCeiling` are not reachable from configuration.**
|
||||||
|
Evidence: `SessionWorkerClientFactory` populates only four of the six `WorkerClientOptions` fields (`Sessions/SessionWorkerClientFactory.cs:83-89`); the other two always use hardcoded defaults (`Workers/WorkerClientOptions.cs:13`, `23`) and have no `WorkerOptions` counterparts.
|
||||||
|
Recommendation: expose them under `MxGateway:Worker:*` or document them as fixed.
|
||||||
|
|
||||||
|
**[Low] `metrics.StreamDisconnected` is always labeled `"Detached"`, even when the stream ends by overflow or worker fault.**
|
||||||
|
Evidence: single call site in the `finally` block (`Grpc/EventStreamService.cs:195`); the fault paths above it (`Grpc/EventStreamService.cs:148-158`) do not differentiate the label.
|
||||||
|
Impact: the disconnect-reason dimension the metric implies is uninformative for diagnosing overflow-vs-fault-vs-client-cancel.
|
||||||
|
Recommendation: record the actual terminal cause (detached / overflow / worker-fault / canceled).
|
||||||
|
|
||||||
|
**[Info] `MaxEventSubscribersPerSession` is a knowingly dead knob in the default single-subscriber configuration** — this is acknowledged and justified in a comment (`Configuration/GatewayOptionsValidator.cs:189-196`) and needs no change; noted here so it is not re-reported.
|
||||||
|
|
||||||
|
## Top 5 recommendations
|
||||||
|
|
||||||
|
1. **Fix the alarm monitor's event consumption** (`Alarms/GatewayAlarmMonitor.cs:228`): attach through the session's `SessionEventDistributor` instead of a second raw `ReadEventsAsync` drain, and add a claimed-once guard on `WorkerClient.ReadEventsAsync` so any future dual-consumer bug fails loudly. This is the only Critical finding and silently corrupts the production alarm feed today.
|
||||||
|
2. **Reap faulted sessions promptly** (`Sessions/SessionManager.cs:255`, `Sessions/GatewaySession.cs:716`): sweep `Faulted` state in `CloseExpiredLeasesAsync` or trigger teardown from `MarkFaulted`, so a faulted session does not pin a worker process and a `MaxSessions` slot for 30 minutes.
|
||||||
|
3. **Apply the documented ACL to the worker pipe** (`Sessions/SessionWorkerClientFactory.cs:158`): use `NamedPipeServerStreamAcl.Create` (or `CurrentUserOnly`) to close the local pipe-squatting denial of service and match `gateway.md`'s pipe-security contract.
|
||||||
|
4. **Enforce the configured sparse-array length bound before allocation** (`Sessions/SparseArrayExpander.cs:65`): implement the `gateway.md`-promised maximum, validated at startup, to remove the memory-exhaustion vector.
|
||||||
|
5. **Decouple event enqueue from the worker read loop and trim per-event costs** (`Workers/WorkerClient.cs:511`, `Grpc/MxAccessGatewayService.cs:155`, `Grpc/MxAccessGrpcMapper.cs:68`, `Workers/WorkerFrameReader.cs:50`): stop command replies and heartbeats from queueing behind a full event channel, then remove the per-event `Stopwatch` allocation, the per-event protobuf clone, and the per-frame buffer allocations.
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
# Worker Process — Architecture Review
|
||||||
|
|
||||||
|
## Scope & method
|
||||||
|
|
||||||
|
This review covers `src/ZB.MOM.WW.MxGateway.Worker` (.NET Framework 4.8, x86): bootstrap (`Bootstrap/`), pipe IPC (`Ipc/`), the STA runtime (`Sta/`), the MXAccess session/command/event layer (`MxAccess/`), and the conversion layer (`Conversion/`). Every finding cites file and line evidence read directly from the working tree on 2026-07-08 (branch `feat/jdk17-client-retarget`). The review is static only — the worker builds exclusively on the Windows x86 host, so no build or test run was performed. `src/ZB.MOM.WW.MxGateway.Worker.Tests` was skimmed for coverage. Reference documents: `gateway.md`, `docs/MxAccessWorkerInstanceDesign.md`, `docs/WorkerSta.md`, `docs/WorkerFrameProtocol.md`, `docs/WorkerConversion.md`.
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
- The STA core is correct: `StaRuntime.ThreadMain` runs the canonical wait/pump/dispatch loop over `MsgWaitForMultipleObjectsEx(QS_ALLINPUT, MWMO_INPUTAVAILABLE)` plus a `PeekMessage`/`TranslateMessage`/`DispatchMessage` drain, with an `AutoResetEvent` wake and a 50 ms idle pump interval (`Sta/StaRuntime.cs:245-251`, `Sta/StaMessagePump.cs:31-59`). This satisfies the architecture's hardest requirement.
|
||||||
|
- COM lifetime discipline is strong throughout: `Marshal.FinalReleaseComObject` on every teardown path, event-sink detach before release, cleanup in MXAccess handle order (UnAdvise → RemoveItem → Unregister), and the wnwrap/subtag alarm consumers release their own RCWs (`MxAccess/MxAccessSession.cs:1250-1288`, `MxAccess/WnWrapAlarmConsumer.cs:554-580`, `MxAccess/LmxSubtagAlarmSource.cs:203-251`).
|
||||||
|
- The most serious defect is a watchdog/`ReadBulk` interaction: a long-running STA command has no way to refresh `LastActivityUtc`, so a legitimate `ReadBulk` over enough uncached tags exceeds the 75-second `HeartbeatStuckCeiling` and self-faults as `StaHung`, after which every completed reply is dropped.
|
||||||
|
- An STA thread that dies after startup (for example a message-pump wait failure) is silent: the captured exception is never logged or reported, and every subsequent `InvokeAsync` task hangs forever.
|
||||||
|
- Commands arriving after graceful shutdown starts are dropped with no reply at all; the gateway's correlation wait can only time out.
|
||||||
|
- Envelope `sequence` is assigned before the writer lock is taken, so concurrently written frames can appear on the wire out of sequence order, violating the "monotonic per sender" rule in `gateway.md`.
|
||||||
|
- The documented outbound write priority (faults > replies > shutdown acks > heartbeats > events) is not implemented; the frame writer is a plain FIFO semaphore.
|
||||||
|
- Event-path allocation costs are avoidable: reflection-based `MXSTATUS_PROXY` field reads on every event, a defensive `Clone()` of every enqueued event, one pipe write + flush per event, and no envelope batching.
|
||||||
|
- Conventions are otherwise very good: MXAccess-aligned names, `sealed` classes, `Async` suffixes, file-scoped namespaces, nonce/credential redaction that actually works (`Bootstrap/WorkerLogRedactor.cs:16-25`); the main blemish is a split `_camelCase` vs `camelCase` private-field style between `Ipc/` and the rest of the worker.
|
||||||
|
- Test coverage is broad (STA scheduling, pump wake behavior, pipe session handshake/watchdog/shutdown races, frame protocol, conversion, event queue, alarm units) but misses STA thread death, wire sequence monotonicity, and the silent post-shutdown command drop.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### Stability
|
||||||
|
|
||||||
|
**S1 — High.** A legitimately long `ReadBulk` self-faults as `StaHung` because no STA command can refresh the activity timestamp.
|
||||||
|
Evidence: `MxAccess/MxAccessSession.cs:919-931` waits up to `timeout` per uncached tag, sequentially per tag (`MxAccess/MxAccessSession.cs:876-888`); the pump step invoked during the wait calls `StaMessagePump.PumpPendingMessages()` directly and never calls `MarkActivity` (`Sta/StaRuntime.cs:90`, `MarkActivity` is private at `Sta/StaRuntime.cs:304-307`); the watchdog fires `StaHung` regardless of an in-flight command once staleness exceeds `HeartbeatStuckCeiling` (default 75 s) (`Ipc/WorkerPipeSession.cs:830-860`, `Ipc/WorkerPipeSessionOptions.cs:19`).
|
||||||
|
Failure scenario: `ReadBulk` with `timeout_ms = 5000` against 20 unreachable tags holds the STA ~100 s with `LastActivityUtc` frozen; at 75 s the worker emits `StaHung`, sets `_state = Faulted`, and from then on drops every completed command reply (`Ipc/WorkerPipeSession.cs:604-607`), so the gateway kills a healthy session. `docs/MxAccessWorkerInstanceDesign.md:688-690` assumes "no legitimate STA command should run that long without periodically refreshing activity", but no refresh mechanism exists.
|
||||||
|
Recommendation: have `StaRuntime.PumpPendingMessages()` (the `pumpStep` used by `ReadBulk`) call `MarkActivity()`, or clamp the total `ReadBulk` duration below the ceiling, or expose an activity-refresh hook to executors.
|
||||||
|
|
||||||
|
**S2 — Medium.** STA thread death after startup is silent and strands all future work.
|
||||||
|
Evidence: `Sta/StaRuntime.cs:255-259` catches any loop exception into `startupException` and sets `startedEvent` — but after startup `Start()` has already returned, so nothing ever observes the exception; it is not logged, not converted to a `WorkerFault`, and `shutdownRequested` stays false, so `InvokeAsync` (`Sta/StaRuntime.cs:165-178`) keeps enqueueing work items into a queue with no consumer and returns tasks that never complete. The only in-loop throw site is the pump wait failure (`Sta/StaMessagePump.cs:38-42`).
|
||||||
|
Failure scenario: `MsgWaitForMultipleObjectsEx` returns `WAIT_FAILED` once; the STA exits, the dispatcher's drain task wedges on the first stuck `InvokeAsync`, heartbeats keep flowing with a frozen `LastStaActivityUtc` and a non-empty `CurrentCommandCorrelationId`, and the worker is only declared hung after `HeartbeatStuckCeiling` — with the true root-cause exception permanently lost.
|
||||||
|
Recommendation: on `ThreadMain` exit, fail all queued and future `InvokeAsync` calls with the captured exception, and surface it (log + `WorkerFault`) instead of storing it in a write-only field.
|
||||||
|
|
||||||
|
**S3 — Medium.** Commands received after shutdown begins are silently dropped with no reply.
|
||||||
|
Evidence: `Ipc/WorkerPipeSession.cs:690-707` — `TryStartCommandTask` returns without any action when `_acceptingCommands` is false; no `WorkerUnavailable` reply, no `LogCommandResultDropped` diagnostic (that log fires only for completed-then-dropped replies, `Ipc/WorkerPipeSession.cs:604-607`).
|
||||||
|
Impact: a command racing `WorkerShutdown` leaves the gateway's correlation wait to expire on its own timeout with no trace. `docs/MxAccessWorkerInstanceDesign.md:697-699` says shutdown should "reject new commands" — the dispatcher layer does reply `WorkerUnavailable` (`Sta/StaCommandDispatcher.cs:117-123`), but this earlier gate replies with nothing.
|
||||||
|
Recommendation: write a `WorkerUnavailable` `WorkerCommandReply` (or at minimum log) for commands refused by the `_acceptingCommands` gate.
|
||||||
|
|
||||||
|
**S4 — Medium.** Envelope `sequence` can appear out of order on the wire.
|
||||||
|
Evidence: `NextSequence()` is called while building the envelope (`Ipc/WorkerPipeSession.cs:1005-1018`), but the writer lock is acquired later inside `WriteAsync` (`Ipc/WorkerFrameWriter.cs:68-77`). Command replies are written from independent per-command tasks (`Ipc/WorkerPipeSession.cs:690-706`) concurrently with the heartbeat and event-drain loops, so task B can take sequence n+1 yet win the write lock before task A's sequence n.
|
||||||
|
Impact: violates the `gateway.md` envelope rule "`sequence` is monotonic per sender" (gateway.md:313); any gateway-side consumer that trusts wire-order monotonicity mis-sorts frames. (Per-event ordering is safe — `WorkerSequence` on `MxEvent` is assigned inside the queue lock, `MxAccess/MxAccessEventQueue.cs:135-143` — but the envelope-level guarantee is broken.)
|
||||||
|
Recommendation: assign the envelope sequence inside the writer's critical section (e.g. a sequence-stamping callback under `_writeLock`).
|
||||||
|
|
||||||
|
**S5 — Medium.** One transient alarm poll failure kills the entire session, including its data subscriptions.
|
||||||
|
Evidence: `MxAccess/MxAccessStaSession.cs:278-291` — any exception from `handler.PollOnce()` records a fault on the shared event queue and permanently stops the poll loop; the drain loop turns that fault into session termination (`Ipc/WorkerPipeSession.cs:344-353` throws after writing the fault). The `FailoverAlarmConsumer` absorbs primary failures only in composite mode (`MxAccess/FailoverAlarmConsumer.cs:278-296`); in the default alarmmgr-only mode (`Ipc/WorkerPipeSession.cs:54` builds the handler with `standbyFactory: null`, and `AlarmCommandHandler.BuildConsumer` returns the bare consumer at `MxAccess/AlarmCommandHandler.cs:228-230`) a single `GetXmlCurrentAlarms2` COM error propagates unwrapped.
|
||||||
|
Failure scenario: one transient E_FAIL from the AVEVA alarm subsystem terminates a client session's `OnDataChange` stream even though the data path was healthy.
|
||||||
|
Recommendation: count consecutive alarm-poll failures (mirroring `FailoverSettings.Threshold`) before declaring the subscription dead, or scope the fault to the alarm feature rather than the whole session.
|
||||||
|
|
||||||
|
**S6 — Low.** Events still queued at graceful shutdown are discarded without a final drain, and the post-`Faulted` reply-drop policy discards legitimate late replies.
|
||||||
|
Evidence: `ShutdownAsync` writes the ack and returns `false` from dispatch (`Ipc/WorkerPipeSession.cs:657-688`, `396-398`); `RunMessageLoopAsync`'s `finally` then cancels the event-drain loop (`Ipc/WorkerPipeSession.cs:287-292`) with whatever remains in `MxAccessEventQueue` unshipped. Reply drops on `_state != Ready` are at `Ipc/WorkerPipeSession.cs:604-607`.
|
||||||
|
Impact: an `OnWriteComplete` raised during cleanup never reaches the gateway; acceptable for a closing session but undocumented — `docs/MxAccessWorkerInstanceDesign.md` does not state that shutdown discards the residual event queue.
|
||||||
|
Recommendation: drain the event queue once after `ShutdownGracefullyAsync` returns and before the ack, or document the discard.
|
||||||
|
|
||||||
|
**S7 — Low.** No `AppDomain.UnhandledException` hook; an exception on an unobserved thread crashes the process without a `WorkerFault` frame.
|
||||||
|
Evidence: `Program.cs:1-4` and `WorkerApplication.Run` (`WorkerApplication.cs:46-141`) install no unhandled-exception or unobserved-task handlers.
|
||||||
|
Impact: the gateway still detects the death via process exit and pipe closure, but the crash cause never reaches the fault channel or the worker log.
|
||||||
|
Recommendation: register `AppDomain.CurrentDomain.UnhandledException` (and `TaskScheduler.UnobservedTaskException`) to log through `IWorkerLogger` before exit.
|
||||||
|
|
||||||
|
**S8 — Low.** Top-level catch blocks log the exception type but never the message.
|
||||||
|
Evidence: `WorkerApplication.cs:112-139` logs only `exception_type` for protocol, pipe, and unexpected failures; similarly `HResultConverter.CreateSafeDiagnosticMessage` reduces every command exception to `Type: HRESULT 0x…` (`Conversion/HResultConverter.cs:46-49`).
|
||||||
|
Impact: root-causing a field failure from worker stderr requires reproducing it; the redactor (`Bootstrap/WorkerLogRedactor.cs`) already exists to make message logging safe.
|
||||||
|
Recommendation: log `exception.Message` (redacted) alongside the type at the process boundary; keep the credential-safe reply shape for IPC replies if that stripping is intentional parity policy.
|
||||||
|
|
||||||
|
Positive observations worth recording: partial pipe reads are handled correctly with a read-exactly loop and explicit EOF detection (`Ipc/WorkerFrameReader.cs:92-113`); gateway hello validation covers protocol version, session id, and nonce (`Ipc/WorkerPipeSession.cs:205-228`); handshake failures always attempt a structured fault before exiting (`Ipc/WorkerPipeSession.cs:192-203`); command ordering into the dispatcher is preserved because `ProcessCommandAsync` enqueues synchronously on the read-loop thread before its first await (`Ipc/WorkerPipeSession.cs:595`, `Sta/StaCommandDispatcher.cs:108-144`); the dispatcher's 128-entry bound provides real backpressure toward MXAccess (`Sta/StaCommandDispatcher.cs:11`, `125-131`); handle bookkeeping strictly follows the record-only-after-COM-success rules (`MxAccess/MxAccessSession.cs:187-307`, registry snapshots are copies so cleanup iteration is safe, `MxAccess/MxAccessHandleRegistry.cs:14-32`); and the value cache is evicted on `RemoveItem` to prevent stale reads across handle reuse (`MxAccess/MxAccessSession.cs:258-262`).
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
**P1 — Medium.** `MXSTATUS_PROXY` conversion reflects on every field of every status of every event.
|
||||||
|
Evidence: `Conversion/MxStatusProxyConverter.cs:22-26` calls `ReadInt32Field` four times per status; each call performs `Type.GetField` + `FieldInfo.GetValue` + `Convert.ToInt32` (`Conversion/MxStatusProxyConverter.cs:83-103`). This runs inside the STA event handler path for every `OnDataChange`.
|
||||||
|
Impact: eight reflection operations plus boxing per event at data-change rates; the status type is always the interop `MXSTATUS_PROXY` struct, so the lookups are fully cacheable.
|
||||||
|
Recommendation: cache the four `FieldInfo` objects per `Type` (a two-entry static dictionary suffices), or cast to the interop struct directly.
|
||||||
|
|
||||||
|
**P2 — Low.** Every accepted event is defensively cloned on enqueue.
|
||||||
|
Evidence: `MxAccess/MxAccessEventQueue.cs:135` — `mxEvent.Clone()` for an event the mapper just built exclusively for this call (`MxAccess/MxAccessBaseEventSink.cs:210-256`); only the value-cache post-publish shares the original.
|
||||||
|
Impact: doubles protobuf allocation on the hottest path in the worker.
|
||||||
|
Recommendation: take ownership of the passed event in the queue and let the value cache store the copy (it already snapshots only value/quality/timestamp/statuses, `MxAccess/MxAccessValueCache.cs:44-57`).
|
||||||
|
|
||||||
|
**P3 — Low.** No event batching per envelope, one flush per event, and a 25 ms drain poll.
|
||||||
|
Evidence: `Ipc/WorkerPipeSession.cs:17-19` (25 ms interval, batch size 128 is only a read batch), `Ipc/WorkerPipeSession.cs:362-367` (one `WriteAsync` per event), `Ipc/WorkerFrameWriter.cs:71-72` (flush per frame). `gateway.md:849` lists "batch events from worker to gateway while preserving order" as the intended optimization; the `WorkerEnvelope` currently carries a single `WorkerEvent`.
|
||||||
|
Impact: at high data-change rates each event costs a semaphore round-trip, a pipe write, and a flush; idle-to-active latency is up to 25 ms.
|
||||||
|
Recommendation: acceptable for v1 parity; when throughput matters, add a repeated-event envelope body or coalesce flushes across a drained batch.
|
||||||
|
|
||||||
|
**P4 — Low.** The frame writer allocates a fresh `byte[]` per frame while the reader pools.
|
||||||
|
Evidence: `Ipc/WorkerFrameWriter.cs:63-66` (`new byte[frameLength]`) versus `Ipc/WorkerFrameReader.cs:55-77` (`ArrayPool<byte>.Shared`).
|
||||||
|
Recommendation: rent the write buffer from the same pool for symmetry.
|
||||||
|
|
||||||
|
### Conventions
|
||||||
|
|
||||||
|
**C1 — Low.** Private-field naming is split between two styles inside one project.
|
||||||
|
Evidence: `Ipc/` and `Bootstrap/` use `_camelCase` (`Ipc/WorkerPipeSession.cs:21-38`, `Ipc/WorkerFrameWriter.cs:13-15`, `Bootstrap/WorkerConsoleLogger.cs:10`), while `Sta/` and `MxAccess/` use bare `camelCase` (`Sta/StaRuntime.cs:10-24`, `MxAccess/MxAccessStaSession.cs:16-27`, `MxAccess/MxAccessSession.cs:11-16`). `docs/style-guides/CSharpStyleGuide.md:30-32` permits the underscore prefix "only when that pattern is already established in the project" — with both patterns established, the project has no single convention.
|
||||||
|
Recommendation: pick one style for the worker and migrate opportunistically; gateway-side code should be the tie-breaker.
|
||||||
|
|
||||||
|
**C2 — Low.** Documentation drift against `docs/WorkerSta.md` and `docs/MxAccessWorkerInstanceDesign.md`.
|
||||||
|
Evidence: the STA thread is named `"MxGateway.Worker.STA"` (`Sta/StaRuntime.cs:61`) but both docs state `ZB.MOM.WW.MxGateway.Worker.STA` (`docs/WorkerSta.md:23,30`, `docs/MxAccessWorkerInstanceDesign.md:254`); `docs/MxAccessWorkerInstanceDesign.md:653-654` says event queue depth and event sequence "are reported as zero until the event queue implementation owns those counters", but `CaptureHeartbeat` populates both from the live queue (`MxAccess/MxAccessStaSession.cs:375-380`).
|
||||||
|
Impact: CLAUDE.md requires docs to change with the source; operators grepping thread dumps for the documented name will miss the STA thread.
|
||||||
|
Recommendation: fix both docs (or rename the thread) in the next change touching this area.
|
||||||
|
|
||||||
|
**C3 — Low.** Boilerplate duplication in the IPC layer.
|
||||||
|
Evidence: seven pairs of trivially identical `CreateEnvelope`/`CreateBaseEnvelope` overloads (`Ipc/WorkerPipeSession.cs:920-1003`); eight constructor overloads on `WorkerPipeClient` (`Ipc/WorkerPipeClient.cs:36-140`).
|
||||||
|
Impact: purely maintenance noise; every new envelope body means two more copy-paste methods.
|
||||||
|
Recommendation: collapse to a single `CreateEnvelope(Action<WorkerEnvelope> setBody)` or a switch on the body message.
|
||||||
|
|
||||||
|
Positive observations: MXAccess-aligned naming is consistently applied (`MxStatusProxy`, `ServerHandle`, `ItemHandle`, `HResult`, event family names match the contract); every class is `sealed`; `Async` suffixes are correct throughout; the net48 constraints are handled cleanly (plain `readonly struct` instead of records, documented at `MxAccess/MxAccessValueCache.cs:165-168`); and empirical COM findings are captured in code comments with dates and doc references (`MxAccess/WnWrapAlarmConsumer.cs:113-119`, `264-274`), which is exactly the "explain why" documentation style the repo mandates.
|
||||||
|
|
||||||
|
### Underdeveloped
|
||||||
|
|
||||||
|
**U1 — Medium.** The documented outbound write priority is not implemented.
|
||||||
|
Evidence: `docs/MxAccessWorkerInstanceDesign.md:606-613` specifies write priority faults > command replies > shutdown acks > heartbeats > events; the worker has no prioritized queue — all writers contend on a single FIFO `SemaphoreSlim` (`Ipc/WorkerFrameWriter.cs:14`, `68-77`), and events are written inline by the drain loop (`Ipc/WorkerPipeSession.cs:362-367`).
|
||||||
|
Impact: with a deep event backlog draining, a `WorkerFault` or command reply queues behind up to 128 event writes; on a slow pipe this delays the gateway's fault reaction.
|
||||||
|
Recommendation: either implement a small priority write scheduler or amend the design doc to state that FIFO ordering was accepted for v1.
|
||||||
|
|
||||||
|
**U2 — Low.** Gateway death exits with the wrong exit code.
|
||||||
|
Evidence: pipe EOF surfaces as `WorkerFrameProtocolException(EndOfStream)` (`Ipc/WorkerFrameReader.cs:104-109`), which `WorkerApplication` catches first and maps to `WorkerExitCode.ProtocolViolation` (6) (`WorkerApplication.cs:110-119`) even though a distinct `PipeConnectionFailed` (5) exists (`Bootstrap/WorkerExitCode.cs:10`) and the in-session fault mapping does distinguish `PipeDisconnected` (`Ipc/WorkerPipeSession.cs:1212-1220`).
|
||||||
|
Impact: post-mortem triage of orphaned workers ("did the gateway die or did the worker misbehave?") reads the wrong signal from the exit code.
|
||||||
|
Recommendation: special-case `WorkerFrameProtocolErrorCode.EndOfStream` to exit `PipeConnectionFailed`.
|
||||||
|
|
||||||
|
**U3 — Low.** Event-queue overflow behavior diverges from the documented sequence, and exits as `UnexpectedFailure`.
|
||||||
|
Evidence: `docs/MxAccessWorkerInstanceDesign.md:615-624` says overflow should "stop accepting new commands" and "let the gateway close or kill the worker"; the implementation instead terminates the session immediately — the drain loop writes the fault then throws (`Ipc/WorkerPipeSession.cs:344-353`), unwinding `RunAsync` into the generic handler and exit code 1 (`WorkerApplication.cs:131-139`).
|
||||||
|
Impact: fail-fast is arguably stronger than documented (acceptable), but the designed fault path is indistinguishable from a crash by exit code.
|
||||||
|
Recommendation: catch the drain-fault termination in `WorkerApplication` and exit with a dedicated code; update the doc to match the implemented policy.
|
||||||
|
|
||||||
|
**U4 — Low.** Command start/end logging with correlation id is absent.
|
||||||
|
Evidence: `docs/MxAccessWorkerInstanceDesign.md:790-791` lists "command start/end with correlation id" among required worker logs; the only per-command log is the dropped-reply diagnostic (`Ipc/WorkerPipeSession.cs:645-655`). `StaCommand.EnqueueTimestamp` is captured (`Sta/StaCommand.cs:36`) but never used for latency measurement.
|
||||||
|
Recommendation: add optional (level-gated) start/end logging in `StaCommandDispatcher.ExecuteQueuedCommandAsync`, which already brackets each command (`Sta/StaCommandDispatcher.cs:265-281`).
|
||||||
|
|
||||||
|
**U5 — Low.** Test-coverage gaps around the failure modes found above.
|
||||||
|
Evidence: `Worker.Tests` covers the pump wake behavior (`Sta/StaMessagePumpTests.cs`), dispatcher ordering/cancellation/shutdown (`Sta/StaCommandDispatcherTests.cs`), the pipe-session handshake, heartbeat, watchdog (including the stuck ceiling), control commands, shutdown races and late-reply drops (`Ipc/WorkerPipeSessionTests.cs:25-1030`), frame protocol, conversion, the event queue, and the alarm units. Not covered: STA thread death mid-run (S2), wire-level envelope sequence monotonicity under concurrent writers (S4), the silent no-reply drop at the `_acceptingCommands` gate (S3), and the `ReadBulk`-exceeds-ceiling false fault (S1).
|
||||||
|
Recommendation: add tests alongside the fixes; the existing fake-runtime harness in `WorkerPipeSessionTests` supports all four.
|
||||||
|
|
||||||
|
Command-surface parity is otherwise complete: all 18 core MXAccess methods, the 11 bulk variants, `ReadBulk` with cached and snapshot paths, the five diagnostics commands, and the five alarm commands are dispatched (`MxAccess/MxAccessCommandExecutor.cs:95-132`), matching the `gateway.md` command list; `OperationComplete` is emitted only from the native handler and nothing synthesizes events (`MxAccess/MxAccessBaseEventSink.cs:160-171`); buffered payloads preserve raw data-type metadata when conversion is incomplete (`MxAccess/MxAccessEventMapper.cs:237-264`, `379-423`).
|
||||||
|
|
||||||
|
## Top 5 recommendations
|
||||||
|
|
||||||
|
1. **Fix the ReadBulk/watchdog false positive (S1).** Make the `pumpStep` used by long-running commands refresh `LastActivityUtc` (a one-line `MarkActivity()` in `StaRuntime.PumpPendingMessages`), or clamp total `ReadBulk` duration below `HeartbeatStuckCeiling`. This is the only path found where a healthy worker declares itself hung and then silently drops all replies.
|
||||||
|
2. **Make STA thread death observable (S2).** On `ThreadMain` exit, complete queued and future `InvokeAsync` calls with the captured exception and emit a `WorkerFault`; never leave the exception in a write-only field.
|
||||||
|
3. **Reply to commands refused during shutdown (S3).** The `_acceptingCommands` gate should produce a `WorkerUnavailable` reply, matching the dispatcher-level rejection the design docs describe.
|
||||||
|
4. **Assign envelope sequence under the write lock (S4)** so the wire honors the "monotonic per sender" envelope rule; add a concurrent-writer test.
|
||||||
|
5. **Cut event hot-path cost (P1, P2).** Cache `MXSTATUS_PROXY` `FieldInfo` lookups and remove the per-event `Clone()` in `MxAccessEventQueue.Enqueue`; both are localized changes with no protocol impact, and they precede any need for envelope batching (P3).
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
# Contracts & IPC Protocol — Architecture Review
|
||||||
|
|
||||||
|
## Scope & method
|
||||||
|
|
||||||
|
This review covers the contracts project (`src/ZB.MOM.WW.MxGateway.Contracts`: `mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto`, `GatewayContractInfo.cs`, `Generated/`), the named-pipe frame protocol on both sides (`src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrame*` + `WorkerClient.cs`; `src/ZB.MOM.WW.MxGateway.Worker/Ipc/*`), and the proto→client codegen pipeline (`clients/proto/proto-inputs.json`, `scripts/publish-client-proto-inputs.ps1`, per-client generation scripts and gradle/tonic/Grpc.Tools configuration). Method: static reading of source, protos, generated output, scripts, and the related docs (`gateway.md`, `docs/Contracts.md`, `docs/WorkerFrameProtocol.md`, `docs/Grpc.md`, `docs/ClientProtoGeneration.md`, `docs/style-guides/ProtobufStyleGuide.md`), plus git history and a byte-level diff of the Galaxy protos. No builds were run (macOS tree); no source files were modified.
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
- The frame protocol core is sound on both sides: length validation before allocation, configurable 16 MiB cap, exact-read loops for partial reads, typed error codes for malformed/oversized/EOF frames, nonce validation in both directions, and disciplined late-reply handling. The disposable-worker fault model makes "desync = kill session" an acceptable recovery strategy.
|
||||||
|
- The single worst concrete defect is in the codegen pipeline: the checked-in client descriptor set (`clients/proto/descriptors/mxaccessgw-client-v1.protoset`) is seven weeks stale — it predates `MxSparseArray`, `ReplayGap`, and the alarm provider-status surface — and nothing (no CI, no test) runs the existing `-Check` mode.
|
||||||
|
- The pipe frame size limit is hard-coded to 16 MiB in the worker while the gateway's is configurable to 256 MiB; the value is never conveyed across the handshake, so raising the gateway config silently produces frames the worker rejects as protocol faults.
|
||||||
|
- `MaxGrpcMessageBytes` and pipe `MaxMessageBytes` share the same 16 MiB default with zero headroom for envelope overhead, and a write-side oversized frame faults the whole session instead of failing the single command.
|
||||||
|
- `WorkerCancel` is a dead contract arm: the worker dispatches it, the gateway never sends it. The cancellation behavior documented in `gateway.md` is half-implemented.
|
||||||
|
- The worker-side frame writer/reader received single-buffer and pooled-buffer optimizations that were never back-ported to the gateway side; the gateway also deep-clones every command twice and every event once on the hot path.
|
||||||
|
- Proto style discipline is good: versioned packages, `UNSPECIFIED` zero enums, `reserved` on retired fields, wire-compat policy headers, credential-field comments. Generated code is in sync with the protos in all five clients and in `Contracts/Generated/`; `galaxy_repository.proto` is wire-identical to the GalaxyRepository package copy (only `csharp_namespace` differs).
|
||||||
|
- Documentation drift exists at the contract boundary: `docs/Grpc.md` says six RPCs (there are seven), `gateway.md`'s `WorkerEnvelope` sketch has the wrong `correlation_id` type and field numbers, and two docs name the wrong Python generated-output directory.
|
||||||
|
- The known codegen fragilities (Python grpcio-tools pin, Java tracked-generated churn, Generated/ commit-after-regen for net48) are guarded by nothing in the repo — no version pins in scripts, no check tasks, no written rule in `docs/Contracts.md`.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### Stability
|
||||||
|
|
||||||
|
**S1 — Medium — The worker's max frame size is hard-coded and cannot follow the gateway's configured limit.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs:15-21` (the `WorkerOptions` ctor always passes `DefaultMaxMessageBytes` = 16 MiB); `src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs:37` and `GatewayOptionsValidator.cs:150-153` (gateway-side `MxGateway:Worker:MaxMessageBytes` is configurable from 1 KiB to 256 MiB) wired at `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionWorkerClientFactory.cs:76`. Neither `GatewayHello` (`mxaccess_worker.proto:41-45`) nor the worker launch arguments carry the value.
|
||||||
|
Failure scenario: an operator raises the gateway limit above 16 MiB for large-array workloads; the gateway now emits frames the worker's reader rejects (`Worker/Ipc/WorkerFrameReader.cs:44-49`) with `MessageTooLarge`, faulting the session, and the worker still cannot emit anything above 16 MiB in the other direction. The configuration appears to work until the first large frame.
|
||||||
|
Recommendation: carry the negotiated max frame size in `GatewayHello`/`WorkerHello` (additive fields) or a launch argument, and fail the handshake on disagreement instead of failing mid-traffic.
|
||||||
|
|
||||||
|
**S2 — Medium — Public gRPC max message size equals the pipe max frame size with zero headroom, and an oversized outbound frame faults the whole session.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Configuration/ProtocolOptions.cs:16` (`MaxGrpcMessageBytes` default 16 MiB) and `WorkerOptions.cs:37` (pipe max 16 MiB); a gRPC-accepted `Invoke` payload near the limit gains `WorkerCommand` + `WorkerEnvelope` overhead (session id, sequence, correlation id, enqueue timestamp, oneof tags) before hitting `Server/Workers/WorkerFrameWriter.cs:49-54`, whose `WorkerFrameProtocolException` escapes the write loop at `Server/Workers/WorkerClient.cs:344-350` and calls `SetFaulted`, which kills the worker process (`WorkerClient.cs:722-749`).
|
||||||
|
Failure scenario: one legitimate, gateway-accepted oversized write tears down the session, all its subscriptions, and its event stream, instead of failing that single command.
|
||||||
|
Recommendation: enforce a public payload cap with explicit headroom below the pipe max, or catch `MessageTooLarge` at the enqueue/write boundary and fail only the offending correlation id.
|
||||||
|
|
||||||
|
**S3 — Medium — `DrainEvents` with `max_events = 0` packs the entire event queue into one reply envelope.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs:172-192` (0 = drain all); the reply is a single `DrainEventsReply` (`mxaccess_gateway.proto:678-680`) written as one frame in `Worker/Ipc/WorkerPipeSession.cs:522-542` and `470-481`. `MxAccessGrpcRequestValidator` imposes no cap on `DrainEventsCommand.max_events` (rules table, `docs/Grpc.md:164-176`).
|
||||||
|
Failure scenario: a deep queue of large array events produces a reply exceeding `MaxMessageBytes`; the control-reply write throws, the exception propagates out of the message loop, and the worker exits — a diagnostics command kills the session.
|
||||||
|
Recommendation: cap the effective drain count worker-side, or chunk the reply across multiple envelopes.
|
||||||
|
|
||||||
|
**S4 — Low — The envelope `sequence` field is write-only; monotonicity is never validated on receive.**
|
||||||
|
Evidence: sequences are assigned at `Server/Workers/WorkerClient.cs:930-936` and `Worker/Ipc/WorkerPipeSession.cs:1005-1018`, but neither validator checks them (`Server/Workers/WorkerEnvelopeValidator.cs:15-39`, `Worker/Ipc/WorkerEnvelopeValidator.cs:12-36`). `gateway.md:312-314` states "`sequence` is monotonic per sender" as a protocol rule.
|
||||||
|
Impact: gap/duplication detection promised by the design is diagnostics-only; ordering integrity rests entirely on pipe FIFO semantics (which is fine for a local pipe, but the rule is unenforced).
|
||||||
|
Recommendation: either validate monotonicity in the envelope validators (cheap) or annotate the proto/doc that `sequence` is diagnostic only.
|
||||||
|
|
||||||
|
**S5 — Low — No protocol version negotiation exists despite a field name that implies it.**
|
||||||
|
Evidence: `GatewayHello.supported_protocol_version` (`mxaccess_worker.proto:42`) is a single value compared for strict equality (`Worker/Ipc/WorkerPipeSession.cs:215-219`); the worker additionally hard-pins its own version to `GatewayContractInfo.WorkerProtocolVersion` at options construction (`Worker/Ipc/WorkerFrameProtocolOptions.cs:65-70`), and every envelope re-checks equality in both validators.
|
||||||
|
Impact: acceptable for a lockstep-deployed pair (and `gateway.md:319` documents mismatch-fails-session), but any future skewed upgrade of gateway and worker requires new machinery; the singular "supported" field cannot express a range.
|
||||||
|
Recommendation: none required now; when the worker protocol first changes, add a min/max supported range rather than bumping the single constant.
|
||||||
|
|
||||||
|
**S6 — Low — The gateway-side frame writer has no internal write lock; frame integrity rests on an undocumented single-writer invariant.**
|
||||||
|
Evidence: `Server/Workers/WorkerFrameWriter.cs` has no synchronization, while the worker's writer serializes writers with a `SemaphoreSlim` (`Worker/Ipc/WorkerFrameWriter.cs:14,68-77`) because its heartbeat, event-drain, and command tasks write concurrently. The gateway is safe only because all writes funnel through the single-reader outbound channel loop (`Server/Workers/WorkerClient.cs:62-69,332-339`).
|
||||||
|
Impact: a future direct call to `_writer.WriteAsync` outside the write loop interleaves the two stream writes (prefix and payload are separate `WriteAsync` calls, `WorkerFrameWriter.cs:59-60`) and corrupts the stream unrecoverably.
|
||||||
|
Recommendation: add the same write lock or an assertion, and document the invariant on the writer.
|
||||||
|
|
||||||
|
**Verified sound (no finding):** length-prefix validation happens before payload allocation on both sides (`Server/Workers/WorkerFrameReader.cs:35-51`, `Worker/Ipc/WorkerFrameReader.cs:36-60`); partial reads use exact-read loops with a typed `EndOfStream` error; malformed protobuf maps to `InvalidEnvelope`; late/unknown command replies are dropped with a debug log (`WorkerClient.cs:565-571`); the worker drops completed-command replies once the state leaves `Ready` with a diagnostic (`WorkerPipeSession.cs:604-608,645-655`); nonces are validated in both directions; pipe connect uses deadline-bounded exponential retry (`Worker/Ipc/WorkerPipeClient.cs:161-213`). Proto compatibility discipline is real: wire-compat policy headers on all three files, `reserved` ranges and names on the retired `session_id` fields (`mxaccess_gateway.proto:912-916,927-930`), `UNSPECIFIED = 0` on every enum, and `ProtocolStatusCode.OK = 1` distinct from the zero value.
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
**P1 — Medium — Redundant deep copies on the command and event hot paths.**
|
||||||
|
Evidence: `MxAccessGrpcMapper.MapCommand` clones the inbound `MxCommand` (documented in `docs/Grpc.md:197-211`), then `WorkerClient.CreateCommandEnvelope` clones the entire `WorkerCommand` a second time (`Server/Workers/WorkerClient.cs:896-903`, `command.Clone()`); `MapEvent` deep-clones every worker event (`Server/Grpc/MxAccessGrpcMapper.cs:64-73`).
|
||||||
|
Impact: each `Invoke` materializes the command graph three times (gRPC parse, mapper clone, worker-client clone) before pipe serialization; each event is parsed once and cloned once. For array-heavy `OnDataChange` streams this is measurable allocation pressure.
|
||||||
|
Recommendation: drop the second clone in `CreateCommandEnvelope` (the mapper's clone already isolates the graph), and evaluate whether `MapEvent` can transfer ownership instead of cloning.
|
||||||
|
|
||||||
|
**P2 — Low — The gateway frame writer serializes each envelope twice and issues two stream writes; the worker side already fixed this.**
|
||||||
|
Evidence: `Server/Workers/WorkerFrameWriter.cs:41-60` calls `CalculateSize()` then `ToByteArray()` (which re-runs size calculation) and writes prefix and payload separately; the worker writer builds one prefixed buffer with a single `WriteTo(Span)` and one write (`Worker/Ipc/WorkerFrameWriter.cs:58-72`, with a comment explaining exactly this rationale).
|
||||||
|
Impact: extra CPU pass and extra pipe write per frame on the higher-volume side of the connection (the gateway writes every command).
|
||||||
|
Recommendation: back-port the worker's single-buffer implementation.
|
||||||
|
|
||||||
|
**P3 — Low — The gateway frame reader allocates a fresh array per frame; the worker rents from `ArrayPool`.**
|
||||||
|
Evidence: `Server/Workers/WorkerFrameReader.cs:50` (`new byte[payloadLength]`) versus `Worker/Ipc/WorkerFrameReader.cs:51-77` (rent/return with a comment noting `ParseFrom` copies).
|
||||||
|
Impact: large event frames (arrays near the 16 MiB cap) allocate LOH buffers per frame on the side that receives the entire event stream.
|
||||||
|
Recommendation: mirror the pooled read.
|
||||||
|
|
||||||
|
**P4 — Low — Worker event delivery is a 25 ms poll with one envelope write and flush per event.**
|
||||||
|
Evidence: `Worker/Ipc/WorkerPipeSession.cs:17-19` (`EventDrainInterval` 25 ms, batch size 128) and `333-368` (per-event `WriteAsync`, each acquiring the writer lock and calling `FlushAsync` inside `Worker/Ipc/WorkerFrameWriter.cs:71-72`).
|
||||||
|
Impact: an idle-to-active latency floor of up to 25 ms per batch, plus per-event flush syscalls under burst; `gateway.md:849-850` lists worker→gateway event batching as a planned optimization that does not exist (there is no multi-event envelope body).
|
||||||
|
Recommendation: acceptable for v1 parity; when event rate targets firm up, add a batched event body (additive `oneof` arm) rather than tightening the poll.
|
||||||
|
|
||||||
|
**P5 — Info — Correlation id is carried twice per reply.**
|
||||||
|
Evidence: `WorkerEnvelope.correlation_id` (`mxaccess_worker.proto:24`) and `MxCommandReply.correlation_id` (`mxaccess_gateway.proto:520`); `CompleteCommand` must fall back from one to the other (`Server/Workers/WorkerClient.cs:559-563`).
|
||||||
|
Impact: two sources of truth for the same value; harmless today, a divergence hazard for future writers.
|
||||||
|
Recommendation: document which is authoritative (the envelope) in the proto comment.
|
||||||
|
|
||||||
|
### Conventions
|
||||||
|
|
||||||
|
**C1 — Medium — `gateway.md`'s Worker Envelope section no longer matches the shipped contract.**
|
||||||
|
Evidence: `gateway.md:291-309` shows `uint64 correlation_id = 4` and body cases `worker_hello = 10; gateway_hello = 11; command = 20; command_reply = 21; event = 22; heartbeat = 23; cancel = 24; shutdown = 25; fault = 26`; the actual contract is `string correlation_id = 4` and `gateway_hello = 10; worker_hello = 11; worker_command = 13; worker_command_reply = 14; worker_cancel = 15; worker_shutdown = 16; worker_shutdown_ack = 17; worker_event = 18; worker_heartbeat = 19; worker_fault = 20` (`mxaccess_worker.proto:20-39`). `WorkerShutdownAck` is absent from the sketch entirely.
|
||||||
|
Impact: field numbers and types in the top-level architecture doc are wrong; anyone implementing an alternate worker (the doc explicitly contemplates a C++ worker, `gateway.md:1077-1099`) from this section produces an incompatible peer. This violates the repo rule that docs change with the source.
|
||||||
|
Recommendation: replace the sketch with the real message or a pointer to `mxaccess_worker.proto`.
|
||||||
|
|
||||||
|
**C2 — Medium — `docs/Grpc.md` says the service has six RPCs; it has seven.**
|
||||||
|
Evidence: `docs/Grpc.md:13` and `:32` ("six in total") omit `QueryActiveAlarms`, which is declared at `mxaccess_gateway.proto:37` and implemented at `src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGatewayService.cs:212`.
|
||||||
|
Impact: the authoritative gRPC-layer doc undercounts the public surface and documents no validation/handler behavior for the missing RPC.
|
||||||
|
Recommendation: update the count, the collaborators table, and add a `QueryActiveAlarms` handler section.
|
||||||
|
|
||||||
|
**C3 — Low — Two docs name the wrong Python generated-output directory.**
|
||||||
|
Evidence: `docs/ClientProtoGeneration.md:81` and `:144-146` (and CLAUDE.md's generated-code bullet) say `clients/python/src/mxgateway/generated`; the manifest (`clients/proto/proto-inputs.json`, `generatedOutputs.python`) and the tree use `clients/python/src/zb_mom_ww_mxgateway/generated`, which is also what `clients/python/generate-proto.ps1` writes.
|
||||||
|
Impact: stale path in the generation guide; a follow-the-doc regeneration writes to a dead directory.
|
||||||
|
Recommendation: fix both docs to the manifest path.
|
||||||
|
|
||||||
|
**C4 — Info (positive) — Generated-code hygiene and Galaxy wire-identity check out.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Contracts/Generated/*.cs` carry `<auto-generated>` headers and contain the newest contract symbols (`MxSparseArray`, 39 hits in `MxaccessGateway.cs`), so the committed output matches the protos; `clients/go/internal/generated`, `clients/java/src/main/generated`, and `clients/python/.../generated` all contain the sparse-array surface and the Java generated tree is clean in git (last regen committed 2026-06-18); Rust generates at build time via `tonic-build` with `clients/rust/src/generated` held as a documented `.gitkeep` placeholder (`clients/rust/src/generated.rs` header). `diff` of `galaxy_repository.proto` against the GalaxyRepository package source (`/Users/dohertj2/Desktop/scadaproj/ZB.MOM.WW.GalaxyRepository/src/ZB.MOM.WW.GalaxyRepository/Protos/galaxy_repository.proto`) shows the only difference is `csharp_namespace` — wire-identical, as the retention comment in the contracts csproj requires (`ZB.MOM.WW.MxGateway.Contracts.csproj:29-33`).
|
||||||
|
|
||||||
|
**C5 — Low — The Generated/-must-be-committed rule for net48 consumers is undocumented and unguarded.**
|
||||||
|
Evidence: `ZB.MOM.WW.MxGateway.Contracts.csproj:26-35` (`Compile Remove="Generated\**\*.cs"` + `Protobuf ... OutputDir="Generated"`) regenerates tracked files on every build; the worker consumes the contracts via `ProjectReference` (`ZB.MOM.WW.MxGateway.Worker.csproj:22`). `docs/Contracts.md:97-98` says only "do not hand-edit"; the operational rule that a proto edit requires regenerating and committing `Generated/` (or the net48 build fails on new types) appears nowhere in the docs, and no test compares `Generated/` to the protos.
|
||||||
|
Impact: same silent-drift class as the descriptor (U1) — the committed C# can lag the protos with nothing failing until a downstream consumer breaks.
|
||||||
|
Recommendation: state the rule in `docs/Contracts.md` and add a freshness check (e.g., a contracts test that reflects over `MxaccessGatewayReflection.Descriptor` and compares against the `.proto` file descriptor).
|
||||||
|
|
||||||
|
### Underdeveloped
|
||||||
|
|
||||||
|
**U1 — High — The published client descriptor set is seven weeks stale and nothing enforces its freshness.**
|
||||||
|
Evidence: `clients/proto/descriptors/mxaccessgw-client-v1.protoset` was last committed 2026-04-30 (`git log`), while `mxaccess_gateway.proto` changed through 2026-06-18 (`MxSparseArray`), 2026-06-16 (`ReplayGap`), and 2026-06-15 (alarm provenance); `strings` over the protoset finds zero occurrences of `MxSparseArray`, `replay_gap`, or `provider_status`. `docs/Contracts.md:127-131` mandates regenerating the descriptor after any proto change; `scripts/publish-client-proto-inputs.ps1` provides a `-Check` mode, but the repo has no CI workflow directory at all and `src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs` validates only manifest versions and path existence — never descriptor content.
|
||||||
|
Impact: the artifact documented as the "stable client input" (`docs/ClientProtoGeneration.md:48-69`) silently misrepresents the contract; any consumer that "prefers a descriptor input" (the doc's own words) generates against a schema missing three shipped features, and grpcurl users on the reflection-disabled deployments get a stale schema.
|
||||||
|
Recommendation: regenerate and commit the descriptor now; add an automated freshness gate (test or CI step) and pin the protoc version it uses.
|
||||||
|
|
||||||
|
**U2 — Medium — `WorkerCancel` is defined and handled but never sent.**
|
||||||
|
Evidence: `mxaccess_worker.proto:71-73` defines it, `Worker/Ipc/WorkerPipeSession.cs:399-401` dispatches it to `_runtimeSession.CancelCommand`, but a repo-wide grep of `src/ZB.MOM.WW.MxGateway.Server` finds zero references — `WorkerClient.InvokeAsync` handles timeout/cancel purely by abandoning the pending correlation (`Server/Workers/WorkerClient.cs:195-213`).
|
||||||
|
Impact: the cancellation contract in `gateway.md:713-719` ("the worker should finish the COM call and discard or log the late reply if the correlation was canceled") is half-implemented: the worker never learns a correlation was canceled, so it always writes the late reply (which the gateway then drops), and any worker-side discard logic behind `CancelCommand` is unreachable.
|
||||||
|
Recommendation: send `WorkerCancel` from the gateway on timeout/caller-cancel, or mark the arm as reserved-for-future in the proto comment and delete the dead worker handling.
|
||||||
|
|
||||||
|
**U3 — Medium — The known codegen fragilities have no in-repo guards.**
|
||||||
|
Evidence: (a) Python — `clients/python/pyproject.toml:41` allows `grpcio-tools>=1.80,<2` and `clients/python/generate-proto.ps1` performs no version check, so regenerating with any newer 1.x stamps a `GRPC_GENERATED_VERSION` above the pinned runtime and breaks pytest (a known, previously-hit failure); (b) Java — `clients/java/zb-mom-ww-mxgateway-client/build.gradle:50` points `generatedFilesBaseDir` at the tracked `src/main/generated`, so every `gradle build` rewrites ~64k lines of tracked output with protobuf-version churn and no check task detects spurious diffs; (c) portability — `clients/go/generate-proto.ps1:8-9` and `clients/python/generate-proto.ps1:7` hard-code `C:\Users\dohertj2\...` tool paths, and `scripts/publish-client-proto-inputs.ps1` resolves only `protoc.exe`, making every generation step single-machine and Windows-only.
|
||||||
|
Impact: contract evolution safety depends on operator memory rather than tooling; a regeneration on a different machine or tool version silently produces incompatible or noisy output.
|
||||||
|
Recommendation: pin exact generator versions in the scripts (fail fast on mismatch), add a Java `checkGeneratedClean`-style task, and resolve tools from PATH with a documented version assertion instead of absolute user paths.
|
||||||
|
|
||||||
|
**U4 — Low — The descriptor `-Check` compares raw bytes, which is protoc-version-sensitive.**
|
||||||
|
Evidence: `scripts/publish-client-proto-inputs.ps1` (`Compare-FileBytes`) byte-compares a descriptor built with `--include_source_info`; source-info bytes differ across protoc releases even for identical schemas.
|
||||||
|
Impact: once a check exists (U1), a protoc upgrade produces a false "stale" failure — or forces a churn commit — unless the protoc version is pinned.
|
||||||
|
Recommendation: pin protoc for descriptor generation, or compare descriptors semantically (drop source info) instead of byte-wise.
|
||||||
|
|
||||||
|
**U5 — Low — Contract surface promised in `gateway.md` but absent: the bidirectional `Session` RPC.**
|
||||||
|
Evidence: `gateway.md:328-345` sketches `rpc Session(stream ClientMessage) returns (stream ServerMessage)` as the "best long-term shape" with a rollout plan whose step 3 is unimplemented; `mxaccess_gateway.proto:17-38` has no such RPC.
|
||||||
|
Impact: intentional phasing, not a defect — but the doc presents it inside the service definition block rather than as future work, compounding C1's staleness.
|
||||||
|
Recommendation: mark it explicitly as not-yet-implemented in `gateway.md`.
|
||||||
|
|
||||||
|
**U6 — Info — Public error detail model stops at status codes plus prose.**
|
||||||
|
Evidence: `MxCommandReply` preserves MXAccess parity detail well (`hresult`, `statuses`, `diagnostic_message`, `mxaccess_gateway.proto:518-529`), but transport-level failures surface only as gRPC status codes with message strings (`docs/Grpc.md:217-241`); no `google.rpc` error details are attached, and `AcknowledgeAlarmReply.status` is a permanently-unset placeholder documented as such (`mxaccess_gateway.proto:940-945`).
|
||||||
|
Impact: machine consumers must parse prose to distinguish sub-causes within a status code; acceptable for the current client set.
|
||||||
|
Recommendation: none now; consider `google.rpc.ErrorInfo` if third-party clients appear.
|
||||||
|
|
||||||
|
## Top 5 recommendations
|
||||||
|
|
||||||
|
1. Regenerate and commit `clients/proto/descriptors/mxaccessgw-client-v1.protoset`, then add an automated freshness gate (a contracts test or CI step running the `-Check` equivalent with a pinned protoc) so the doc-mandated regeneration step cannot be skipped silently again (U1, U4, C5).
|
||||||
|
2. Make the pipe frame limit a negotiated value: carry it (or at least assert agreement) in the `GatewayHello`/`WorkerHello` exchange, keep `MaxGrpcMessageBytes` below the pipe max by an explicit headroom margin, and convert write-side `MessageTooLarge` from a session-killing fault into a per-command failure (S1, S2, S3).
|
||||||
|
3. Close the cancellation gap: either send `WorkerCancel` from `WorkerClient` on timeout/cancel so the worker's `CancelCommand` path is reachable, or reserve the arm and remove the dead handling, updating `gateway.md` either way (U2).
|
||||||
|
4. Back-port the worker's frame I/O optimizations to the gateway (single-buffer write, pooled read) and remove the redundant second `Clone` in `WorkerClient.CreateCommandEnvelope` (P1, P2, P3).
|
||||||
|
5. Fix the contract-boundary doc drift in one pass: `gateway.md` envelope sketch and unimplemented `Session` RPC, `docs/Grpc.md` RPC count and missing `QueryActiveAlarms` section, and the Python generated-directory path in `docs/ClientProtoGeneration.md`/CLAUDE.md; document the Generated/ regen-and-commit rule in `docs/Contracts.md` (C1, C2, C3, C5, U5).
|
||||||
@@ -0,0 +1,186 @@
|
|||||||
|
# Security, Dashboard & Observability — Architecture Review
|
||||||
|
|
||||||
|
## Scope & method
|
||||||
|
|
||||||
|
This review covers `src/ZB.MOM.WW.MxGateway.Server`: `Security/` (API-key authentication glue, gRPC authorization interceptor, scope resolver, constraint enforcement, audit, TLS), `Dashboard/` (LDAP login, cookie/hub-token authentication, SignalR hubs, admin services, redaction), `Metrics/`, `Diagnostics/`, and the security-relevant parts of `Configuration/`. Method: static reading of the actual source on the macOS tree (no builds, no runtime probes, no source modifications), cross-checked against `gateway.md`, `glauth.md`, `docs/Authentication.md`, `docs/Authorization.md`, `docs/GatewayDashboardDesign.md`, `docs/GatewayConfiguration.md`, `docs/Metrics.md`, and `docs/Diagnostics.md`. Every finding cites a verified `path:line`.
|
||||||
|
|
||||||
|
An important structural fact for this review: the core API-key pipeline (parser, peppered HMAC hasher, `FixedTimeEquals` compare, SQLite stores, verifier, migrator) no longer lives in this repository. It moved to the shared `ZB.MOM.WW.Auth.ApiKeys` 0.1.2 package (`src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj:11`, remark in `src/ZB.MOM.WW.MxGateway.Server/Security/Authentication/AuthStoreServiceCollectionExtensions.cs:16-23`). Claims about hashing and timing-safety in this report are therefore sourced from `docs/Authentication.md` and the package's test coverage in this repo, not from readable implementation code.
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
- The confirmed lead is real: `src/ZB.MOM.WW.MxGateway.Server/C:\ProgramData\MxGateway\gateway-auth.db` is a literal filename containing backslashes — a genuine SQLite auth database (schema v2: `api_keys`, `api_key_audit`, `schema_version`) created in the source tree because the Windows-absolute default path in `AuthenticationOptions.cs:9` is treated as a relative filename on macOS. It contains **zero key rows and zero audit rows**, and it is untracked (caught by the `*.db` gitignore rule), so no hash material leaked — this time.
|
||||||
|
- The gRPC authorization design is sound: one global interceptor, fail-closed scope fallback to `admin`, distinct `Unauthenticated`/`PermissionDenied`, and an ambient identity accessor consumed by the constraint enforcer and browse-scope provider.
|
||||||
|
- The scope resolver is missing an arm for `QueryActiveAlarmsRequest`, so that RPC silently demands the `admin` scope; the two tests that appear to cover it actually construct `StreamAlarmsRequest`, masking the gap.
|
||||||
|
- `AllowAnonymousLocalhost` (default `true`) satisfies **every** dashboard authorization requirement on loopback — including `AdminOnly` — not just Viewer. Service-layer re-checks currently prevent anonymous key/session management, but the policy layer is a single-line mistake away from anonymous local admin.
|
||||||
|
- The dashboard cookie is named `MxGatewayDashboard`; `gateway.md`, `docs/GatewayDashboardDesign.md`, and `CLAUDE.md` all still claim `__Host-MxGatewayDashboard`. The `__Host-` browser-enforced protections documented everywhere are not actually in effect.
|
||||||
|
- Secret-logging discipline is good: LDAP passwords and the pepper are redacted from effective config, the Serilog redaction seam masks API keys, and no logging call site was found emitting passwords, secrets, or credentials.
|
||||||
|
- Per-RPC authentication costs a SQLite read **and** a `last_used_utc` write on every call, with no caching; this is the gateway's per-call throughput ceiling.
|
||||||
|
- `mxgateway.heartbeats.failed` is tagged with `session_id`, an unbounded-cardinality label on an exported counter.
|
||||||
|
- Underdeveloped areas: no API-key expiry, no rate limiting or lockout on either auth surface, hub bearer tokens are irrevocable for 30 minutes, dashboard Close/Kill actions bypass the canonical audit store, and the per-session EventsHub ACL is an acknowledged TODO.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
**SEC-1 · Medium — Windows-absolute default paths become relative files on non-Windows; a real auth DB materialized inside the source tree.**
|
||||||
|
Evidence: the stray file `src/ZB.MOM.WW.MxGateway.Server/C:\ProgramData\MxGateway\gateway-auth.db` (32 KB, SQLite 3.x, tables `api_keys`/`api_key_audit`/`schema_version`, zero rows in both data tables); default in `src/ZB.MOM.WW.MxGateway.Server/Configuration/AuthenticationOptions.cs:9` and `src/ZB.MOM.WW.MxGateway.Server/appsettings.json:17`.
|
||||||
|
Mechanism: on Unix, `C:\ProgramData\MxGateway\gateway-auth.db` contains no path separators, so the shared `AuthSqliteConnectionFactory` (which "ensures the parent directory exists" per `docs/Authentication.md:110`) sees no parent directory and SQLite creates the whole string as a single filename relative to the content root, which `GatewayApplication.ResolveContentRootPath` resolves to the server project directory (`src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:132-157`). The migration hosted service (`RunMigrationsOnStartup` default `true`, `AuthenticationOptions.cs:15`) or a CLI run then creates the schema. The startup validator does not catch this: `AddIfInvalidPath` only requires `Path.GetFullPath` to succeed (`src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs:40-43,383-410`), and the string is a valid relative path on Unix.
|
||||||
|
Impact: (a) the auth DB location silently depends on host OS and working directory, so keys created on one launch path are invisible on another; (b) had a key been created on this tree, its peppered hash, key id, scopes, and audit rows (with remote addresses) would sit inside the repo protected only by the generic `*.db` gitignore entry; (c) the same defect class applies to `src/ZB.MOM.WW.MxGateway.Server/Configuration/TlsOptions.cs:11-12` (`SelfSignedCertPath` — a **private-key PFX** would be written into the tree if an HTTPS endpoint were configured on a non-Windows host) and `appsettings.json:80` (`SnapshotCachePath`).
|
||||||
|
Recommendation: derive defaults from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` + `Path.Combine`, and make `GatewayOptionsValidator` fail startup when `Path.IsPathRooted` is false for `SqlitePath`/`SelfSignedCertPath`. Delete the stray file and add an explicit ignore (or a test) that fails if a `*.db` ever appears under `src/`.
|
||||||
|
|
||||||
|
**SEC-2 · Medium — `AllowAnonymousLocalhost` satisfies the Admin requirement, not just Viewer.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardAuthorizationHandler.cs:32-37` calls `context.Succeed(requirement)` for any loopback request before the role loop runs, and the same handler serves `DashboardAuthorizationRequirement.AdminOnly` and `AnyDashboardRole`. The same applies to `Authentication:Mode=Disabled` at lines 25-30, which succeeds every requirement for **remote** requests too.
|
||||||
|
Impact: any endpoint or component gated solely by the `MxGateway.Dashboard.Admin` policy is authorized for an anonymous local process (or for everyone when auth is disabled). Today the destructive surfaces are saved by service-layer re-checks that require an authenticated principal with the Admin role (`Dashboard/DashboardApiKeyAuthorization.cs:10-18`, `Dashboard/DashboardApiKeyManagementService.cs:35-37`, `Dashboard/DashboardSessionAdminService.cs:33-39`), so the docs' claim that anonymous localhost is read-only holds — but only by defense-in-depth, not by the policy. Additionally, the loopback bypass grants anonymous local processes full SignalR hub access (`HubClientsPolicy` uses the same requirement): the snapshot hub pushes the API-key inventory (key ids, scopes, constraints) and effective configuration every second (`Dashboard/DashboardSnapshotService.cs:102-103`), and `EventsHub.SubscribeSession` lets them join any session's raw `MxEvent` feed (`Dashboard/Hubs/EventsHub.cs:47-55`), including tag values.
|
||||||
|
Recommendation: make the loopback bypass satisfy only the Viewer requirement (check `requirement.RequiredRoles` before succeeding), and decide explicitly whether anonymous loopback should reach the hubs at all. Note the loopback test trusts `Connection.RemoteIpAddress` (`DashboardAuthorizationHandler.cs:56-61`); if a reverse proxy or forwarded-headers middleware is ever added, revisit.
|
||||||
|
|
||||||
|
**SEC-3 · Medium — Dashboard cookie lost its `__Host-` prefix; four documents still promise it.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardAuthenticationDefaults.cs:38` (`CookieName = "MxGatewayDashboard"`) versus `gateway.md:211`, `docs/GatewayDashboardDesign.md:425`, `docs/GatewayProcessDesign.md:686`, `docs/ImplementationPlanGateway.md:454`, and the project `CLAUDE.md`. Only `docs/GatewayConfiguration.md:170` reflects the real name.
|
||||||
|
Impact: the `__Host-` prefix's browser-enforced guarantees (Secure required, no `Domain`, `Path=/`) are gone. The cookie is still HttpOnly/SameSite=Strict/SecurePolicy-controlled via `ZbCookieDefaults.Apply` (`Dashboard/DashboardServiceCollectionExtensions.cs:93-109`), and `RequireHttpsCookie=false` support plus the configurable `CookieName` (`Configuration/DashboardOptions.cs:39,50`) are legitimate reasons the prefix was dropped — but the security docs now overstate the cookie's protections.
|
||||||
|
Recommendation: either restore `__Host-` as the default when `RequireHttpsCookie` is true, or update `gateway.md`/design docs/`CLAUDE.md` in one change to describe the actual cookie contract ("update docs in the same change as the source" is a stated repo rule).
|
||||||
|
|
||||||
|
**SEC-4 · Medium — `DisableLogin` has no production guard.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardServiceCollectionExtensions.cs:63-90` swaps in `DashboardAutoLoginAuthenticationHandler` under the cookie scheme for **all** clients (remote included, `Dashboard/DashboardAutoLoginAuthenticationHandler.cs:56-62`); `GatewayOptionsValidator.ValidateDashboard` (`Configuration/GatewayOptionsValidator.cs:219-253`) never inspects `DisableLogin`, and the warning only fires on first `GatewayOptions` resolution.
|
||||||
|
Impact: a single copied config flag turns the entire dashboard — including API-key CRUD and worker Kill — into an unauthenticated admin surface on a network-exposed port (both deployed hosts bind 0.0.0.0).
|
||||||
|
Recommendation: fail startup (or force the flag off) when `IHostEnvironment.IsProduction()` and `DisableLogin` is true; at minimum, add a validator error so the misconfiguration is fail-fast like every other section.
|
||||||
|
|
||||||
|
**SEC-5 · Medium — Hub bearer tokens are irrevocable and travel in query strings.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/HubTokenService.cs:29,44-52` (30-minute data-protected token, no jti/revocation state, roles frozen at issue time) and `Dashboard/HubTokenAuthenticationHandler.cs:59-61` (accepts `?access_token=` on the WebSocket upgrade).
|
||||||
|
Impact: logout (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:136-155`) clears the cookie but every previously minted hub token stays valid for up to 30 minutes; a token captured from an access log or proxy grants live snapshot/alarm/event access. The query-string carriage is the standard SignalR pattern but puts bearer material where request logging can see it.
|
||||||
|
Recommendation: keep the lifetime short (or shorten to ~5 minutes given the factory refreshes per reconnect, `docs/GatewayDashboardDesign.md:497-499`), and confirm no request-path logging captures query strings (Serilog request logging is not currently enabled; keep it that way or scrub `access_token`).
|
||||||
|
|
||||||
|
**SEC-6 · Medium — LDAP is plaintext-by-default with a committed service-account password.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Configuration/LdapOptions.cs:49-61` (defaults `Transport=None`, `AllowInsecure=true`, `ServiceAccountPassword = "serviceaccount123"`), `appsettings.json:21-33` (same values checked into the repo), `glauth.md:30,327` (dev LDAPS disabled; "binding sends passwords cleartext on the wire").
|
||||||
|
Impact: every dashboard login sends the operator's password in cleartext to `10.100.0.35:3893`, and the LDAP service-account credential is in source control. This is a documented dev posture (the shadow-options rationale at `LdapOptions.cs:20-28` is explicit that the shared library is secure-by-default), and the validator does enforce the `Transport=None ⇒ AllowInsecure` consistency rule (`GatewayOptionsValidator.cs:82-85`) — but nothing distinguishes dev from prod at runtime.
|
||||||
|
Recommendation: for production deployment docs, require `Transport=Ldaps`/`StartTls` + `AllowInsecure=false` and move `ServiceAccountPassword` to env-var/secret configuration; consider an `IsProduction` startup check mirroring SEC-4. LDAP injection risk is delegated to the shared `ZB.MOM.WW.Auth.Ldap` provider (bind-then-search per `Dashboard/DashboardAuthenticator.cs:41-47`); its escaping cannot be verified from this repo — flag for review in the donor repo.
|
||||||
|
|
||||||
|
**SEC-7 · Low — Credential-bearing command list omits the secured-bulk variants.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Diagnostics/GatewayLogRedactor.cs:11-16` names `AuthenticateUser`, `WriteSecured`, `WriteSecured2` but not `WriteSecuredBulk`/`WriteSecured2Bulk`, which exist as command kinds (`Security/Authorization/GatewayGrpcScopeResolver.cs:41-45`).
|
||||||
|
Impact: currently latent — `RedactCommandValue`/`IsCredentialBearingCommand` have no call sites in the server outside the redactor itself (verified by grep), so no value logging occurs at all. If value logging is ever wired up per `docs/Diagnostics.md:124-148`, secured-bulk payloads (which carry credentials) would pass the unconditional-redaction check.
|
||||||
|
Recommendation: add the two bulk names now, and add a test asserting every `WriteSecured*` command kind is credential-bearing.
|
||||||
|
|
||||||
|
**SEC-8 · Low — `/metrics` and `/health` are mapped with no visible authorization; a metric leaks session ids.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:196-197` maps `MapZbHealth()`/`MapZbMetrics()` (shared packages) with no `RequireAuthorization`; `Metrics/GatewayMetrics.cs:354` tags `mxgateway.heartbeats.failed` with `session_id`.
|
||||||
|
Impact: if the packages do not enforce auth internally (not verifiable from this repo), an unauthenticated scraper on the gRPC/dashboard port can read gateway telemetry, including live session identifiers usable with the anonymous-localhost hub surface (SEC-2).
|
||||||
|
Recommendation: verify the shared packages' endpoint auth; if anonymous, either bind metrics to a loopback-only endpoint or replace the `session_id` tag (see PERF-2).
|
||||||
|
|
||||||
|
**SEC-9 · Low — GET `/logout` skips antiforgery.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardEndpointRouteBuilderExtensions.cs:53-58` (comment acknowledges the choice).
|
||||||
|
Impact: a third-party page can sign a dashboard operator out (nuisance-level CSRF; no state beyond the session is affected). POST login/logout correctly validate antiforgery (`:104,140`).
|
||||||
|
Recommendation: acceptable as documented; consider a confirmation interstitial if it ever grows side effects.
|
||||||
|
|
||||||
|
**SEC-10 · Low — CLI accepts the pepper as a command-line argument.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Program.cs:37-40` (`command.Pepper` → `MxGateway:ApiKeyPepper`).
|
||||||
|
Impact: the pepper lands in shell history and the process command line visible to other local users.
|
||||||
|
Recommendation: prefer an environment variable or prompt; document the risk in the CLI help.
|
||||||
|
|
||||||
|
**SEC-11 · Info — Positive observations.**
|
||||||
|
Verified: the interceptor treats every unrecognized request type as `admin` (fail-closed, `Security/Authorization/GatewayGrpcScopeResolver.cs:28`); authentication failures are opaque to clients (`Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:66-78`); the effective-config surface redacts the pepper name and LDAP password (`Configuration/GatewayConfigurationProvider.cs:20,30`); the Galaxy connection string is rebuilt field-by-field for display (`Dashboard/DashboardConnectionStringDisplay.cs:14-24`); the Serilog seam masks identity-bearing properties globally (`Diagnostics/GatewayLogRedactorSeam.cs:17-27`); self-signed PFX generation hardens file permissions before writing private-key bytes and clears the buffer (`Security/Tls/SelfSignedCertificateProvider.cs:160-193`); scope strings are validated against the canonical catalog on every creation path (`Security/Authorization/GatewayScopes.cs:20-36`, `Dashboard/DashboardApiKeyManagementService.cs:297-304`); dashboard key-management inputs constrain key ids to a safe character set (`Dashboard/DashboardApiKeyManagementService.cs:309-321`); the login flow returns one generic failure message for every failure mode (`Dashboard/DashboardAuthenticator.cs:26,34-67`); `SanitizeReturnUrl` blocks open redirects (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:196-207`). No grep hit was found for any logging call emitting passwords, secrets, peppers, or credentials.
|
||||||
|
|
||||||
|
### Stability
|
||||||
|
|
||||||
|
**STA-1 · Medium — `QueryActiveAlarmsRequest` is missing from the scope resolver, and the tests that claim to cover it test the wrong request type.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/GatewayGrpcScopeResolver.cs:15-29` has no `QueryActiveAlarmsRequest` arm, so the RPC (`Grpc/MxAccessGatewayService.cs:212-213`, proto `mxaccess_gateway.proto:37`) falls to the `_ => GatewayScopes.Admin` default. The two tests named `ServerStreamingServerHandler_QueryActiveAlarms…` construct `new StreamAlarmsRequest()` instead (`src/ZB.MOM.WW.MxGateway.Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:322-359`), so they pass while asserting the intended `events:read` behavior against the wrong message. `docs/Authorization.md`'s scope table (lines 200-215) omits the RPC entirely.
|
||||||
|
Impact: fail-closed, so not a vulnerability — but a client holding `events:read` (the documented alarm/event scope) receives `PermissionDenied` demanding `admin` when calling `QueryActiveAlarms`, contradicting the stated design that alarm snapshot data shares the event surface.
|
||||||
|
Recommendation: add `QueryActiveAlarmsRequest => GatewayScopes.EventsRead`, fix both tests to use the real request type, and add the row to `docs/Authorization.md`.
|
||||||
|
|
||||||
|
**STA-2 · Low — The interceptor does not override client-streaming or duplex handlers.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:21-47` overrides only `UnaryServerHandler` and `ServerStreamingServerHandler`. All current RPCs are unary or server-streaming (verified against both `.proto` files), so nothing bypasses auth today.
|
||||||
|
Impact: a future duplex/client-streaming RPC would run with **no** authentication and no scope check, silently.
|
||||||
|
Recommendation: override `ClientStreamingServerHandler` and `DuplexStreamingServerHandler` to throw `Unimplemented` (or run the same auth path with an `admin` fallback scope) so the failure mode is loud.
|
||||||
|
|
||||||
|
**STA-3 · Low — Pepper-unavailable detection matches library exception message text.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardApiKeyManagementService.cs:21,281-282` catches `InvalidOperationException` whose `Message` contains `"pepper unavailable"`.
|
||||||
|
Impact: a wording change in `ZB.MOM.WW.Auth.ApiKeys` turns the friendly "pepper is not configured" result into an unhandled exception on the Blazor circuit.
|
||||||
|
Recommendation: ask the library to expose a typed exception (the pre-cutover code had `ApiKeyPepperUnavailableException` per `docs/Authentication.md:68`) and catch that.
|
||||||
|
|
||||||
|
**STA-4 · Low — Canonical audit store re-issues `CREATE TABLE IF NOT EXISTS` on every write and read.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Security/Audit/SqliteCanonicalAuditStore.cs:54,94,131-136`.
|
||||||
|
Impact: an extra round-trip per audit operation and a schema definition that lives outside the migrator, so a future column change has no migration path — the `IF NOT EXISTS` silently keeps the old shape.
|
||||||
|
Recommendation: move `audit_event` creation into the startup migration path and drop the per-call ensure.
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
**PERF-1 · Medium — Every authenticated RPC performs a SQLite read plus a `last_used_utc` write; there is no verification cache.**
|
||||||
|
Evidence: the interceptor calls `IApiKeyVerifier.VerifyAsync` per call (`Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:69-71`); the verifier contract is find-by-id → hash → compare → `MarkKeyUsedAsync` per `docs/Authentication.md:72-96,122`, which explicitly notes the write "runs on every authenticated request". The interceptor additionally deserializes the constraints JSON per call (`Security/Authentication/GatewayApiKeyIdentityMapper.cs:41`).
|
||||||
|
Impact: WAL and busy-timeout make this correct, but a per-call database **write** is the gateway's throughput ceiling on the hot path (bulk reads at high frequency are the primary workload), causes continuous WAL churn on `gateway-auth.db`, and makes auth-store latency a tail-latency contributor to every RPC.
|
||||||
|
Recommendation: add a short-TTL (5-30 s) in-memory verification cache keyed by key id + presented-hash, invalidated on revoke/rotate (all mutations flow through in-process `ApiKeyAdminCommands`), and coalesce `last_used_utc` updates to at most one write per key per minute.
|
||||||
|
|
||||||
|
**PERF-2 · Low — `session_id` as a metric tag is unbounded cardinality.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Metrics/GatewayMetrics.cs:354` (`mxgateway.heartbeats.failed` tagged `session_id`); `docs/Metrics.md:47` documents it.
|
||||||
|
Impact: every session ever opened mints a new exporter time series; long-running gateways with session churn bloat Prometheus/OTLP storage. (The in-memory `EventsBySession` map is correctly pruned on session close, `GatewayMetrics.cs:310-313` — the problem is only the exported tag.)
|
||||||
|
Recommendation: drop the tag (keep the aggregate counter) and rely on the dashboard snapshot/log scope for per-session attribution.
|
||||||
|
|
||||||
|
**PERF-3 · Low — The snapshot publisher works every second regardless of audience, including a SQLite query per tick.**
|
||||||
|
Evidence: `src/ZB.MOM.WW.MxGateway.Server/Dashboard/DashboardSnapshotService.cs:134-166` (hosted publisher loop), `:256-279` (`RefreshApiKeySummariesAsync` calls `IApiKeyAdminStore.ListAsync` on every tick, gated and 2-s-timeboxed), `:76-105` (`GetSnapshot` rebuilds session/worker/metric/fault/config projections each tick). The Galaxy breakdown is properly memoized by sequence (`:107-129`).
|
||||||
|
Impact: constant background SQLite reads and allocation churn on an idle gateway; the full snapshot (including API-key summaries and effective configuration) is serialized to all hub clients every second.
|
||||||
|
Recommendation: refresh the API-key summary on a slower cadence (or on mutation, since all mutations are in-process), and consider skipping publication when the hub has no connections.
|
||||||
|
|
||||||
|
### Conventions
|
||||||
|
|
||||||
|
**CON-1 · Medium — The dashboard design doc's configuration sample now fails startup validation.**
|
||||||
|
Evidence: `docs/GatewayDashboardDesign.md:515-519` shows `"GroupToRole": { "GwAdmin": "Admin", … }`, but `GatewayOptionsValidator.cs:233-238` accepts only `DashboardRoles.Admin` = `"Administrator"` (`Dashboard/DashboardRoles.cs:14`) or `"Viewer"` (ordinal compare). The live `appsettings.json:66-69` correctly uses `"Administrator"`.
|
||||||
|
Impact: an operator copying the documented sample gets a fail-fast boot error; the doc also still describes the role as `Admin` throughout (e.g. lines 412-413, 434).
|
||||||
|
Recommendation: update the doc sample and prose to the canonical `Administrator` value (the rename is otherwise well-annotated in `glauth.md:79-83`).
|
||||||
|
|
||||||
|
**CON-2 · Low — `docs/Authentication.md` documents implementation types that no longer exist in this repository.**
|
||||||
|
Evidence: the doc presents `ApiKeyParser`, `ApiKeySecretGenerator`, `SqliteApiKeyStore`, `AuthStoreServiceCollectionExtensions.AddSqliteAuthStore()` with a parameterless signature and code excerpts (`docs/Authentication.md:9-28,36-48,253-272`) — the real registration is the package-delegating two-parameter method (`Security/Authentication/AuthStoreServiceCollectionExtensions.cs:41-105`), and the store/verifier code lives in `ZB.MOM.WW.Auth.ApiKeys`. The doc also states the library's `api_key_audit` table is written on every denial (`:122,131`), but the audit store override redirects all writes to `audit_event`, leaving `api_key_audit` unused (`Security/Audit/CanonicalForwardingApiKeyAuditStore.cs:21-25`).
|
||||||
|
Impact: violates the repo's "no stale prose" documentation rule; a maintainer auditing hashing or storage from the doc will look for code that is not there.
|
||||||
|
Recommendation: rewrite `docs/Authentication.md` as a consumer-side doc: token format, options binding, the audit-store override, and a pointer to the donor library for internals.
|
||||||
|
|
||||||
|
**CON-3 · Info — UI-stack rule verified compliant.**
|
||||||
|
Evidence: `wwwroot/lib/` contains only `bootstrap/css/bootstrap.min.css` and `bootstrap/js/bootstrap.bundle.min.js`; grep for MudBlazor/Radzen/Syncfusion/Telerik across `.csproj`/`.razor`/`.cs` returns nothing. The shared `ZB.MOM.WW.Theme` package provides CSS only (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:178-179`).
|
||||||
|
|
||||||
|
**CON-4 · Low — Validator coverage gaps against `docs/GatewayConfiguration.md`.**
|
||||||
|
Evidence: `GatewayOptionsValidator.ValidateDashboard` (`Configuration/GatewayOptionsValidator.cs:219-253`) validates `GroupToRole`, snapshot interval, and the two limits, but ignores `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie`, and `CookieName` (all documented at `docs/GatewayConfiguration.md:169-177`). A `CookieName` beginning with `__Host-` combined with `RequireHttpsCookie=false` produces a cookie browsers silently drop — no startup diagnosis.
|
||||||
|
Impact: silent misconfiguration classes the validator was built to prevent.
|
||||||
|
Recommendation: add the `__Host-`/`RequireHttpsCookie` consistency check and the SEC-4 production guard; validate `AutoLoginUser` is non-blank-or-null semantics explicitly.
|
||||||
|
|
||||||
|
**CON-5 · Low — Effective-config view omits the riskiest dashboard flags.**
|
||||||
|
Evidence: `Configuration/GatewayConfigurationProvider.cs:57-64` projects `EffectiveDashboardConfiguration` without `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie`, or `CookieName`; TLS and Alarms sections are absent from `EffectiveGatewayConfiguration` entirely.
|
||||||
|
Impact: the Settings page cannot show an operator that login is disabled — the one flag they most need to see.
|
||||||
|
Recommendation: add the missing fields (values are non-secret).
|
||||||
|
|
||||||
|
### Underdeveloped
|
||||||
|
|
||||||
|
**UND-1 · Medium — No API-key expiry.**
|
||||||
|
Evidence: the `api_keys` schema has `created_utc`/`last_used_utc`/`revoked_utc` only (confirmed from the stray DB's `sqlite_master` and `docs/Authentication.md:128-130`); no expiry field, no expiry check in the documented verification flow, no dashboard staleness surfacing beyond `LastUsedUtc` display.
|
||||||
|
Impact: keys live until an operator remembers to revoke; a leaked key is valid indefinitely.
|
||||||
|
Recommendation: add optional `expires_utc` in the shared library and enforce it in the verifier; surface age/staleness warnings on the API Keys page.
|
||||||
|
|
||||||
|
**UND-2 · Medium — No rate limiting or lockout on either authentication surface.**
|
||||||
|
Evidence: the gRPC interceptor verifies unconditionally per call (`GatewayGrpcAuthorizationInterceptor.cs:54-91`); `/auth/login` has no throttle (`DashboardEndpointRouteBuilderExtensions.cs:38-43,99-134`); the only brake is dev GLAuth's per-IP 3-fail lockout (`glauth.md:35`), which production AD will not replicate and which a shared NAT makes hazardous anyway (`glauth.md:323-325`).
|
||||||
|
Impact: unbounded online guessing of API-key secrets (mitigated by 256-bit secrets, but each guess costs the gateway a SQLite read) and unthrottled LDAP credential stuffing relayed to the directory.
|
||||||
|
Recommendation: add ASP.NET Core rate limiting on `/auth/login` and a cheap per-peer failure counter (or fixed-window limiter) in front of `VerifyAsync`.
|
||||||
|
|
||||||
|
**UND-3 · Medium — Dashboard session/worker Close and Kill bypass the canonical audit store.**
|
||||||
|
Evidence: `Dashboard/DashboardSessionAdminService.cs:64-69,129-134` record the action via `ILogger` only; API-key operations write `AuditEvent`s through `IAuditWriter` (`Dashboard/DashboardApiKeyManagementService.cs:242-264`).
|
||||||
|
Impact: destructive operational actions (killing a worker mid-production) leave no durable, queryable audit row; the log line is subject to log rotation and is invisible to the dashboard's recent-audit view.
|
||||||
|
Recommendation: emit `dashboard-close-session`/`dashboard-kill-worker` `AuditEvent`s through the existing `IAuditWriter`, mirroring the API-key pattern (actor resolution helpers already exist).
|
||||||
|
|
||||||
|
**UND-4 · Low — Per-session EventsHub ACL is an acknowledged TODO.**
|
||||||
|
Evidence: `Dashboard/Hubs/EventsHub.cs:29-44` (`TODO(per-session-acl)`) — any Viewer (and anonymous localhost per SEC-2) can subscribe to any session's raw event feed, which bypasses the per-gRPC-subscriber filtering (`docs/GatewayDashboardDesign.md:170`).
|
||||||
|
Impact: acceptable per the in-code rationale for v1, but it is the single seam where tag values reach the least-privileged principals; combine with `ShowTagValues=false` expectations and it can surprise operators.
|
||||||
|
Recommendation: keep the TODO but tie it to the tracked per-session-ACL work item; consider redacting event values in the dashboard mirror when `ShowTagValues` is false.
|
||||||
|
|
||||||
|
**UND-5 · Low — Audit trail has no retention or pruning.**
|
||||||
|
Evidence: `audit_event` is append-only with no cleanup path (`Security/Audit/SqliteCanonicalAuditStore.cs`); constraint denials append per denied bulk entry (`docs/Authorization.md:194-198`).
|
||||||
|
Impact: a misconfigured constrained client hammering denied reads grows the auth DB without bound.
|
||||||
|
Recommendation: add a retention sweep (age- or row-count-based) or document the operational expectation to archive.
|
||||||
|
|
||||||
|
**UND-6 · Low — Dashboard `GatewayStatus` is hardcoded.**
|
||||||
|
Evidence: `Dashboard/DashboardSnapshotService.cs:17,96` (`GatewayStatus: HealthyStatus` constant).
|
||||||
|
Impact: the home page's headline status never reflects the registered health checks (e.g. `AuthStoreHealthCheck` unhealthy while the banner says Healthy).
|
||||||
|
Recommendation: project `HealthCheckService` results into the snapshot.
|
||||||
|
|
||||||
|
**UND-7 · Info — Value-logging feature is unwired.**
|
||||||
|
Evidence: `GatewayLogRedactor.RedactCommandValue`/`IsCredentialBearingCommand` have no call sites in the server (grep-verified); the opt-in value-logging flag described in `docs/Diagnostics.md:124-148` has no configuration knob in `GatewayOptions`.
|
||||||
|
Impact: currently the safest possible state (no values are logged anywhere); the doc implies a capability that does not exist end-to-end.
|
||||||
|
Recommendation: either wire the flag or trim the doc to match (see SEC-7 before wiring).
|
||||||
|
|
||||||
|
## Top 5 recommendations
|
||||||
|
|
||||||
|
1. **Make filesystem defaults cross-platform and fail-fast** (SEC-1): replace the three Windows-literal defaults (`AuthenticationOptions.SqlitePath`, `TlsOptions.SelfSignedCertPath`, Galaxy `SnapshotCachePath`) with `SpecialFolder`-derived paths, reject non-rooted paths in `GatewayOptionsValidator`, and delete the stray `C:\ProgramData\MxGateway\gateway-auth.db` file from the source tree.
|
||||||
|
2. **Constrain the loopback bypass to the Viewer requirement** (SEC-2) so `AllowAnonymousLocalhost` can never satisfy `AdminOnly`, and decide deliberately whether anonymous loopback should reach the SignalR hubs and the API-key inventory in the snapshot payload.
|
||||||
|
3. **Fix the `QueryActiveAlarms` scope gap and its mislabeled tests** (STA-1): map `QueryActiveAlarmsRequest => events:read`, correct the two tests that construct `StreamAlarmsRequest`, and update the `docs/Authorization.md` scope table.
|
||||||
|
4. **Add production guards for the dev bypasses** (SEC-4, CON-4): fail startup when `DisableLogin=true` in a Production environment, and validate the `CookieName`/`RequireHttpsCookie` combination; reconcile the `__Host-` cookie documentation (SEC-3) in the same change.
|
||||||
|
5. **Cache API-key verification and batch `last_used_utc` writes** (PERF-1), and drop the `session_id` tag from `mxgateway.heartbeats.failed` (PERF-2), removing the per-RPC database write from the hot path and the unbounded metric cardinality in one observability-focused change.
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
# Language Clients — Architecture Review
|
||||||
|
|
||||||
|
## Scope & method
|
||||||
|
|
||||||
|
This review covers the five official client libraries and their CLIs: `clients/dotnet`, `clients/go`, `clients/java`, `clients/python`, `clients/rust`, plus `clients/proto` and the client-facing docs (`docs/ClientLibrariesDesign.md`, `docs/ClientPackaging.md`, `docs/ClientBehaviorFixtures.md`, `docs/CrossLanguageSmokeMatrix.md`). All clients consume the shared protos in `src/ZB.MOM.WW.MxGateway.Contracts/Protos`. The review is static (macOS tree, no builds run): every handwritten source file in each client library was read, CLIs were spot-checked, generated directories were verified to exist and were excluded from style review. All findings cite `path:line` in the repository root. The Java client was reviewed specifically for JDK 17 compatibility following the `feat/jdk17-client-retarget` work.
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
- All five clients implement the same core shape — connect, open/close session, typed Register/AddItem/Advise/Write helpers, raw `Invoke` escape hatch, event streaming with a resume cursor, alarm RPCs, and Galaxy browse with a lazy walker — and all five load the shared behavior fixtures in tests. Overall maturity is high and unusually uniform for a five-language surface.
|
||||||
|
- The Java JDK 21→17 retarget is correct in build config (`toolchain 17` + `options.release = 17`) and no JDK 21+ APIs remain in source; however three docs still say Java 21 (`clients/java/README.md:354`, `clients/java/JavaClientDesign.md:34`, `docs/ClientPackaging.md:193`), violating the repo's docs-in-same-commit rule.
|
||||||
|
- The published Rust crate cannot build outside this repository: `build.rs` resolves protos at `../../src/ZB.MOM.WW.MxGateway.Contracts/Protos` and packaging runs `cargo package/publish --no-verify`, so the defect is never caught. This is the most serious packaging finding.
|
||||||
|
- The Go `Session.Events()` path silently cancels the stream and closes the results channel when its 16-slot buffer fills — the consumer cannot distinguish a backpressure disconnect from normal stream end. This is the most serious runtime finding.
|
||||||
|
- The Rust client validates only `protocol_status` on `invoke`; it never inspects `hresult` or the `MXSTATUS_PROXY` array, so a reply with an OK protocol envelope but failing per-item statuses reads as success — every other client raises. Conversely, .NET/Go/Java treat *any* nonzero HRESULT as failure (positive success codes like `S_FALSE` misclassified); only Python uses the correct `hresult < 0` COM semantics.
|
||||||
|
- No client exposes typed helpers for `WriteSecured`/`WriteSecured2` (single-item), `AuthenticateUser`, `ArchestrAUserToId`, `AdviseSupervisory`, `AddBufferedItem`, `SetBufferedUpdateInterval`, `Suspend`, or `Activate`, even though the wire contract supports all of them (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto:150-160`). The secured-write parity path is reachable only via raw `Invoke`.
|
||||||
|
- Reconnect-with-replay is supported everywhere via `after_worker_sequence`, and four of five clients can re-attach a session wrapper to an existing session id; .NET cannot (its `MxGatewaySession` constructor is internal with no factory).
|
||||||
|
- Version constants drift in three clients: Go reports `0.1.0-dev`, Rust `version.rs` says `0.1.0-dev` while `Cargo.toml` says `0.1.2`, Python `version.py` says `0.1.0` while `pyproject.toml` says `0.1.2`. Only Java (0.2.0) is consistent.
|
||||||
|
- TLS posture is intentionally lenient but inconsistent across languages: .NET/Go/Java skip verification by default, Python does a blocking trust-on-first-use certificate pin, Rust is strict pin-only. The docs acknowledge this, but the Go CLI cannot even opt into strict validation.
|
||||||
|
- `docs/ClientPackaging.md` has drifted badly from the Python client's actual naming (package, module paths, CLI module) and the .NET solution filename.
|
||||||
|
|
||||||
|
## Cross-client parity matrix
|
||||||
|
|
||||||
|
| Capability | .NET | Go | Java | Python | Rust |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| Open/close session (typed + raw) | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Re-attach wrapper to existing session id | **No** (internal ctor) | Yes (`NewSessionForID`) | Yes (`forSessionId`) | Yes (ctor) | Yes (`client.session()`) |
|
||||||
|
| Register / AddItem / AddItem2 / Advise / UnAdvise / RemoveItem | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Unregister typed helper | **No** | Yes | Yes | Yes | **No** |
|
||||||
|
| Write / Write2 typed | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| WriteSecured / WriteSecured2 single-item typed | No | No | No | No | No |
|
||||||
|
| AuthenticateUser / ArchestrAUserToId typed | No | No | No | No | No |
|
||||||
|
| AdviseSupervisory / buffered / Suspend / Activate typed | No | No | No | No | No |
|
||||||
|
| Bulk add/advise/remove/unadvise/subscribe/unsubscribe | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| WriteBulk / Write2Bulk / WriteSecured(2)Bulk | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| ReadBulk | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Sparse array write helper (`WriteArrayElements`) | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Client-side 1000-item bulk cap | **No** | Yes | **No** | Yes | Yes |
|
||||||
|
| Event stream (idiomatic primitive) | `IAsyncEnumerable` | channel | iterator + observer | async iterator | `Stream` |
|
||||||
|
| Resume cursor (`after_worker_sequence`) | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Automatic reconnect loop / `ReplayGap` handling | No | No | No | No | No |
|
||||||
|
| Alarm RPCs (Ack / StreamAlarms / QueryActiveAlarms) | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Galaxy browse + lazy walker + WatchDeployEvents | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
| Typed auth errors (Unauthenticated vs PermissionDenied) | Yes | **No** | Yes | Yes | Yes |
|
||||||
|
| MXAccess HRESULT/status-array validation | Yes (`!=0`) | Yes (`!=0`) | Yes (`!=0`) | Yes (`<0`, correct) | **No** |
|
||||||
|
| Automatic transient retry | Yes (Polly) | No | No | No | No |
|
||||||
|
| TLS default posture | skip-verify | skip-verify | skip-verify | TOFU pin | strict pin-only |
|
||||||
|
| Version constant matches package version | n/a (no version in csproj) | **No** (`0.1.0-dev`) | Yes (0.2.0) | **No** (0.1.0 vs 0.1.2) | **No** (`0.1.0-dev` vs 0.1.2) |
|
||||||
|
| Fixture-driven unit tests | Yes | Yes | Yes | Yes | Yes |
|
||||||
|
|
||||||
|
## Findings — .NET (`clients/dotnet`)
|
||||||
|
|
||||||
|
**D1 — Medium.** A session wrapper cannot be reconstructed from an existing session id, so reconnect-with-replay after client restart loses all typed helpers. `MxGatewaySession`'s constructor is `internal` (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewaySession.cs:19`) and no factory equivalent to Go's `NewSessionForID` (`clients/go/mxgateway/session.go:70`) or Java's `forSessionId` exists. Impact: the gateway's `DetachGraceSeconds`/replay features are usable only through the raw stub. Recommendation: add a public `MxGatewayClient.AttachSession(string sessionId)` factory.
|
||||||
|
|
||||||
|
**D2 — Medium.** `MxGatewaySession.DisposeAsync` calls `CloseAsync()` with no cancellation token or timeout (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewaySession.cs:881-885`); when the gateway is unreachable, `await using` disposal blocks for the full retry-pipeline budget and then throws from disposal, masking the original exception. Recommendation: swallow or time-bound close failures in the disposal path.
|
||||||
|
|
||||||
|
**D3 — Medium.** The retry budget self-defeats on timeouts: `ExecuteSafeUnaryAsync` caps the whole retry pipeline with `CancelAfter(Options.DefaultCallTimeout)` while each attempt also gets a `DefaultCallTimeout` deadline (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewayClient.cs:305-316` with `:269-303`), so a first attempt that ends in `DeadlineExceeded` exhausts the outer budget and the configured retries (`MxGatewayClientRetryPolicy.cs:62-67` lists `DeadlineExceeded` as retryable) never run. Recommendation: give the outer budget headroom (e.g., attempts × delay) or drop `DeadlineExceeded` from the retryable set.
|
||||||
|
|
||||||
|
**D4 — Medium.** `EnsureMxAccessSuccess` treats any nonzero HRESULT as failure (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxCommandReplyExtensions.cs:32`), misclassifying positive COM success codes (e.g. `S_FALSE = 1`); Python's `hresult < 0` (`clients/python/src/zb_mom_ww_mxgateway/errors.py:133`) is the correct COM semantics. Same defect in Go (`clients/go/mxgateway/errors.go:121`) and Java (`.../client/MxGatewayErrors.java:50`). Impact: a parity-preserving gateway reply carrying a success HRESULT other than 0 throws in three clients and passes in one. Recommendation: align all clients on `hresult < 0`.
|
||||||
|
|
||||||
|
**D5 — Low.** No `<Version>` property in `clients/dotnet/ZB.MOM.WW.MxGateway.Client/ZB.MOM.WW.MxGateway.Client.csproj:14-28`; the published NuGet version (0.1.2) is supplied out-of-band at pack time, so the source tree does not record what ships. Recommendation: add `<Version>` to the csproj.
|
||||||
|
|
||||||
|
**D6 — Low.** Duplicate `InternalsVisibleTo` declared in both `Properties/AssemblyInfo.cs:3` and the csproj `AssemblyAttribute` block (`ZB.MOM.WW.MxGateway.Client.csproj:36-40`). Harmless but redundant; keep one.
|
||||||
|
|
||||||
|
**D7 — Low.** With `UseTls` and no CA file, the default (`RequireCertificateValidation = false`) installs an accept-all certificate callback (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewayClient.cs:361-364`). Documented and intentional for the internal-tool posture (`MxGatewayClientOptions.cs:31-36`), but it means `--tls` without a CA gives no server authentication. See cross-cutting C4.
|
||||||
|
|
||||||
|
Otherwise the .NET client is in good shape: retries are correctly restricted to idempotent commands (`MxGatewayClientRetryPolicy.cs:45-50`), the stream enumerators map `RpcException` per read and honor `EnumeratorCancellation` (`GrpcMxGatewayClientTransport.cs:78-109`), close is idempotent under a lock (`MxGatewaySession.cs:42-67`), style follows `CSharpStyleGuide.md` (sealed types, `Async` suffix, file-scoped namespaces), and the test project covers options, TLS handler, session, alarms, Galaxy, values, statuses, and CLI redaction.
|
||||||
|
|
||||||
|
## Findings — Go (`clients/go`)
|
||||||
|
|
||||||
|
**G1 — High.** `Session.Events()`/`EventsAfter()` silently terminate the stream on consumer backpressure: when the 16-slot results channel is full, `sendEventResult` cancels the stream and returns without queuing any terminal error, and the goroutine closes the channel (`clients/go/mxgateway/session.go:751-768`, spawn at `:700-742`). The consumer sees a closed channel — indistinguishable from graceful server end — and events are lost with no signal. The documented contract ("until … a terminal error is sent", `session.go:675-677`) does not mention this. Impact: silent data loss for any consumer that stalls for 16 events. Recommendation: enqueue a sentinel `EventResult{Err: ErrSlowConsumer}` before closing (guarantee one reserved slot), or drop the buffered-cancel variant in favor of the blocking `SubscribeEvents` path, which correctly rides gRPC flow control.
|
||||||
|
|
||||||
|
**G2 — Medium.** No typed auth error mapping: all RPC failures are wrapped in the generic `GatewayError` (`clients/go/mxgateway/errors.go:9-34`; e.g. `client.go:110-133`), so distinguishing `Unauthenticated` from `PermissionDenied` requires `status.Code(errors.Unwrap(err))`. `docs/ClientLibrariesDesign.md:153` requires the two be treated distinctly, and the other four clients expose typed auth errors. Recommendation: add `AuthenticationError`/`AuthorizationError` wrappers (or sentinel errors) in `errors.go`.
|
||||||
|
|
||||||
|
**G3 — Medium.** `Dial` uses deprecated `grpc.DialContext` with `grpc.WithBlock()` (`clients/go/mxgateway/client.go:60-68`); grpc-go ≥1.63 deprecates both in favor of `grpc.NewClient` with lazy connection. Impact: future grpc-go upgrades and `go vet`/staticcheck noise; blocking dial also hides per-RPC connection errors semantics that the rest of the ecosystem now expects. Recommendation: migrate to `grpc.NewClient` and surface readiness via a first `Ping`.
|
||||||
|
|
||||||
|
**G4 — Medium.** The CLI cannot opt into strict TLS validation: `dialForCommand` never sets `Options.RequireCertificateValidation` and no flag exists for it (`clients/go/cmd/mxgw-go/main.go:1151-1158`), so every non-CA-pinned TLS run of `mxgw-go` is skip-verify even though the library supports strictness (`clients/go/mxgateway/client.go:231-241`). Recommendation: add `-require-certificate-validation`.
|
||||||
|
|
||||||
|
**G5 — Low.** `ClientVersion = "0.1.0-dev"` (`clients/go/mxgateway/version.go:5-7`) is stale relative to the tagged module releases published via `scripts/tag-go-module.ps1`. Recommendation: bump on release as part of the tagging script.
|
||||||
|
|
||||||
|
**G6 — Low.** `newCorrelationID` swallows `crypto/rand` errors and returns an empty correlation id (`clients/go/mxgateway/session.go:786-792`); a fallback (timestamp counter) would preserve traceability.
|
||||||
|
|
||||||
|
**G7 — Low.** Nil-vs-empty asymmetry: `WriteBulk`/`ReadBulk` short-circuit an empty (non-nil) slice locally (`session.go:399-407`, `:524-532`) while `AddItemBulk`/`SubscribeBulk` send an empty command to the wire (`session.go:255-274`). Harmless but inconsistent within one file.
|
||||||
|
|
||||||
|
Otherwise the Go client is idiomatic: contexts propagate everywhere, `%w`-style wrapping via `Unwrap`, correct capped-timeout merging in `callContext` (`client.go:190-205`), CLI redacts the API key before JSON output (`cmd/mxgw-go/main.go:1162-1177`), and test coverage (session, TLS, galaxy, alarms, fixtures) is broad.
|
||||||
|
|
||||||
|
## Findings — Java (`clients/java`)
|
||||||
|
|
||||||
|
**J1 — Verified (no defect).** The JDK 17 retarget is complete and correct: `java.toolchain.languageVersion = 17` plus `options.release = 17` (`clients/java/build.gradle:20-31`) guarantees both language level and API surface are 17-bounded. A scan of all handwritten sources found no JDK 21+ APIs (no `SequencedCollection`/`getFirst`/`reversed()`, no virtual threads, no `ScopedValue`, no string templates, no `Math.clamp`); the language features present (records `MxStatuses.java:45`, pattern `instanceof`, `switch` expressions `MxGatewayErrors.java:17-25`) are all ≤17. Gradle/grpc/protobuf dependency versions (`build.gradle:5-11`) are 17-compatible.
|
||||||
|
|
||||||
|
**J2 — Medium.** Docs still claim Java 21 after the retarget: `clients/java/README.md:354` ("Java 21 Gradle toolchain"), `clients/java/JavaClientDesign.md:34-35`, and `docs/ClientPackaging.md:193`. CLAUDE.md's same-commit docs rule is violated on the retarget's own branch. Recommendation: sweep all three in this branch before merge.
|
||||||
|
|
||||||
|
**J3 — Medium.** The event-stream buffer is hardcoded to 16 with no configuration: `MxGatewayClient.streamEvents` constructs `new MxEventStream(16)` (`clients/java/zb-mom-ww-mxgateway-client/src/main/java/com/zb/mom/ww/mxgateway/client/MxGatewayClient.java:248`), and overflow cancels the RPC (`.../client/MxEventStream.java:124-132`). Unlike Go, the failure is at least surfaced as `MxGatewayException("…queue overflowed")`, but a consumer that stalls for 16 events is disconnected even though gRPC has native flow control that could simply be leaned on. Recommendation: make capacity an option and/or use `disableAutoRequestWithInitial`-style manual flow control instead of cancel-on-overflow.
|
||||||
|
|
||||||
|
**J4 — Low.** `MxEventStream` is a single-consumer iterator with unsynchronized `next` state (`MxEventStream.java:31`, `:65-92`); this constraint is not documented. Add a Javadoc note.
|
||||||
|
|
||||||
|
**J5 — Low.** `close()` initiates `shutdown()` without awaiting termination (`MxGatewayClient.java:346-351`); acceptable given `closeAndAwaitTermination()` exists (`:360-367`), but try-with-resources users can leak a channel briefly at JVM exit. Consider making `close()` await a short bound.
|
||||||
|
|
||||||
|
Otherwise the Java client is the most complete: blocking, future, and async stub variants with correct deadline layering (unary vs stream, `MxGatewayClient.java:418-430`), a full typed exception hierarchy with credential redaction (`MxGatewayErrors.java:13-29`, `MxGatewaySecrets.java`), the only in-process end-to-end CLI test harness (`zb-mom-ww-mxgateway-cli/src/test/java/.../InProcessGatewayHarness.java`), and consistent 0.2.0 versioning between Gradle (`build.gradle:16`) and `MxGatewayClientVersion.java:12`.
|
||||||
|
|
||||||
|
## Findings — Python (`clients/python`)
|
||||||
|
|
||||||
|
**P1 — Medium.** The default TLS path performs a blocking, unverified certificate fetch and pins it (trust-on-first-use) (`clients/python/src/zb_mom_ww_mxgateway/options.py:108-160`), and silently defaults the SNI override to `localhost` (`options.py:150-154`). This is documented and bounded (probe timeout, `asyncio.to_thread` in `client.py:61-63`), but it is the only client that opens a second out-of-band TCP+TLS connection per channel, and TOFU is vulnerable to first-contact interception. Recommendation: document the MITM window in the README threat model and prefer `ca_file` in examples.
|
||||||
|
|
||||||
|
**P2 — Low.** Version mismatch: `pyproject.toml` says `0.1.2` (`clients/python/pyproject.toml:9`) while `version.py` says `__version__ = "0.1.0"` (`clients/python/src/zb_mom_ww_mxgateway/version.py:3`), so `mxgw-py version` reports the wrong version. Recommendation: derive one from the other (e.g. `importlib.metadata.version`).
|
||||||
|
|
||||||
|
**P3 — Low.** `Session.close()` is not concurrency-safe (no lock around `_closed`, `session.py:38-55`) and repeated closes return a locally synthesized `CloseSessionReply` rather than the cached server reply — divergent from .NET/Go which cache the real reply. Minor; align with a cached-reply pattern.
|
||||||
|
|
||||||
|
**P4 — Low.** Circular-import workaround: `from .client import GatewayClient` at the bottom of `session.py:590` (`# noqa: E402`). Works, but a `TYPE_CHECKING` import plus a string annotation would remove the runtime cycle.
|
||||||
|
|
||||||
|
Otherwise the Python client is strong: correct `hresult < 0` semantics (`errors.py:133`), fully typed (`from __future__ import annotations`, precise unions), GIL-friendly pure-asyncio design with stream iterators that cancel the call on generator exit (`client.py:230-263`), packaging correctly capped at `setuptools <77` for the ≤2.3 metadata constraint (`pyproject.toml:2-5`), and the largest per-client test suite (13 test modules including regression files for prior review findings).
|
||||||
|
|
||||||
|
## Findings — Rust (`clients/rust`)
|
||||||
|
|
||||||
|
**R1 — High.** The published crate cannot build outside this repository. `build.rs` resolves the protos two directories above the crate (`clients/rust/build.rs:8-16`: "clients/rust must live two levels below the repository root") and `src/generated.rs` is only `tonic::include_proto!` of build output (`clients/rust/src/generated.rs:16-40`); `Cargo.toml` does not vendor the `.proto` files into the package. The packaging script masks this by using `cargo package --no-verify` and `cargo publish --no-verify` (`scripts/pack-clients.ps1:190-211`). Any consumer of the Gitea-published `zb-mom-ww-mxgateway-client 0.1.2` fails in `build.rs` at first `cargo build`. Recommendation: copy the three protos into the crate (e.g. `clients/rust/protos/`, listed in `include`), fall back to them when the repo path is absent, and remove `--no-verify` so `cargo package` verifies buildability.
|
||||||
|
|
||||||
|
**R2 — High.** `invoke` validates only `protocol_status` and never inspects `hresult` or the `MXSTATUS_PROXY` array: `ensure_command_success` checks `code == Ok` only (`clients/rust/src/error.rs:214-226`, used by `client.rs:177-179`). Every other client performs a second MXAccess-level check (e.g. `MxCommandReplyExtensions.cs:27-41`, `errors.go:117-130`, `MxGatewayErrors.java:46-58`, `errors.py:122-148`), because a reply can carry an OK protocol envelope with failing per-item statuses. Impact: a Rust `session.write(...)` can report success while MXAccess rejected the write; there is also no distinct MxAccess error variant to catch. Recommendation: add an `ensure_mxaccess_success` pass (hresult `< 0` + statuses) and an `Error::MxAccess` variant.
|
||||||
|
|
||||||
|
**R3 — Low.** `CLIENT_VERSION = "0.1.0-dev"` with a doc comment claiming it "Mirrors Cargo.toml" while `Cargo.toml` says `0.1.2` (`clients/rust/src/version.rs:6-7` vs `clients/rust/Cargo.toml:3`). Recommendation: `pub const CLIENT_VERSION: &str = env!("CARGO_PKG_VERSION");`.
|
||||||
|
|
||||||
|
**R4 — Low.** No `unregister` typed helper (session surface at `clients/rust/src/session.rs:119-231` covers register/add/advise/remove but not Unregister); Go/Java/Python have it. Add for symmetry.
|
||||||
|
|
||||||
|
**R5 — Low.** The CLI is a single 2,699-line `main.rs` (`clients/rust/crates/mxgw-cli/src/main.rs`) — the largest single file in the client tree; the Windows stack-size workaround it forced (`clients/rust/.cargo/config.toml:1-19`) is itself evidence the command enum has outgrown one module. Split subcommands into modules.
|
||||||
|
|
||||||
|
Otherwise the Rust client is clean: clippy-conscious generated-module allowances (`generated.rs:18`, `#![warn(missing_docs)]` in `lib.rs:12`), a well-structured `thiserror` enum with boxed `tonic::Status`, credential scrubbing in error messages with a unit test (`error.rs:256-289`), cheap `Clone` client over a shared channel, correct unary-vs-stream timeout split (`client.rs:280-293`), and bulk caps enforced client-side (`session.rs:29`).
|
||||||
|
|
||||||
|
## Findings — cross-cutting
|
||||||
|
|
||||||
|
**X1 — High.** The MXAccess command parity gap is uniform: no client exposes typed helpers for single-item `WriteSecured`/`WriteSecured2`, `AuthenticateUser`, `ArchestrAUserToId`, `AdviseSupervisory`, `AddBufferedItem`, `SetBufferedUpdateInterval`, `Suspend`, or `Activate`, although the contract defines all of them (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto:150-160`) and `gateway.md` documents `AdviseSupervisory` as a precondition for user-attributed plain writes. Only the .NET, Go, and Python CLIs offer `advise-supervisory` via hand-built raw commands (`clients/go/cmd/mxgw-go/main.go:364-391`, `clients/dotnet/ZB.MOM.WW.MxGateway.Client.Cli/MxGatewayClientCli.cs:456-470`, `clients/python/src/zb_mom_ww_mxgateway_cli/commands.py:287-291`); Java and Rust CLIs lack even that. Impact: the secured-write path (AuthenticateUser → WriteSecured) and the buffered-event family (`OnBufferedDataChange`) cannot be exercised through any typed client API. Recommendation: add the missing typed session helpers to all five clients, starting with `adviseSupervisory`, `writeSecured`, and `authenticateUser`.
|
||||||
|
|
||||||
|
**X2 — Medium.** No client handles the `ReplayGap` sentinel or offers a reconnect loop. All five expose the `after_worker_sequence` resume cursor (e.g. `MxGatewaySession.cs:865-876`, `session.go:682-688`, `MxGatewaySession.java:718-722`, `session.py:571-582`, `session.rs:648`), which matches the "no client-side reconnect" v1 non-goal in `docs/ClientLibrariesDesign.md:64-70`, but since the gateway shipped `DetachGraceSeconds` + replay, none of the clients document or type the `ReplayGap` event a resuming consumer must expect (`grep ReplayGap clients/` → no handwritten hits). Recommendation: at minimum document gap detection per client README; longer term add a resume helper.
|
||||||
|
|
||||||
|
**X3 — Medium.** `docs/ClientPackaging.md` has drifted from reality: Python package named `mxaccess-gateway-client` and generated dir `src/mxgateway/generated` (`docs/ClientPackaging.md:159-160`) vs actual `zb-mom-ww-mxaccess-gateway-client` (`clients/python/pyproject.toml:8`) and `src/zb_mom_ww_mxgateway/generated`; CLI module `python -m mxgateway_cli` (`ClientPackaging.md:187`) vs actual `zb_mom_ww_mxgateway_cli`; .NET solution `ZB.MOM.WW.MxGateway.Client.sln` (`ClientPackaging.md:51-52`) vs actual `.slnx`; Java 21 (`:193`, see J2). `docs/ClientLibrariesDesign.md:410` repeats the stale Python generated path. Recommendation: one doc sweep commit.
|
||||||
|
|
||||||
|
**X4 — Medium.** TLS default posture is inconsistent across the five clients: accept-any-cert in .NET (`MxGatewayClient.cs:361-364`), Go (`client.go:236-240`, `InsecureSkipVerify`), and Java (`MxGatewayClient.java:387-395`, `InsecureTrustManagerFactory`); TOFU pinning in Python (`options.py:133-154`); strict pin-only in Rust. `docs/CrossLanguageSmokeMatrix.md:58-66` documents the divergence, but the practical result is that the same `--tls`-without-CA invocation authenticates the server in one language, half-authenticates in another, and not at all in three. Recommendation: converge on the Python TOFU model or at least emit a one-line warning when verification is disabled.
|
||||||
|
|
||||||
|
**X5 — Low.** Client-side bulk caps differ: Go/Python/Rust enforce 1,000 items locally (`session.go:19`, `session.py:11`, `session.rs:29`) while .NET and Java send unbounded lists and rely on the gateway. Harmless but produces different error types for the same oversized call. Align (either all enforce or none).
|
||||||
|
|
||||||
|
**X6 — Low.** Event-stream backpressure semantics differ by language: Go buffered-16/silent-cancel (G1), Java buffered-16/error-cancel (J3), .NET/Python/Rust unbuffered (native gRPC flow control, pushing backpressure to the gateway where the documented fail-fast policy applies). The per-language behavior under a slow consumer is a parity-relevant observable and belongs in `docs/ClientBehaviorFixtures.md`.
|
||||||
|
|
||||||
|
**X7 — Low.** Generated-code hygiene is good everywhere: `clients/go/internal/generated`, `clients/java/src/main/generated` (6 tracked files), `clients/python/src/zb_mom_ww_mxgateway/generated` exist and match the manifest; Rust generates into `OUT_DIR` with a `.gitkeep` placeholder; `clients/dotnet/generated` is intentionally absent because the .NET client references the Contracts project directly (`ZB.MOM.WW.MxGateway.Client.csproj:4`), which `docs/ClientPackaging.md:38-40` correctly describes. Python `build/` and `.pytest_cache/` exist on disk but are not git-tracked.
|
||||||
|
|
||||||
|
## Top 5 recommendations
|
||||||
|
|
||||||
|
1. **Fix Rust crate publishability (R1):** vendor the three `.proto` files into the crate, add a repo-path fallback in `build.rs`, and drop `--no-verify` from `scripts/pack-clients.ps1` so `cargo package` proves the crate builds standalone.
|
||||||
|
2. **Fix Go silent stream termination (G1):** reserve a slot for a terminal `EventResult{Err: ...}` before closing the `Events()` channel so slow-consumer disconnects are observable, or deprecate the buffered-cancel path in favor of `SubscribeEvents`.
|
||||||
|
3. **Add MXAccess-level reply validation to Rust and align HRESULT semantics everywhere (R2, D4):** Rust must check `hresult` and the status array; .NET/Go/Java should switch from `hresult != 0` to `hresult < 0` to match COM semantics and Python.
|
||||||
|
4. **Close the typed-command parity gap (X1):** add `adviseSupervisory`, single-item `writeSecured`/`writeSecured2`, and `authenticateUser` helpers to all five session APIs (the wire already supports them), then the buffered-item family.
|
||||||
|
5. **One doc-and-version sweep on the retarget branch (J2, X3, G5, P2, R3):** update the three "Java 21" references, the Python naming in `ClientPackaging.md`/`ClientLibrariesDesign.md`, the `.sln` → `.slnx` reference, and reconcile the Go/Rust/Python version constants with their published package versions.
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
# Testing, Documentation & Underdeveloped Areas — Architecture Review
|
||||||
|
|
||||||
|
## Scope & method
|
||||||
|
|
||||||
|
This review covers test architecture (`src/ZB.MOM.WW.MxGateway.Tests`, `src/ZB.MOM.WW.MxGateway.Worker.Tests`, `src/ZB.MOM.WW.MxGateway.IntegrationTests`, `scripts/`), documentation currency (`gateway.md`, `docs/Sessions.md`, `docs/GatewayConfiguration.md`, `docs/DesignDecisions.md`, `docs/GatewayTesting.md`, CLAUDE.md, repo-root working artifacts), and repo-wide underdeveloped areas (TODO sweep, half-shipped features, operational gaps). Method is static reading of code and docs on the macOS tree; no test suites were run. Generated code, `bin/`, and `obj/` are excluded. All paths are relative to `/Users/dohertj2/Desktop/MxAccessGateway`.
|
||||||
|
|
||||||
|
## Executive summary
|
||||||
|
|
||||||
|
- Test coverage of the gateway core is strong and unusually well-layered: 46 gateway test classes, 34 worker test classes, and an 8-scenario opt-in live MXAccess suite map cleanly onto session lifecycle, pipe framing, event fan-out/replay, auth, and dashboard. The `FakeWorkerHarness` reuses the production `WorkerFrameReader`/`WorkerFrameWriter`, so fake tests exercise real frame validation rather than a mock transport.
|
||||||
|
- There is **no CI configuration anywhere in the repository** — no `.github/`, `.gitea/`, or pipeline files. Every documented verification step (per-task filtered `dotnet test`, five client toolchains, opt-in live matrices) is manual operator discipline; nothing prevents a cross-component regression from landing on `main` untested.
|
||||||
|
- The session-resilience epic is half-shipped: 12 of 28 tasks are merged (`oldtasks.md`), leaving reconnect owner re-validation (a security gap), client-side `ReplayGap` handling in all five clients, per-session dashboard ACL, and orphan-worker reattach pending. The server emits a reconnect protocol no client yet understands.
|
||||||
|
- Fake-vs-real fidelity has one structural blind spot by design: `FakeWorkerHarness` cannot simulate STA behavior, COM faults, process crash, or MXAccess semantics — those live only in the opt-in `WorkerLiveMxAccessSmokeTests` and dev-rig probes, which never run automatically.
|
||||||
|
- Timing-based tests are mostly disciplined (`ManualTimeProvider`, cancellation-token-bounded polls), but a handful of real-clock sleeps followed by negative assertions remain (`WorkerClientTests.cs:433`, `SessionManagerTests.cs:402`) and are latent flakes under load. The known macOS `SessionWorkerClientFactoryFakeWorkerTests` timeout failures are pre-existing and environmental.
|
||||||
|
- Documentation currency is better than typical: `docs/GatewayConfiguration.md` defaults match `Configuration/*.cs` exactly, and `docs/Sessions.md`/`docs/DesignDecisions.md` reflect the shipped reconnect/fan-out behavior. The stale spots are concentrated in `gateway.md` (design-era `WorkerEnvelope` sketch, single-subscriber policy line, never-built `Session` bidi RPC) and one materially wrong sentence in CLAUDE.md about default retention.
|
||||||
|
- Repo-root hygiene is mixed: `stillpending.md` and `oldtasks.md` are tracked, annotated snapshots that still serve as the de-facto backlog (there is no issue tracker in evidence); the five untracked `*-docs-{issues,fixed,final}.md` artifacts (15k+ lines) are dead local working files; `code-reviews/` + `REVIEW-PROCESS.md` are a completed, self-consistent review system with zero open findings.
|
||||||
|
- Operationally the gateway is under-instrumented for a service: one health check (auth-store) with no Galaxy SQL, LDAP, alarm-monitor, or worker-launchability checks; hand-rolled NSSM deployment with config living in service env vars outside the repo; and no version stamping on the server/worker assemblies while client packages drift (Java 0.2.0 vs 0.1.2 everywhere else).
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### Coverage map
|
||||||
|
|
||||||
|
| Component | Unit | Fake-worker integration | Live integration | Gap notes |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Session lifecycle (open/close/lease/detach-grace) | `SessionManagerTests`, `GatewaySessionTests` | `GatewayEndToEndFakeWorkerSmokeTests`, `SessionWorkerClientFactoryFakeWorkerTests` | `WorkerLiveMxAccessSmokeTests` | `SessionLeaseMonitorHostedService` timer loop itself untested (sweep logic covered via `CloseExpiredLeasesAsync` tests at `SessionManagerTests.cs:859-930`) |
|
||||||
|
| Worker launch / validation / orphan cleanup | `WorkerProcessLauncherTests`, `WorkerExecutableValidatorTests`, `OrphanWorkerTerminatorTests` | scripted fake launcher | live x86 launch | `OrphanWorkerCleanupHostedService` wrapper untested; reattach not implemented (epic Phase 5) |
|
||||||
|
| Pipe framing / IPC envelope | `WorkerFrameProtocolTests` (both projects), `WorkerClientTests`, `WorkerPipeClientTests`, `WorkerPipeSessionTests` | `FakeWorkerHarnessTests` incl. malformed/oversized frames | live | good — malformed payloads and oversized headers explicitly scripted |
|
||||||
|
| Worker crash / heartbeat / fault | `WorkerClientTests` with `ManualTimeProvider` | fault frames via harness | abnormal-exit kill test | real process crash covered only opt-in |
|
||||||
|
| Event streaming / fan-out / replay | `EventStreamServiceTests` (15 tests), `SessionEventDistributorTests` (22) | `GatewayEndToEndMultiSubscriberTests` | stream phase of live smoke | no end-to-end reconnect/replay fake-worker test (epic Task 15 pending); no client handles `ReplayGap` (Task 14) |
|
||||||
|
| Backpressure | distributor/stream tests | overflow via harness | e2e script drain-loop workaround | policy interplay (`FailFast` vs multi-subscriber degrade) unit-covered only |
|
||||||
|
| Auth (API keys, scopes, audit) | `ApiKeyVerifierTests`, `SqliteAuthStoreTests`, `GatewayGrpcAuthorizationInterceptorTests`, `ConstraintEnforcerTests`, audit tests | — | e2e script auth-rejection phase (opt-in) | good |
|
||||||
|
| Dashboard | ~17 test classes under `Gateway/Dashboard/` | — | `DashboardLdapLiveTests` (opt-in) | `DashboardLiveDataService`, `EventsHub` hub methods, `AlarmsHubPublisher`, `DashboardHubConnectionFactory` have no direct tests |
|
||||||
|
| Alarms (consumer, failover, monitor) | `FailoverAlarmConsumerTests`, `SubtagAlarmStateMachineTests`, `GatewayAlarmMonitorProviderModeTests`, `AlarmFailoverEndToEndTests` | fake alarm service | dev-rig probes (`[Fact(Skip=...)]`) | live failover undrivable on rig; `provider_switches` metric never live-exercised; subtag Clear unvalidated |
|
||||||
|
| Worker STA / COM / conversion | `StaMessagePumpTests`, `StaCommandDispatcherTests`, `MxAccessCommandExecutorTests`, `VariantConverterTests`, etc. | — | `MxAccessLiveComCreationTests` (opt-in) | multi-sample `OnBufferedDataChange` conversion never observed live (`stillpending.md` §3.2) |
|
||||||
|
| Galaxy browse | `GalaxyFilterInputSafetyTests`, `GalaxyRepositoryHostWiringTests` | — | `GalaxyRepositoryLiveTests` (opt-in) | core browse logic now lives in the external `ZB.MOM.WW.GalaxyRepository` package |
|
||||||
|
| Clients (×5) | dotnet 13 / go 8 / rust 3 / python 13 / java 7 test files | — | `scripts/run-client-e2e-tests.ps1` only | no per-client wire-level integration tests; all client↔gateway behavior unverified in default runs |
|
||||||
|
|
||||||
|
### Test Coverage
|
||||||
|
|
||||||
|
- **High — Reconnect/replay has no end-to-end test and no client-side consumer.** The server emits the `ReplayGap` sentinel and replays the ring (`docs/Sessions.md` "Reconnect and replay"), unit-covered in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/EventStreamServiceTests.cs` and `MxAccessGatewayServiceTests.cs`, but the epic's fake-worker reconnect integration test (Task 15) and client `ReplayGap` handling in all five clients (Task 14) are pending per `oldtasks.md:40-44`. Impact: the shipped, default-on reconnect protocol (DetachGraceSeconds=30, ReplayBufferCapacity=1024) is unproven end-to-end and unusable by every official client. Recommendation: land Tasks 14–15 before advertising reconnect; until then document it as server-only.
|
||||||
|
- **High — Reconnect owner re-validation is not implemented.** Epic Task 13 ("Owner re-validation on reconnect", `oldtasks.md:40`) is pending, so nothing ties a resuming `StreamEvents` call to the API key that opened the session beyond the `event` scope and knowledge of the session id. Impact: with fan-out or detach-grace enabled, any event-scoped key that learns a session id can attach to another key's session and receive its replayed data. Recommendation: prioritize Task 13; it is a security control, not a resilience feature.
|
||||||
|
- **Medium — Real-worker control/COM behavior is only verified opt-in.** All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (`stillpending.md` §1.1), but the default suite exercises `Ping`/`GetWorkerInfo`/`DrainEvents`/`ShutdownWorker` only through `FakeWorkerHarness.RespondToControlCommandAsync` (`src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs:417-513`), which returns canned replies. Impact: a worker-side regression in these paths is invisible until someone sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`. Recommendation: schedule the live suite on the windev box on a cadence (see CI finding).
|
||||||
|
- **Medium — Dashboard live-data path untested.** `DashboardLiveDataService` (owns the shared lazily-opened gateway session backing `/browse` live values, per `gateway.md:128-131`) has no test class; `EventsHub`/`AlarmsHubPublisher` hub methods are likewise untested. Impact: the one dashboard component that holds a real worker session and can fault it has no regression net. Recommendation: add a fake-worker-backed test for session reuse, fault recovery, and disposal.
|
||||||
|
- **Low — Hosted-service wrappers untested.** `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` delegate to well-tested cores but their timer/startup wiring has no tests. Impact: low; failure mode is obvious at startup.
|
||||||
|
|
||||||
|
### Test Quality
|
||||||
|
|
||||||
|
- **Medium — Real-clock sleeps with negative assertions are latent flakes.** `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs:433` sleeps 150 ms of wall time ("give the heartbeat monitor a few real check-intervals") then asserts state did *not* change; `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/SessionManagerTests.cs:402` races a 50 ms delayed state flip against a 500 ms bounded wait. 47 `Thread.Sleep`/`Task.Delay` occurrences exist across 22 test files; most are benign (cancellation-bounded polls, `Timeout.InfiniteTimeSpan` hang simulators, `ManualTimeProvider` used for the actual clock), but the fixed-real-time ones can pass spuriously or fail under load. Impact: intermittent failures erode trust in an already manual test regime. Recommendation: convert the negative-assertion sleeps to `ManualTimeProvider`-driven check-interval pumping or polling-until-stable.
|
||||||
|
- **Medium — Full-suite orphaned testhost processes remain an unfixed hygiene defect.** CLAUDE.md ("Source Update Workflow") documents that the full gateway suite "leaves orphaned testhost processes" and mandates filtered runs as a workaround. Impact: the workaround is procedural, so any future CI or an unaware contributor inherits zombie processes; it also suggests undisposed pipe servers or hosted services in some fixture. Recommendation: identify the leaking fixture (likely a `NamedPipeServerStream` or hosted-service test without disposal) rather than institutionalizing the workaround. The macOS `SessionWorkerClientFactoryFakeWorkerTests` timeout-message failures are pre-existing platform noise (wrong timeout message on macOS) and are correctly excluded from this finding.
|
||||||
|
- **Low — `FakeWorkerHarness` fidelity is good within its charter.** It shares `WorkerFrameReader`/`WorkerFrameWriter`/`WorkerEnvelope` with production (`FakeWorkerHarness.cs:36-37`), scripts malformed payloads and oversized headers (`:519-553`), heartbeats, faults, and shutdown acks, and mirrors real control-command reply shapes. What it structurally cannot represent — STA pumping, COM HRESULT semantics, process exit codes, event timing under load — is honestly delegated to the live suite by `docs/GatewayTesting.md`. No action needed beyond keeping its canned replies in lockstep with `WorkerPipeSession` (the §1.1 history shows this drifted once: green fakes masked an unimplemented real worker for months).
|
||||||
|
- **Low — The e2e script embeds a workaround for a real product sharp edge.** `scripts/run-client-e2e-tests.ps1` (1,715 lines) must interleave `StreamEvents` drains every 15 advised tags because advising without a consumer overflows the worker event channel and faults the session under `FailFast` (`docs/GatewayTesting.md` "Client E2E Scripts" phase 3). Impact: the test harness papering over this documents that real clients doing bulk-advise-then-stream will fault sessions. Recommendation: treat this as product feedback (e.g., subscribe-time channel policy), not just a script detail.
|
||||||
|
|
||||||
|
### CI & Operations
|
||||||
|
|
||||||
|
- **High — No CI exists.** There is no `.github/`, `.gitea/`, `azure-pipelines.yml`, `Jenkinsfile`, or any pipeline file in the repository. `docs/ImplementationPlanIndex.md` names a Gitea repo and a `packaging-and-ops` milestone, but nothing automated runs the .NET, Go, Rust, Python, or Java suites on push. Impact: the multi-language, multi-target (net10/net48-x86) build matrix is exactly the kind that silently breaks; the repo's own history (net48 CS0246 on unregenerated protos, Java generated-file churn) shows cross-component breakage is routine. Recommendation: a minimal Gitea Actions (or runner-on-windev) pipeline that builds `NonWindows.slnx` + runs the gateway suite on every push, with a Windows job for the x86 worker, would catch the majority class of regressions; add a scheduled opt-in live-MXAccess job on the dev rig.
|
||||||
|
- **Medium — Health checks cover only the auth store.** `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:71-75` registers a single `AuthStoreHealthCheck`; there is no readiness check for Galaxy SQL reachability, LDAP bind, alarm-monitor session health, or worker-executable presence/launchability, even though each is a documented startup dependency. Impact: `/health` reports ready while the gateway cannot open a session or browse Galaxy. Recommendation: add tagged checks for worker exe validation (cheap, reuses `WorkerExecutableValidator`) and Galaxy/LDAP with caching.
|
||||||
|
- **Medium — Deployment and upgrade are undocumented in-repo and hand-rolled.** Deployments are NSSM-wrapped services with configuration held in NSSM environment variables (acknowledged in `docs/GatewayConfiguration.md` "Host Endpoints" and the project memory notes); `scripts/` contains packaging and e2e scripts but no deploy/upgrade script, and there is no runbook for the `MxAccessGw`→`OtOpcUa` service-dependency dance. Impact: deploys are tribal knowledge; a second operator cannot reproduce them from the repo. Recommendation: commit a `docs/Deployment.md` plus a publish/deploy script that captures the NSSM env-var config as code.
|
||||||
|
- **Medium — No version discipline on server artifacts; client versions drift.** The Server and Worker csproj files carry no `Version` property (assemblies stamp 1.0.0), Contracts is `0.1.2` (`src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj:10`), Python and Rust clients are `0.1.2` (`clients/python/pyproject.toml:9`, `clients/rust/Cargo.toml:3`), while Java is `0.2.0` (`clients/java/build.gradle:16`) — and the Java bump required a separately-maintained `CLIENT_VERSION` constant (commit `b6a0d90` fixed a mismatch). Impact: support cannot correlate a deployed gateway or a client wheel to a commit; duplicated version constants have already drifted once. Recommendation: single-source versions (Directory.Build.props for the .NET side; generate client version constants at build) and stamp the server assembly informational version with the git SHA.
|
||||||
|
- **Low — Log rotation is configured but minimal.** `src/ZB.MOM.WW.MxGateway.Server/appsettings.json:8-11` rolls the file sink daily (`logs/mxgateway-.log`, a path relative to the service working directory) with no explicit `retainedFileCountLimit` or size cap (Serilog defaults to 31 files, unbounded size per day). Impact: acceptable, but a high-rate event day can produce an unbounded single file. Recommendation: set `fileSizeLimitBytes` + `rollOnFileSizeLimit` and an absolute log path for the service deployment.
|
||||||
|
|
||||||
|
### Documentation Currency
|
||||||
|
|
||||||
|
- **Medium — CLAUDE.md misstates the default retention behavior.** CLAUDE.md ("Repository-Specific Conventions") says "Default config preserves the original single-subscriber, no-retention behavior," but the code defaults enable retention: `DetachGraceSeconds = 30` (`src/ZB.MOM.WW.MxGateway.Server/Configuration/SessionOptions.cs:47`) and `ReplayBufferCapacity = 1024` / `ReplayRetentionSeconds = 300` (`Configuration/EventOptions.cs:11-31`). Only `AllowMultipleEventSubscribers` defaults off. Impact: an agent or operator following CLAUDE.md will assume detached sessions die immediately and no events are buffered. Recommendation: reword to "single-subscriber by default; detach-grace and replay retention are on by default."
|
||||||
|
- **Medium — gateway.md carries design-era sketches that no longer match the wire contract.** The `WorkerEnvelope` snippet (`gateway.md:291-309`) shows `uint64 correlation_id = 4` and body tags `command = 20 … fault = 26` with no `WorkerShutdownAck`; the actual proto uses `string correlation_id = 4` and `gateway_hello = 10 … worker_fault = 20` including `worker_shutdown_ack = 17` (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:19-38`). Similarly `gateway.md:738` still states the gateway "allow[s] one active client event subscriber per session, reject[s] a second subscriber" as unconditional policy, contradicted by the config-gated fan-out described later in the same file (`:1177-1180`) and by `docs/Sessions.md`. The public-API sketch (`gateway.md:328-334`) includes `rpc Session(stream ClientMessage)` which was never implemented (the real service has no bidi RPC — `mxaccess_gateway.proto:18-37`). Impact: gateway.md is the mandated pre-change reading; internally inconsistent sections cost every reader a reconciliation pass. Recommendation: replace the envelope sketch with a pointer to the proto, mark the single-subscriber paragraph as the default mode, and label `Session` as unbuilt future work.
|
||||||
|
- **Low — GatewayConfiguration.md, Sessions.md, DesignDecisions.md, and GatewayTesting.md are current.** Spot-checks confirm every table default in `docs/GatewayConfiguration.md` matches `Configuration/{SessionOptions,EventOptions,DashboardOptions,AuthenticationOptions,WorkerOptions}.cs`; `docs/DesignDecisions.md:63-118` correctly records the superseded reconnect/fan-out constraints; `docs/GatewayTesting.md` "all eight tests" matches the 8 `[LiveMxAccessFact]` attributes in `WorkerLiveMxAccessSmokeTests.cs`. The only nit: the Configuration Shape JSON block (`docs/GatewayConfiguration.md:12-83`) omits `DisableLogin`/`AutoLoginUser`/`CookieName`, `Tls`, and the `Alarms:Fallback` block that the tables below it document.
|
||||||
|
- **Medium — Repo-root working artifacts need triage.** Classification by evidence:
|
||||||
|
- `MxAccessGateway-docs-{issues,fixed,final}.md` and `MxGatewayClient-docs-{issues,fixed}.md` — **untracked** (gitignored via `*-docs-issues.md`, `.gitignore:152`; the others simply never committed), 15,416 lines in the largest. These are dead local outputs of a finished docs-review pass; delete or move out of the working tree so greps and agents stop tripping over them.
|
||||||
|
- `stillpending.md` (tracked, last touched 2026-06-25) — a 2026-06-15 audit snapshot with resolution annotations that currently serves as the project's only backlog register. Living but structurally a snapshot; either promote it to a maintained register (drop "Generated: … Commit:" framing) or migrate open items (§1.3, §1.4, §3.x, epic remainder) to the Gitea tracker that `docs/ImplementationPlanIndex.md` describes.
|
||||||
|
- `oldtasks.md` (tracked, 2026-06-16) — explicitly a "human-readable mirror" of `docs/plans/2026-06-15-session-resilience.md.tasks.json`; keep only until the epic resumes, then delete.
|
||||||
|
- `REVIEW-PROCESS.md` + `code-reviews/` — a coherent, generated-index review system, all 351 findings resolved, `Server`/`Tests` re-reviewed 2026-06-25. Living docs; keep.
|
||||||
|
- `A2-galaxyrepository-adoption-handoff.md` — completed-migration handoff referenced from CLAUDE.md; archive candidate under `docs/plans/` once its cross-repo follow-ups land.
|
||||||
|
|
||||||
|
### Underdeveloped Areas
|
||||||
|
|
||||||
|
- **High — Session-resilience epic is 16/28 tasks unfinished with security-relevant remainder.** Pending per `oldtasks.md:39-63`: owner re-validation (Task 13), client ReplayGap handling (14), reconnect integration test (15), per-session dashboard ACL (16–19), orphan-worker reattach (20–28, incl. the `EnableOrphanReattach` flag that does not exist yet). Impact: the repo advertises reconnect and fan-out as shipped while their trust boundary and client support are absent. Recommendation: finish Phase 3 (13–15) as one unit; re-scope Phases 4–5 explicitly if they are no longer planned.
|
||||||
|
- **Medium — Dashboard EventsHub has no per-session ACL.** `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHub.cs:39` (`TODO(per-session-acl)`): any authenticated dashboard Viewer may join any `session:{id}` group and observe that session's event stream. The remark documents why this is accepted today; it is the only production-source TODO in the repo. Impact: acceptable for single-tenant dashboards, wrong the moment `GroupToRole` admits low-trust viewers. Recommendation: covered by epic Phase 4; keep the TODO until then.
|
||||||
|
- **Medium — `Dashboard:ShowTagValues` is a dead flag.** It is bound (`Configuration/DashboardOptions.cs:62`), projected into effective configuration, and displayed on the settings page (`Dashboard/Components/Pages/SettingsPage.razor:65`), but gates no value-display behavior anywhere; `docs/GatewayConfiguration.md:174` calls it "Reserved". Impact: operators toggling it see no effect. Recommendation: implement or remove; a config option that renders on the settings page implies function.
|
||||||
|
- **Medium — Vendor-gated alarm parity residuals are silently lossy.** The 8-arg `AlarmAckByName` operator `domain`/`full_name` fields are accepted on the wire and discarded (`src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs:261-278`, vendor stub returns −55); `AlarmAckByGUID` is `E_NOTIMPL` on this AVEVA build; the `provider_switches` metric's reason tagging has never been exercised live (`stillpending.md` §1.3–1.4, §3.4–3.5). Impact: contract fields that do nothing, invisible to callers. Recommendation: surface the drop in the ack reply diagnostic message until AVEVA implements the stub.
|
||||||
|
- **Low — The bidirectional `Session` RPC from the original design was never built.** `gateway.md:328-345` presents it as the "best long-term shape"; the service stopped at unary + server-streaming. Impact: none functionally; it is only a docs-currency and roadmap-clarity issue (see Documentation Currency).
|
||||||
|
- **Low — Client wire behavior has no automated verification.** All five clients have unit tests (13/8/3/13/7 test files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1` against a live rig. Impact: a gateway contract change can pass every default suite and break all five clients (mitigated somewhat by shared-proto codegen and `CrossLanguageSmokeMatrixTests` shape checks). Recommendation: an in-process fake gateway (the Java CLI already has `InProcessGatewayHarness` per `stillpending.md` §8) is the cheapest pattern to replicate per client.
|
||||||
|
|
||||||
|
## Top 5 recommendations
|
||||||
|
|
||||||
|
1. **Stand up minimal CI.** One pipeline that builds `NonWindows.slnx` and runs the gateway suite per push, a Windows/windev job for the x86 worker + worker tests, and a scheduled opt-in live-MXAccess run. This single change converts most findings above from "process risk" to "caught automatically."
|
||||||
|
2. **Finish session-resilience Phase 3 (Tasks 13–15).** Owner re-validation is a security control for a default-on retention window; client `ReplayGap` handling and the reconnect integration test make the already-shipped server protocol real.
|
||||||
|
3. **Fix the two documentation defects that actively mislead:** the CLAUDE.md "no-retention default" sentence and gateway.md's stale `WorkerEnvelope`/single-subscriber/`Session` sections (replace sketches with pointers to the protos and `docs/Sessions.md`).
|
||||||
|
4. **Triage the repo root:** delete the untracked `*-docs-*.md` artifacts, retire `oldtasks.md` into the plan's tasks.json, and either maintain `stillpending.md` as a living register or migrate its open items into the Gitea tracker the plans already assume.
|
||||||
|
5. **Harden operations:** add Galaxy/LDAP/worker-executable health checks next to the existing auth-store check, commit a deployment runbook + script capturing the NSSM env-var configuration, single-source versions across server, contracts, and clients, and fix (rather than route around) the orphaned-testhost fixture leak.
|
||||||
@@ -0,0 +1,256 @@
|
|||||||
|
# MxAccessGateway — Remediation Tracking
|
||||||
|
|
||||||
|
Master progress tracker for the 2026-07-08 architecture review. Generated 2026-07-09.
|
||||||
|
|
||||||
|
Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation designs are linked below and hold the full **Finding / Impact / Design / Implementation / Verification** for every entry here.
|
||||||
|
|
||||||
|
## How to use this document
|
||||||
|
|
||||||
|
- Each finding has a stable ID (`GWC-01`, `WRK-01`, …) that never changes. Cite it in commits, branches, and PRs (e.g. `fix(GWC-01): split alarm feed`).
|
||||||
|
- The **Status** column is the single source of truth for progress. Update it in the same change that lands the fix.
|
||||||
|
- Do the work in **roadmap-tier order** (P0 → P1 → P2), respecting the `Dep` (depends-on) column within a tier.
|
||||||
|
- When a fix lands: flip Status to `Done`, and per the repo rule, update the affected docs in the same commit.
|
||||||
|
|
||||||
|
**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action).
|
||||||
|
|
||||||
|
## Severity roll-up
|
||||||
|
|
||||||
|
Every finding from all six domain reports, including Lows and informational/positive notes (which exceed the headline "106" because the domain docs cover each sub-finding individually).
|
||||||
|
|
||||||
|
| Domain | Doc | Critical | High | Medium | Low | Info | Total |
|
||||||
|
|---|---|:-:|:-:|:-:|:-:|:-:|:-:|
|
||||||
|
| Gateway core | [10-gateway-core.md](10-gateway-core.md) | 1 | 2 | 7 | 12 | 1 | 23 |
|
||||||
|
| Worker | [20-worker.md](20-worker.md) | — | 1 | 6 | 13 | — | 20 |
|
||||||
|
| Contracts & IPC | [30-contracts-ipc.md](30-contracts-ipc.md) | — | 1 | 8 | 13 | — | 22 |
|
||||||
|
| Security & dashboard | [40-security-dashboard.md](40-security-dashboard.md) | — | — | 12 | 15 | 3 | 30 |
|
||||||
|
| Clients | [50-clients.md](50-clients.md) | — | 4 | 13 | 17 | — | 34 |
|
||||||
|
| Testing, docs & gaps | [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | — | 4 | 13 | 7 | — | 24 |
|
||||||
|
| **Total** | | **1** | **12** | **59** | **77** | **4** | **153** |
|
||||||
|
|
||||||
|
## Roadmap-tier roll-up
|
||||||
|
|
||||||
|
Tiers follow the roadmap in `00-overall.md`. Work top-down; within a tier, `Dep` gives ordering.
|
||||||
|
|
||||||
|
| Tier | Meaning | Count | Findings |
|
||||||
|
|---|---|:-:|---|
|
||||||
|
| **P0** | Correctness & safety — before wider rollout | 8 | GWC-01, GWC-02, GWC-03, WRK-01, CLI-01, CLI-03, TST-02, TST-12 |
|
||||||
|
| **P1** | Process & hardening — next few weeks | 26 | GWC-04, WRK-07, IPC-01, IPC-02, IPC-03, IPC-04, IPC-09, IPC-19, IPC-20, SEC-01, SEC-02, SEC-04, SEC-05, SEC-06, SEC-07, SEC-08, SEC-10, SEC-11, SEC-12, SEC-20, CLI-02, TST-03, TST-05, TST-08 |
|
||||||
|
| **P2** | Completeness & polish | 39 | GWC-06/07/08/14/15, WRK-06/11/12/15, IPC-05/06/07/13/14/15/17/21, SEC-03/09/22/25/30, CLI-04/12/15/16/18/21/26/29, TST-01/04/11/13/14/15/23/24 |
|
||||||
|
| **—** | Not individually in the roadmap (mostly Low/Info) | 80 | see per-domain registers below |
|
||||||
|
|
||||||
|
### P0 — do first (8)
|
||||||
|
|
||||||
|
| ID | Sev | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|---|---|---|
|
||||||
|
| GWC-01 | Critical | M | — | Not started | Alarm monitor and distributor pump both drain the single worker event channel |
|
||||||
|
| GWC-02 | High | M | — | Not started | Faulted sessions are never swept (worker + slot pinned up to 30 min) |
|
||||||
|
| GWC-03 | High | S | — | Not started | Documented sparse-array max-length bound is unimplemented |
|
||||||
|
| WRK-01 | High | M | — | Not started | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
|
||||||
|
| CLI-01 | High | M | — | Not started | Go `Session.Events()` silently closes stream on 16-slot overflow |
|
||||||
|
| CLI-03 | High | M | — | Not started | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
|
||||||
|
| TST-02 | High | M | TST-04 | Not started | Reconnect owner re-validation not implemented (security control) |
|
||||||
|
| TST-12 | Medium | S | — | Not started | CLAUDE.md misstates default retention behaviour (defaults are on, not off) |
|
||||||
|
|
||||||
|
## Finding registers by domain
|
||||||
|
|
||||||
|
Full design + implementation for each row lives in the linked domain doc under its ID heading.
|
||||||
|
|
||||||
|
### Gateway core — [10-gateway-core.md](10-gateway-core.md)
|
||||||
|
|
||||||
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|:-:|---|---|---|
|
||||||
|
| GWC-01 | Critical | P0 | M | — | Not started | Alarm monitor and distributor pump both drain the single worker event channel |
|
||||||
|
| GWC-02 | High | P0 | M | — | Not started | Faulted sessions are never swept |
|
||||||
|
| GWC-03 | High | P0 | S | — | Not started | Documented sparse-array max-length bound is unimplemented |
|
||||||
|
| GWC-04 | Medium | P1 | M | — | Not started | Full event channel stalls the worker read loop behind command replies |
|
||||||
|
| GWC-05 | Medium | — | S | — | Not started | Worker pipe created with no ACL / no CurrentUserOnly |
|
||||||
|
| GWC-06 | Medium | P2 | S | GWC-07,08 | Not started | Stopwatch allocated per streamed event |
|
||||||
|
| GWC-07 | Medium | P2 | S | GWC-06,08 | Not started | Every mapped event is deep-cloned |
|
||||||
|
| GWC-08 | Medium | P2 | M | GWC-06,07 | Not started | Pipe framing allocates per frame and writes twice |
|
||||||
|
| GWC-09 | Medium | — | M | — | Not started | Startup probe is a no-op; its retry pipeline can never retry |
|
||||||
|
| GWC-10 | Medium | — | S | — | Not started | Envelope `sequence` monotonicity specified but not enforced |
|
||||||
|
| GWC-11 | Low | — | S | — | Not started | `_workerClient` written under lock but read lock-free |
|
||||||
|
| GWC-12 | Low | — | S | — | Not started | `WorkerClient.DisposeAsync` not safe against double-dispose |
|
||||||
|
| GWC-13 | Low | — | M | — | Not started | Worker-ready wait is a 25 ms poll loop |
|
||||||
|
| GWC-14 | Low | P2 | M | — | Not started | Replay ring is a `LinkedList` with a node alloc per event |
|
||||||
|
| GWC-15 | Low | P2 | S | — | Not started | Per-event gauge reads `ChannelReader.Count` every event |
|
||||||
|
| GWC-16 | Low | — | S | — | Not started | `Invoke` resolves the session twice per command |
|
||||||
|
| GWC-17 | Low | — | M | — | Not started | Sessions layer throws `Grpc.Core.RpcException` (layering leak) |
|
||||||
|
| GWC-18 | Low | — | S | — | Not started | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` |
|
||||||
|
| GWC-19 | Low | — | S | — | Not started | Dead `GatewaySession.KillWorker(string)` + stale doc |
|
||||||
|
| GWC-20 | Low | — | S | — | Not started | Heartbeat config semantics conflated (send vs check interval) |
|
||||||
|
| GWC-21 | Low | — | S | GWC-04 | Not started | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable |
|
||||||
|
| GWC-22 | Low | — | S | — | Not started | `StreamDisconnected` always labeled `"Detached"` |
|
||||||
|
| GWC-23 | Info | — | — | — | N/A | `MaxEventSubscribersPerSession` dead in default single-subscriber mode |
|
||||||
|
|
||||||
|
### Worker — [20-worker.md](20-worker.md)
|
||||||
|
|
||||||
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|:-:|---|---|---|
|
||||||
|
| WRK-01 | High | P0 | M | — | Not started | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
|
||||||
|
| WRK-02 | Medium | — | M | — | Not started | STA thread death after startup is silent; future work hangs forever |
|
||||||
|
| WRK-03 | Medium | — | S | WRK-02 | Not started | Commands after shutdown starts are dropped with no reply |
|
||||||
|
| WRK-04 | Medium | — | M | — | Not started | Envelope `sequence` can appear out of order on the wire |
|
||||||
|
| WRK-05 | Medium | — | M | — | Not started | One transient alarm-poll failure kills the whole session |
|
||||||
|
| WRK-06 | Medium | P2 | S | — | Not started | `MXSTATUS_PROXY` conversion reflects per field, per event |
|
||||||
|
| WRK-07 | Medium | P1 | M | WRK-04 | Not started | Documented outbound write priority not implemented |
|
||||||
|
| WRK-08 | Low | — | S | — | Not started | Residual event queue discarded at graceful shutdown, undocumented |
|
||||||
|
| WRK-09 | Low | — | S | — | Not started | No `AppDomain.UnhandledException` hook |
|
||||||
|
| WRK-10 | Low | — | S | — | Not started | Top-level catch logs exception type but never message |
|
||||||
|
| WRK-11 | Low | P2 | S | — | Not started | Every accepted event is defensively cloned on enqueue |
|
||||||
|
| WRK-12 | Low | P2 | M | WRK-04 | Not started | No event batching per envelope; one flush per event; 25 ms poll |
|
||||||
|
| WRK-13 | Low | — | S | — | Not started | Frame writer allocates a fresh buffer per frame; reader pools |
|
||||||
|
| WRK-14 | Low | — | M | — | Not started | Private-field naming split `_camelCase` vs `camelCase` |
|
||||||
|
| WRK-15 | Low | P2 | S | — | Not started | Doc drift: STA thread name and heartbeat-counter note |
|
||||||
|
| WRK-16 | Low | — | S | — | Not started | Boilerplate duplication in IPC envelope/ctor overloads |
|
||||||
|
| WRK-17 | Low | — | S | — | Not started | Gateway death exits with wrong exit code (6 not 5) |
|
||||||
|
| WRK-18 | Low | — | S | — | Not started | Event-queue overflow exits as generic `UnexpectedFailure` |
|
||||||
|
| WRK-19 | Low | — | S | — | Not started | Command start/end logging with correlation id absent |
|
||||||
|
| WRK-20 | Low | — | M | WRK-01..04 | Not started | Test-coverage gaps for the failure modes above |
|
||||||
|
|
||||||
|
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
|
||||||
|
|
||||||
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|:-:|---|---|---|
|
||||||
|
| IPC-01 | High | P1 | M | — | Not started | Published client descriptor set 7 weeks stale; nothing enforces freshness |
|
||||||
|
| IPC-02 | Medium | P1 | M | IPC-03 | Not started | Worker max frame size hard-coded, cannot follow gateway config |
|
||||||
|
| IPC-03 | Medium | P1 | M | IPC-02 | Not started | gRPC max == pipe max with zero headroom; oversized write faults whole session |
|
||||||
|
| IPC-04 | Medium | P1 | S | IPC-03 | Not started | `DrainEvents max_events=0` packs entire queue into one reply frame |
|
||||||
|
| IPC-05 | Medium | P2 | S | — | Not started | Redundant deep copies on command and event hot paths |
|
||||||
|
| IPC-06 | Medium | P2 | S | — | Not started | `gateway.md` Worker Envelope sketch no longer matches the contract |
|
||||||
|
| IPC-07 | Medium | P2 | S | — | Not started | `docs/Grpc.md` says six RPCs; there are seven |
|
||||||
|
| IPC-08 | Medium | — | M | — | Not started | `WorkerCancel` defined and handled but never sent |
|
||||||
|
| IPC-09 | Medium | P1 | M | IPC-01 | Not started | Known codegen fragilities have no in-repo guards |
|
||||||
|
| IPC-10 | Low | — | S | — | Not started | Envelope `sequence` is write-only; monotonicity never validated |
|
||||||
|
| IPC-11 | Low | — | S | — | Not started | No protocol version negotiation despite `supported_protocol_version` name |
|
||||||
|
| IPC-12 | Low | — | S | — | Not started | Gateway frame writer has no write lock; integrity rests on undocumented invariant |
|
||||||
|
| IPC-13 | Low | P2 | S | IPC-05 | Not started | Gateway writer serializes twice and issues two stream writes |
|
||||||
|
| IPC-14 | Low | P2 | S | IPC-05 | Not started | Gateway reader allocates fresh array per frame; worker rents from `ArrayPool` |
|
||||||
|
| IPC-15 | Low | P2 | M | — | Not started | Worker event delivery 25 ms poll with per-event write+flush, no batching |
|
||||||
|
| IPC-16 | Low | — | S | — | N/A | Correlation id carried twice per reply (Info) |
|
||||||
|
| IPC-17 | Low | P2 | S | — | Not started | Two docs name the wrong Python generated-output directory |
|
||||||
|
| IPC-18 | Low | — | S | IPC-01 | N/A | Generated-code hygiene verified good — preserve under change (positive) |
|
||||||
|
| IPC-19 | Low | P1 | S | IPC-01 | Not started | Generated/-must-be-committed rule for net48 undocumented and unguarded |
|
||||||
|
| IPC-20 | Low | P1 | S | IPC-01 | Not started | Descriptor `-Check` byte-compares source-info bytes; protoc-version-sensitive |
|
||||||
|
| IPC-21 | Low | P2 | S | IPC-06 | Not started | `gateway.md` presents unimplemented `Session` RPC as live |
|
||||||
|
| IPC-22 | Low | — | S | — | N/A | Public error-detail model stops at status codes plus prose (Info) |
|
||||||
|
|
||||||
|
### Security & dashboard — [40-security-dashboard.md](40-security-dashboard.md)
|
||||||
|
|
||||||
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|:-:|---|---|---|
|
||||||
|
| SEC-01 | Medium | P1 | M | — | Not started | Windows-absolute default paths become relative files off-Windows |
|
||||||
|
| SEC-02 | Medium | P1 | S | — | Not started | `AllowAnonymousLocalhost` satisfies AdminOnly, not just Viewer |
|
||||||
|
| SEC-03 | Medium | P2 | S | — | Not started | Dashboard cookie lost its `__Host-` prefix; four docs still promise it |
|
||||||
|
| SEC-04 | Medium | P1 | S | SEC-03 | Not started | `DisableLogin` has no production guard |
|
||||||
|
| SEC-05 | Medium | P1 | M | — | Not started | Hub bearer tokens irrevocable for 30 min, carried in query string |
|
||||||
|
| SEC-06 | Medium | P1 | M | — | Not started | LDAP plaintext-by-default with a committed service password |
|
||||||
|
| SEC-07 | Medium | P1 | S | — | Not started | `QueryActiveAlarmsRequest` missing from scope resolver; tests mislabeled |
|
||||||
|
| SEC-08 | Medium | P1 | L | — | Not started | Per-RPC SQLite read + `last_used_utc` write; no verification cache |
|
||||||
|
| SEC-09 | Medium | P2 | S | — | Not started | Dashboard design-doc `GroupToRole` sample now fails startup validation |
|
||||||
|
| SEC-10 | Medium | P1 | L | — | Not started | No API-key expiry |
|
||||||
|
| SEC-11 | Medium | P1 | M | SEC-08 | Not started | No rate limiting or lockout on either auth surface |
|
||||||
|
| SEC-12 | Medium | P1 | M | — | Not started | Dashboard Close/Kill bypass the canonical audit store |
|
||||||
|
| SEC-13 | Low | — | S | SEC-30 | Not started | Redactor's credential-command list omits secured-bulk variants |
|
||||||
|
| SEC-14 | Low | — | S | SEC-02,20 | Not started | `/metrics` + `/health` unauthenticated; a metric leaks session ids |
|
||||||
|
| SEC-15 | Low | — | S | — | Not started | GET `/logout` skips antiforgery |
|
||||||
|
| SEC-16 | Low | — | S | — | Not started | CLI accepts the pepper as a command-line argument |
|
||||||
|
| SEC-17 | Low | — | S | — | Not started | Interceptor does not override client-streaming/duplex handlers |
|
||||||
|
| SEC-18 | Low | — | S | — | Not started | Pepper-unavailable detection matches library message text |
|
||||||
|
| SEC-19 | Low | — | S | — | Not started | Canonical audit store re-issues `CREATE TABLE` per write/read |
|
||||||
|
| SEC-20 | Low | P1 | S | — | Not started | `session_id` metric tag is unbounded cardinality |
|
||||||
|
| SEC-21 | Low | — | M | — | Not started | Snapshot publisher works every second regardless of audience |
|
||||||
|
| SEC-22 | Low | P2 | M | — | Not started | `docs/Authentication.md` documents types no longer in this repo |
|
||||||
|
| SEC-23 | Low | — | S | SEC-03,04 | Not started | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` |
|
||||||
|
| SEC-24 | Low | — | S | SEC-04 | Not started | Effective-config view omits the riskiest dashboard flags |
|
||||||
|
| SEC-25 | Low | P2 | M | SEC-02 | Not started | Per-session EventsHub ACL is an acknowledged TODO |
|
||||||
|
| SEC-26 | Low | — | M | — | Not started | Audit trail has no retention or pruning |
|
||||||
|
| SEC-27 | Low | — | S | — | Not started | Dashboard `GatewayStatus` is hardcoded Healthy |
|
||||||
|
| SEC-28 | Info | — | S | — | N/A | Positive security observations (preserve under change) |
|
||||||
|
| SEC-29 | Info | — | S | — | N/A | UI-stack rule verified compliant |
|
||||||
|
| SEC-30 | Info | P2 | S | SEC-13 | Not started | Value-logging feature is unwired end-to-end |
|
||||||
|
|
||||||
|
### Clients — [50-clients.md](50-clients.md)
|
||||||
|
|
||||||
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|:-:|---|---|---|
|
||||||
|
| CLI-01 | High | P0 | M | — | Not started | Go `Session.Events()` silently closes stream on 16-slot overflow |
|
||||||
|
| CLI-02 | High | P1 | M | — | Not started | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) |
|
||||||
|
| CLI-03 | High | P0 | M | — | Not started | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
|
||||||
|
| CLI-04 | High | P2 | L | CLI-15 | Not started | Typed-command parity gap across all five clients |
|
||||||
|
| CLI-05 | Medium | — | S | — | Not started | .NET session cannot be re-attached to an existing session id |
|
||||||
|
| CLI-06 | Medium | — | S | — | Not started | .NET `DisposeAsync` blocks/throws on unreachable gateway |
|
||||||
|
| CLI-07 | Medium | — | S | — | Not started | .NET retry budget self-defeats on `DeadlineExceeded` |
|
||||||
|
| CLI-08 | Medium | — | S | CLI-03 | Not started | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) |
|
||||||
|
| CLI-09 | Medium | — | M | — | Not started | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) |
|
||||||
|
| CLI-10 | Medium | — | M | — | Not started | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` |
|
||||||
|
| CLI-11 | Medium | — | S | — | Not started | Go CLI cannot opt into strict TLS validation |
|
||||||
|
| CLI-12 | Medium | P2 | S | — | Not started | Java docs still say "Java 21" after the JDK 17 retarget |
|
||||||
|
| CLI-13 | Medium | — | M | — | Not started | Java event-stream buffer hardcoded 16, cancel-on-overflow |
|
||||||
|
| CLI-14 | Medium | — | S | — | Not started | Python default TLS is blocking TOFU pin with silent `localhost` SNI |
|
||||||
|
| CLI-15 | Medium | P2 | M | — | Not started | No client handles `ReplayGap` or offers a reconnect helper |
|
||||||
|
| CLI-16 | Medium | P2 | S | CLI-12 | Not started | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` |
|
||||||
|
| CLI-17 | Medium | — | M | CLI-14 | Not started | TLS default posture inconsistent across the five clients |
|
||||||
|
| CLI-18 | Low | P2 | S | — | Not started | .NET csproj records no `<Version>` |
|
||||||
|
| CLI-19 | Low | — | S | — | Not started | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) |
|
||||||
|
| CLI-20 | Low | — | S | CLI-17 | Not started | .NET `--tls` without CA installs accept-all callback |
|
||||||
|
| CLI-21 | Low | P2 | S | — | Not started | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases |
|
||||||
|
| CLI-22 | Low | — | S | — | Not started | Go `newCorrelationID` swallows `crypto/rand` error → empty id |
|
||||||
|
| CLI-23 | Low | — | S | — | Not started | Go nil-vs-empty bulk short-circuit asymmetry |
|
||||||
|
| CLI-24 | Low | — | S | — | Not started | Java `MxEventStream` single-consumer constraint undocumented |
|
||||||
|
| CLI-25 | Low | — | S | — | Not started | Java `close()` does not await channel termination |
|
||||||
|
| CLI-26 | Low | P2 | S | — | Not started | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) |
|
||||||
|
| CLI-27 | Low | — | S | — | Not started | Python `Session.close()` not concurrency-safe; synthesizes reply |
|
||||||
|
| CLI-28 | Low | — | S | — | Not started | Python circular-import workaround at end of `session.py` |
|
||||||
|
| CLI-29 | Low | P2 | S | — | Not started | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) |
|
||||||
|
| CLI-30 | Low | — | S | CLI-04 | Not started | Rust has no `unregister` typed helper |
|
||||||
|
| CLI-31 | Low | — | M | — | Not started | Rust CLI is a single 2,699-line `main.rs` |
|
||||||
|
| CLI-32 | Low | — | S | — | Not started | Client-side bulk caps differ (.NET/Java unbounded) |
|
||||||
|
| CLI-33 | Low | — | S | CLI-01,13 | Not started | Per-language event backpressure semantics undocumented |
|
||||||
|
| CLI-34 | Low | — | S | — | Not started | Python `build/`/`.pytest_cache/` present on disk (untracked) |
|
||||||
|
|
||||||
|
### Testing, docs & gaps — [60-testing-docs-gaps.md](60-testing-docs-gaps.md)
|
||||||
|
|
||||||
|
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||||
|
|---|---|:-:|:-:|---|---|---|
|
||||||
|
| TST-01 | High | P2 | L | TST-04 | Not started | Reconnect/replay has no e2e test and no client consumer |
|
||||||
|
| TST-02 | High | P0 | M | TST-04 | Not started | Reconnect owner re-validation not implemented |
|
||||||
|
| TST-03 | High | P1 | M | — | Not started | No CI exists |
|
||||||
|
| TST-04 | High | P2 | L | — | Not started | Session-resilience epic 16/28 tasks unfinished |
|
||||||
|
| TST-05 | Medium | P1 | S | TST-03 | Not started | Real-worker control/COM paths verified opt-in only |
|
||||||
|
| TST-06 | Medium | — | M | — | Not started | Dashboard live-data path untested |
|
||||||
|
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
|
||||||
|
| TST-08 | Medium | P1 | M | — | Not started | Full-suite orphaned testhost processes |
|
||||||
|
| TST-09 | Medium | — | M | — | Not started | Health checks cover only the auth store |
|
||||||
|
| TST-10 | Medium | — | M | — | Not started | Deployment/upgrade undocumented and hand-rolled |
|
||||||
|
| TST-11 | Medium | P2 | M | — | Not started | No version discipline on server; client versions drift |
|
||||||
|
| TST-12 | Medium | P0 | S | — | Not started | CLAUDE.md misstates default retention behaviour |
|
||||||
|
| TST-13 | Medium | P2 | S | — | Not started | gateway.md carries stale design-era sketches |
|
||||||
|
| TST-14 | Medium | P2 | S | — | Not started | Repo-root working artifacts need triage |
|
||||||
|
| TST-15 | Medium | P2 | M | TST-04 | Not started | Dashboard EventsHub has no per-session ACL |
|
||||||
|
| TST-16 | Medium | — | S | — | Not started | `Dashboard:ShowTagValues` is a dead flag |
|
||||||
|
| TST-17 | Medium | — | S | — | Not started | Vendor-gated alarm parity residuals silently lossy |
|
||||||
|
| TST-18 | Low | — | S | — | Not started | Hosted-service wrappers untested |
|
||||||
|
| TST-19 | Low | — | S | — | Not started | Keep FakeWorkerHarness canned replies in lockstep |
|
||||||
|
| TST-20 | Low | — | S | — | Not started | E2E script papers over a real advise-without-consumer sharp edge |
|
||||||
|
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
|
||||||
|
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
|
||||||
|
| TST-23 | Low | P2 | S | — | Not started | Bidirectional `Session` RPC never built |
|
||||||
|
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification |
|
||||||
|
|
||||||
|
## Cross-cutting clusters
|
||||||
|
|
||||||
|
Findings the review flagged as one coordinated design pass — sequence them together rather than piecemeal.
|
||||||
|
|
||||||
|
- **Size / backpressure topology:** IPC-02, IPC-03, IPC-04 + GWC-04 + WRK-12/WRK-07 + CLI-13/CLI-32/CLI-33. Convey `MaxMessageBytes` to the worker, bound `DrainEvents`, decouple event backlog from command replies, make client buffer-overflow behavior uniform and loud.
|
||||||
|
- **Event hot-path performance:** GWC-06/07/08/14/15 + WRK-06/11/12/13 + IPC-05/13/14/15. Kill double clones, cache the status converter, batch event frames, pool gateway frame buffers.
|
||||||
|
- **Silent-failure hardening:** GWC-02, WRK-01/02/03, CLI-01, CLI-03. Every death or drop must produce an observable error or fault frame.
|
||||||
|
- **CI + codegen freshness (unlocks the rest):** TST-03 + IPC-01/09/19/20 + CLI-34. Stand up minimal CI so the descriptor/codegen/version guards actually run.
|
||||||
|
- **Doc-drift sweep:** IPC-06/07/17/21 + SEC-03/09/22 + CLI-12/16 + TST-12/13/14 + GWC-19 + WRK-15. Correct load-bearing docs to match shipped behavior.
|
||||||
|
- **Session-resilience epic:** TST-01/02/04/15 + SEC-25 + CLI-15. Finish owner re-validation, client `ReplayGap` handling, per-session event ACL, and the replay e2e test.
|
||||||
|
|
||||||
|
## Change log
|
||||||
|
|
||||||
|
| Date | Change |
|
||||||
|
|---|---|
|
||||||
|
| 2026-07-09 | Initial tracking doc generated from the six domain remediation designs. All 153 findings `Not started`. |
|
||||||
@@ -0,0 +1,447 @@
|
|||||||
|
# Gateway Server Core — Remediation Design & Implementation
|
||||||
|
|
||||||
|
Source review: [10-gateway-core.md](../10-gateway-core.md) · Generated: 2026-07-09
|
||||||
|
|
||||||
|
This document turns the Gateway Server Core review into buildable remediation entries. Every cited `path:line` was re-verified against the working tree; all citations still match at the time of writing. Findings are grouped most-severe first, then in the source report's own order (Stability → Performance → Conventions → Underdeveloped). The one Critical (alarm-feed split), both Highs (faulted-session reaping, sparse-array bound), and one Medium (event-backlog stall) carry roadmap tiers from `00-overall.md`; the performance cluster is the P2 hot-path pass.
|
||||||
|
|
||||||
|
## Finding index
|
||||||
|
|
||||||
|
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||||
|
|----|-----|-------|---------|--------|-----------|-------|
|
||||||
|
| GWC-01 | Critical | Alarm monitor and distributor pump both drain the single worker event channel | P0 | M | — | Alarms/GatewayAlarmMonitor.cs, Sessions/GatewaySession.cs, Workers/WorkerClient.cs |
|
||||||
|
| GWC-02 | High | Faulted sessions are never swept | P0 | M | — | Sessions/GatewaySession.cs, Sessions/SessionManager.cs |
|
||||||
|
| GWC-03 | High | Documented sparse-array max-length bound is unimplemented | P0 | S | — | Sessions/SparseArrayExpander.cs, Configuration/EventOptions.cs, Configuration/GatewayOptionsValidator.cs |
|
||||||
|
| GWC-04 | Medium | Full event channel stalls the worker read loop behind command replies | P1 | M | — | Workers/WorkerClient.cs |
|
||||||
|
| GWC-05 | Medium | Worker pipe created with no ACL / no CurrentUserOnly | — | S | — | Sessions/SessionWorkerClientFactory.cs |
|
||||||
|
| GWC-06 | Medium | Stopwatch allocated per streamed event | P2 | S | GWC-07, GWC-08 | Grpc/MxAccessGatewayService.cs |
|
||||||
|
| GWC-07 | Medium | Every mapped event is deep-cloned | P2 | S | GWC-06, GWC-08 | Grpc/MxAccessGrpcMapper.cs, Sessions/GatewaySession.cs |
|
||||||
|
| GWC-08 | Medium | Pipe framing allocates per frame and writes twice | P2 | M | GWC-06, GWC-07 | Workers/WorkerFrameReader.cs, Workers/WorkerFrameWriter.cs |
|
||||||
|
| GWC-09 | Medium | Startup probe is a no-op; its retry pipeline can never retry | — | M | — | Workers/WorkerProcessStartedProbe.cs, Workers/WorkerProcessLauncher.cs, Configuration/WorkerOptions.cs |
|
||||||
|
| GWC-10 | Medium | Envelope `sequence` monotonicity is specified but not enforced | — | S | — | Workers/WorkerEnvelopeValidator.cs, Workers/WorkerClient.cs |
|
||||||
|
| GWC-11 | Low | `_workerClient` written under lock but read lock-free | — | S | — | Sessions/GatewaySession.cs |
|
||||||
|
| GWC-12 | Low | `WorkerClient.DisposeAsync` not safe against double-dispose | — | S | — | Workers/WorkerClient.cs |
|
||||||
|
| GWC-13 | Low | Worker-ready wait is a 25 ms poll loop | — | M | — | Sessions/GatewaySession.cs |
|
||||||
|
| GWC-14 | Low | Replay ring is a `LinkedList` with a node alloc per event | P2 | M | — | Sessions/SessionEventDistributor.cs |
|
||||||
|
| GWC-15 | Low | Per-event gauge reads `ChannelReader.Count` every event | P2 | S | — | Grpc/EventStreamService.cs |
|
||||||
|
| GWC-16 | Low | `Invoke` resolves the session twice per command | — | S | — | Grpc/MxAccessGatewayService.cs, Sessions/SessionManager.cs |
|
||||||
|
| GWC-17 | Low | Sessions layer throws `Grpc.Core.RpcException` | — | M | — | Sessions/SparseArrayExpander.cs, Sessions/GatewaySession.cs, Grpc/MxAccessGatewayService.cs |
|
||||||
|
| GWC-18 | Low | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` | — | S | — | Sessions/GatewaySession.cs |
|
||||||
|
| GWC-19 | Low | Dead `GatewaySession.KillWorker(string)` + stale doc | — | S | — | Sessions/GatewaySession.cs, docs/Sessions.md |
|
||||||
|
| GWC-20 | Low | Heartbeat config semantics conflated (send vs check interval) | — | S | — | Configuration/WorkerOptions.cs, Sessions/SessionWorkerClientFactory.cs, Workers/WorkerClient.cs |
|
||||||
|
| GWC-21 | Low | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable | — | S | GWC-04 | Sessions/SessionWorkerClientFactory.cs, Configuration/WorkerOptions.cs |
|
||||||
|
| GWC-22 | Low | `StreamDisconnected` always labeled `"Detached"` | — | S | — | Grpc/EventStreamService.cs |
|
||||||
|
| GWC-23 | Info | `MaxEventSubscribersPerSession` dead in default single-subscriber mode | — | — | — | Configuration/GatewayOptionsValidator.cs |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-01 — Alarm monitor and distributor pump both drain the single worker event channel `Critical` · `P0`
|
||||||
|
|
||||||
|
**Finding.** `GatewayAlarmMonitor.RunMonitorAsync` drains worker events directly through `_sessionManager.ReadEventsAsync(session.SessionId, …)` (`Alarms/GatewayAlarmMonitor.cs:228-231`) → `GatewaySession.ReadEventsAsync` (`Sessions/GatewaySession.cs:1427-1440`) → `WorkerClient.ReadEventsAsync` (`Workers/WorkerClient.cs:227-236`). But `MarkReady` starts the dashboard mirror (`Sessions/GatewaySession.cs:433-437`), whose distributor pump source `MapWorkerEventsAsync` also calls `ReadEventsAsync` (`Sessions/GatewaySession.cs:701-710`). `WorkerClient._events` is a single channel created with `SingleReader = false` (`Workers/WorkerClient.cs:70-77`), so the two concurrent `ReadAllAsync` enumerators each receive a random subset of events.
|
||||||
|
|
||||||
|
**Impact.** In every production deployment `IDashboardEventBroadcaster` is registered and injected, so the pump starts at session-ready — including on the alarm monitor's own session. Roughly half of `OnAlarmTransition` events land in the dashboard mirror instead of `ApplyTransition`. `ReconcileLoopAsync` eventually repairs Raise/Clear presence, but `ApplyReconcile` broadcasts only presence deltas (`Alarms/GatewayAlarmMonitor.cs:511-550`), so a stolen Acknowledge transition never reaches `StreamAlarms` subscribers — clients show unacked alarms indefinitely. Provider-mode-change events can also be stolen, delaying degraded-state visibility. `docs/Sessions.md:196` documents the exact race single-subscriber mode exists to prevent; the monitor recreates it internally.
|
||||||
|
|
||||||
|
**Design.** Make the alarm monitor a *distributor subscriber* rather than a second raw channel drain, so the single worker channel keeps exactly one consumer (the distributor pump) and the pump fans every event to both the dashboard mirror and the alarm monitor. Add an internal (non-counted) registration path on the session mirroring the existing dashboard mirror lease (`isInternal: true`, `Sessions/GatewaySession.cs:582`), so the alarm subscriber does not consume one of `MaxEventSubscribersPerSession` and a slow reconcile never faults the session.
|
||||||
|
|
||||||
|
To make any *future* dual-consumer regression fail loudly rather than silently split events, additionally flip `WorkerClient._events` to `SingleReader = true` and add a claimed-once guard in `WorkerClient.ReadEventsAsync` (throw if a second enumerator is created). After this fix the distributor pump is the only direct caller, so `SingleReader = true` is correct and the guard converts the bug class into an immediate exception.
|
||||||
|
|
||||||
|
Alternative rejected: giving the alarm monitor its own second worker channel (fan the worker frame to two channels in `EnqueueWorkerEventAsync`) — doubles per-event copy cost on the hot path and duplicates the backpressure/overflow accounting the distributor already owns.
|
||||||
|
|
||||||
|
Parity note: no synthesized events and no MXAccess behavior change — this only fixes *delivery* of events the worker already emits.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/GatewaySession.cs`: add an internal-subscriber accessor the alarm monitor can use, e.g. `AttachInternalEventSubscriber()` that runs `EnsureDistributorCreated` / `distributor.Register(isInternal: true)` / `StartPumpIfRequested` (the same sequence `StartDashboardMirror` uses at `:581-583`) and returns an `IAsyncEnumerable<MxEvent>` (or the lease + its channel reader). Reuse `MapWorkerEventsAsync`'s public `MxEvent` shape so the monitor's existing `MxEvent.BodyCase` switch (`Alarms/GatewayAlarmMonitor.cs:232-246`) is unchanged.
|
||||||
|
- `Sessions/SessionManager.cs`: expose the new subscriber path (e.g. `IAsyncEnumerable<MxEvent> ReadAlarmEventsAsync(sessionId, ct)`) alongside the existing `ReadEventsAsync`, or repoint the monitor at the distributor via the session directly.
|
||||||
|
- `Alarms/GatewayAlarmMonitor.cs:228-231`: consume the distributor subscriber stream (mapped `MxEvent`) instead of `_sessionManager.ReadEventsAsync` (raw `WorkerEvent`). `workerEvent.Event` reads become direct `MxEvent` reads.
|
||||||
|
- `Workers/WorkerClient.cs:70-77`: set `SingleReader = true`; in `ReadEventsAsync` (`:227`) add an `Interlocked.CompareExchange` claimed-once guard that throws `InvalidOperationException` on a second enumerator.
|
||||||
|
- Tests: extend `GatewayAlarmMonitorTests` (or add `GatewayAlarmMonitorDistributorTests`) driving the `FakeWorkerHarness` with a session that has the dashboard mirror active, asserting every `OnAlarmTransition` (including Acknowledge) reaches `StreamAlarms`; add a `WorkerClientTests` case asserting a second `ReadEventsAsync` enumerator throws.
|
||||||
|
- Docs: note in `docs/Sessions.md` that the alarm monitor attaches as an internal distributor subscriber; update `gateway.md` alarm section if it describes the direct drain.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewayAlarmMonitor"` and `--filter "FullyQualifiedName~WorkerClient"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-02 — Faulted sessions are never swept `High` · `P0`
|
||||||
|
|
||||||
|
**Finding.** `MarkFaulted` only flips state and records the reason (`Sessions/GatewaySession.cs:716-728`); it does not kill the worker, stop the distributor, or notify the registry. `CloseExpiredLeasesAsync` sweeps only lease-expired or detach-grace-expired sessions (`Sessions/SessionManager.cs:273-277`); there is no `State == Faulted` branch. Detach-grace deliberately skips faulted sessions (`Sessions/GatewaySession.cs:2022-2028`), so the only exit is lease expiry at `DefaultLeaseSeconds` = 1800 s. In the FailFast single-subscriber overflow case (`Sessions/GatewaySession.cs:690-694`) the worker stays healthy and keeps pumping events while the session is permanently unusable (`EvaluateReadyUnderLock` fails every command).
|
||||||
|
|
||||||
|
**Impact.** A burst-slow client that overflows its queue faults the session; for up to 30 minutes the gateway pins one of `MaxSessions` (default 64) slots and a running x86 MXAccess worker with live COM subscriptions no client can use or close (clients rarely call `CloseSession` on a faulted session). A handful of such faults exhausts session capacity.
|
||||||
|
|
||||||
|
**Design.** Sweep `Faulted` sessions in `CloseExpiredLeasesAsync` so the existing serialized teardown (`TryBeginCloseIfExpired` → `CloseSessionCoreAsync`) reaps them promptly, reusing the proven TOCTOU-safe close gate rather than adding a second teardown path. Add a distinct close reason (`FaultedReason`) so operators can distinguish fault reaps from lease/detach reaps in logs and metrics. Optionally add a short `FaultedGraceSeconds` (default 0) so a monitoring client can still observe the fault via `GetSessionStatus` before the slot is reclaimed; recommend default-immediate to bound blast radius, with the grace knob as the tunable.
|
||||||
|
|
||||||
|
Alternative considered: have `MarkFaulted` synchronously kill the worker and schedule teardown (as `WorkerClient.SetFaulted` does on its side). Rejected as the primary fix because `MarkFaulted` is called from inside the distributor overflow handler and the gRPC pump (`Grpc/EventStreamService.cs:155`) under async contexts where a synchronous kill + registry mutation risks re-entrancy against the close gate; the sweeper path already centralizes teardown ordering. `MarkFaulted` can still *nudge* the sweeper (e.g. set an eligibility timestamp), but the reap stays in `CloseExpiredLeasesAsync`.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/GatewaySession.cs`: add `IsFaultedReapable(now)` (true when `_state == Faulted`, gated by optional `FaultedGraceSeconds` measured from a fault timestamp stamped in `MarkFaulted`).
|
||||||
|
- `Sessions/SessionManager.cs:273-277`: extend the reason ladder — `session.IsLeaseExpired(now) ? LeaseExpiredReason : session.IsFaultedReapable(now) ? FaultedReason : session.IsDetachGraceExpired(now) ? DetachGraceExpiredReason : null`. Ensure `TryBeginCloseIfExpired` treats a faulted session as eligible.
|
||||||
|
- `Configuration/SessionOptions.cs`: add `FaultedGraceSeconds` (default 0); validate `>= 0` in `Configuration/GatewayOptionsValidator.cs`.
|
||||||
|
- Tests: `SessionManagerTests`/`SessionLeaseSweeperTests` — fault a session, advance `TimeProvider`, assert the sweeper closes it and kills the worker; assert a non-faulted leased session is untouched.
|
||||||
|
- Docs: `docs/Sessions.md` (sweeper reasons) and `docs/GatewayConfiguration.md` (`FaultedGraceSeconds`) in the same commit.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionManager"` and `--filter "FullyQualifiedName~Sweeper"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-03 — Documented sparse-array max-length bound is unimplemented `High` · `P0`
|
||||||
|
|
||||||
|
**Finding.** `gateway.md:532` lists "`total_length` exceeds the gateway-configured maximum array length" as an `InvalidArgument` rejection, but `SparseArrayExpander.Expand` validates only `totalLength > (uint)Array.MaxLength` (`Sessions/SparseArrayExpander.cs:65-69`) — ~2.1 billion elements. No configuration option for the bound exists in `Configuration/`.
|
||||||
|
|
||||||
|
**Impact.** A single authorized `Write` carrying `total_length = 500_000_000` forces materialization of a multi-GB `MxArray` (`Sessions/SparseArrayExpander.cs:98-232`) before the 16 MB `MaxMessageBytes` frame check finally rejects the serialized result in `WorkerFrameWriter` (`Workers/WorkerFrameWriter.cs:49-54`) — a memory-exhaustion vector reachable through the normal command path.
|
||||||
|
|
||||||
|
**Design.** Add the documented configurable cap under `MxGateway:Events` (or a new `MxGateway:Sparse` section) and enforce it in `SparseArrayExpander.Expand` *before* allocation, throwing the same `InvalidArgument` the surrounding validators already use. Validate the bound at startup in `GatewayOptionsValidator`. A conservative default (e.g. 1_000_000 elements — comfortably above realistic MXAccess array writes yet far below the frame-size ceiling) matches the parity contract: MXAccess itself has no multi-hundred-million-element array use case, so the cap rejects only abusive inputs and does not alter valid behavior.
|
||||||
|
|
||||||
|
The expander is constructed per invocation from `NormalizeOutboundCommand` (`Sessions/GatewaySession.cs:984-1063`); thread the configured max in through its constructor or an `Expand(..., int maxTotalLength)` parameter sourced from `EventOptions`/a new option object the session already holds.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/EventOptions.cs` (or new `SparseArrayOptions`): add `MaxSparseArrayLength` (default 1_000_000).
|
||||||
|
- `Configuration/GatewayOptionsValidator.cs`: validate `MaxSparseArrayLength >= 1` and `<= Array.MaxLength`.
|
||||||
|
- `Sessions/SparseArrayExpander.cs:65-69`: check `totalLength > maxSparseArrayLength` first, with a message naming the configured cap; keep the `Array.MaxLength` guard as a backstop.
|
||||||
|
- Wire the value from options through `GatewaySession.NormalizeOutboundCommand` into the expander.
|
||||||
|
- Tests: `SparseArrayExpanderTests` — assert `total_length` above the cap throws `InvalidArgument` before allocation; assert a value below the cap still expands. Add a `GatewayOptionsValidatorTests` case for the new bound.
|
||||||
|
- Docs: confirm `gateway.md:532` wording matches the option name; add the key to `docs/GatewayConfiguration.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SparseArrayExpander"` and `--filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-04 — Full event channel stalls the worker read loop behind command replies `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** `ReadLoopAsync` awaits `DispatchEnvelopeAsync` inline (`Workers/WorkerClient.cs:358-362`), and the `WorkerEvent` branch awaits `EnqueueWorkerEventAsync`, which blocks in `WriteAsync` for up to `EventChannelFullModeTimeout` (default 5 s) when `_events` is full (`Workers/WorkerClient.cs:511-553`, `Workers/WorkerClientOptions.cs:13`).
|
||||||
|
|
||||||
|
**Impact.** With no event consumer attached or a stalled distributor, each incoming event costs up to 5 s of read-loop stall before the fault fires. A `WorkerCommandReply` queued behind an event frame is not dispatched, so an in-flight `InvokeAsync` can hit `CommandTimeout` (`Workers/WorkerClient.cs:187-213`) even though the worker replied in time; heartbeats behind the stall feed the very watchdog interplay the code tries to compensate for (`Workers/WorkerClient.cs:394-424`).
|
||||||
|
|
||||||
|
**Design.** Decouple event enqueue from the read loop so replies and heartbeats are never blocked by an event backlog. The class already has a dedicated `WriteLoopAsync` for outbound envelopes (`Workers/WorkerClient.cs:332-338`); mirror that for events. In `DispatchEnvelopeAsync`, the `WorkerEvent` branch should hand the event to `_events` via a **non-blocking** `TryWrite`; when the channel is full, start (or continue) a short bounded backpressure window on a *separate* task rather than blocking the read loop. Command-reply, heartbeat, fault, and shutdown-ack branches dispatch synchronously and immediately, so they can never queue behind events.
|
||||||
|
|
||||||
|
Concretely: keep the `EventChannelFullModeTimeout` semantics but move the waiting `WriteAsync` off the read loop — either (a) route events through a small internal event channel drained by a dedicated writer task that performs the timed `WriteAsync` and faults on timeout, or (b) on a `TryWrite` miss, record a monotonic "backlog since" timestamp and fault only once the backlog persists past the timeout, all without awaiting inside `ReadLoopAsync`. Option (a) is cleaner and symmetrical with the existing write loop; recommend it.
|
||||||
|
|
||||||
|
This is the gateway half of the coordinated backpressure/size pass in `00-overall.md` P1.8 (co-designed with the worker and contracts backlog findings); the fault model (`ProtocolViolation` on sustained overflow) is unchanged.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Workers/WorkerClient.cs`: add an internal unbounded-or-small event staging channel + `EventWriteLoopAsync` (parallel to `WriteLoopAsync`); `DispatchEnvelopeAsync`'s `WorkerEvent` branch does a non-awaiting hand-off; the writer task owns the timed `WriteAsync`/fault. Register the new task in `WaitForBackgroundTasksAsync` and complete it in `DisposeAsync`.
|
||||||
|
- Tests: `WorkerClientTests` — with `_events` full and no consumer, assert a `WorkerCommandReply` arriving after an event is still dispatched promptly (command does not time out); assert sustained overflow still faults with the existing `ProtocolViolation` message. Reuse the fake pipe harness.
|
||||||
|
- Docs: note the event-vs-reply decoupling in `docs/WorkerFrameProtocol.md` / `docs/MxAccessWorkerInstanceDesign.md` if they describe the read loop.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-05 — Worker pipe created with no ACL / no CurrentUserOnly `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `SessionWorkerClientFactory.CreatePipe` uses the plain `NamedPipeServerStream` constructor with `PipeOptions.Asynchronous` only (`Sessions/SessionWorkerClientFactory.cs:158-166`). `gateway.md:282-283` requires the pipe be "ACL restricted to the gateway identity and the launched worker identity, no anonymous access".
|
||||||
|
|
||||||
|
**Impact.** Any local process can connect to `mxaccess-gateway-{pid}-{sessionId}` before the real worker; with `maxNumberOfServerInstances: 1` the legitimate worker then never connects, so `OpenSession` fails on startup timeout — a trivially repeatable local denial of service. The startup nonce (`Workers/WorkerClient.cs:632-637`) prevents impersonation but not connection stealing.
|
||||||
|
|
||||||
|
**Design.** Create the pipe with an explicit DACL limited to the service identity via `NamedPipeServerStreamAcl.Create` (from `System.IO.Pipes.AccessControl`). Because workers run as the gateway identity (`docs/DesignDecisions.md`), `PipeOptions.CurrentUserOnly` is the simpler equivalent and is the recommended first choice — it restricts the pipe to the creating user's SID with no manual `PipeSecurity` construction. Keep `PipeOptions.Asynchronous`. This is Windows-only surface; the pipe factory is not exercised on the macOS `NonWindows.slnx`, so guard behind the existing Windows compilation/runtime path.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/SessionWorkerClientFactory.cs:158-166`: `return NamedPipeServerStreamAcl.Create(pipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, 0, 0, pipeSecurity);` with a `PipeSecurity` granting only the current user full control and denying everyone else — or, if acceptable, `new NamedPipeServerStream(name, …, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly)`.
|
||||||
|
- Tests: worker-pipe tests run on Windows only; add a `SessionWorkerClientFactory` test asserting a second connector as a different principal is rejected (or, minimally, that the pipe is created with restricted ACL). Document why it is skipped on non-Windows.
|
||||||
|
- Docs: none needed beyond confirming `gateway.md:282-283` now matches code.
|
||||||
|
|
||||||
|
**Verification.** On the Windows host: `dotnet build src/ZB.MOM.WW.MxGateway.Server`; run the factory/pipe tests. On macOS the change is not compiled into `NonWindows.slnx`; document the skip.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-06 — Stopwatch allocated per streamed event `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `MxAccessGatewayService.StreamEvents` runs `Stopwatch stopwatch = Stopwatch.StartNew()` inside the per-event loop (`Grpc/MxAccessGatewayService.cs:155-157`), one heap allocation per event per subscriber on the highest-volume gateway path.
|
||||||
|
|
||||||
|
**Impact.** Avoidable GC pressure under alarm bursts — the exact load the system exists to handle.
|
||||||
|
|
||||||
|
**Design.** Replace the `Stopwatch` object with the allocation-free timestamp API: `long ts = Stopwatch.GetTimestamp();` before the write and `metrics.RecordEventStreamSend(family, Stopwatch.GetElapsedTime(ts))` after. Behavior and the recorded metric are identical. Co-designed with GWC-07 and GWC-08 as the P2 hot-path pass.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Grpc/MxAccessGatewayService.cs:155-157`: swap to `Stopwatch.GetTimestamp()` / `Stopwatch.GetElapsedTime(ts)`.
|
||||||
|
- Tests: existing `MxAccessGatewayService`/`EventStreamService` streaming tests cover behavior; no new assertion needed beyond a green run.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~StreamEvents"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-07 — Every mapped event is deep-cloned `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `MxAccessGrpcMapper.MapEvent` returns `workerEvent.Event?.Clone()` (`Grpc/MxAccessGrpcMapper.cs:64-73`), invoked once per event by the distributor pump via `MapWorkerEventsAsync` (`Sessions/GatewaySession.cs:701-710`).
|
||||||
|
|
||||||
|
**Impact.** A full protobuf deep copy (including value arrays) per event, even though the enclosing `WorkerEvent` is discarded immediately after mapping and nothing else retains the inner message.
|
||||||
|
|
||||||
|
**Design.** Transfer ownership of `workerEvent.Event` instead of cloning — return the inner message directly. The `WorkerEvent` wrapper is created by the frame reader per received frame and is not retained after `MapEvent`, so no other consumer aliases the inner `MxEvent`. Keep a comment stating the ownership-transfer invariant so a future second consumer of the `WorkerEvent` restores the clone. This is safe today precisely because GWC-01 makes the distributor pump the single consumer of the worker channel.
|
||||||
|
|
||||||
|
Depends on GWC-01 (single-consumer guarantee) to be strictly safe; co-designed with GWC-06/GWC-08.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Grpc/MxAccessGrpcMapper.cs:64-73`: `return workerEvent.Event ?? new MxEvent { Family = MxEventFamily.Unspecified, RawStatus = "…" };` (drop `.Clone()`), with an ownership-transfer comment.
|
||||||
|
- Tests: `MxAccessGrpcMapperTests` — assert the returned reference is the same instance as `workerEvent.Event` (ownership transferred) and the unspecified-family fallback path is unchanged.
|
||||||
|
- Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~MxAccessGrpcMapper"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-08 — Pipe framing allocates per frame and writes twice `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** The reader allocates `new byte[sizeof(uint)]` and `new byte[payloadLength]` per frame (`Workers/WorkerFrameReader.cs:32,50`); the writer allocates `new byte[sizeof(uint)]` plus `envelope.ToByteArray()` and performs two `WriteAsync` calls (`Workers/WorkerFrameWriter.cs:56-60`).
|
||||||
|
|
||||||
|
**Impact.** Per-frame GC pressure proportional to event rate and two pipe syscalls per outbound frame. The worker side already uses `ArrayPool`; the gateway side does not.
|
||||||
|
|
||||||
|
**Design.** On the write path, serialize length + payload into a single pooled buffer and issue one `WriteAsync`: rent `4 + payloadLength` from `ArrayPool<byte>.Shared`, write the little-endian length prefix into the first four bytes, serialize the envelope into the remainder via `envelope.WriteTo(new CodedOutputStream(...))` or `envelope.WriteTo(span)`, write once, then return the buffer. On the read path, rent a buffer sized to the frame instead of `new byte[payloadLength]`, parse from the rented span, and return it. Preserve the existing length-validation-before-allocation order (`Workers/WorkerFrameReader.cs:43-48`) — the `MaxMessageBytes` check must still gate the rent size. Co-designed with GWC-06/GWC-07.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Workers/WorkerFrameWriter.cs:56-60`: rent `4 + payloadLength`, fill prefix + serialized payload, single `WriteAsync(buffer, 0, total)`, `try/finally` return to pool.
|
||||||
|
- `Workers/WorkerFrameReader.cs:32,50`: rent buffers (or reuse a per-reader scratch buffer for the 4-byte prefix), parse from the rented region, return in `finally`. Keep exact-read semantics (`ReadExactlyOrThrowAsync`).
|
||||||
|
- Tests: `WorkerFrameReaderTests`/`WorkerFrameWriterTests` round-trip must stay green; add a large-payload round-trip and an oversize-rejection test to confirm the validation order is preserved. These also protect the `FakeWorkerHarness` which reuses this frame code.
|
||||||
|
- Docs: `docs/WorkerFrameProtocol.md` if it describes the two-write layout.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerFrame"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-09 — Startup probe is a no-op; its retry pipeline can never retry `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `WorkerProcessStartedProbe.WaitUntilReadyAsync` does one instantaneous `HasExited` check and throws `WorkerProcessLaunchException` on failure (`Workers/WorkerProcessStartedProbe.cs:6-19`); `ShouldRetryStartupProbe` explicitly excludes `WorkerProcessLaunchException` and `OperationCanceledException` from retry (`Workers/WorkerProcessLauncher.cs:291-299`). The Polly pipeline with exponential backoff and jitter (`Workers/WorkerProcessLauncher.cs:264-289`) therefore executes exactly one attempt in every case, so `StartupProbeRetryAttempts` and `StartupProbeRetryDelayMilliseconds` (`Configuration/WorkerOptions.cs:19-22`, validated at `Configuration/GatewayOptionsValidator.cs:119-126`) have no observable effect.
|
||||||
|
|
||||||
|
**Impact.** Two configuration options and their validation are dead; operators may tune them expecting an effect. Actual worker readiness is instead established later by the pipe-connect + nonce handshake, so nothing is *broken*, but the surface is misleading.
|
||||||
|
|
||||||
|
**Design.** Two viable options; recommend **(B) remove the dead surface** unless a real readiness signal is planned, because the pipe handshake already gates readiness and a genuine probe would duplicate it.
|
||||||
|
|
||||||
|
- (A) Implement a real readiness probe with retryable transient failures: have the probe poll a real signal (e.g. pipe-created / worker "ready" marker) and throw a *retryable* exception type (not `WorkerProcessLaunchException`) on transient not-ready, so the Polly pipeline actually retries. Then the two options become live.
|
||||||
|
- (B) Delete the retry pipeline and the two options: replace `CreateStartupProbePipeline` usage with a single direct `WaitUntilReadyAsync` call, remove `StartupProbeRetryAttempts` / `StartupProbeRetryDelayMilliseconds` from `WorkerOptions` and their validator branch, and update `docs/WorkerProcessLauncher.md` in the same commit.
|
||||||
|
|
||||||
|
Open question the review cannot settle: whether a future probe (COM-init readiness) is planned. If yes, do (A); if no, do (B). Recommended: (B) now, reintroduce a real probe if/when COM-readiness signaling lands.
|
||||||
|
|
||||||
|
**Implementation (B).**
|
||||||
|
- `Workers/WorkerProcessLauncher.cs`: drop `CreateStartupProbePipeline` and call the probe directly; remove `ShouldRetryStartupProbe`.
|
||||||
|
- `Configuration/WorkerOptions.cs:19-22` and `Configuration/GatewayOptionsValidator.cs:119-126`: remove the two options and their validation.
|
||||||
|
- Tests: `WorkerProcessLauncherTests` — remove/adjust retry-count assertions; assert an exited-before-ready worker fails fast once.
|
||||||
|
- Docs: `docs/WorkerProcessLauncher.md`, `docs/GatewayConfiguration.md` (drop the two keys) in the same commit.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerProcessLauncher"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-10 — Envelope `sequence` monotonicity specified but not enforced `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `gateway.md:314` states "`sequence` is monotonic per sender", but `WorkerEnvelopeValidator.Validate` checks only protocol version, session id, and body presence (`Workers/WorkerEnvelopeValidator.cs:15-39`). Nothing on the gateway side detects out-of-order, duplicated, or replayed frames from a misbehaving worker.
|
||||||
|
|
||||||
|
**Impact.** A worker bug that reorders or repeats frames is invisible; the event-ordering guarantee rests solely on the worker's writer discipline. Latent — depends on a worker defect — hence Medium.
|
||||||
|
|
||||||
|
**Design.** Track the last-received `sequence` per `WorkerClient` connection and fault the client with `ProtocolViolation` on a regression (non-increasing sequence), matching the existing fault model (`Workers/WorkerClient.cs:376-380`). Enforce in the read loop after `Validate`, not inside the static validator (which has no per-connection state). Allow the very first frame to establish the baseline. Whether strict `+1` monotonicity or merely strictly-increasing is required depends on whether the worker sends a continuous sequence across all envelope types — recommend strictly-increasing (regression → fault) as the safe superset, since gaps are legal if any frame type is unsequenced.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Workers/WorkerClient.cs`: add `ulong _lastReceivedSequence` (per connection); in `ReadLoopAsync`/`DispatchEnvelopeAsync` after validation, if `envelope.Sequence <= _lastReceivedSequence` (and not the first frame) call `SetFaulted(ProtocolViolation, …)`; else advance. Keep the check off the hot allocation path.
|
||||||
|
- Tests: `WorkerClientTests` — feed a decreasing/duplicate sequence via the fake pipe and assert the client faults with `ProtocolViolation`; assert normal increasing sequences pass.
|
||||||
|
- Docs: confirm `gateway.md:314` matches; note enforcement point.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-11 — `_workerClient` written under lock but read lock-free `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `_workerClient` is written under `_syncRoot` in `AttachWorkerClient` (`Sessions/GatewaySession.cs:368-376`) but read lock-free in `CloseAsync` (`:1470`), `DisposeAsync` (`:1762`), `KillWorker` (`:1617`), and `WorkerProcessId` (`:268`). Reference reads are atomic and the manager's call ordering makes a torn interleaving unlikely, but the discipline documented for `_state` is not applied to the field.
|
||||||
|
|
||||||
|
**Impact.** Latent consistency risk only; no observed failure.
|
||||||
|
|
||||||
|
**Design.** Apply the class's own locking contract: either read `_workerClient` under `_syncRoot` on those paths or mark the field `volatile`. Recommend `volatile` — the reads are single-reference snapshots and lock acquisition on the close/dispose paths is otherwise unnecessary; `volatile` documents the shared-field intent with minimal churn. (`KillWorker` itself is dead — see GWC-19 — so that read disappears if GWC-19 is applied.)
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/GatewaySession.cs`: mark `_workerClient` `volatile` (or wrap the four reads in `lock (_syncRoot)`).
|
||||||
|
- Tests: covered by existing session lifecycle tests; no new assertion.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-12 — `WorkerClient.DisposeAsync` not safe against double-dispose `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `DisposeAsync` uses a plain `if (_disposed) return; _disposed = true;` with no interlock (`Workers/WorkerClient.cs:296-303`). Two concurrent disposals would both run kill/complete/dispose, and the second `_stopCts.Cancel()` after `_stopCts.Dispose()` (`:305`, `:328`) would throw `ObjectDisposedException`.
|
||||||
|
|
||||||
|
**Impact.** Latent only — `SessionManager.RemoveSessionAsync`'s registry `TryRemove` gate makes the session's `DisposeAsync` single-shot in practice.
|
||||||
|
|
||||||
|
**Design.** Use `Interlocked.Exchange` to make disposal single-shot, matching the pattern the lease classes already use. Minimal, no behavior change on the single-dispose path.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Workers/WorkerClient.cs:296-303`: `if (Interlocked.Exchange(ref _disposedFlag, 1) == 1) return;` (int field), keeping the remainder unchanged.
|
||||||
|
- Tests: `WorkerClientTests` — call `DisposeAsync` concurrently twice and assert no throw.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-13 — Worker-ready wait is a 25 ms poll loop `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `GetReadyWorkerClientAsync` polls with `Task.Delay(25 ms)` up to `WorkerReadyWaitTimeoutMs` (`Sessions/GatewaySession.cs:1841-1909`). It is default-off (`Configuration/SessionOptions.cs:69`), bounded, and testable via `TimeProvider`, so impact is minor; it is still a poll on the command hot path when enabled.
|
||||||
|
|
||||||
|
**Impact.** Minor latency/CPU when the option is enabled; none by default.
|
||||||
|
|
||||||
|
**Design.** If the option sees real use, replace the poll with a `TaskCompletionSource` pulsed on worker state transitions (the worker client already raises state changes internally). Register a waiter under `_syncRoot` in `GetReadyWorkerClientAsync` when the state is transient, and complete it when the worker becomes Ready or terminal, with the existing timeout as a `CancelAfter`. Keep the existing zero-timeout fail-fast path byte-for-byte. Because this is default-off and low-value, this is a defer-until-used item; the recommended action is to leave the poll but document it as a known trade-off unless the option is turned on in a real deployment.
|
||||||
|
|
||||||
|
**Implementation (if pursued).**
|
||||||
|
- `Sessions/GatewaySession.cs`: add a state-change signal (`TaskCompletionSource`) set on worker Ready/terminal transitions; await it with the timeout instead of the poll loop.
|
||||||
|
- Tests: `GatewaySessionTests` using `TimeProvider` — assert readiness is observed without the 25 ms granularity delay.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-14 — Replay ring is a `LinkedList` with a node alloc per event `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** The distributor replay ring is a `LinkedList<ReplayEntry>` (`Sessions/SessionEventDistributor.cs:108`), appended per event under `_replayLock` (`:766-793`). Capacity is fixed (`ReplayBufferCapacity`, default 1024).
|
||||||
|
|
||||||
|
**Impact.** A node allocation and poor cache locality per event on the fan-out hot path — exactly the shape a circular array serves allocation-free.
|
||||||
|
|
||||||
|
**Design.** Replace the `LinkedList` with a fixed-size ring array (`ReplayEntry[]` + head/count) sized to `ReplayBufferCapacity`. Keep the `_replayLock` protocol and the ascending-`WorkerSequence` ordering invariant (`:100-108`) unchanged; append overwrites the oldest slot, and `TryGetReplayFrom` binary-searches or linear-scans from the front. Age-based eviction (`_ageEvictionEnabled`) maps to advancing the tail index. Part of the P2 hot-path pass; behavior (replay contents, gap detection) must be identical.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/SessionEventDistributor.cs`: swap the backing store to a ring array with head/tail/count; update append (`:766-793`) and `TryGetReplayFrom` accordingly; keep capacity-0-disables-retention and retention-eviction semantics.
|
||||||
|
- Tests: `SessionEventDistributorTests` (replay) must stay green — replay-from-sequence, gap detection, capacity overflow eviction, and age eviction; add a wraparound test crossing the ring boundary.
|
||||||
|
- Docs: none (internal).
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionEventDistributor"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-15 — Per-event gauge reads `ChannelReader.Count` every event `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `EventStreamService` reconciles the queue-depth gauge by reading `subscriber.Reader.Count` on every event (`Grpc/EventStreamService.cs:173-179`). Bounded-channel `Count` acquires the channel's internal lock; combined with the metric adjustment this adds measurable per-event overhead at high rates.
|
||||||
|
|
||||||
|
**Impact.** Minor per-event cost on the streaming hot path.
|
||||||
|
|
||||||
|
**Design.** Sample the backlog periodically instead of per event — e.g. every N events (a cheap counter modulo) or on a time interval — and reconcile the gauge then. The gauge is a diagnostic backlog indicator, not an exact-per-event quantity, so coarser sampling is acceptable. Keep the terminal reconcile in the `finally` (`:189-193`) so the gauge returns to zero on disconnect. Part of the P2 hot-path pass.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Grpc/EventStreamService.cs:173-179`: gate the `Count`/`AdjustGrpcEventStreamQueueDepth` reconciliation behind `if ((++counter % SampleEvery) == 0)`; retain the `finally` zeroing.
|
||||||
|
- Tests: existing streaming tests confirm the gauge zeroes on disconnect; add a case asserting the gauge is still reconciled at least at stream end.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~EventStream"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-16 — `Invoke` resolves the session twice per command `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `Invoke` calls `ResolveSession(request.SessionId)` (`Grpc/MxAccessGatewayService.cs:104`) then `sessionManager.InvokeAsync(request.SessionId, …)` (`:122-124`) → `GetRequiredSession` (`Sessions/SessionManager.cs:161`) — a redundant `ConcurrentDictionary` lookup on every command.
|
||||||
|
|
||||||
|
**Impact.** Small but per-command overhead.
|
||||||
|
|
||||||
|
**Design.** Add an `ISessionManager.InvokeAsync(GatewaySession session, WorkerCommand, ct)` overload (or pass the already-resolved session through) so the service reuses the session it resolved for constraint application. Keep the existing `InvokeAsync(sessionId, …)` overload for other callers. No behavior change.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/SessionManager.cs`: add the session-typed overload; the existing one resolves then delegates to it.
|
||||||
|
- `Grpc/MxAccessGatewayService.cs:122-124`: pass `session`.
|
||||||
|
- Tests: `SessionManagerTests` — assert both overloads behave identically; existing `MxAccessGatewayService` invoke tests stay green.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionManager"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-17 — Sessions layer throws `Grpc.Core.RpcException` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `SparseArrayExpander.Invalid` creates `new RpcException(new Status(StatusCode.InvalidArgument, …))` (`Sessions/SparseArrayExpander.cs:283-284`), invoked from `GatewaySession.NormalizeOutboundCommand` (`Sessions/GatewaySession.cs:984-1063`), so `GatewaySession.InvokeAsync` — a transport-agnostic session API also used by the alarm monitor — throws a gRPC type. `gateway.md:1063-1065` places translation at the gRPC layer.
|
||||||
|
|
||||||
|
**Impact.** Layering leak: a non-gRPC caller (alarm monitor, tests) receives a gRPC exception type; convention deviation from the style guide's layer boundaries.
|
||||||
|
|
||||||
|
**Design.** Throw a domain exception from the expander/session — reuse `SessionManagerException` with a `SessionManagerErrorCode` that maps to `InvalidArgument` (the codebase already routes these through `MxAccessGatewayService.MapException`). Add the mapping so the gRPC surface still returns `InvalidArgument`. Coordinate with GWC-03, which also adds an `InvalidArgument` rejection through the same path — both should use the domain exception. Verify no caller currently catches `RpcException` specifically from this path (the `Invoke` catch already rethrows non-`RpcException` via `MapException` at `:135`, so domain exceptions are handled).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/SparseArrayExpander.cs:283-284`: return a domain exception (e.g. `SessionManagerException(SessionManagerErrorCode.InvalidArgument, message)`); rename `Invalid` accordingly. Add the error code if absent.
|
||||||
|
- `Grpc/MxAccessGatewayService.cs` `MapException`: ensure the new code maps to `StatusCode.InvalidArgument`.
|
||||||
|
- Tests: `SparseArrayExpanderTests` assert the domain exception type; `MxAccessGatewayService`/mapping tests assert it surfaces as `InvalidArgument` over gRPC.
|
||||||
|
- Docs: confirm `gateway.md:1063-1065` layering statement now holds.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SparseArrayExpander"` and `--filter "FullyQualifiedName~MxAccessGatewayService"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-18 — `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `public sealed class GatewaySession` declares no interfaces (`Sessions/GatewaySession.cs:13`) but exposes a public `async ValueTask DisposeAsync()` (`:1672`). `await using` does not compile against the type; disposal is discoverable only by convention.
|
||||||
|
|
||||||
|
**Impact.** Convention deviation; latent maintainability risk.
|
||||||
|
|
||||||
|
**Design.** Declare `: IAsyncDisposable` on the class. The method already matches the interface signature, so this is a declaration-only change with no behavior impact.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/GatewaySession.cs:13`: `public sealed class GatewaySession : IAsyncDisposable`.
|
||||||
|
- Tests: existing lifecycle tests compile/run unchanged; optionally add an `await using` usage in a test.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-19 — Dead `GatewaySession.KillWorker(string)` + stale doc `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `GatewaySession.KillWorker(string)` (`Sessions/GatewaySession.cs:1615-1619`) has no callers (grep across `src/` and `clients/` finds none; the gated `KillWorkerWithCloseGateAsync` is what `SessionManager.KillWorkerAsync` uses). `docs/Sessions.md:57` still states `KillWorkerAsync` "calls `GatewaySession.KillWorker` directly".
|
||||||
|
|
||||||
|
**Impact.** Dead public method and a doc that describes a non-existent call path — violates the repo's docs-with-source rule.
|
||||||
|
|
||||||
|
**Design.** Delete the dead method and correct the doc in the same commit, per the repo convention. Confirm no reflection/DI reference exists (none found — it is a plain public method). This also removes one of the lock-free `_workerClient` reads flagged in GWC-11.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sessions/GatewaySession.cs:1615-1619`: delete `KillWorker`.
|
||||||
|
- `docs/Sessions.md:57`: reword to describe `KillWorkerWithCloseGateAsync` (the actual forceful path).
|
||||||
|
- Tests: none reference it; a full build confirms no dangling call site.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-20 — Heartbeat config semantics conflated (send vs check interval) `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `WorkerOptions.HeartbeatIntervalSeconds` is documented as "the interval in seconds for worker heartbeats" (`Configuration/WorkerOptions.cs:30-31`, default 5) but is bound to the gateway-side `HeartbeatCheckInterval` (`Sessions/SessionWorkerClientFactory.cs:86`), whose own default is 1 s (`Workers/WorkerClientOptions.cs:10`). Production checks every 5 s while unit-constructed clients check every 1 s, and the option name does not describe what it controls. Separately, `HeartbeatLoopAsync` uses raw `Task.Delay` (`Workers/WorkerClient.cs:400`) rather than the injected `TimeProvider`, unlike the rest of the class.
|
||||||
|
|
||||||
|
**Impact.** Misleading configuration surface; the option name implies the worker's *send* cadence but controls the gateway's *check* cadence. The raw `Task.Delay` makes the heartbeat loop the one loop not virtualizable under `TimeProvider` for tests.
|
||||||
|
|
||||||
|
**Design.** Rename the option to describe what it controls — add `HeartbeatCheckIntervalSeconds` (and correct the XML doc), keeping the old key as a deprecated alias for one release if config compatibility matters, or rename outright since this is pre-1.0. Fix the doc comment to state it is the gateway-side check interval. Separately, route `HeartbeatLoopAsync`'s delay through `_timeProvider` (e.g. a `PeriodicTimer` from `TimeProvider.CreateTimer` or `Task.Delay(interval, _timeProvider, token)`) for consistency and testability.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/WorkerOptions.cs:30-31`: rename to `HeartbeatCheckIntervalSeconds`; fix the XML doc.
|
||||||
|
- `Sessions/SessionWorkerClientFactory.cs:86`: bind the renamed option.
|
||||||
|
- `Workers/WorkerClient.cs:400`: use the injected `TimeProvider` for the loop delay.
|
||||||
|
- Tests: `WorkerClientTests` heartbeat-watchdog tests should drive the loop via `FakeTimeProvider` once the delay is virtualized; assert the check-interval semantics.
|
||||||
|
- Docs: `docs/GatewayConfiguration.md` (key rename) and any heartbeat description in `docs/MxAccessWorkerInstanceDesign.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"` and `--filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-21 — `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `SessionWorkerClientFactory` populates only four of the six `WorkerClientOptions` fields (`Sessions/SessionWorkerClientFactory.cs:83-89`); `EventChannelFullModeTimeout` and `HeartbeatStuckCeiling` always use the hardcoded defaults (`Workers/WorkerClientOptions.cs:13,23`) and have no `WorkerOptions` counterparts.
|
||||||
|
|
||||||
|
**Impact.** Two operationally significant tunables (event-backlog fault window and the stuck-command watchdog ceiling) cannot be adjusted without a rebuild.
|
||||||
|
|
||||||
|
**Design.** Expose both under `MxGateway:Worker:*` (`EventChannelFullModeTimeoutMilliseconds`, `HeartbeatStuckCeilingSeconds`) with defaults matching the current constants (5000 ms, 75 s), bind them in the factory, and validate ranges in `GatewayOptionsValidator`. Coordinate with GWC-04: once the event enqueue is decoupled from the read loop, `EventChannelFullModeTimeout` becomes the fault window for the dedicated event writer task, and making it configurable is the natural companion. Alternatively, if the values should stay fixed, document them as non-configurable in `docs/GatewayConfiguration.md` — but given GWC-04, exposing them is preferred.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/WorkerOptions.cs`: add the two options with defaults.
|
||||||
|
- `Configuration/GatewayOptionsValidator.cs`: validate `> 0` (ceiling `>=` grace, matching the watchdog logic).
|
||||||
|
- `Sessions/SessionWorkerClientFactory.cs:83-89`: bind both.
|
||||||
|
- Tests: `GatewayOptionsValidatorTests` for the new bounds; `WorkerClientTests` confirming the configured timeout drives the overflow fault.
|
||||||
|
- Docs: `docs/GatewayConfiguration.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewayOptionsValidator"` and `--filter "FullyQualifiedName~WorkerClient"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-22 — `StreamDisconnected` always labeled `"Detached"` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** The single `metrics.StreamDisconnected(...)` call site hardcodes `"Detached"` in the `finally` block (`Grpc/EventStreamService.cs:195`); the fault/overflow paths above it (`:148-158`) do not differentiate the label.
|
||||||
|
|
||||||
|
**Impact.** The disconnect-reason dimension the metric implies is uninformative for diagnosing overflow-vs-fault-vs-client-cancel.
|
||||||
|
|
||||||
|
**Design.** Record the actual terminal cause. Track a terminal-reason variable in the stream method, set it to `worker-fault` in the `WorkerClientException` catch (`:148-158`), `canceled` on `OperationCanceledException` tied to the caller's token, `overflow` if the subscriber channel completed via overflow, and default `detached` for a clean end; pass it to `StreamDisconnected` in the `finally`. No new metric — only the existing label dimension becomes meaningful.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Grpc/EventStreamService.cs`: introduce a `string terminalReason = "detached";` updated on each terminal branch; use it at `:195`.
|
||||||
|
- Tests: `EventStreamService` tests asserting the label reflects fault vs cancel vs clean end.
|
||||||
|
- Docs: `docs/Metrics.md` (or wherever the metric's label values are enumerated) if the reason set is documented — note this is in the metrics domain; coordinate with that owner.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~EventStream"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## GWC-23 — `MaxEventSubscribersPerSession` dead in default single-subscriber mode `Info` · `—`
|
||||||
|
|
||||||
|
**Finding.** `MaxEventSubscribersPerSession` is a knowingly dead knob under the default single-subscriber configuration; this is acknowledged and justified in a comment (`Configuration/GatewayOptionsValidator.cs:189-196`).
|
||||||
|
|
||||||
|
**Impact / action.** None. Recorded so it is not re-reported. No change proposed — the option becomes live when `AllowMultipleEventSubscribers` is enabled, and the validator comment already explains the intent. Leave as-is.
|
||||||
|
|
||||||
|
**Verification.** N/A (no change).
|
||||||
@@ -0,0 +1,335 @@
|
|||||||
|
# Worker Process — Remediation Design & Implementation
|
||||||
|
|
||||||
|
Source review: [20-worker.md](../20-worker.md) · Generated: 2026-07-09
|
||||||
|
|
||||||
|
The worker's STA pump and COM-teardown discipline are correct; the risk is concentrated at the IPC edges — a long `ReadBulk` that self-faults, silent thread/command deaths, and a wire-sequence race — plus a small event hot-path allocation cluster. Every fix below stays on the STA for COM work and respects net48 constraints (no init-only properties, no positional records). The worker builds and tests only on the Windows x86 host, so verification commands assume that host.
|
||||||
|
|
||||||
|
## Finding index
|
||||||
|
|
||||||
|
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||||
|
|----|-----|-------|---------|--------|-----------|-------|
|
||||||
|
| WRK-01 | High | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped | P0 | M | — | MxAccess/MxAccessSession.cs, Sta/StaRuntime.cs, Ipc/WorkerPipeSession.cs |
|
||||||
|
| WRK-02 | Medium | STA thread death after startup is silent; future work hangs forever | — | M | — | Sta/StaRuntime.cs |
|
||||||
|
| WRK-03 | Medium | Commands after shutdown starts are dropped with no reply | — | S | WRK-02 | Ipc/WorkerPipeSession.cs |
|
||||||
|
| WRK-04 | Medium | Envelope `sequence` can appear out of order on the wire | — | M | — | Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs |
|
||||||
|
| WRK-05 | Medium | One transient alarm-poll failure kills the whole session | — | M | — | MxAccess/MxAccessStaSession.cs, MxAccess/AlarmCommandHandler.cs |
|
||||||
|
| WRK-06 | Medium | `MXSTATUS_PROXY` conversion reflects per field, per event | P2 | S | — | Conversion/MxStatusProxyConverter.cs |
|
||||||
|
| WRK-07 | Medium | Documented outbound write priority not implemented | P1 | M | WRK-04 | Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs |
|
||||||
|
| WRK-08 | Low | Residual event queue discarded at graceful shutdown, undocumented | — | S | — | Ipc/WorkerPipeSession.cs, docs/MxAccessWorkerInstanceDesign.md |
|
||||||
|
| WRK-09 | Low | No `AppDomain.UnhandledException` hook | — | S | — | WorkerApplication.cs, Program.cs |
|
||||||
|
| WRK-10 | Low | Top-level catch logs exception type but never message | — | S | — | WorkerApplication.cs, Conversion/HResultConverter.cs |
|
||||||
|
| WRK-11 | Low | Every accepted event is defensively cloned on enqueue | P2 | S | — | MxAccess/MxAccessEventQueue.cs |
|
||||||
|
| WRK-12 | Low | No event batching per envelope; one flush per event; 25 ms poll | P2 | M | WRK-04 | Ipc/WorkerPipeSession.cs, Ipc/WorkerFrameWriter.cs |
|
||||||
|
| WRK-13 | Low | Frame writer allocates a fresh buffer per frame; reader pools | P2 | S | — | Ipc/WorkerFrameWriter.cs |
|
||||||
|
| WRK-14 | Low | Private-field naming split `_camelCase` vs `camelCase` | — | M | — | Ipc/*, Bootstrap/*, Sta/*, MxAccess/* |
|
||||||
|
| WRK-15 | Low | Doc drift: STA thread name and heartbeat-counter note | P2 | S | — | docs/WorkerSta.md, docs/MxAccessWorkerInstanceDesign.md |
|
||||||
|
| WRK-16 | Low | Boilerplate duplication in IPC envelope/ctor overloads | — | S | — | Ipc/WorkerPipeSession.cs, Ipc/WorkerPipeClient.cs |
|
||||||
|
| WRK-17 | Low | Gateway death exits with wrong exit code (6 not 5) | — | S | — | WorkerApplication.cs |
|
||||||
|
| WRK-18 | Low | Event-queue overflow exits as generic `UnexpectedFailure` | — | S | — | Ipc/WorkerPipeSession.cs, WorkerApplication.cs, docs/MxAccessWorkerInstanceDesign.md |
|
||||||
|
| WRK-19 | Low | Command start/end logging with correlation id absent | — | S | — | Sta/StaCommandDispatcher.cs |
|
||||||
|
| WRK-20 | Low | Test-coverage gaps for the failure modes above | — | M | WRK-01..04 | Worker.Tests/Ipc, Worker.Tests/Sta |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-01 — Long `ReadBulk` self-faults as `StaHung`; all replies then dropped `High` · `P0`
|
||||||
|
|
||||||
|
**Finding.** `ReadOneTag` reads uncached tags one at a time, waiting up to `timeout` per tag via `valueCache.TryWaitForUpdate(..., pumpStep, ...)` (`MxAccess/MxAccessSession.cs:876-889`, `918-931`). The `pumpStep` handed down is `StaRuntime.PumpPendingMessages()`, which pumps Windows messages but never refreshes the activity timestamp (`Sta/StaRuntime.cs:83-90`; `MarkActivity` is private, `Sta/StaRuntime.cs:304-307`). The watchdog suppresses `StaHung` only while `staleFor <= HeartbeatStuckCeiling` (default 75 s); past the ceiling it faults even with a command in flight (`Ipc/WorkerPipeSession.cs:830-860`, `Ipc/WorkerPipeSessionOptions.cs`). Once faulted, every completed reply is dropped at the `_state != Ready` gate (`Ipc/WorkerPipeSession.cs:604-607`).
|
||||||
|
|
||||||
|
**Impact.** A legitimate `ReadBulk` with `timeout_ms=5000` over ~20 unreachable tags holds the STA ~100 s with `LastActivityUtc` frozen. At 75 s the worker emits `StaHung`, sets `_state = Faulted`, and thereafter silently drops every command reply, so the gateway kills a healthy session. This is the one path where a healthy worker declares itself hung. Also called out as P0 item 4 in the roadmap.
|
||||||
|
|
||||||
|
**Design.** The design doc's watchdog contract assumes "no legitimate STA command should run that long without periodically refreshing activity" (`docs/MxAccessWorkerInstanceDesign.md:688-690`) — but no refresh mechanism exists. The minimal, correct fix is to make the pump step used by long-running STA commands *also* refresh activity: have `StaRuntime.PumpPendingMessages()` call `MarkActivity()` after pumping. Because the pump step is invoked on every wait iteration inside `TryWaitForUpdate` while the STA legitimately holds the thread, activity stays fresh for the whole in-flight command; the moment the command stops pumping (a genuine hang) staleness accrues and the watchdog still fires correctly. This preserves the watchdog's purpose (detect a *stuck* STA) while removing the false positive (a *busy* STA).
|
||||||
|
|
||||||
|
Rejected alternatives: (a) clamping total `ReadBulk` duration below `HeartbeatStuckCeiling` — changes MXAccess parity (per-tag `timeout` is the contract) and still breaks for large tag counts; (b) exposing a public activity-refresh hook threaded through every executor — larger surface, easy to forget on new commands. Refreshing inside the shared `pumpStep` covers all current and future long-running STA commands in one place. Parity is unaffected: no MXAccess behavior changes, only the local liveness signal.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sta/StaRuntime.cs`: in `PumpPendingMessages()`, capture the pump count, call `MarkActivity()`, return the count. (`MarkActivity` is already private on the same class — no visibility change.)
|
||||||
|
- Confirm every long-hold command routes its wait through this method: `ReadBulk` does (`MxAccessSession.cs:930` passes `pumpStep`, wired from `StaRuntime.PumpPendingMessages` at the executor). No new config.
|
||||||
|
- Optional defense-in-depth: document in `docs/GatewayConfiguration.md` that operators running very long bulk ops may raise `HeartbeatStuckCeiling` (already the doc's stated escape hatch, `docs/MxAccessWorkerInstanceDesign.md:689-690`).
|
||||||
|
- Tests: add `StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity`, and an integration-style test in `Worker.Tests/Ipc/WorkerPipeSessionTests` using the fake runtime to prove a >75 s simulated in-flight command that keeps pumping does not emit `StaHung` and its reply is delivered.
|
||||||
|
- Docs: adjust `docs/MxAccessWorkerInstanceDesign.md:688-690` to state that the pump step refreshes activity, so the "no legitimate command runs that long" caveat is now enforced mechanically rather than assumed.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` then `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests|FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-02 — STA thread death after startup is silent; future work hangs forever `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `ThreadMain` catches any loop exception into the write-only `startupException` field and sets `startedEvent` (`Sta/StaRuntime.cs:255-259`). After `Start()` has returned, nothing observes that field: the exception is never logged, never turned into a `WorkerFault`, and `shutdownRequested` stays false, so `InvokeAsync` keeps enqueuing work into a queue with no consumer and returns tasks that never complete (`Sta/StaRuntime.cs:165-179`). The only in-loop throw site is a pump-wait failure (`Sta/StaMessagePump.cs:38-42`).
|
||||||
|
|
||||||
|
**Impact.** If `MsgWaitForMultipleObjectsEx` returns `WAIT_FAILED` once, the STA exits; the dispatcher's drain wedges on the first stuck `InvokeAsync`; heartbeats keep flowing with a frozen `LastStaActivityUtc` and a pinned `CurrentCommandCorrelationId`; the worker is only declared hung after the 75 s ceiling, and the true root cause is permanently lost. Matches cross-cutting theme 1 (silent failure modes) in `00-overall.md`.
|
||||||
|
|
||||||
|
**Design.** Post-startup thread death must (1) fail all queued and future `InvokeAsync` calls deterministically and (2) surface the cause. Introduce a `terminalException` field set in the `ThreadMain` `catch`/`finally`, and a `Faulted` state distinct from graceful shutdown. In the `finally`, after `CancelQueuedCommands()`, transition to a terminal-faulted state so `InvokeAsync` returns `Task.FromException` (using the captured exception) for all subsequent calls, and `CancelQueuedCommands` completes already-queued items faulted rather than cancelled when the exit was abnormal. Add an optional `Action<Exception>? onTerminalFault` callback (constructor-injected, defaulted null) that `MxAccessStaSession` wires to record a `WorkerFault` on the event queue — reusing the existing fault→drain→IPC path (`MxAccessStaSession.RecordFault`, `Ipc/WorkerPipeSession.cs:344-353`) so the gateway sees a real fault frame instead of a slow watchdog timeout. Keep the callback optional to avoid disturbing the many unit tests that construct `StaRuntime` directly.
|
||||||
|
|
||||||
|
Rejected: polling `IsRunning` from the dispatcher — racy and still loses the exception. A push callback plus terminal task-completion is deterministic.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Sta/StaRuntime.cs`: add `private Exception? terminalException;` and a `volatile bool terminated;` (net48 — plain field, no init-only). In `ThreadMain`'s `catch`, store the exception; in `finally`, set `terminated = true` before `stoppedEvent.Set()`. In `InvokeAsync<T>`, after the `shutdownRequested` check, if `terminated` return `Task.FromException<T>(terminalException ?? new StaRuntimeShutdownException())`. Make `CancelQueuedCommands` fault (not cancel) items when `terminalException is not null`.
|
||||||
|
- Add constructor param `Action<Exception>? onTerminalFault = null`; invoke it once from `finally` when `terminalException is not null`.
|
||||||
|
- `MxAccess/MxAccessStaSession.cs`: pass a callback that calls `eventQueue.RecordFault(...)` with category `StaHung` (or a new `StaTerminated` if the contract enum allows — otherwise reuse `StaHung` and set a descriptive `DiagnosticMessage`).
|
||||||
|
- Tests: `Worker.Tests/Sta/StaRuntimeTests` — inject a message pump stub that throws on `WaitForWorkOrMessages`; assert the terminal callback fires, `InvokeAsync` after death returns a faulted task, and queued items fault.
|
||||||
|
- Docs: note in `docs/WorkerSta.md` that abnormal STA exit faults pending/future work and emits a fault frame.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` then `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-03 — Commands after shutdown starts are dropped with no reply `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `TryStartCommandTask` returns silently when `_acceptingCommands` is false (`Ipc/WorkerPipeSession.cs:690-707`): no `WorkerUnavailable` reply, not even the `LogCommandResultDropped` diagnostic (which fires only for completed-then-dropped replies, `Ipc/WorkerPipeSession.cs:604-607`, `645-655`). The dispatcher layer *does* reply `WorkerUnavailable` when it is the one shutting down (`Sta/StaCommandDispatcher.cs:117-123`), but this earlier gate short-circuits before reaching it.
|
||||||
|
|
||||||
|
**Impact.** A command racing `WorkerShutdown` leaves the gateway's correlation wait to expire on its own timeout with no trace, exactly the silent-drop pattern `docs/MxAccessWorkerInstanceDesign.md:697-699` ("reject new commands") intends to avoid.
|
||||||
|
|
||||||
|
**Design.** Make the gate *loud*: when `_acceptingCommands` is false, write a `WorkerCommandReply` carrying `ProtocolStatusCode.WorkerUnavailable` for that correlation id (mirroring `StaCommandDispatcher.CreateRejectedReply`), then return. This matches the dispatcher-level rejection and gives the gateway an immediate, correlated failure. Guard the write with the same `_state`/`TryWriteFault`-style tolerance so a half-closed pipe during shutdown does not throw. Co-designed with WRK-02 (both close silent-death holes).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Ipc/WorkerPipeSession.cs`: in the `!_acceptingCommands` branch of `TryStartCommandTask`, build a rejection reply (reuse a small helper `CreateRejectedCommandReply(correlationId, method, WorkerUnavailable, "Worker is shutting down.")`) and enqueue a best-effort `_writer.WriteAsync(CreateEnvelope(reply), ...)` via a fire-and-forget observed task (same pattern as `ObserveCommandTaskAsync`). At minimum, log a `WorkerCommandRefusedDuringShutdown` diagnostic even if the write is skipped.
|
||||||
|
- Tests: `Worker.Tests/Ipc/WorkerPipeSessionTests` — after triggering shutdown, feed a command envelope and assert a `WorkerUnavailable` reply (or the diagnostic) is observed on the fake writer; covers the WRK-20 gap for S3.
|
||||||
|
- Docs: none beyond the existing shutdown sequence, which already promises rejection.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-04 — Envelope `sequence` can appear out of order on the wire `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `NextSequence()` is called while *building* the envelope in `CreateBaseEnvelope()` (`Ipc/WorkerPipeSession.cs:1005-1018`), but the write lock is acquired later, inside `WorkerFrameWriter.WriteAsync` (`Ipc/WorkerFrameWriter.cs:68-77`). Command replies are written from independent per-command tasks (`Ipc/WorkerPipeSession.cs:690-706`) concurrently with the heartbeat and event-drain loops, so task B can take sequence n+1 yet win the write lock before task A's sequence n. (Per-event ordering is safe — `WorkerSequence` is stamped inside the queue lock, `MxAccess/MxAccessEventQueue.cs:135-143` — but the envelope-level guarantee is not.)
|
||||||
|
|
||||||
|
**Impact.** Violates the `gateway.md` envelope rule "`sequence` is monotonic per sender"; any gateway consumer trusting wire-order monotonicity mis-sorts frames.
|
||||||
|
|
||||||
|
**Design.** Assign the envelope sequence *inside* the writer's critical section so the number and the write are atomic. Change `WorkerFrameWriter.WriteAsync` to accept a sequence-stamping callback (`Action<WorkerEnvelope>` or a `Func<ulong>` that the writer invokes under `_writeLock` to set `envelope.Sequence`) rather than reading a pre-stamped value. `CreateBaseEnvelope` stops calling `NextSequence()`; the writer stamps `Sequence` immediately before serialization, under the lock. This keeps a single `_nextSequence` counter and removes the window entirely. Co-designed with WRK-07 (both restructure the write path) — sequence stamping and priority scheduling should land together to avoid two passes over the writer.
|
||||||
|
|
||||||
|
Rejected: making `NextSequence` `Interlocked` (already is) does not help — the reordering is between *assignment* and *write*, not in the increment. The atomic region must span both.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Ipc/WorkerFrameWriter.cs`: add an overload `WriteAsync(WorkerEnvelope envelope, Func<ulong> sequenceProvider, CancellationToken)` that, after acquiring `_writeLock`, sets `envelope.Sequence = sequenceProvider()` *before* `Validate`/`CalculateSize`/`WriteTo`. (Serialize inside the lock now, since the payload depends on the stamped sequence.)
|
||||||
|
- `Ipc/WorkerPipeSession.cs`: `CreateBaseEnvelope()` no longer sets `Sequence`; every `_writer.WriteAsync(CreateEnvelope(...), ...)` call site passes `NextSequence` as the provider. Keep `NextSequence()` on the session (single owner of `_nextSequence`).
|
||||||
|
- Tests: `Worker.Tests/Ipc/WorkerPipeSessionTests` — a concurrent-writer test that fires N reply writes and M event writes in parallel through a capturing stream and asserts observed `Sequence` values are strictly monotonic in wire order (covers WRK-20 gap for S4). A `WorkerFrameWriterTests` unit test that the provider runs under the lock.
|
||||||
|
- Docs: none — restores the documented invariant.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-05 — One transient alarm-poll failure kills the whole session `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** Any exception from the alarm poll loop is recorded as a fault on the shared event queue and permanently stops the loop (`MxAccess/MxAccessStaSession.cs:278-291`); the drain loop turns that fault into full session termination (`Ipc/WorkerPipeSession.cs:344-353`). Failover only absorbs primary failures in composite mode (`MxAccess/FailoverAlarmConsumer.cs`); the default `WorkerPipeSession` builds the alarm handler with `standbyFactory: null` (`Ipc/WorkerPipeSession.cs:54`), so `AlarmCommandHandler.BuildConsumer` returns a bare consumer and a single `GetXmlCurrentAlarms2` COM error propagates unwrapped.
|
||||||
|
|
||||||
|
**Impact.** One transient `E_FAIL` from the AVEVA alarm subsystem terminates a client's healthy `OnDataChange` data stream even though the data path never failed — a whole-session death from an isolated alarm blip.
|
||||||
|
|
||||||
|
**Design.** Two viable scopes; recommend the smaller, parity-safe one:
|
||||||
|
|
||||||
|
1. **Recommended — bounded retry before declaring the alarm subscription dead.** Count *consecutive* poll failures against a threshold (mirror `FailoverSettings.Threshold`, default e.g. 3) with a short back-off; reset the counter on any successful poll. Only after the threshold record the fault. This tolerates transient COM errors without touching the data path and matches the existing failover semantics.
|
||||||
|
2. Scope the fault to the alarm feature (stop alarm delivery, keep data subscriptions and the session alive). Larger change — requires a per-feature fault channel the drain loop does not currently model, and risks masking a genuinely dead provider. Defer.
|
||||||
|
|
||||||
|
Parity note: this does not synthesize or suppress alarm *events* — it only changes how many consecutive *poll infrastructure* failures constitute "the subscription is dead." A persistent failure still faults the session, preserving fail-fast.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxAccess/MxAccessStaSession.cs`: in the poll loop, maintain `int consecutiveFailures`; on catch, increment and only call `eventQueue.RecordFault(...)` + `return` when `consecutiveFailures >= threshold`; otherwise log a warning, `await Task.Delay(backoff)`, and continue. Reset on success. Keep the STA-affinity `InvalidOperationException` (from `EnsureOnAlarmConsumerThread`) as an *immediate* fault (it is a programming-error regression, not transient) — distinguish it before the counting branch.
|
||||||
|
- Config: add `AlarmPollFailureThreshold` and `AlarmPollBackoff` to the alarm handler options (thread through `AlarmCommandHandler`); default threshold to match `FailoverSettings.Threshold`.
|
||||||
|
- Tests: `Worker.Tests` alarm units — a consumer stub that throws N-1 times then succeeds keeps the session alive; N consecutive throws faults it; an affinity `InvalidOperationException` faults immediately.
|
||||||
|
- Docs: document the threshold in `docs/GatewayConfiguration.md` and note the transient-tolerance behavior in `docs/MxAccessWorkerInstanceDesign.md` alarm section.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` then `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~Alarm"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-06 — `MXSTATUS_PROXY` conversion reflects per field, per event `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `MxStatusProxyConverter.Convert` calls `ReadInt32Field` four times per status (`Conversion/MxStatusProxyConverter.cs:22-26`); each does `Type.GetField` + `FieldInfo.GetValue` + `Convert.ToInt32` (`:83-103`). This runs on the STA event path for every status of every `OnDataChange`.
|
||||||
|
|
||||||
|
**Impact.** Eight reflection ops plus boxing per event under data-change/alarm bursts — the exact load the gateway exists to handle. The status type is always the interop `MXSTATUS_PROXY` struct, so the field lookups are fully cacheable. Roadmap P2 item 14 (event hot-path pass).
|
||||||
|
|
||||||
|
**Design.** Cache the four `FieldInfo` objects keyed by `Type` in a small static `ConcurrentDictionary<Type, (FieldInfo success, category, detectedBy, detail)>`, resolved once per type and reused. `GetValue` + `Convert.ToInt32` still run per event (unavoidable via reflection over a late-bound COM RCW), but the `GetField` metadata scan — the expensive part — is eliminated. A direct cast to the interop struct type is faster still but couples the converter to the interop assembly and its exact struct shape; the review notes the type is stable, but the cached-`FieldInfo` approach keeps the converter interop-agnostic and testable with the existing plain-CLR test doubles. Recommend cached `FieldInfo`.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Conversion/MxStatusProxyConverter.cs`: add a static cache; a private `GetFields(Type)` that populates it once (throwing the same `MxStatusConversionException` if a field is missing) and `Convert` reads the four cached `FieldInfo`s. Behavior and exceptions unchanged.
|
||||||
|
- Tests: existing `MxStatusProxyConverter` tests must still pass (same outputs); add one asserting two conversions of the same type reuse cached metadata (e.g. via a type whose `GetField` is instrumented, or simply a throughput/regression test).
|
||||||
|
- Docs: none (internal perf).
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxStatusProxyConverter"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-07 — Documented outbound write priority not implemented `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** `docs/MxAccessWorkerInstanceDesign.md:600-613` specifies write priority faults > command replies > shutdown acks > heartbeats > events; the worker has no prioritized queue — all writers contend on a single FIFO `SemaphoreSlim` in `WorkerFrameWriter` (`Ipc/WorkerFrameWriter.cs:14`, `68-77`), and events are written inline by the drain loop (`Ipc/WorkerPipeSession.cs:362-367`).
|
||||||
|
|
||||||
|
**Impact.** With a deep event backlog draining (up to 128 per batch), a `WorkerFault` or command reply queues behind those event writes; on a slow pipe this delays the gateway's fault reaction. Relates to roadmap P1 item 8 (backpressure/size topology).
|
||||||
|
|
||||||
|
**Design.** Two acceptable outcomes; the review explicitly allows either. Recommend the *documented-decision* path first, with a small scheduler as the follow-up:
|
||||||
|
|
||||||
|
1. **Minimal now:** amend the design doc to state that FIFO write ordering was accepted for v1 (the single `_writeLock` serializes but does not prioritize), and that event batching (WRK-12) plus the sequence fix (WRK-04) bound the worst-case delay. This removes the doc/code contradiction immediately.
|
||||||
|
2. **Full fix (recommended for P1):** introduce a priority write scheduler in `WorkerFrameWriter` — a small set of per-priority queues drained newest-priority-first under `_writeLock`, or a `Channel`-per-priority merged by a single writer task. Faults and replies jump ahead of queued events. Must preserve per-sender sequence monotonicity — co-designed with WRK-04, since sequence is now stamped inside the lock at actual write time, priority reordering before stamping keeps sequence consistent with wire order automatically.
|
||||||
|
|
||||||
|
The full scheduler is the correct end state; if effort is constrained, ship option 1 in the same commit that lands WRK-04/WRK-12 and file option 2.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Ipc/WorkerFrameWriter.cs`: add an internal priority-ordered pending set drained by the lock holder; or expose `WriteAsync(envelope, priority, sequenceProvider, ct)`. `Ipc/WorkerPipeSession.cs` tags each write site with a priority (fault=0 … event=4).
|
||||||
|
- Tests: `Worker.Tests/Ipc/WorkerFrameWriterTests` — enqueue a fault behind many event writes on a blocked stream, unblock, assert the fault frame emerges first while sequences remain monotonic.
|
||||||
|
- Docs: update `docs/MxAccessWorkerInstanceDesign.md:600-613` to describe the implemented scheduler (or the accepted-FIFO decision if option 1).
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-08 — Residual event queue discarded at graceful shutdown, undocumented `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `ShutdownAsync` writes the ack and returns; `RunMessageLoopAsync`'s `finally` cancels the event-drain loop (`Ipc/WorkerPipeSession.cs:287-292`) with whatever remains in `MxAccessEventQueue` unshipped; late replies on `_state != Ready` are dropped (`Ipc/WorkerPipeSession.cs:604-607`). An `OnWriteComplete` raised during cleanup never reaches the gateway. Acceptable for a closing session but not stated in `docs/MxAccessWorkerInstanceDesign.md`.
|
||||||
|
|
||||||
|
**Design.** Cheapest correct action: **document the discard** as intended v1 behavior — a session tearing down does not guarantee delivery of events queued after `WorkerShutdown`. Optionally (if a downstream wants last-gasp events) drain the queue once after `ShutdownGracefullyAsync` returns and before writing the ack, bounded by the grace period. Recommend documenting now; the final drain is a small enhancement to schedule only if a client needs it.
|
||||||
|
|
||||||
|
**Implementation.** Add a paragraph to `docs/MxAccessWorkerInstanceDesign.md` shutdown section. If implementing the drain: in `ShutdownAsync`, after `ShutdownGracefullyAsync`, call `DrainEvents` once and write each before the ack. Test: `WorkerPipeSessionTests` asserting either the documented discard or the final-drain delivery.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-09 — No `AppDomain.UnhandledException` hook `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `Program.cs:1-4` and `WorkerApplication.Run` (`WorkerApplication.cs:46-141`) install no unhandled-exception or unobserved-task handlers (confirmed: no `AppDomain`/`UnhandledException` reference exists anywhere in the worker). An exception on an unobserved thread crashes the process with no `WorkerFault` and no log.
|
||||||
|
|
||||||
|
**Design.** Register `AppDomain.CurrentDomain.UnhandledException` and `TaskScheduler.UnobservedTaskException` at process start, logging the (redacted) exception through `IWorkerLogger` before exit. The gateway still detects death via process exit + pipe closure, so this is diagnostics-only; keep it minimal.
|
||||||
|
|
||||||
|
**Implementation.** In `WorkerApplication.Run` (or a tiny bootstrap in `Program.cs`), before parsing args, subscribe both handlers and log `WorkerUnhandledException` / `WorkerUnobservedTaskException` with `WorkerLogRedactor`-scrubbed message + type. Test: unit test the handler delegate logs and does not throw. Docs: mention in `docs/MxAccessWorkerInstanceDesign.md` logging list.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86`; `dotnet test ...Worker.Tests... -p:Platform=x86`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-10 — Top-level catch logs exception type but never message `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `WorkerApplication.cs:112-139` logs only `exception_type` for protocol, pipe, and unexpected failures; `HResultConverter.CreateSafeDiagnosticMessage` reduces every command exception to `Type: HRESULT 0x…` (`Conversion/HResultConverter.cs:46-49`).
|
||||||
|
|
||||||
|
**Design.** Log `exception.Message` (routed through `Bootstrap/WorkerLogRedactor` — which already scrubs nonces/credentials, `WorkerLogRedactor.cs:16-25`) alongside the type at the *process boundary*. Keep the credential-safe reply shape for IPC *replies* unchanged if the HRESULT-only stripping is intentional parity/secret policy — the fix is scoped to worker stderr/log, not the wire reply.
|
||||||
|
|
||||||
|
**Implementation.** In the three `WorkerApplication.Run` catch blocks, add `["exception_message"] = WorkerLogRedactor.Redact(exception.Message)`. Leave `HResultConverter` reply text as-is (document the intent in a code comment). Test: `WorkerApplicationTests` (or add) asserting the redacted message is present. Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-11 — Every accepted event is defensively cloned on enqueue `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `MxAccessEventQueue.Enqueue` calls `mxEvent.Clone()` (`MxAccess/MxAccessEventQueue.cs:135`) for an event the mapper built exclusively for this call (`MxAccess/MxAccessBaseEventSink.cs:210-256`); only the value-cache post-publish shares the original.
|
||||||
|
|
||||||
|
**Design.** Take ownership of the passed event in the queue (stamp `WorkerSequence`/`WorkerTimestamp` on it directly) and let the value cache store the copy — the cache already snapshots only value/quality/timestamp/statuses (`MxAccess/MxAccessValueCache.cs:44-57`), so it does not need the full `MxEvent` alias. This halves protobuf allocation on the hottest path. Requires confirming no caller reuses the passed `MxEvent` after enqueue (the mapper builds a fresh one per event — safe). Roadmap P2 item 14.
|
||||||
|
|
||||||
|
**Implementation.** Remove the `Clone()`; mutate the incoming `mxEvent` in place under the queue lock. Audit `MxAccessBaseEventSink` call sites to confirm single-ownership. Tests: existing event-queue tests must still pass; add one asserting the enqueued instance is the same reference passed in and that the value cache's stored copy is independent. Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxAccessEventQueue|FullyQualifiedName~EventSink"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-12 — No event batching per envelope; one flush per event; 25 ms poll `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** The drain loop writes one `WriteAsync` per event (`Ipc/WorkerPipeSession.cs:362-367`), the writer flushes per frame (`Ipc/WorkerFrameWriter.cs:71-72`), and the poll interval is 25 ms (`Ipc/WorkerPipeSession.cs:17`). Each event costs a semaphore round-trip, a pipe write, and a flush; idle-to-active latency up to 25 ms. `gateway.md` lists event batching as the intended optimization.
|
||||||
|
|
||||||
|
**Design.** Acceptable for v1 parity (the review agrees). When throughput matters: either (a) add a repeated-event `WorkerEnvelope` body (a contracts/proto change — coordinate with domain 30 IPC, cross-references `IPC` findings; must preserve per-event order and sequence semantics), or (b) keep one event per envelope but coalesce *flushes* across a drained batch (flush once after writing the batch), which needs no proto change and captures most of the benefit. Recommend (b) now, (a) as a coordinated cross-domain change. Depends on WRK-04 (sequence stamping under the lock) so a coalesced batch keeps monotonic sequences.
|
||||||
|
|
||||||
|
**Implementation.** (Option b) `WorkerFrameWriter`: add a `WriteBatchAsync(IEnumerable<WorkerEnvelope>, ...)` that writes all frames then flushes once; drain loop calls it per drained batch. Tests: `WorkerFrameWriterTests` asserting one flush per batch and correct framing. Docs: note the flush-coalescing in `docs/WorkerFrameProtocol.md`; if option (a), update `.proto` and regenerate per CLAUDE.md contracts rule.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-13 — Frame writer allocates a fresh buffer per frame; reader pools `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `WorkerFrameWriter` does `new byte[frameLength]` per frame (`Ipc/WorkerFrameWriter.cs:63-66`) while `WorkerFrameReader` rents from `ArrayPool<byte>.Shared` (`Ipc/WorkerFrameReader.cs:55-77`).
|
||||||
|
|
||||||
|
**Design.** Rent the write buffer from `ArrayPool<byte>.Shared` for symmetry; return it in a `finally` after the write completes. Because `ArrayPool` may return an oversized buffer, pass explicit `(0, frameLength)` to `WriteAsync` (already does) and never leak the buffer's tail. Trivial, self-contained. Roadmap P2 item 14.
|
||||||
|
|
||||||
|
**Implementation.** `WorkerFrameWriter.WriteAsync`: `byte[] frame = ArrayPool<byte>.Shared.Rent(frameLength);` … `try { … } finally { ArrayPool<byte>.Shared.Return(frame); }`. Note the return must occur after the awaited write completes (it does — inside the same method). Tests: existing `WorkerFrameWriterTests` framing tests cover correctness; no behavior change. Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-14 — Private-field naming split `_camelCase` vs `camelCase` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `Ipc/` and `Bootstrap/` use `_camelCase` (`Ipc/WorkerPipeSession.cs:21-38`, `Ipc/WorkerFrameWriter.cs:13-15`, `Bootstrap/WorkerConsoleLogger.cs:10`); `Sta/` and `MxAccess/` use bare `camelCase` (`Sta/StaRuntime.cs:10-24`, `MxAccess/MxAccessStaSession.cs:16-27`). `docs/style-guides/CSharpStyleGuide.md:30-32` permits the underscore prefix "only when already established" — both are established, so the worker has no single convention.
|
||||||
|
|
||||||
|
**Design.** Pick one style project-wide and migrate opportunistically (not in one churn commit). The gateway server is the tie-breaker: adopt whatever the gateway uses so the whole solution converges. This is a mechanical rename with no behavior change; do it file-by-file as those files are touched for other findings to keep diffs reviewable.
|
||||||
|
|
||||||
|
**Implementation.** Decide the target (check `src/ZB.MOM.WW.MxGateway.Server` private-field style), record it in `docs/style-guides/CSharpStyleGuide.md`, and rename incrementally. Tests: build only. Docs: the style-guide note.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` (analyzers/`TreatWarningsAsErrors` must stay green).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-15 — Doc drift: STA thread name and heartbeat-counter note `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** The STA thread is named `"MxGateway.Worker.STA"` (`Sta/StaRuntime.cs:61`) but `docs/WorkerSta.md:23,30` and `docs/MxAccessWorkerInstanceDesign.md:254` say `ZB.MOM.WW.MxGateway.Worker.STA`. And `docs/MxAccessWorkerInstanceDesign.md:653-654` says event-queue depth and sequence "are reported as zero until the event queue implementation owns those counters," but `CaptureHeartbeat` now populates both from the live queue (`MxAccess/MxAccessStaSession.cs:375-380`).
|
||||||
|
|
||||||
|
**Design.** Docs must match source (CLAUDE.md rule). Choose: either rename the thread to the documented `ZB.MOM.WW.MxGateway.Worker.STA` (operators grep thread dumps for it) or update the docs to the actual name — recommend renaming the thread to the fully-qualified documented name for operability, in the same commit that fixes the heartbeat-counter note. Roadmap P2 item 15 (doc-drift sweep).
|
||||||
|
|
||||||
|
**Implementation.** Either edit `Sta/StaRuntime.cs:61` thread `Name` to `ZB.MOM.WW.MxGateway.Worker.STA`, or edit the two docs. Fix `docs/MxAccessWorkerInstanceDesign.md:653-654` to state the counters are live. Tests: if renaming, update any test asserting the thread name. Docs: as above.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86`; `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntime"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-16 — Boilerplate duplication in IPC envelope/ctor overloads `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Seven near-identical `CreateEnvelope`/`CreateBaseEnvelope` overload pairs (`Ipc/WorkerPipeSession.cs:920-1003`) and eight `WorkerPipeClient` constructor overloads (`Ipc/WorkerPipeClient.cs:36-140`). Maintenance noise — each new body means two more copy-paste methods.
|
||||||
|
|
||||||
|
**Design.** Collapse the envelope overloads to a single `CreateEnvelope(Action<WorkerEnvelope> setBody)` (or a switch on the body message type); keep the correlation-id special case for `WorkerCommandReply` inside the setter. Reduce the client constructors to one primary constructor with the rest chaining via defaulted parameters (net48 supports optional params — no init-only needed). Pure refactor, no behavior change. Fold into WRK-04/WRK-07 since those already rework `CreateBaseEnvelope`.
|
||||||
|
|
||||||
|
**Implementation.** Refactor `WorkerPipeSession` envelope factories and `WorkerPipeClient` ctors. Tests: existing IPC tests must pass unchanged; they are the regression guard. Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSession|FullyQualifiedName~WorkerPipeClient"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-17 — Gateway death exits with wrong exit code (6 not 5) `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Pipe EOF surfaces as `WorkerFrameProtocolException(EndOfStream)` (`Ipc/WorkerFrameReader.cs:104-109`), which `WorkerApplication.Run` catches first and maps to `ProtocolViolation` (6) (`WorkerApplication.cs:110-119`) even though `PipeConnectionFailed` (5) exists (`Bootstrap/WorkerExitCode.cs:10`) and the in-session fault mapping already distinguishes pipe disconnect.
|
||||||
|
|
||||||
|
**Design.** Special-case `WorkerFrameProtocolErrorCode.EndOfStream` in the `WorkerFrameProtocolException` catch to return `PipeConnectionFailed` (5) — EOF means the gateway went away, not that the worker misbehaved. Improves orphan-worker post-mortem triage.
|
||||||
|
|
||||||
|
**Implementation.** In `WorkerApplication.cs:110-119`, branch on `exception.ErrorCode == WorkerFrameProtocolErrorCode.EndOfStream` → log + return `PipeConnectionFailed`; else `ProtocolViolation`. Tests: `WorkerApplicationTests` — inject a pipe client throwing `EndOfStream` and assert exit code 5. Docs: `docs/WorkerFrameProtocol.md` / exit-code table if one exists.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-18 — Event-queue overflow exits as generic `UnexpectedFailure` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `docs/MxAccessWorkerInstanceDesign.md:615-624` says overflow should "stop accepting new commands" and "let the gateway close or kill the worker"; the implementation instead terminates immediately — the drain loop writes the fault then throws (`Ipc/WorkerPipeSession.cs:344-353`), unwinding into the generic handler and exit code 1 (`WorkerApplication.cs:131-139`). The designed fault path is indistinguishable from a crash by exit code.
|
||||||
|
|
||||||
|
**Design.** The implemented fail-fast is arguably stronger than the doc and acceptable; the defect is *observability*. Give overflow a dedicated exit code (e.g. add `EventQueueOverflow` to `WorkerExitCode`) and catch the drain-fault termination in `WorkerApplication` to return it, then update the doc to describe the implemented immediate-terminate policy. Recommend keeping fail-fast (do not weaken to "stop accepting commands") and aligning the doc + exit code to it.
|
||||||
|
|
||||||
|
**Implementation.** Add `WorkerExitCode.EventQueueOverflow`; wrap the drain-fault throw in a typed exception (`WorkerEventQueueOverflowTerminationException`) so `WorkerApplication.Run` can map it. Tests: `WorkerPipeSessionTests` overflow path asserts the dedicated code. Docs: rewrite `docs/MxAccessWorkerInstanceDesign.md:615-624` to match.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests|FullyQualifiedName~WorkerApplication"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-19 — Command start/end logging with correlation id absent `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `docs/MxAccessWorkerInstanceDesign.md:790-791` lists "command start/end with correlation id" among required worker logs; the only per-command log is the dropped-reply diagnostic (`Ipc/WorkerPipeSession.cs:645-655`). `StaCommand.EnqueueTimestamp` is captured (`Sta/StaCommand.cs`) but never used for latency.
|
||||||
|
|
||||||
|
**Design.** Add optional, level-gated start/end logging in `StaCommandDispatcher.ExecuteQueuedCommandAsync`, which already brackets each command (`Sta/StaCommandDispatcher.cs:265-281`). Log correlation id + method at start; at end log outcome + latency (`now - EnqueueTimestamp`). Gate at a verbose/debug level so production noise is opt-in.
|
||||||
|
|
||||||
|
**Implementation.** Inject the optional `IWorkerLogger` into `StaCommandDispatcher` (or pass through the session); emit `WorkerCommandStarted`/`WorkerCommandCompleted` with correlation id, method, latency, outcome. Tests: `StaCommandDispatcherTests` asserting both logs fire with the correlation id. Docs: confirm the log names in `docs/MxAccessWorkerInstanceDesign.md` logging list.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaCommandDispatcher"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## WRK-20 — Test-coverage gaps for the failure modes above `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `Worker.Tests` covers the pump wake behavior, dispatcher ordering/cancellation/shutdown, handshake/heartbeat/watchdog (incl. the stuck ceiling), control commands, shutdown races, late-reply drops, frame protocol, conversion, event queue, and alarm units. Not covered: STA thread death mid-run (WRK-02), wire-level envelope sequence monotonicity under concurrent writers (WRK-04), the silent no-reply drop at the `_acceptingCommands` gate (WRK-03), and the `ReadBulk`-exceeds-ceiling false fault (WRK-01).
|
||||||
|
|
||||||
|
**Design.** Add the four tests *alongside* their fixes (named in each entry above) using the existing fake-runtime harness in `WorkerPipeSessionTests`, which already supports all four. This is not a separate work item so much as the acceptance criterion for WRK-01..04 — tracked here so it is not dropped.
|
||||||
|
|
||||||
|
**Implementation.** Tests to add: `StaRuntimeTests.ThreadDeath_FaultsPendingAndFutureWork` (WRK-02); `WorkerPipeSessionTests.ConcurrentWriters_SequenceIsMonotonic` (WRK-04); `WorkerPipeSessionTests.CommandAfterShutdown_RepliesWorkerUnavailable` (WRK-03); `WorkerPipeSessionTests.LongReadBulk_DoesNotFaultWhilePumping` + `StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity` (WRK-01). Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86` (full worker suite once, after the batch lands, per the targeted-tests-then-phase-suite rule).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cross-references
|
||||||
|
|
||||||
|
- WRK-01 is roadmap P0 item 4; WRK-06/11/12/13 are the worker slice of P2 item 14 (event hot-path); WRK-15 is part of P2 item 15 (doc-drift sweep); WRK-07 relates to P1 item 8 (backpressure/size topology).
|
||||||
|
- WRK-04 and WRK-07 rework the write path together; WRK-16 folds into that refactor. WRK-12 depends on WRK-04's under-lock sequence stamping.
|
||||||
|
- WRK-02 and WRK-03 both close silent-failure edges (cross-cutting theme 1 in `00-overall.md`); the same theme spans gateway and client findings — coordinate the "every death/drop is observable" pattern across domains.
|
||||||
@@ -0,0 +1,409 @@
|
|||||||
|
# Contracts & IPC Protocol — Remediation Design & Implementation
|
||||||
|
|
||||||
|
Source review: [30-contracts-ipc.md](../30-contracts-ipc.md) · Generated: 2026-07-09
|
||||||
|
|
||||||
|
The frame protocol and proto evolution hygiene are sound; the remediable risk clusters in three seams: an unenforced codegen/descriptor freshness gate, an un-negotiated size/backpressure topology that converts legitimate large payloads into whole-session death, and contract-boundary documentation drift. All `path:line` citations below were re-verified against the working tree; corrections are noted inline where the review's lines had shifted.
|
||||||
|
|
||||||
|
## Finding index
|
||||||
|
|
||||||
|
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||||
|
|----|-----|-------|---------|--------|-----------|-------|
|
||||||
|
| IPC-01 | High | Published client descriptor set is 7 weeks stale, nothing enforces freshness | P1 | M | — | clients/proto/descriptors/mxaccessgw-client-v1.protoset, scripts/publish-client-proto-inputs.ps1, src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs |
|
||||||
|
| IPC-02 | Medium | Worker max frame size hard-coded, cannot follow gateway config | P1 | M | IPC-03 | src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs, Contracts/Protos/mxaccess_worker.proto |
|
||||||
|
| IPC-03 | Medium | gRPC max == pipe max with zero headroom; oversized write faults whole session | P1 | M | IPC-02 | src/ZB.MOM.WW.MxGateway.Server/Configuration/ProtocolOptions.cs, Workers/WorkerClient.cs, Workers/WorkerFrameWriter.cs |
|
||||||
|
| IPC-04 | Medium | `DrainEvents max_events=0` packs entire queue into one reply frame | P1 | S | IPC-03 | src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs, Grpc/MxAccessGrpcRequestValidator.cs |
|
||||||
|
| IPC-05 | Medium | Redundant deep copies on command and event hot paths | P2 | S | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs, Grpc/MxAccessGrpcMapper.cs |
|
||||||
|
| IPC-06 | Medium | `gateway.md` Worker Envelope sketch no longer matches the contract | P2 | S | — | gateway.md |
|
||||||
|
| IPC-07 | Medium | `docs/Grpc.md` says six RPCs; there are seven | P2 | S | — | docs/Grpc.md |
|
||||||
|
| IPC-08 | Medium | `WorkerCancel` defined and handled but never sent | — | M | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs, Worker/Ipc/WorkerPipeSession.cs, Contracts/Protos/mxaccess_worker.proto |
|
||||||
|
| IPC-09 | Medium | Known codegen fragilities have no in-repo guards | P1 | M | IPC-01 | clients/python/pyproject.toml, clients/java/.../build.gradle, clients/*/generate-proto.ps1, scripts/publish-client-proto-inputs.ps1 |
|
||||||
|
| IPC-10 | Low | Envelope `sequence` is write-only; monotonicity never validated | — | S | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerEnvelopeValidator.cs, Worker/Ipc/WorkerEnvelopeValidator.cs, gateway.md |
|
||||||
|
| IPC-11 | Low | No protocol version negotiation despite `supported_protocol_version` name | — | S | — | Contracts/Protos/mxaccess_worker.proto, Worker/Ipc/WorkerPipeSession.cs |
|
||||||
|
| IPC-12 | Low | Gateway frame writer has no write lock; integrity rests on an undocumented invariant | — | S | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameWriter.cs |
|
||||||
|
| IPC-13 | Low | Gateway writer serializes twice and issues two stream writes | P2 | S | IPC-05 | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameWriter.cs |
|
||||||
|
| IPC-14 | Low | Gateway reader allocates a fresh array per frame; worker rents from `ArrayPool` | P2 | S | IPC-05 | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameReader.cs |
|
||||||
|
| IPC-15 | Low | Worker event delivery is a 25 ms poll with per-event write+flush, no batching | P2 | M | — | src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs, Contracts/Protos/mxaccess_worker.proto |
|
||||||
|
| IPC-16 | Low | Correlation id carried twice per reply (Info) | — | S | — | Contracts/Protos/mxaccess_worker.proto, mxaccess_gateway.proto |
|
||||||
|
| IPC-17 | Low | Two docs name the wrong Python generated-output directory | P2 | S | — | docs/ClientProtoGeneration.md, CLAUDE.md |
|
||||||
|
| IPC-18 | Low | Generated-code hygiene verified good — preserve under change (positive) | — | S | IPC-01 | src/ZB.MOM.WW.MxGateway.Contracts/Generated/ |
|
||||||
|
| IPC-19 | Low | Generated/-must-be-committed rule for net48 is undocumented and unguarded | P1 | S | IPC-01 | docs/Contracts.md, ZB.MOM.WW.MxGateway.Contracts.csproj |
|
||||||
|
| IPC-20 | Low | Descriptor `-Check` byte-compares source-info-bearing bytes; protoc-version-sensitive | P1 | S | IPC-01 | scripts/publish-client-proto-inputs.ps1 |
|
||||||
|
| IPC-21 | Low | `gateway.md` presents unimplemented `Session` RPC inside the live service block | P2 | S | IPC-06 | gateway.md, Contracts/Protos/mxaccess_gateway.proto |
|
||||||
|
| IPC-22 | Low | Public error-detail model stops at status codes plus prose (Info) | — | S | — | Contracts/Protos/mxaccess_gateway.proto, docs/Grpc.md |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-01 — Published client descriptor set is 7 weeks stale and nothing enforces freshness `High` · `P1`
|
||||||
|
|
||||||
|
**Finding.** `clients/proto/descriptors/mxaccessgw-client-v1.protoset` was last committed 2026-04-30 (`git log` verified: commit `0f88a95`), while `mxaccess_gateway.proto` changed through 2026-06-18 (`MxSparseArray`, commit `8ac9a33`), 2026-06-16 (`ReplayGap`), and 2026-06-15 (alarm provenance). `strings` over the protoset returns zero hits for `MxSparseArray`, `replay_gap`, or `provider_status` — confirmed stale. `docs/Contracts.md:127` mandates regenerating the descriptor after any proto change, `scripts/publish-client-proto-inputs.ps1` has a `-Check` mode (verified at line 3, 70, 88), but no CI workflow exists and `ClientProtoInputTests.cs` validates only manifest versions and path existence (verified lines 12-34: `Manifest_DeclaresCurrentProtocolVersionsAndExistingInputs` checks `schemaVersion`, protocol versions, `File.Exists`, `Directory.Exists` — never descriptor content).
|
||||||
|
|
||||||
|
**Impact.** The artifact documented as the "stable client input" (`docs/ClientProtoGeneration.md:48`) silently misrepresents the contract. Any consumer that prefers a descriptor input generates against a schema missing three shipped features; grpcurl users on the reflection-disabled deployments (per the deployment memory, gRPC reflection is off on both hosts) get a stale schema. This is the domain's single worst concrete defect and appears in the overall roadmap under P1 item 7.
|
||||||
|
|
||||||
|
**Design.** Two parts: (1) regenerate and commit the descriptor now; (2) add an automated freshness gate so the doc-mandated regeneration cannot be skipped silently again. The gate belongs in the gateway test project (`ClientProtoInputTests`) rather than only in the `-Check` script, because the test runs in the NonWindows build that the P1 CI will exercise, and it does not depend on `protoc` being on the runner. The most robust check compares the *descriptor's own descriptor set* against a set rebuilt from the current `.proto` sources at test time — but that reintroduces the protoc dependency and the byte-sensitivity of IPC-20. The lighter, sufficient check: reflect over the in-process `MxaccessGatewayReflection.Descriptor` / `MxaccessWorkerReflection.Descriptor` (already compiled into `Contracts`) and assert every message and field name present in the live descriptor also appears in the committed protoset's `FileDescriptorSet` (parsed with `FileDescriptorSet.Parser`). This catches "protoset missing a symbol the contract has" — exactly the stale-descriptor failure mode — without invoking protoc or being sensitive to source-info bytes. Co-designed with IPC-19 (same freshness-drift class for `Generated/`) and IPC-20 (protoc pinning for the script path).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Regenerate: run `scripts/publish-client-proto-inputs.ps1` (no `-Check`) on a box with the pinned protoc (IPC-20), commit the refreshed `.protoset`.
|
||||||
|
- Extend `src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs`: add `Descriptor_ContainsEveryContractMessageAndField`. Load the protoset bytes, `FileDescriptorSet.Parser.ParseFrom`, build the set of `{file}.{message}.{field}` names; enumerate `MxaccessGatewayReflection.Descriptor.MessageTypes` (recursively for nested types) and `MxaccessWorkerReflection.Descriptor`; assert each contract symbol is present in the protoset set. Fail with the missing symbol name.
|
||||||
|
- Wire the same assertion into the P1 CI job (see IPC-09) as a redundant guard.
|
||||||
|
- Config/proto surface: none changes.
|
||||||
|
- Docs: note in `docs/ClientProtoGeneration.md` that the freshness test guards the descriptor and that a red test means "regenerate and commit the protoset."
|
||||||
|
|
||||||
|
**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~ClientProtoInputTests` (runs on macOS NonWindows tree). After regenerating, confirm `strings clients/proto/descriptors/mxaccessgw-client-v1.protoset | grep MxSparseArray` returns hits.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-02 — Worker max frame size is hard-coded and cannot follow the gateway's configured limit `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** `WorkerFrameProtocolOptions`' `WorkerOptions` ctor always passes `DefaultMaxMessageBytes` = 16 MiB (verified `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs:15-21`). The gateway-side `MxGateway:Worker:MaxMessageBytes` is configurable (default 16 MiB, `Server/Configuration/WorkerOptions.cs:37`; validated 1 KiB–256 MiB per the review's `GatewayOptionsValidator` reference). Neither `GatewayHello` (`mxaccess_worker.proto:41-45`) nor the worker launch arguments carry the value, so the two limits are set independently and never reconciled.
|
||||||
|
|
||||||
|
**Impact.** An operator who raises the gateway limit above 16 MiB for large-array workloads gets a gateway that emits frames the worker's reader rejects with `MessageTooLarge` (`Worker/Ipc/WorkerFrameReader.cs:44-48`), faulting the session; the worker still cannot emit anything above 16 MiB in the return direction. The config appears to work until the first large frame. Part of the P1 backpressure-topology pass (roadmap item 8).
|
||||||
|
|
||||||
|
**Design.** Convey the negotiated max frame size in the handshake and fail fast on disagreement, rather than failing mid-traffic. Add `uint32 max_frame_bytes = 4;` to `GatewayHello` (additive, next free tag). The worker reads it during handshake and constructs its `WorkerFrameProtocolOptions.MaxMessageBytes` from the negotiated value instead of the hard-coded default; if the value is 0 (older gateway) it falls back to `DefaultMaxMessageBytes`. Because the worker and gateway are lockstep-deployed (one-worker-per-session invariant; gateway launches the worker), a launch-argument would also work, but the `GatewayHello` field keeps the value on the wire where an alternate-language worker (contemplated in `gateway.md`) can read it. Reject at handshake if the value exceeds a sane worker ceiling. This is a proto change, so it triggers the regen-and-commit rule (IPC-19) and the descriptor refresh (IPC-01). Co-designed with IPC-03; the negotiated value must sit *above* the public gRPC cap plus envelope headroom.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Contracts/Protos/mxaccess_worker.proto`: add `uint32 max_frame_bytes = 4;` to `GatewayHello`. Regenerate `Contracts/Generated/` and commit (net48 rule).
|
||||||
|
- Gateway: populate the field when building `GatewayHello` (in the worker-client handshake path) from `WorkerOptions.MaxMessageBytes`.
|
||||||
|
- Worker: in `WorkerPipeSession` handshake (around the version check at `:215`), pass the received `max_frame_bytes` into the `WorkerFrameProtocolOptions` construction, replacing the `DefaultMaxMessageBytes` argument in the `WorkerOptions` ctor overload. Keep the `> 0` guard already in the all-parameters ctor.
|
||||||
|
- Tests: `WorkerFrameProtocolTests` / handshake tests — assert the worker adopts the gateway's advertised limit and that a frame between the old (16 MiB) and new limit now succeeds; a fake-worker gateway test asserting handshake carries the value.
|
||||||
|
- Docs: `docs/WorkerFrameProtocol.md` and `docs/GatewayConfiguration.md` — document that `MaxMessageBytes` is negotiated to the worker via `GatewayHello`.
|
||||||
|
|
||||||
|
**Verification.** Contracts changed → regenerate, then `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`; `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86` and `dotnet test ...Worker.Tests -p:Platform=x86` (Windows host); `dotnet test ...Tests --filter FullyQualifiedName~Handshake`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-03 — Public gRPC max message size equals the pipe max with zero headroom; oversized outbound frame faults the whole session `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** `MaxGrpcMessageBytes` defaults to 16 MiB (`Server/Configuration/ProtocolOptions.cs:16`) and the pipe `MaxMessageBytes` defaults to the same 16 MiB (`WorkerOptions.cs:37`). A gRPC-accepted `Invoke` payload near the limit gains `WorkerCommand` + `WorkerEnvelope` overhead (session id, sequence, correlation id, enqueue timestamp, oneof tags) before hitting `WorkerFrameWriter.cs:49-54`, whose `WorkerFrameProtocolException` escapes the write loop at `WorkerClient.cs:344-350` and calls `SetFaulted`, which kills the worker process (verified `WorkerClient.cs:722-749`: `SetFaulted` → `KillOwnedProcess` → `CompletePendingCommands`).
|
||||||
|
|
||||||
|
**Impact.** One legitimate, gateway-accepted oversized write tears down the session, all its subscriptions, and its event stream — instead of failing that single command. Roadmap item 8.
|
||||||
|
|
||||||
|
**Design.** Two coordinated changes. (1) Headroom: keep `MaxGrpcMessageBytes` strictly below the negotiated pipe max (IPC-02) by an explicit envelope-overhead margin. Rather than a second magic constant, derive the accepted public payload cap as `pipeMax − EnvelopeOverheadReserve` (a named constant, e.g. 64 KiB, sufficient for the fixed envelope fields plus timestamp/oneof framing), and validate at startup (`GatewayOptionsValidator`) that `MaxGrpcMessageBytes ≤ pipeMax − reserve`, failing config otherwise. (2) Per-command failure: at the enqueue/write boundary, catch `WorkerFrameProtocolException` with `ErrorCode == MessageTooLarge` and fail only the offending correlation id (complete its `PendingCommand` with a `WorkerClientErrorCode` → `ResourceExhausted`/`InvalidArgument` gRPC status) instead of letting it reach `SetFaulted`. Because all writes funnel through the single `WriteLoopAsync` (`WorkerClient.cs:332-339`), the cleanest seam is to pre-check `envelope.CalculateSize()` against `MaxMessageBytes` in `CreateCommandEnvelope`/enqueue and reject there synchronously in `InvokeAsync`, before the frame ever enters the outbound channel — this keeps the write loop's remaining exceptions genuinely fatal (a mid-frame `MessageTooLarge` there really is a desync). Co-designed with IPC-02 and IPC-04 (the three form the size/backpressure pass). Does not touch MXAccess parity — this is a transport cap, not a COM behavior.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Server/Configuration/ProtocolOptions.cs` / `GatewayOptionsValidator`: add the headroom invariant; document `EnvelopeOverheadReserve`.
|
||||||
|
- `Server/Workers/WorkerClient.cs`: in `InvokeAsync` before `EnqueueAsync`, size-check the built envelope; on overshoot, throw `WorkerClientException(PayloadTooLarge, ...)` mapped to gRPC `ResourceExhausted` in `MxAccessGatewayService` — do not enqueue, do not fault. Add `WorkerClientErrorCode.PayloadTooLarge`.
|
||||||
|
- Keep the write-loop `MessageTooLarge` → `SetFaulted` path for genuine desync (now unreachable for legitimate command payloads).
|
||||||
|
- Tests: `WorkerClientTests` — an over-cap `Invoke` fails that one command with the mapped status and leaves the session `Ready` and other pending commands intact (assert worker not killed). `GatewayOptionsValidatorTests` — headroom invariant rejects `MaxGrpcMessageBytes ≥ pipeMax`.
|
||||||
|
- Docs: `docs/GatewayConfiguration.md` (headroom rule), `docs/Grpc.md` (Invoke now returns `ResourceExhausted` for oversized payloads).
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewayOptionsValidatorTests"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-04 — `DrainEvents` with `max_events = 0` packs the entire event queue into one reply envelope `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** `MxAccessEventQueue` treats `0` as drain-all (review cite `Worker/MxAccess/MxAccessEventQueue.cs:172-192`); the reply is a single `DrainEventsReply` written as one frame (`Worker/Ipc/WorkerPipeSession.cs` control-reply path). `MxAccessGrpcRequestValidator` imposes no cap on `DrainEventsCommand.max_events`.
|
||||||
|
|
||||||
|
**Impact.** A deep queue of large array events produces a reply exceeding `MaxMessageBytes`; the control-reply write throws, the exception propagates out of the message loop, and the worker exits — a diagnostics command kills the session. Roadmap item 8.
|
||||||
|
|
||||||
|
**Design.** Cap the effective drain worker-side rather than chunking (chunking `DrainEvents` needs a streaming reply arm the control-command path doesn't have; a bounded single reply is simpler and sufficient for a diagnostics RPC). Introduce a `MaxDrainEventsPerReply` worker constant (e.g. `WorkerPipeSession` alongside the existing `EventDrainInterval`/batch-size constants at `:17-19`), and interpret `max_events = 0` as "up to `MaxDrainEventsPerReply`", never unbounded. Optionally surface a `has_more`/`remaining` field on `DrainEventsReply` (additive) so a caller can drain iteratively. Also add an upper-bound validation in `MxAccessGrpcRequestValidator` for the public `DrainEventsCommand.max_events`. This keeps parity untouched (drain is a gateway diagnostics feature, not an MXAccess behavior). Co-designed with IPC-03 (both prevent one accepted request from producing a session-killing frame).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Worker/Ipc/WorkerPipeSession.cs`: clamp the requested count to `MaxDrainEventsPerReply`; treat 0 as the clamp value.
|
||||||
|
- (Optional, additive) `Contracts/Protos/mxaccess_gateway.proto`: add `uint32 remaining = N;` to `DrainEventsReply` — regen + commit if taken.
|
||||||
|
- `Server/Grpc/MxAccessGrpcRequestValidator`: reject `max_events` above the documented ceiling with `InvalidArgument`.
|
||||||
|
- Tests: `MxAccessEventQueueTests` / worker session tests — a queue larger than the cap drains at most `MaxDrainEventsPerReply` per reply and never builds an over-`MaxMessageBytes` frame; validator test for the public cap.
|
||||||
|
- Docs: `docs/Grpc.md` DrainEvents rules row; `docs/WorkerFrameProtocol.md` if the reply gains `remaining`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86` + `dotnet test ...Worker.Tests -p:Platform=x86 --filter FullyQualifiedName~MxAccessEventQueue`; gateway validator test on macOS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-05 — Redundant deep copies on the command and event hot paths `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `MxAccessGrpcMapper.MapCommand` clones the inbound `MxCommand`, then `WorkerClient.CreateCommandEnvelope` clones the whole `WorkerCommand` a second time (verified `WorkerClient.cs:902`, `envelope.WorkerCommand = command.Clone()`); `MapEvent` deep-clones every worker event (verified `MxAccessGrpcMapper.cs:68`, `workerEvent.Event?.Clone()`). So each `Invoke` materializes the command graph three times (gRPC parse, mapper clone, worker-client clone); each event is parsed once and cloned once.
|
||||||
|
|
||||||
|
**Impact.** Measurable allocation pressure on array-heavy `OnDataChange` streams — the exact load the gateway exists to handle (overall theme 7). Not a correctness bug.
|
||||||
|
|
||||||
|
**Design.** Drop the second `Clone()` in `CreateCommandEnvelope`: the mapper's clone already isolates the graph from the caller-owned gRPC message, and the envelope is built and owned entirely inside `WorkerClient`, so no aliasing hazard remains once the envelope is enqueued. For `MapEvent`, evaluate ownership transfer: the `WorkerEvent` is parsed fresh from the pipe frame in the read loop and is not retained after mapping, so the mapper can move `workerEvent.Event` into the outbound `MxEvent` graph rather than cloning — but confirm no code path re-reads `workerEvent` after `MapEvent` (the fan-out distributor may share one `MxEvent` across subscribers, which is read-only and safe). Recommend dropping the command clone unconditionally (clear win, no aliasing) and gating the event ownership-transfer on a quick audit of the fan-out path (coordinate with the gateway-core distributor findings). Parity-neutral. Co-designed with IPC-13/IPC-14 as the frame-I/O + allocation pass (roadmap item 14).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Server/Workers/WorkerClient.cs:902`: `envelope.WorkerCommand = command;` (remove `.Clone()`), with a comment that the mapper already isolated the graph.
|
||||||
|
- `Server/Grpc/MxAccessGrpcMapper.cs:68`: keep the clone only if the audit shows the event is aliased downstream; otherwise transfer ownership.
|
||||||
|
- Tests: existing `WorkerClientTests`/mapper tests must still pass (no behavior change); add an assertion that the enqueued envelope's command is reference-equal to the mapper output where the clone was removed, to lock the intent.
|
||||||
|
- Docs: `docs/Grpc.md:197-211` (the clone-count description) must be corrected to match the reduced copies.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient|FullyQualifiedName~MxAccessGrpcMapper"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-06 — `gateway.md`'s Worker Envelope section no longer matches the shipped contract `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `gateway.md:291-309` shows `uint64 correlation_id = 4` and body cases `command = 20; command_reply = 21; event = 22; heartbeat = 23; cancel = 24; shutdown = 25; fault = 26`. The actual contract (`mxaccess_worker.proto:20-39`) is `string correlation_id = 4` and `worker_command = 13; worker_command_reply = 14; worker_cancel = 15; worker_shutdown = 16; worker_shutdown_ack = 17; worker_event = 18; worker_heartbeat = 19; worker_fault = 20`. `WorkerShutdownAck` is absent from the doc sketch entirely.
|
||||||
|
|
||||||
|
**Impact.** The top-level architecture doc's field numbers and types are wrong; anyone implementing an alternate worker (the doc contemplates a C++ worker) from this section produces an incompatible peer. Violates the repo's docs-change-with-source rule. Roadmap item 15.
|
||||||
|
|
||||||
|
**Design.** Replace the hand-maintained protobuf sketch with either the verbatim current message or, preferably, a short prose description plus a pointer to `mxaccess_worker.proto` as the single source of truth — hand-copied proto in docs is exactly what drifted. Keep the "Rules" list but reconcile it (`correlation_id` is a string; note `WorkerShutdownAck` and `WorkerReady`). Do this in the same sweep as IPC-07, IPC-17, IPC-21, IPC-22.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `gateway.md:288-320`: replace the code block with the real oneof (or a pointer), add `WorkerReady`/`WorkerShutdownAck`, fix `correlation_id` type.
|
||||||
|
- No source/proto/config change.
|
||||||
|
|
||||||
|
**Verification.** Doc-only; no build. Cross-check the doc block against `mxaccess_worker.proto:20-39` after editing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-07 — `docs/Grpc.md` says the service has six RPCs; it has seven `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `docs/Grpc.md:13` and `:32` say "six in total" and omit `QueryActiveAlarms`. The proto declares seven RPCs (`mxaccess_gateway.proto:18-37`: `OpenSession`, `CloseSession`, `Invoke`, `StreamEvents`, `AcknowledgeAlarm`, `StreamAlarms`, `QueryActiveAlarms`); `QueryActiveAlarms` is implemented in `MxAccessGatewayService`.
|
||||||
|
|
||||||
|
**Impact.** The authoritative gRPC-layer doc undercounts the public surface and documents no validation/handler behavior for the missing RPC. Note the cross-domain link: the security review found `QueryActiveAlarms` also missing from the scope resolver (SEC domain) — the doc omission masked that gap. Roadmap item 15.
|
||||||
|
|
||||||
|
**Design.** Update the count (six → seven) at both cites, add `QueryActiveAlarms` to the collaborators/RPC table, and add a handler section describing its validation, streaming behavior, and required scope. Flag to the security-domain remediation that the same RPC needs a scope-resolver entry.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/Grpc.md:13,32`: fix count and RPC list; add `QueryActiveAlarms` row and handler subsection.
|
||||||
|
- No source change in this domain (scope resolver fix is SEC).
|
||||||
|
|
||||||
|
**Verification.** Doc-only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-08 — `WorkerCancel` is defined and handled but never sent `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `WorkerCancel` is defined (`mxaccess_worker.proto:71-73`) and dispatched by the worker to `_runtimeSession.CancelCommand` (verified `Worker/Ipc/WorkerPipeSession.cs:399-401`), but no gateway code sends it — `WorkerClient` handles timeout/cancel purely by abandoning the pending correlation (verified `WorkerClient.cs:195-213`: on timeout it calls `RemovePendingCommandAsFailed` and throws, never emitting a `WorkerCancel`).
|
||||||
|
|
||||||
|
**Impact.** The cancellation contract in `gateway.md:713-719` ("the worker should finish the COM call and discard or log the late reply if the correlation was canceled") is half-implemented: the worker never learns a correlation was canceled, so it always writes the late reply (which the gateway then drops at `WorkerClient.cs:565-571`), and the worker-side discard logic behind `CancelCommand` is unreachable. Not in the roadmap.
|
||||||
|
|
||||||
|
**Design.** Decide between two coherent end states; recommend **wiring it up** because the worker handling already exists and the late-reply-drop path is real work being wasted. On timeout or caller-cancel in `InvokeAsync`, enqueue a `WorkerCancel` envelope carrying the correlation id (add `string correlation_id`-based routing — the envelope already has `CorrelationId`, and the worker's `CancelCommand(envelope.CorrelationId)` already reads it, so no proto change is needed; `WorkerCancel.reason` is the human note). The worker then discards the late reply instead of writing it, saving a frame and a pointless gateway-side drop. Keep the abandon-the-pending-correlation behavior as the gateway's own timeout resolution — `WorkerCancel` is best-effort cleanup, not a synchronous ack. The alternative (mark the arm reserved-for-future and delete the worker handling) loses working behavior and still needs a `gateway.md` edit; reject it. This does not synthesize events and does not alter MXAccess parity — the COM call still completes; only the late reply is suppressed. Because the worker must keep pumping the STA, `CancelCommand` must remain a non-blocking mark-and-discard (confirm it is).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Server/Workers/WorkerClient.cs`: add a `CreateCancelEnvelope(correlationId, reason)` (mirrors `CreateShutdownEnvelope`) and enqueue it in the timeout and cancellation-token branches of `InvokeAsync` (`:195-213`) after removing the pending command. Best-effort: swallow enqueue failures if the session is already terminal.
|
||||||
|
- Worker: confirm `MxAccessSession.CancelCommand` marks the correlation for late-reply discard and that `WorkerPipeSession`'s reply-write path checks that mark (it already drops replies once state leaves `Ready` at `:604-608`; extend to "or correlation was canceled").
|
||||||
|
- Tests: `WorkerClientTests` — a timed-out `Invoke` enqueues a `WorkerCancel` with the right correlation id; worker session test — a canceled correlation suppresses the late reply.
|
||||||
|
- Docs: `gateway.md:713-719` — mark the cancellation path as implemented; `docs/WorkerFrameProtocol.md` — document `WorkerCancel` semantics.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` + `dotnet test ...Tests --filter FullyQualifiedName~WorkerClient`; worker side `-p:Platform=x86`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-09 — Known codegen fragilities have no in-repo guards `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** (a) Python — `clients/python/pyproject.toml:15,41` allow `grpcio>=1.80,<2` and `grpcio-tools>=1.80,<2`; `clients/python/generate-proto.ps1` does no version check, so regenerating with any newer 1.x stamps a `GRPC_GENERATED_VERSION` above the pinned runtime and breaks pytest (a previously-hit failure per the memory). (b) Java — `clients/java/zb-mom-ww-mxgateway-client/build.gradle:50` points `generatedFilesBaseDir` at the tracked `src/main/generated`, so every `gradle build` rewrites the tracked output with protobuf-version churn and no task detects spurious diffs. (c) Portability — `clients/python/generate-proto.ps1:7` and `clients/go/generate-proto.ps1:8-9` hard-code `C:\Users\dohertj2\...` tool paths (verified), making generation single-machine and Windows-only.
|
||||||
|
|
||||||
|
**Impact.** Contract evolution safety depends on operator memory. A regeneration on a different machine or tool version silently produces incompatible or noisy output. Roadmap item 7 (CI) and the "everything guarded by operator memory" theme.
|
||||||
|
|
||||||
|
**Design.** Three targeted guards, each fail-fast:
|
||||||
|
- Python: pin exact generator versions used for regeneration. Add a version assertion at the top of `generate-proto.ps1` that reads `grpcio-tools.__version__` and refuses to run if it differs from the pinned baseline (grpcio 1.80.0 / protobuf 6.31.1 per the memory). Keep the `pyproject.toml` runtime range but document that regeneration must use the exact pin.
|
||||||
|
- Java: add a `checkGeneratedClean` Gradle task (git-diff the `src/main/generated` tree after generation and fail on non-empty diff when no `.proto` changed) so the ~64k-line spurious churn is caught in review rather than committed. Alternatively, and preferred long-term, move `generatedFilesBaseDir` to `$buildDir` and stop tracking generated Java — but that is a larger change; the check task is the minimal guard.
|
||||||
|
- Portability: resolve `protoc`/`python`/plugins from `PATH` with a documented version assertion (the `publish-client-proto-inputs.ps1` `Resolve-Protoc` at lines 15-25 already does PATH-first with a documented fallback — mirror that pattern in the per-client scripts) instead of absolute user paths.
|
||||||
|
|
||||||
|
Coordinate with IPC-01/IPC-20 (descriptor freshness + protoc pin) so the CI job that runs the freshness gate also runs the codegen-clean checks.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/python/generate-proto.ps1`: PATH-resolve python; assert `grpcio-tools` version equals the pin, else throw.
|
||||||
|
- `clients/go/generate-proto.ps1`: PATH-resolve `protoc`, `protoc-gen-go`, `protoc-gen-go-grpc`.
|
||||||
|
- `clients/java/.../build.gradle`: add `checkGeneratedClean` task; wire into CI (not into every local build).
|
||||||
|
- CI (new, per roadmap item 7): a job that regenerates and runs the clean/freshness checks.
|
||||||
|
- Docs: `docs/ClientProtoGeneration.md` — record the exact pinned generator versions and the check tasks; `docs/Contracts.md` — cross-link.
|
||||||
|
|
||||||
|
**Verification.** Per-client: `python -m pytest` from `clients/python` after a pinned regen; `gradle checkGeneratedClean` from `clients/java` on a Windows/JDK-17 host; `go build ./...` from `clients/go`. Note the Java build must run on windev (Mac has no JRE per the memory) via an isolated `origin/<branch>` worktree.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-10 — Envelope `sequence` is write-only; monotonicity is never validated on receive `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Sequences are assigned on send (verified `WorkerClient.cs:934`, `Interlocked.Increment(ref _nextSequence)`; worker side at `WorkerPipeSession.cs:1005-1018`), but neither validator checks them (verified `WorkerEnvelopeValidator.cs:15-39` checks only protocol version, session id, and non-empty body; the worker validator likewise). `gateway.md:312` states "`sequence` is monotonic per sender" as a protocol rule.
|
||||||
|
|
||||||
|
**Impact.** Gap/duplication detection promised by the design is diagnostics-only; ordering integrity rests entirely on pipe FIFO semantics (fine for a local named pipe, but the rule is unenforced). Low.
|
||||||
|
|
||||||
|
**Design.** Choose the cheap enforcement or the honest annotation. Recommend **annotate** rather than enforce: on a local named pipe FIFO is guaranteed, and adding per-sender monotonicity tracking to the validators introduces state and a new fault mode for no real safety gain (a reordering would already be a kernel-level pipe bug). Mark `sequence` as diagnostic-only in the proto comment and soften `gateway.md`'s "rule" to "diagnostic aid." If enforcement is later wanted, it belongs in the validators as a per-sender last-seen check, faulting on regression.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Contracts/Protos/mxaccess_worker.proto:23`: comment that `sequence` is a monotonic diagnostic counter, not validated on receive (regen + commit — comment-only proto changes still regenerate).
|
||||||
|
- `gateway.md:312`: reword the rule.
|
||||||
|
|
||||||
|
**Verification.** Regenerate contracts; `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` to confirm the comment-only regen is clean.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-11 — No protocol version negotiation despite a field name that implies it `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `GatewayHello.supported_protocol_version` (`mxaccess_worker.proto:42`) is a single value compared for strict equality (`Worker/Ipc/WorkerPipeSession.cs:215-219`); the worker also hard-pins its version to `GatewayContractInfo.WorkerProtocolVersion` (=1, verified `GatewayContractInfo.cs:15`) at options construction (`WorkerFrameProtocolOptions.cs:65-70`), and every envelope re-checks equality in both validators.
|
||||||
|
|
||||||
|
**Impact.** Acceptable for a lockstep-deployed pair (`gateway.md:319` documents mismatch-fails-session), but the singular "supported" field cannot express a range; any future skewed upgrade needs new machinery. Low — no action required now.
|
||||||
|
|
||||||
|
**Design.** No change now. When the worker protocol first changes (bumps past 1), replace the single field with a `min`/`max` supported range in `GatewayHello`/`WorkerHello` and negotiate the highest common version rather than bumping the single constant. Record this as the intended evolution path in the proto comment so the next editor doesn't just increment the constant.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Contracts/Protos/mxaccess_worker.proto`: add a comment on `supported_protocol_version` describing the min/max-range migration when a second version appears (regen + commit).
|
||||||
|
- No code change.
|
||||||
|
|
||||||
|
**Verification.** Regenerate; NonWindows build clean.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-12 — Gateway frame writer has no write lock; integrity rests on an undocumented single-writer invariant `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `Server/Workers/WorkerFrameWriter.cs` has no synchronization (verified: no lock/semaphore; `WriteAsync` at `:34-61` issues prefix and payload as two separate `_stream.WriteAsync` calls at `:59-60`). The worker's writer serializes with a `SemaphoreSlim` (verified `Worker/Ipc/WorkerFrameWriter.cs:14,68-77`) because its heartbeat, event-drain, and command tasks write concurrently. The gateway is safe only because all writes funnel through the single-reader outbound channel loop (`WorkerClient.cs:332-339`).
|
||||||
|
|
||||||
|
**Impact.** A future direct `_writer.WriteAsync` call outside the write loop interleaves the two stream writes and corrupts the frame stream unrecoverably. Latent, not live. Low.
|
||||||
|
|
||||||
|
**Design.** Defense in depth: either (a) collapse the two writes into one (adopt the worker's single-buffer approach — this is IPC-13, which also removes the interleaving surface entirely), or (b) add the same `SemaphoreSlim` write lock plus an XML-doc invariant note. Recommend doing IPC-13 (single write) *and* documenting the single-writer invariant on the class, which together make interleaving impossible even under a future stray caller. A bare assertion is weaker than a single atomic write.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Fold into IPC-13's single-buffer rewrite; add a class-level `<remarks>` stating "all frames must be written through `WorkerClient.WriteLoopAsync`; the writer is not internally synchronized for concurrent callers" (or, if a lock is added, document that it is).
|
||||||
|
- Tests: covered by IPC-13's frame round-trip tests.
|
||||||
|
|
||||||
|
**Verification.** See IPC-13.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-13 — Gateway writer serializes each envelope twice and issues two stream writes `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `Server/Workers/WorkerFrameWriter.cs:41,60`: `CalculateSize()` then `ToByteArray()` (which re-runs size calculation internally) and separate prefix/payload writes at `:59-60`. The worker writer already builds one prefixed buffer with a single `WriteTo(Span)` and one write (verified `Worker/Ipc/WorkerFrameWriter.cs:63-72`, with a comment explaining exactly this).
|
||||||
|
|
||||||
|
**Impact.** Extra CPU pass and extra pipe write per frame on the higher-volume side (the gateway writes every command). Low; roadmap item 14.
|
||||||
|
|
||||||
|
**Design.** Back-port the worker's single-buffer implementation verbatim: allocate `sizeof(uint) + payloadLength`, write the little-endian length prefix, `envelope.WriteTo(new Span<byte>(frame, 4, payloadLength))`, one `_stream.WriteAsync`. This also closes IPC-12's interleaving surface. Keep the existing size/`MessageTooLarge`/empty-payload guards (already present at `:41-54`). Consider adding the `FlushAsync` the worker does — verify the gateway's underlying pipe stream doesn't already auto-flush before adding it.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Server/Workers/WorkerFrameWriter.cs:56-60`: replace the two-array/two-write block with the single-buffer/single-write block from `Worker/Ipc/WorkerFrameWriter.cs:63-72`.
|
||||||
|
- Tests: `WorkerFrameWriterTests` / frame round-trip tests must still pass (byte-identical wire output); add a test asserting a single write path if a mock stream is available.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter FullyQualifiedName~WorkerFrame`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-14 — Gateway reader allocates a fresh array per frame; the worker rents from `ArrayPool` `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `Server/Workers/WorkerFrameReader.cs:50`: `byte[] payload = new byte[payloadLength];` per frame. The worker rents/returns from `ArrayPool<byte>.Shared` with a comment that `ParseFrom` copies (verified `Worker/Ipc/WorkerFrameReader.cs:55-77`).
|
||||||
|
|
||||||
|
**Impact.** Large event frames (arrays near the cap) allocate LOH buffers per frame on the side that receives the entire event stream. Low; roadmap item 14.
|
||||||
|
|
||||||
|
**Design.** Mirror the pooled read: `int length = checked((int)payloadLength); byte[] payload = ArrayPool<byte>.Shared.Rent(length);` read exactly `length`, `WorkerEnvelope.Parser.ParseFrom(payload, 0, length)`, `Return` in `finally`. The gateway reader already reads-exactly via `ReadExactlyAsync`; keep that. Note `ParseFrom(payload, 0, length)` is required (the rented buffer may be larger than `length`).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Server/Workers/WorkerFrameReader.cs:50-56`: adopt the rent/return pattern from the worker reader; use the length-bounded `ParseFrom` overload.
|
||||||
|
- Tests: `WorkerFrameReaderTests` round-trip must pass; add a large-frame test to exercise the pool path.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test ...Tests --filter FullyQualifiedName~WorkerFrame`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-15 — Worker event delivery is a 25 ms poll with per-event write+flush, no batching `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `Worker/Ipc/WorkerPipeSession.cs:17-19` sets `EventDrainInterval` 25 ms and batch size 128; the drain loop writes each event with its own `WriteAsync` (each acquiring the writer lock and `FlushAsync`, verified `Worker/Ipc/WorkerFrameWriter.cs:71-72`). `gateway.md:849-850` lists worker→gateway event batching as a planned optimization that doesn't exist — there is no multi-event envelope body.
|
||||||
|
|
||||||
|
**Impact.** Idle-to-active latency floor up to 25 ms per batch, plus per-event flush syscalls under burst. Acceptable for v1 parity; roadmap item 14.
|
||||||
|
|
||||||
|
**Design.** Do not tighten the poll (that trades latency for CPU wakeups). When event-rate targets firm up, add a batched event body as an *additive* `oneof` arm — a `WorkerEventBatch { repeated MxEvent events = 1; }` message and a `worker_event_batch = N` case in `WorkerEnvelope` — so the drain loop packs up to N events (bounded by `MaxMessageBytes`, coordinate with IPC-03/IPC-04) into one frame with one flush. The gateway read loop unpacks the batch and dispatches each event to the distributor. This preserves the "don't synthesize events" invariant (the worker still forwards only real events; batching is framing, not fabrication) and the one-worker-per-session invariant. Keep single-event `worker_event` for compatibility during rollout. This is deferred work, not a v1 fix — flagged so the batching claim in `gateway.md` is either implemented or re-marked as future.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- (When scheduled) `Contracts/Protos/mxaccess_worker.proto`: add `WorkerEventBatch` + `worker_event_batch` oneof arm (regen + commit + descriptor refresh).
|
||||||
|
- `Worker/Ipc/WorkerPipeSession.cs`: drain up to the batch size into one `WorkerEventBatch` frame under the frame-size cap.
|
||||||
|
- `Server/Workers/WorkerClient.cs` read dispatch: handle the batch arm.
|
||||||
|
- Now (doc-only, immediate): correct `gateway.md:849-850` to mark event batching as not-yet-implemented.
|
||||||
|
- Tests: event round-trip under batch; ordering preserved.
|
||||||
|
|
||||||
|
**Verification.** Immediate: doc edit. When implemented: contracts regen + `dotnet build NonWindows.slnx`, worker `-p:Platform=x86` tests, gateway fake-worker event tests.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-16 — Correlation id is carried twice per reply `Low (Info)` · `—`
|
||||||
|
|
||||||
|
**Finding.** `WorkerEnvelope.correlation_id` (`mxaccess_worker.proto:24`) and `MxCommandReply.correlation_id` (review cite `mxaccess_gateway.proto:520`) both carry the value; `CompleteCommand` falls back from the envelope to the inner reply (verified `WorkerClient.cs:559-562`: uses `envelope.CorrelationId`, then `envelope.WorkerCommandReply.Reply?.CorrelationId`).
|
||||||
|
|
||||||
|
**Impact.** Two sources of truth for one value; harmless today, a divergence hazard for a future writer that sets one but not the other. Info.
|
||||||
|
|
||||||
|
**Design.** No wire change. Document that the **envelope** `correlation_id` is authoritative and the inner reply copy is the MXAccess-parity echo; the fallback in `CompleteCommand` exists only for defensive robustness. Add a proto comment on both fields naming the envelope as canonical.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Contracts/Protos/mxaccess_worker.proto:24` and `mxaccess_gateway.proto` `MxCommandReply.correlation_id`: cross-referencing comments (regen + commit).
|
||||||
|
|
||||||
|
**Verification.** Regenerate; NonWindows build clean.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-17 — Two docs name the wrong Python generated-output directory `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `docs/ClientProtoGeneration.md:80,145` say `clients/python/src/mxgateway/generated`; the manifest (`clients/proto/proto-inputs.json:28`) and the tree use `clients/python/src/zb_mom_ww_mxgateway/generated`, which is also what `clients/python/generate-proto.ps1` writes. CLAUDE.md's generated-code bullet repeats the wrong path.
|
||||||
|
|
||||||
|
**Impact.** A follow-the-doc regeneration writes to a dead directory. Low; roadmap item 15.
|
||||||
|
|
||||||
|
**Design.** Fix both docs (and the CLAUDE.md bullet) to the manifest path `clients/python/src/zb_mom_ww_mxgateway/generated`. Trivial, no source change.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/ClientProtoGeneration.md:80,145`: correct the path.
|
||||||
|
- `CLAUDE.md` generated-code bullet: correct `clients/python/src/mxgateway/generated` → `clients/python/src/zb_mom_ww_mxgateway/generated`.
|
||||||
|
|
||||||
|
**Verification.** Doc-only; grep to confirm no `src/mxgateway/generated` remains.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-18 — Generated-code hygiene verified good — preserve under change (positive) `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** No defect. Verified positive: `Contracts/Generated/*.cs` carry `<auto-generated>` headers and contain the newest symbols (`MxSparseArray`); all five client generated trees contain the sparse-array surface; `galaxy_repository.proto` is wire-identical to the GalaxyRepository package copy (only `csharp_namespace` differs), consistent with the retention comment in `ZB.MOM.WW.MxGateway.Contracts.csproj:29-33` that keeps it as the language-client codegen source.
|
||||||
|
|
||||||
|
**Impact.** None — this is a property to preserve, not a fix. Recorded so it is protected under change.
|
||||||
|
|
||||||
|
**Design.** No remediation. The freshness gates in IPC-01 (descriptor) and IPC-19 (`Generated/` net48 rule) are what keep this property true; the galaxy wire-identity should be asserted by a small test if the two copies ever diverge in practice (a byte-diff test comparing the two `.proto` files modulo `csharp_namespace`), but this is optional given the csproj comment already documents the invariant.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Optional: a `ContractsTests` case diffing `galaxy_repository.proto` against the package source path modulo `csharp_namespace`. Not required.
|
||||||
|
|
||||||
|
**Verification.** N/A (no change).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-19 — The `Generated/`-must-be-committed rule for net48 consumers is undocumented and unguarded `Low` · `P1`
|
||||||
|
|
||||||
|
**Finding.** Verified `ZB.MOM.WW.MxGateway.Contracts.csproj:26-35` does `Compile Remove="Generated\**\*.cs"` + `Protobuf ... OutputDir="Generated"`, regenerating tracked files on every build; the worker consumes contracts via `ProjectReference`. `docs/Contracts.md:97-98` says only "do not hand-edit." The operational rule — a proto edit requires regenerating and committing `Generated/`, or the net48 worker build fails CS0246 on new types (confirmed by the repo memory `project_proto_codegen_regen`) — appears nowhere, and no test compares `Generated/` to the protos.
|
||||||
|
|
||||||
|
**Impact.** Same silent-drift class as IPC-01: committed C# can lag the protos with nothing failing until a downstream net48 consumer breaks. Low; roadmap item 7.
|
||||||
|
|
||||||
|
**Design.** Document the rule and lean on the IPC-01 freshness test for enforcement. State in `docs/Contracts.md` (near the existing "do not hand-edit" line) that after any `.proto` edit you must regenerate `Contracts/Generated/` and commit it, and why (net48 worker has no first-class regen-on-restore; a stale `Generated/` fails CS0246). The IPC-01 reflection-based test (comparing `MxaccessGatewayReflection.Descriptor` against the committed protoset) already indirectly catches a proto edited without regenerating the descriptor; for `Generated/` specifically, the .NET build itself compiles against the freshly regenerated output on the net10 side, so the practical guard is the net48 worker build in CI (IPC-09/roadmap item 7) plus the documented rule.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/Contracts.md:97`: add the regenerate-and-commit rule with the net48 rationale.
|
||||||
|
- Optionally reference the memory-known `del Generated/*.cs` force-regen trick in the doc.
|
||||||
|
- CI (roadmap item 7): the Windows net48 worker build job is the real enforcement.
|
||||||
|
|
||||||
|
**Verification.** Doc-only for the rule; the net48 build (`dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86` on Windows) is what proves a stale `Generated/` would fail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-20 — Descriptor `-Check` byte-compares source-info-bearing bytes; protoc-version-sensitive `Low` · `P1`
|
||||||
|
|
||||||
|
**Finding.** Verified `scripts/publish-client-proto-inputs.ps1`: `-Check` (line 70) rebuilds the descriptor with `--include_source_info` (line 78) and `Compare-FileBytes` (line 36, 88) byte-compares. Source-info bytes differ across protoc releases even for identical schemas.
|
||||||
|
|
||||||
|
**Impact.** Once a check exists (IPC-01), a protoc upgrade produces a false "stale" failure or forces a churn commit unless protoc is pinned. Low; roadmap item 7.
|
||||||
|
|
||||||
|
**Design.** Two options, do both. (1) Pin protoc for descriptor generation: `Resolve-Protoc` (lines 15-25) already prefers PATH and documents a fallback; add a version assertion (`protoc --version` must equal the pinned toolchain in `docs/ToolchainLinks.md`, protoc 34.1 per CLAUDE.md) and fail fast on mismatch. (2) Make the in-process freshness test (IPC-01) the *primary* gate since it compares semantically (symbol presence) rather than byte-wise, sidestepping source-info sensitivity entirely; keep the script's byte-`-Check` as a stricter secondary that runs only with the pinned protoc. Alternatively drop `--include_source_info` and compare descriptors without source info for a version-tolerant byte compare — but source info is useful for grpcurl UX, so prefer pin + semantic test.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `scripts/publish-client-proto-inputs.ps1`: add a protoc-version assertion in `Resolve-Protoc`; document the pin.
|
||||||
|
- Rely on IPC-01's semantic test as the CI gate.
|
||||||
|
- Docs: `docs/ClientProtoGeneration.md` / `docs/Contracts.md` — record the pinned protoc version for descriptor generation.
|
||||||
|
|
||||||
|
**Verification.** Run `scripts/publish-client-proto-inputs.ps1 -Check` with the pinned protoc on Windows; the IPC-01 test on macOS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-21 — `gateway.md` presents the unimplemented `Session` RPC inside the live service block `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `gateway.md:328-345` sketches `rpc Session(stream ClientMessage) returns (stream ServerMessage)` inside the `service MxAccessGateway` block as "the best long-term shape," with a rollout plan whose step 3 is unimplemented; `mxaccess_gateway.proto:18-37` has no such RPC.
|
||||||
|
|
||||||
|
**Impact.** Intentional phasing, not a defect — but presenting it inside the service definition (alongside the five real RPCs) compounds IPC-06's staleness and reads as shipped surface. Low; roadmap item 15.
|
||||||
|
|
||||||
|
**Design.** Move the `Session` RPC out of the live service block into a clearly-labeled "Future work / not yet implemented" subsection, keeping the rollout plan. Do this in the same doc sweep as IPC-06/IPC-07. No proto or code change.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `gateway.md:328-345`: relabel; separate the bidi `Session` sketch from the shipped RPC list (which should also be reconciled to the actual seven RPCs — coordinate with IPC-07).
|
||||||
|
|
||||||
|
**Verification.** Doc-only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## IPC-22 — Public error-detail model stops at status codes plus prose `Low (Info)` · `—`
|
||||||
|
|
||||||
|
**Finding.** `MxCommandReply` preserves MXAccess parity detail well (`hresult`, `statuses`, `diagnostic_message`, review cite `mxaccess_gateway.proto:518-529`), but transport-level failures surface only as gRPC status codes with message strings; no `google.rpc` error details are attached, and `AcknowledgeAlarmReply.status` is a permanently-unset placeholder documented as such (verified `mxaccess_gateway.proto:938-945`: `MxStatusProxy status = 5` with a comment that the by-name/by-GUID ack path produces only the int32 return code and the field is left UNSET; clients must read `hresult`/`protocol_status`).
|
||||||
|
|
||||||
|
**Impact.** Machine consumers must parse prose to distinguish sub-causes within a gRPC status code. Acceptable for the current in-repo client set. Info.
|
||||||
|
|
||||||
|
**Design.** No change now. If third-party clients appear, attach `google.rpc.ErrorInfo` (a stable `reason` enum + `metadata`) to transport failures so consumers branch on a machine-readable reason instead of message strings, and either populate or formally reserve `AcknowledgeAlarmReply.status`. The `status` placeholder is already honestly documented (parity-correct: the worker ack path genuinely produces only the int32), so leave it as-is unless the ack path grows a structured status. Recording the recommendation without acting keeps parity intact.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- None now. Future: add `google.rpc.ErrorInfo` details in `MxAccessGatewayService` exception mapping; document reason codes in `docs/Grpc.md:217-241`.
|
||||||
|
|
||||||
|
**Verification.** N/A (no change).
|
||||||
@@ -0,0 +1,537 @@
|
|||||||
|
# Security, Dashboard & Observability — Remediation Design & Implementation
|
||||||
|
|
||||||
|
Source review: [40-security-dashboard.md](../40-security-dashboard.md) · Generated: 2026-07-09
|
||||||
|
|
||||||
|
This document turns every finding in the Security/Dashboard/Observability review into a buildable remediation entry. The crypto and interceptor foundations are sound; the work here is policy tightening (loopback/`DisableLogin`/scope gaps), cross-platform path hygiene, hot-path auth cost, and documentation reconciliation. IDs are re-sequenced under the domain prefix `SEC-NN` (most-severe first); each entry names the original review tag (e.g. *review STA-1*) so evidence stays traceable. Roadmap tiers follow `00-overall.md`: the P1 "Security policy pass" (item 9) and "cross-platform paths" (item 10), and the P2 per-session-ACL (item 12) and documentation-drift sweep (item 15). Findings not itemized in that roadmap but belonging to the same security-hardening theme (§theme 6) are tagged P1 with a note; the remainder are `—`.
|
||||||
|
|
||||||
|
## Finding index
|
||||||
|
|
||||||
|
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||||
|
|----|-----|-------|---------|--------|-----------|-------|
|
||||||
|
| SEC-01 | Medium | Windows-absolute default paths become relative files off-Windows | P1 | M | — | Configuration/AuthenticationOptions.cs, Configuration/TlsOptions.cs, Configuration/GatewayOptionsValidator.cs, appsettings.json |
|
||||||
|
| SEC-02 | Medium | `AllowAnonymousLocalhost` satisfies AdminOnly, not just Viewer | P1 | S | — | Dashboard/DashboardAuthorizationHandler.cs |
|
||||||
|
| SEC-03 | Medium | Dashboard cookie lost its `__Host-` prefix; four docs still promise it | P2 | S | — | Dashboard/DashboardAuthenticationDefaults.cs, gateway.md, docs/GatewayDashboardDesign.md, CLAUDE.md |
|
||||||
|
| SEC-04 | Medium | `DisableLogin` has no production guard | P1 | S | SEC-03 | Dashboard/DashboardServiceCollectionExtensions.cs, Configuration/GatewayOptionsValidator.cs |
|
||||||
|
| SEC-05 | Medium | Hub bearer tokens irrevocable for 30 min, carried in query string | P1 | M | — | Dashboard/HubTokenService.cs, Dashboard/HubTokenAuthenticationHandler.cs, Dashboard/DashboardEndpointRouteBuilderExtensions.cs |
|
||||||
|
| SEC-06 | Medium | LDAP plaintext-by-default with a committed service password | P1 | M | — | Configuration/LdapOptions.cs, appsettings.json, docs/GatewayConfiguration.md |
|
||||||
|
| SEC-07 | Medium | `QueryActiveAlarmsRequest` missing from scope resolver; tests mislabeled | P1 | S | — | Security/Authorization/GatewayGrpcScopeResolver.cs, Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs, docs/Authorization.md |
|
||||||
|
| SEC-08 | Medium | Per-RPC SQLite read + `last_used_utc` write; no verification cache | P1 | L | — | Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs, Security/Authentication/GatewayApiKeyIdentityMapper.cs |
|
||||||
|
| SEC-09 | Medium | Dashboard design-doc `GroupToRole` sample now fails startup validation | P2 | S | — | docs/GatewayDashboardDesign.md |
|
||||||
|
| SEC-10 | Medium | No API-key expiry | P1 | L | — | (shared `ZB.MOM.WW.Auth.ApiKeys`), Dashboard/DashboardApiKeyManagementService.cs |
|
||||||
|
| SEC-11 | Medium | No rate limiting or lockout on either auth surface | P1 | M | SEC-08 | Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs, Dashboard/DashboardEndpointRouteBuilderExtensions.cs |
|
||||||
|
| SEC-12 | Medium | Dashboard Close/Kill bypass the canonical audit store | P1 | M | — | Dashboard/DashboardSessionAdminService.cs |
|
||||||
|
| SEC-13 | Low | Redactor's credential-command list omits secured-bulk variants | — | S | SEC-30 | Diagnostics/GatewayLogRedactor.cs |
|
||||||
|
| SEC-14 | Low | `/metrics` + `/health` unauthenticated; a metric leaks session ids | — | S | SEC-02, SEC-20 | GatewayApplication.cs, Metrics/GatewayMetrics.cs |
|
||||||
|
| SEC-15 | Low | GET `/logout` skips antiforgery | — | S | — | Dashboard/DashboardEndpointRouteBuilderExtensions.cs |
|
||||||
|
| SEC-16 | Low | CLI accepts the pepper as a command-line argument | — | S | — | Program.cs |
|
||||||
|
| SEC-17 | Low | Interceptor does not override client-streaming/duplex handlers | — | S | — | Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs |
|
||||||
|
| SEC-18 | Low | Pepper-unavailable detection matches library message text | — | S | — | Dashboard/DashboardApiKeyManagementService.cs |
|
||||||
|
| SEC-19 | Low | Canonical audit store re-issues `CREATE TABLE` per write/read | — | S | — | Security/Audit/SqliteCanonicalAuditStore.cs |
|
||||||
|
| SEC-20 | Low | `session_id` metric tag is unbounded cardinality | P1 | S | — | Metrics/GatewayMetrics.cs, docs/Metrics.md |
|
||||||
|
| SEC-21 | Low | Snapshot publisher works every second regardless of audience | — | M | — | Dashboard/DashboardSnapshotService.cs |
|
||||||
|
| SEC-22 | Low | `docs/Authentication.md` documents types no longer in this repo | P2 | M | — | docs/Authentication.md |
|
||||||
|
| SEC-23 | Low | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` | — | S | SEC-03, SEC-04 | Configuration/GatewayOptionsValidator.cs |
|
||||||
|
| SEC-24 | Low | Effective-config view omits the riskiest dashboard flags | — | S | SEC-04 | Configuration/GatewayConfigurationProvider.cs |
|
||||||
|
| SEC-25 | Low | Per-session EventsHub ACL is an acknowledged TODO | P2 | M | SEC-02 | Dashboard/Hubs/EventsHub.cs |
|
||||||
|
| SEC-26 | Low | Audit trail has no retention or pruning | — | M | — | Security/Audit/SqliteCanonicalAuditStore.cs |
|
||||||
|
| SEC-27 | Low | Dashboard `GatewayStatus` is hardcoded Healthy | — | S | — | Dashboard/DashboardSnapshotService.cs |
|
||||||
|
| SEC-28 | Info | Positive security observations (preserve under change) | — | S | — | (multiple) |
|
||||||
|
| SEC-29 | Info | UI-stack rule verified compliant | — | S | — | wwwroot/lib/ |
|
||||||
|
| SEC-30 | Info | Value-logging feature is unwired end-to-end | P2 | S | SEC-13 | Diagnostics/GatewayLogRedactor.cs, docs/Diagnostics.md |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-01 — Windows-absolute default paths become relative files off-Windows `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-1)* Three option defaults are Windows-absolute string literals: `AuthenticationOptions.SqlitePath` (`Configuration/AuthenticationOptions.cs:9`, mirrored `appsettings.json:17`), `TlsOptions.SelfSignedCertPath` (`Configuration/TlsOptions.cs:11-12`), and Galaxy `SnapshotCachePath` (`appsettings.json:80`). On Unix these contain no path separator, so SQLite/PFX writers treat the whole string `C:\ProgramData\MxGateway\gateway-auth.db` as a single filename relative to the content root — which is how the stray auth DB materialized under `src/ZB.MOM.WW.MxGateway.Server/`. The validator does not catch it: `AddIfInvalidPath` only requires `Path.GetFullPath` to *succeed* (`Configuration/GatewayOptionsValidator.cs:383-410`), and a bare filename is a valid relative path.
|
||||||
|
|
||||||
|
**Impact.** The auth DB (and, if an HTTPS endpoint is configured, a private-key PFX) silently lands in the working directory, its location varying by host OS and launch CWD. Keys created under one CWD are invisible under another. A materialized key DB in the tree is protected only by the generic `*.db` gitignore; a PFX would not be.
|
||||||
|
|
||||||
|
**Design.** Derive defaults from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` + `Path.Combine`, which resolves to `C:\ProgramData` on Windows and `/usr/share` (or the container equivalent) elsewhere, and make the validator *reject* non-rooted paths for these keys so a relative override fails fast rather than writing into CWD. Rejected alternative: keeping the literals and only fixing the validator — that still leaves a broken default for any non-Windows run and diverges appsettings from code. Rooting enforcement is chosen over auto-rooting because the paths are security-sensitive (auth DB, private key) and silent relocation is worse than a boot error.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/AuthenticationOptions.cs:9`, `Configuration/TlsOptions.cs:11-12`: replace the literal initializers with a static helper, e.g. `Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), "MxGateway", "gateway-auth.db")`. Keep `appsettings.json` values as the explicit Windows deployment path (they override the code default on the production hosts).
|
||||||
|
- `Configuration/GatewayOptionsValidator.cs`: add an `AddIfNotRooted` check (`!Path.IsPathRooted(value)`) applied to `SqlitePath`, `SelfSignedCertPath`, and Galaxy `SnapshotCachePath`, alongside the existing `AddIfInvalidPath`.
|
||||||
|
- Delete the stray `src/ZB.MOM.WW.MxGateway.Server/C:\ProgramData\MxGateway\gateway-auth.db` file and add a guard test that fails if any `*.db` appears under `src/` (a repo-hygiene test in `ZB.MOM.WW.MxGateway.Tests`).
|
||||||
|
- Tests: extend `GatewayOptionsValidatorTests` with a non-rooted-path case per key; add the tree-hygiene test.
|
||||||
|
- Docs: note the derived defaults in `docs/GatewayConfiguration.md` and `docs/Authentication.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-02 — `AllowAnonymousLocalhost` satisfies AdminOnly, not just Viewer `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-2)* `Dashboard/DashboardAuthorizationHandler.cs:32-37` calls `context.Succeed(requirement)` for any loopback request before the role loop, and the same handler serves both `AdminOnly` and `AnyDashboardRole`. Lines 25-30 do the same for `Authentication.Mode == Disabled` — for **remote** requests too. `AllowAnonymousLocalhost` defaults `true` (`Configuration/DashboardOptions.cs:9`).
|
||||||
|
|
||||||
|
**Impact.** Anything gated solely by `MxGateway.Dashboard.Admin` is authorized for an anonymous local process; `HubClientsPolicy` uses the same requirement, so a loopback process reaches every SignalR hub — the snapshot hub pushes the API-key inventory and effective config every second, and `EventsHub.SubscribeSession` joins any session's raw `MxEvent` feed. Destructive surfaces are saved today only by service-layer re-checks (`Dashboard/DashboardApiKeyAuthorization.cs`, `Dashboard/DashboardSessionAdminService.cs:33-39`), i.e. defense-in-depth, not the policy.
|
||||||
|
|
||||||
|
**Design.** Restrict the loopback bypass to the Viewer requirement: succeed only when `requirement.RequiredRoles` contains `DashboardRoles.Viewer` (the read-only requirement) and not when it is the Admin-only requirement. Decide deliberately whether anonymous loopback reaches hubs at all — recommended: keep hub read access for loopback (dashboard-on-the-box is the primary local-ops UX) but never Admin. This preserves the documented "anonymous localhost is read-only" contract at the policy layer instead of by accident. Note the loopback test trusts `Connection.RemoteIpAddress` (`:56-61`); if forwarded-headers middleware is ever added, this must be revisited — capture that as a comment.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/DashboardAuthorizationHandler.cs`: in the loopback branch, succeed only if `requirement.RequiredRoles.Contains(DashboardRoles.Viewer)` (the `AnyDashboardRole` requirement includes Viewer; `AdminOnly` does not). Leave the `Mode == Disabled` branch as-is but document that it is a global kill-switch, or gate it behind the same Viewer check for consistency (open question: whether `Disabled` should still grant admin — recommend no).
|
||||||
|
- Tests: add cases to `DashboardAuthorizationHandlerTests` asserting loopback satisfies `AnyDashboardRole` but is denied `AdminOnly`.
|
||||||
|
- Docs: reconcile `docs/GatewayDashboardDesign.md` and `CLAUDE.md`'s "anonymous localhost bypasses auth on loopback" wording to say "read-only".
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~DashboardAuthorizationHandler"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-03 — Dashboard cookie lost its `__Host-` prefix; four docs still promise it `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-3)* `Dashboard/DashboardAuthenticationDefaults.cs:38` sets `CookieName = "MxGatewayDashboard"`, but `gateway.md:211`, `docs/GatewayDashboardDesign.md:425`, `docs/GatewayProcessDesign.md:686`, `docs/ImplementationPlanGateway.md:454`, and `CLAUDE.md` all still claim `__Host-MxGatewayDashboard`. Only `docs/GatewayConfiguration.md:170` is correct.
|
||||||
|
|
||||||
|
**Impact.** The `__Host-` browser guarantees (Secure required, no `Domain`, `Path=/`) are not in effect. The cookie is still HttpOnly/SameSite=Strict/SecurePolicy-controlled via `ZbCookieDefaults.Apply` (`Dashboard/DashboardServiceCollectionExtensions.cs:93-109`), and `RequireHttpsCookie=false` plus the configurable `CookieName` are legitimate reasons the prefix was dropped — so this is primarily doc drift, but security docs overstate protections.
|
||||||
|
|
||||||
|
**Design.** Prefer restoring the guarantee conditionally over a pure doc fix: when `RequireHttpsCookie` is true and no explicit `CookieName` override is set, default to `__Host-MxGatewayDashboard`; otherwise keep the plain name (the `__Host-` prefix is incompatible with `SameAsRequest`/HTTP). This is a small change in the PostConfigure at `DashboardServiceCollectionExtensions.cs:119-133` and recovers the documented protection for the default secure deployment. Either way, update the four stale docs in the *same* change (repo rule).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/DashboardServiceCollectionExtensions.cs`: in the `CookieAuthenticationOptions` PostConfigure, when `RequireHttpsCookie` and `CookieName` is null/blank, set `cookieOptions.Cookie.Name = "__Host-MxGatewayDashboard"`. Guard: never apply `__Host-` when `SecurePolicy != Always` (would be silently dropped — this ties to SEC-23's consistency check).
|
||||||
|
- `Dashboard/DashboardAuthenticationDefaults.cs`: keep `MxGatewayDashboard` as the non-secure fallback constant.
|
||||||
|
- Docs: update `gateway.md`, `docs/GatewayDashboardDesign.md`, `docs/GatewayProcessDesign.md`, `docs/ImplementationPlanGateway.md`, `CLAUDE.md` to describe the actual conditional contract.
|
||||||
|
- Tests: `DashboardServiceCollectionExtensionsTests` (or a cookie-options resolution test) asserting the name flips with `RequireHttpsCookie`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the dashboard-config test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-04 — `DisableLogin` has no production guard `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-4)* `Dashboard/DashboardServiceCollectionExtensions.cs:63-90` swaps in `DashboardAutoLoginAuthenticationHandler` under the cookie scheme for **all** clients when `MxGateway:Dashboard:DisableLogin` is true; the handler authenticates every request, remote included (`Dashboard/DashboardAutoLoginAuthenticationHandler.cs:56-62`). `GatewayOptionsValidator.ValidateDashboard` (`Configuration/GatewayOptionsValidator.cs:219-253`) never inspects `DisableLogin`, and the warning fires only on first `GatewayOptions` resolution (not guaranteed at process start).
|
||||||
|
|
||||||
|
**Impact.** One copied config flag turns the whole dashboard — API-key CRUD and worker Kill included — into an unauthenticated admin surface on a 0.0.0.0-bound port.
|
||||||
|
|
||||||
|
**Design.** Fail fast in production: in `ValidateDashboard`, when `IHostEnvironment.IsProduction()` and `DisableLogin` is true, emit a validator error so startup aborts like every other misconfiguration. Rejected alternative: silently forcing the flag off — hides operator intent and diverges config from behavior. The dev warning stays for non-production. Co-design with SEC-03 (cookie reconciliation) and SEC-23 (validator coverage) so the dashboard-security fixes land together.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/GatewayOptionsValidator.cs`: inject/accept the environment name (the validator already runs at startup; thread `IHostEnvironment` in, or pass an `isProduction` flag) and add the check in `ValidateDashboard`.
|
||||||
|
- Tests: `GatewayOptionsValidatorTests` — `DisableLogin=true` + Production → invalid; + Development → valid.
|
||||||
|
- Docs: `docs/GatewayConfiguration.md` DisableLogin row notes the production hard-stop.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-05 — Hub bearer tokens irrevocable for 30 min, carried in query string `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-5)* `Dashboard/HubTokenService.cs:29,44-52` mints a 30-minute data-protected token with no jti/revocation state and roles frozen at issue; `Dashboard/HubTokenAuthenticationHandler.cs:59-61` accepts `?access_token=` on the WebSocket upgrade. Logout (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:136-155`) clears the cookie but not outstanding tokens.
|
||||||
|
|
||||||
|
**Impact.** A token captured from an access log or proxy grants live snapshot/alarm/event access for up to 30 minutes after logout; role changes/revocations do not take effect until expiry.
|
||||||
|
|
||||||
|
**Design.** Two independent, low-cost mitigations: (1) shorten `TokenLifetime` to ~5 minutes — the factory refreshes per reconnect (`docs/GatewayDashboardDesign.md:497-499`), so the shorter window is transparent to clients and bounds exposure; (2) confirm no request-path logging captures query strings (Serilog request logging is not enabled — keep it off, or scrub `access_token` if it is ever added). Full server-side revocation (a jti denylist keyed on the data-protection payload) is the heavier option; recommend deferring it and relying on the short lifetime, since the tokens are already encrypted and single-purpose. State the open question: if per-session ACLs (SEC-25) land, tokens will need role/session binding and revocation becomes worthwhile.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/HubTokenService.cs:29`: `TokenLifetime = TimeSpan.FromMinutes(5)`.
|
||||||
|
- Add an XML-doc note at `HubTokenAuthenticationHandler` that query-string carriage is the SignalR pattern and must not be request-logged.
|
||||||
|
- Docs: `docs/GatewayDashboardDesign.md` token section — update the lifetime and the logout-vs-token caveat.
|
||||||
|
- Tests: `HubTokenServiceTests` — token invalid after lifetime; issue/validate round-trip.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~HubTokenService"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-06 — LDAP plaintext-by-default with a committed service password `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-6)* `Configuration/LdapOptions.cs:49-61` defaults `Transport=None`, `AllowInsecure=true`, `ServiceAccountPassword="serviceaccount123"`; `appsettings.json:21-33` ships the same. `glauth.md:30,327` confirms dev LDAPS is disabled and binds send cleartext. The validator enforces the `Transport=None ⇒ AllowInsecure` consistency rule (`GatewayOptionsValidator.cs:82-85`) but nothing distinguishes dev from prod.
|
||||||
|
|
||||||
|
**Impact.** Every dashboard login sends the operator's password cleartext to `10.100.0.35:3893`, and a service-account credential is in source control.
|
||||||
|
|
||||||
|
**Design.** These are deliberate dev defaults (the shadow-options rationale at `LdapOptions.cs:20-28` is explicit that the shared library is secure-by-default). The fix is a production posture, not a default flip: (1) require `Transport=Ldaps`/`StartTls` + `AllowInsecure=false` in production deployment docs, and add an `IsProduction()` validator check (mirroring SEC-04) that rejects `Transport=None` in production; (2) move `ServiceAccountPassword` out of `appsettings.json` to env-var/secret configuration on the deployed hosts and rotate the committed value. LDAP-injection escaping is delegated to `ZB.MOM.WW.Auth.Ldap` (bind-then-search, `Dashboard/DashboardAuthenticator.cs:41-47`) and cannot be verified here — flag for review in the donor repo.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/GatewayOptionsValidator.cs`: in `ValidateLdap`, when Production and `Transport == None`, emit an error (co-locate with SEC-04's env plumbing).
|
||||||
|
- Deployment: keep `serviceaccount123` only for local GLAuth dev; document env-var override (`MxGateway__Ldap__ServiceAccountPassword`) for the NSSM-wrapped hosts; rotate the dev credential's reuse.
|
||||||
|
- Docs: `docs/GatewayConfiguration.md` Ldap section and a production hardening note referencing `glauth.md`.
|
||||||
|
- Tests: `GatewayOptionsValidatorTests` — `Transport=None` + Production → invalid.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the validator test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-07 — `QueryActiveAlarmsRequest` missing from scope resolver; tests mislabeled `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review STA-1)* `Security/Authorization/GatewayGrpcScopeResolver.cs:15-29` has arms for `StreamAlarmsRequest` (→ `EventsRead`) and `AcknowledgeAlarmRequest` but **no** `QueryActiveAlarmsRequest` arm, so that RPC (`Grpc/MxAccessGatewayService.cs`, proto `mxaccess_gateway.proto`) falls to the `_ => GatewayScopes.Admin` default. The two `ServerStreamingServerHandler_QueryActiveAlarms…` tests construct `new StreamAlarmsRequest()` instead (`Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:322-359`), so they pass while testing the wrong message. `docs/Authorization.md`'s scope table omits the RPC.
|
||||||
|
|
||||||
|
**Impact.** Fail-closed (not a vulnerability) but a client holding `events:read` gets `PermissionDenied` demanding `admin` on `QueryActiveAlarms`, contradicting the stated design that alarm snapshot data shares the event surface.
|
||||||
|
|
||||||
|
**Design.** Add the missing arm `QueryActiveAlarmsRequest => GatewayScopes.EventsRead`, consistent with `StreamAlarmsRequest`. Fix both tests to use the real request type so they actually exercise the arm. Add the row to `docs/Authorization.md`. No parity or backward-compat concern — it only relaxes an over-strict requirement to the documented scope.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Security/Authorization/GatewayGrpcScopeResolver.cs:22`: add `QueryActiveAlarmsRequest => GatewayScopes.EventsRead,` next to `StreamAlarmsRequest`.
|
||||||
|
- `Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:322-359`: change `new StreamAlarmsRequest()` to `new QueryActiveAlarmsRequest()` in both `QueryActiveAlarms` tests; keep the `events:read`-passes / missing-scope-denies assertions.
|
||||||
|
- Docs: `docs/Authorization.md` scope table — add `QueryActiveAlarms → events:read`; also `docs/Grpc.md` if it omits the RPC.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayGrpcAuthorizationInterceptor"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-08 — Per-RPC SQLite read + `last_used_utc` write; no verification cache `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review PERF-1)* The interceptor calls `IApiKeyVerifier.VerifyAsync` per call (`Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:69-71`); the verifier contract is find-by-id → hash → compare → `MarkKeyUsedAsync` per `docs/Authentication.md`, and the `last_used_utc` write "runs on every authenticated request". The interceptor also deserializes the constraints JSON per call (`Security/Authentication/GatewayApiKeyIdentityMapper.cs:41`).
|
||||||
|
|
||||||
|
**Impact.** WAL + busy-timeout keep it correct, but a per-call database **write** is the throughput ceiling on the hot path (bulk reads at high frequency are the primary workload), causing continuous WAL churn and making auth-store latency a per-RPC tail-latency contributor.
|
||||||
|
|
||||||
|
**Design.** Add a short-TTL (5–30 s) in-memory verification cache keyed by key id + presented-hash, invalidated on revoke/rotate — all mutations flow through in-process `ApiKeyAdminCommands`, so invalidation is a direct call, not a cross-process concern. Coalesce `last_used_utc` updates to at most one write per key per minute (track last-flushed-at in the cache entry). Cache the parsed constraints on the identity so the per-call JSON deserialize disappears. This is `L` effort and likely needs a small surface addition in `ZB.MOM.WW.Auth.ApiKeys` (a `MarkKeyUsed` batching hook or a verifier that returns the raw stored record so the gateway can cache it) — recommend coordinating with the donor library rather than shadowing its store. Open question: whether the cache lives gateway-side (wraps `IApiKeyVerifier`) or in the shared library; recommend gateway-side wrapper first (no library release dependency), keyed on a constant-time hash of the presented secret to avoid caching plaintext.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- New gateway-side `CachingApiKeyVerifier : IApiKeyVerifier` decorator registered in `Security/Authentication/AuthStoreServiceCollectionExtensions.cs`; memory cache (`IMemoryCache`) with per-entry TTL; invalidation entry point called from `ApiKeyAdminCommands` on create/revoke/rotate.
|
||||||
|
- Coalesce `MarkKeyUsedAsync`: only forward when the cached entry's last-marked timestamp is older than the coalesce window.
|
||||||
|
- `GatewayApiKeyIdentityMapper.cs:41`: cache the deserialized constraints on the mapped identity.
|
||||||
|
- Tests: `CachingApiKeyVerifierTests` — cache hit avoids store read; revoke invalidates; `last_used` coalesced to ≤1/min. Load-style assertion optional.
|
||||||
|
- Docs: `docs/Authentication.md` and `docs/Metrics.md` — document the cache TTL/coalesce and any new config knob.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~CachingApiKeyVerifier"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-09 — Dashboard design-doc `GroupToRole` sample fails startup validation `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** *(review CON-1)* `docs/GatewayDashboardDesign.md:515-519` shows `"GroupToRole": { "GwAdmin": "Admin", … }`, but `GatewayOptionsValidator.cs:233-238` accepts only `DashboardRoles.Admin` = `"Administrator"` (`Dashboard/DashboardRoles.cs:14`) or `"Viewer"`. Live `appsettings.json:66-69` correctly uses `"Administrator"`. The doc also calls the role `Admin` in prose (lines 412-413, 434).
|
||||||
|
|
||||||
|
**Impact.** An operator copying the documented sample gets a fail-fast boot error.
|
||||||
|
|
||||||
|
**Design.** Doc-only fix: update the sample and prose to the canonical `Administrator` value (the rename is annotated in `glauth.md:79-83`). No code change — the validator is correct.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/GatewayDashboardDesign.md`: change the `GroupToRole` sample values to `"Administrator"`/`"Viewer"` and align role prose.
|
||||||
|
|
||||||
|
**Verification.** Doc change; no build. Cross-check the sample against `Dashboard/DashboardRoles.cs`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-10 — No API-key expiry `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-1)* The `api_keys` schema carries `created_utc`/`last_used_utc`/`revoked_utc` only (confirmed from the stray DB `sqlite_master` and `docs/Authentication.md`); no expiry field, no expiry check in the verification flow, no staleness surfacing beyond `LastUsedUtc`.
|
||||||
|
|
||||||
|
**Impact.** Keys live until an operator revokes; a leaked key is valid indefinitely.
|
||||||
|
|
||||||
|
**Design.** Add optional `expires_utc` to the shared `ZB.MOM.WW.Auth.ApiKeys` schema + verifier (an expired key verifies as failed, opaque to the client like every other auth failure). Surface age/staleness on the dashboard API Keys page. This is `L` because the store/verifier live in the donor package — coordinate a schema migration there; the gateway consumes it. Gateway-side work: expose an optional `--expires` on `apikey create`, display expiry/staleness in `DashboardApiKeyManagementService` summaries. Open question: default expiry (recommend none — opt-in — to preserve current behavior; backward-compatible).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Shared library: `expires_utc` column + migration + verifier check + `ApiKeyPepperUnavailableException`-style typing (see SEC-18).
|
||||||
|
- Gateway: `Program.cs` apikey create option; `Dashboard/DashboardApiKeyManagementService.cs` summary projection; API Keys Razor page staleness badge.
|
||||||
|
- Docs: `docs/Authentication.md` token lifecycle + `docs/Authorization.md` if scopes interplay.
|
||||||
|
- Tests: verifier expiry test in the library; gateway summary test.
|
||||||
|
|
||||||
|
**Verification.** After the library bump: `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~DashboardApiKeyManagement"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-11 — No rate limiting or lockout on either auth surface `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-2)* The gRPC interceptor verifies unconditionally per call (`GatewayGrpcAuthorizationInterceptor.cs:54-91`); `/auth/login` has no throttle (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:38-43,99-134`). The only brake is dev GLAuth's per-IP 3-fail lockout (`glauth.md:35`), absent in production AD and hazardous behind shared NAT.
|
||||||
|
|
||||||
|
**Impact.** Unbounded online guessing of API-key secrets (each guess costs a SQLite read — compounds with SEC-08) and unthrottled LDAP credential stuffing relayed to the directory.
|
||||||
|
|
||||||
|
**Design.** Add ASP.NET Core rate limiting on `/auth/login` (fixed-window per remote IP) and a cheap per-peer failure counter in front of `VerifyAsync` (sliding-window limiter keyed on peer + presented key id) that short-circuits with `ResourceExhausted`/`Unauthenticated` before the store read. Co-design with SEC-08 (both sit on the auth hot path; the cache and the limiter share the peer key). Recommend the built-in `Microsoft.AspNetCore.RateLimiting` middleware for the dashboard and an in-process counter (bounded LRU) for gRPC to avoid a dependency in the interceptor. Note the NAT caveat from `glauth.md:323-325`: prefer per-key-id over pure per-IP for the gRPC counter.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/DashboardEndpointRouteBuilderExtensions.cs`: apply a named rate-limiter policy to the login POST route.
|
||||||
|
- `Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs` (or a decorator): per-peer failure counter checked before `VerifyAsync`; reset on success.
|
||||||
|
- Config: new `MxGateway:Security:LoginRateLimit*` / `ApiKeyFailureLimit*` keys bound in `GatewayOptions`, validated.
|
||||||
|
- Tests: interceptor test asserting Nth failed attempt short-circuits; dashboard integration test for 429 on burst.
|
||||||
|
- Docs: `docs/GatewayConfiguration.md`, `docs/Authorization.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the interceptor + dashboard test filters.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-12 — Dashboard Close/Kill bypass the canonical audit store `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-3)* `Dashboard/DashboardSessionAdminService.cs:64-69,129-134` record close/kill via `ILogger` only, whereas API-key operations write `AuditEvent`s through `IAuditWriter` (`Dashboard/DashboardApiKeyManagementService.cs:242-264`).
|
||||||
|
|
||||||
|
**Impact.** Destructive operational actions (killing a worker mid-production) leave no durable, queryable audit row — invisible to the dashboard's recent-audit view and subject to log rotation.
|
||||||
|
|
||||||
|
**Design.** Emit `dashboard-close-session` / `dashboard-kill-worker` `AuditEvent`s through the existing `IAuditWriter`, mirroring the API-key pattern. Actor resolution helpers already exist in the service (`ResolveActor`, `ResolveRemoteAddress`). Keep the `ILogger` line too (operational tail). No new store — reuse `SqliteCanonicalAuditStore`.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/DashboardSessionAdminService.cs`: inject `IAuditWriter`; after a successful close/kill, write an `AuditEvent` with actor, session id, remote address, and `AlreadyClosed`. Failure paths write a denied/failed event as the API-key path does.
|
||||||
|
- Tests: `DashboardSessionAdminServiceTests` — successful close/kill produces the expected `AuditEvent`; assert against a fake `IAuditWriter`.
|
||||||
|
- Docs: `docs/Authorization.md` audit-event catalog — add the two action names.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~DashboardSessionAdmin"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-13 — Redactor's credential-command list omits secured-bulk variants `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-7)* `Diagnostics/GatewayLogRedactor.cs:11-16` lists `AuthenticateUser`, `WriteSecured`, `WriteSecured2` but not `WriteSecuredBulk`/`WriteSecured2Bulk`, which exist as command kinds (`Security/Authorization/GatewayGrpcScopeResolver.cs:41-45`). Currently latent — `RedactCommandValue`/`IsCredentialBearingCommand` have no call sites (grep-verified), so no value logging occurs.
|
||||||
|
|
||||||
|
**Impact.** If value logging is ever wired (SEC-30), secured-bulk payloads (credential-bearing) would pass the unconditional-redaction check.
|
||||||
|
|
||||||
|
**Design.** Add the two bulk names now — trivial and forward-safe. Add a test asserting every `WriteSecured*` command kind is treated as credential-bearing, so the redactor cannot drift from the command catalog again.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Diagnostics/GatewayLogRedactor.cs:13-15`: add `"WriteSecuredBulk"`, `"WriteSecured2Bulk"` to `SensitiveCommandMethods`.
|
||||||
|
- Tests: `GatewayLogRedactorTests` — enumerate `MxCommandKind` names starting with `WriteSecured` and assert `IsCredentialBearingCommand` is true for each.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayLogRedactor"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-14 — `/metrics` + `/health` unauthenticated; a metric leaks session ids `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-8)* `GatewayApplication.cs:196-197` maps `MapZbHealth()`/`MapZbMetrics()` (shared packages) with no `RequireAuthorization`; `Metrics/GatewayMetrics.cs:354` tags `mxgateway.heartbeats.failed` with `session_id`.
|
||||||
|
|
||||||
|
**Impact.** If the shared packages don't enforce auth internally (not verifiable here), an unauthenticated scraper on the gRPC/dashboard port can read telemetry including live session identifiers usable against the anonymous-localhost hub surface (SEC-02).
|
||||||
|
|
||||||
|
**Design.** Verify the shared packages' endpoint auth; if anonymous, either bind metrics to a loopback-only endpoint or gate with `RequireAuthorization`. Combine with SEC-20 (drop the `session_id` tag) so the leak closes even if the endpoint stays open. Depends-on SEC-02 (hub surface) and SEC-20 (tag).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Confirm `MapZbMetrics`/`MapZbHealth` auth behavior in the shared package; if open, add `.RequireAuthorization()` or bind to a separate loopback Kestrel endpoint.
|
||||||
|
- Docs: `docs/Metrics.md`/`docs/Diagnostics.md` state the endpoint's auth posture.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; manual `curl` of `/metrics` on a running gateway to confirm the posture.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-15 — GET `/logout` skips antiforgery `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-9)* `Dashboard/DashboardEndpointRouteBuilderExtensions.cs:53-58` intentionally skips antiforgery on GET `/logout` (comment acknowledges it). POST login/logout validate antiforgery (`:104,140`).
|
||||||
|
|
||||||
|
**Impact.** A third-party page can sign a dashboard operator out (nuisance CSRF; no state beyond the session).
|
||||||
|
|
||||||
|
**Design.** Acceptable as documented — logout is self-destructive with no side effects. No code change; if logout ever grows side effects, add a confirmation interstitial. Recorded so the tracking doc captures the accepted risk.
|
||||||
|
|
||||||
|
**Implementation.** None (documented acceptance). Optionally add a one-line note to `docs/GatewayDashboardDesign.md`.
|
||||||
|
|
||||||
|
**Verification.** N/A.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-16 — CLI accepts the pepper as a command-line argument `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-10)* `Program.cs:37-40` maps `command.Pepper` → `MxGateway:ApiKeyPepper`.
|
||||||
|
|
||||||
|
**Impact.** The pepper lands in shell history and the process command line visible to other local users.
|
||||||
|
|
||||||
|
**Design.** Prefer an environment variable (`MxGateway__ApiKeyPepper`) or an interactive prompt over the CLI arg; keep the arg for scripted use but document the exposure in `--help`. Low-risk, small change.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Program.cs`: fall back to the `MxGateway__ApiKeyPepper` env var when `--pepper` is absent; note the risk in the apikey subcommand help text.
|
||||||
|
- Docs: CLAUDE.md apikey example already uses scopes; add a pepper-via-env note in `docs/Authentication.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; run `apikey --help`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-17 — Interceptor does not override client-streaming/duplex handlers `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review STA-2)* `Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:21-47` overrides only `UnaryServerHandler` and `ServerStreamingServerHandler`. All current RPCs are unary or server-streaming (verified against both `.proto` files), so nothing bypasses auth today.
|
||||||
|
|
||||||
|
**Impact.** A future duplex/client-streaming RPC would run with no authentication and no scope check, silently.
|
||||||
|
|
||||||
|
**Design.** Override `ClientStreamingServerHandler` and `DuplexStreamingServerHandler` to run the same auth path (fail-closed to `admin` via the resolver default) — a loud, safe default so any future streaming RPC is authenticated by construction rather than accidentally open.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs`: add the two overrides calling `AuthenticateAndAuthorizeAsync` then the continuation, mirroring the existing two.
|
||||||
|
- Tests: `GatewayGrpcAuthorizationInterceptorTests` — a client-streaming/duplex handler is denied without a key.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the interceptor test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-18 — Pepper-unavailable detection matches library message text `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review STA-3)* `Dashboard/DashboardApiKeyManagementService.cs:21,281-282` catches `InvalidOperationException` whose `Message` contains `"pepper unavailable"`.
|
||||||
|
|
||||||
|
**Impact.** A wording change in `ZB.MOM.WW.Auth.ApiKeys` turns the friendly "pepper not configured" result into an unhandled exception on the Blazor circuit.
|
||||||
|
|
||||||
|
**Design.** Ask the library to expose a typed `ApiKeyPepperUnavailableException` (the pre-cutover code had one per `docs/Authentication.md:68`) and catch that instead of string-matching. Coordinate with SEC-10 (also a library change). Until the library ships it, keep the string match but add a test pinning the expected message so drift is caught in CI.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Shared library: add/restore the typed exception.
|
||||||
|
- `Dashboard/DashboardApiKeyManagementService.cs`: catch the typed exception; drop the message match.
|
||||||
|
- Tests: `DashboardApiKeyManagementServiceTests` — pepper-unavailable yields the friendly result, not an unhandled throw.
|
||||||
|
|
||||||
|
**Verification.** After library bump: `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the management-service test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-19 — Canonical audit store re-issues `CREATE TABLE` per write/read `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review STA-4)* `Security/Audit/SqliteCanonicalAuditStore.cs:54,94,131-136` runs `CREATE TABLE IF NOT EXISTS audit_event` on every write and read.
|
||||||
|
|
||||||
|
**Impact.** An extra round-trip per audit op, and a schema defined outside the migrator — a future column change has no migration path (the `IF NOT EXISTS` silently keeps the old shape).
|
||||||
|
|
||||||
|
**Design.** Move `audit_event` creation into the startup migration path (alongside the api-key schema migrator) and drop the per-call ensure. Ties to UND-5/SEC-26 (retention) — do both in the audit-store pass.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Security/Audit/SqliteCanonicalAuditStore.cs`: remove per-call `CREATE TABLE`; ensure the migration hosted service creates `audit_event`.
|
||||||
|
- Tests: `SqliteCanonicalAuditStoreTests` — write/read succeed post-migration; no `CREATE TABLE` on the hot path (assert via a command-counting fake or integration check).
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~CanonicalAudit"`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-20 — `session_id` metric tag is unbounded cardinality `Low` · `P1`
|
||||||
|
|
||||||
|
**Finding.** *(review PERF-2)* `Metrics/GatewayMetrics.cs:354` tags `mxgateway.heartbeats.failed` with `session_id`; `docs/Metrics.md:47` documents it. The in-memory `EventsBySession` map is correctly pruned on close (`:310-313`) — only the exported tag is the problem.
|
||||||
|
|
||||||
|
**Impact.** Every session ever opened mints a new exporter time series; long-running gateways with churn bloat Prometheus/OTLP storage.
|
||||||
|
|
||||||
|
**Design.** Drop the `session_id` tag from the exported counter (keep the aggregate). Per-session attribution stays available via the dashboard snapshot and log scope. Co-design with SEC-08 (both land in the observability pass) and closes the leak flagged in SEC-14.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Metrics/GatewayMetrics.cs:354`: `_heartbeatFailuresCounter.Add(1)` without the tag.
|
||||||
|
- Docs: `docs/Metrics.md:47` — remove the `session_id` dimension.
|
||||||
|
- Tests: `GatewayMetricsTests` (if present) — counter increments without a `session_id` tag.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the metrics test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-21 — Snapshot publisher works every second regardless of audience `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review PERF-3)* `Dashboard/DashboardSnapshotService.cs:134-166` runs the publisher loop; `:256-279` (`RefreshApiKeySummariesAsync`) calls `IApiKeyAdminStore.ListAsync` every tick (gated, 2 s-timeboxed); `:76-105` rebuilds session/worker/metric/fault/config projections each tick. The Galaxy breakdown is properly memoized by sequence (`:107-129`).
|
||||||
|
|
||||||
|
**Impact.** Constant background SQLite reads and allocation churn on an idle gateway; the full snapshot (API-key summaries + effective config) is serialized to all hub clients every second.
|
||||||
|
|
||||||
|
**Design.** Refresh the API-key summary on a slower cadence or on mutation (all mutations are in-process via `ApiKeyAdminCommands`, so an invalidation hook is exact — shares the mechanism with SEC-08's cache invalidation). Skip publication when the hub has no connections. Keep the 1 s cadence for live session/metric projections.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/DashboardSnapshotService.cs`: gate `RefreshApiKeySummariesAsync` behind a mutation flag (set by `ApiKeyAdminCommands`) or a longer interval; short-circuit the publish when `IHubContext` has zero clients.
|
||||||
|
- Tests: `DashboardSnapshotServiceTests` — key summary refreshes on mutation, not every tick; no publish with zero connections.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the snapshot-service test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-22 — `docs/Authentication.md` documents types no longer in this repo `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** *(review CON-2)* The doc presents `ApiKeyParser`, `ApiKeySecretGenerator`, `SqliteApiKeyStore`, and a parameterless `AddSqliteAuthStore()` (`docs/Authentication.md:9-28,36-48,253-272`), but the real registration is the package-delegating two-parameter method (`Security/Authentication/AuthStoreServiceCollectionExtensions.cs:41-105`) and the store/verifier live in `ZB.MOM.WW.Auth.ApiKeys`. The doc also says `api_key_audit` is written on every denial (`:122,131`), but the override redirects all writes to `audit_event`, leaving `api_key_audit` unused (`Security/Audit/CanonicalForwardingApiKeyAuditStore.cs:21-25`).
|
||||||
|
|
||||||
|
**Impact.** Violates the "no stale prose" rule; a maintainer auditing hashing/storage from the doc looks for code that isn't there.
|
||||||
|
|
||||||
|
**Design.** Rewrite `docs/Authentication.md` as a consumer-side doc: token format, options binding, the audit-store override (`api_key_audit` → `audit_event`), and a pointer to the donor library for internals. Fold in SEC-10/SEC-18 (typed exception, expiry) as they land.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/Authentication.md`: rewrite per above; remove code excerpts of moved types.
|
||||||
|
|
||||||
|
**Verification.** Doc change; cross-check each referenced type against the current tree.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-23 — Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review CON-4)* `Configuration/GatewayOptionsValidator.cs:219-253` validates `GroupToRole`, snapshot interval, and the two limits, but ignores `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie`, and `CookieName` (all documented at `docs/GatewayConfiguration.md:169-177`). A `CookieName` beginning `__Host-` with `RequireHttpsCookie=false` yields a cookie browsers silently drop — no startup diagnosis.
|
||||||
|
|
||||||
|
**Impact.** Silent misconfiguration classes the validator exists to prevent.
|
||||||
|
|
||||||
|
**Design.** Add a `__Host-`/`RequireHttpsCookie` consistency check (reject `CookieName` starting `__Host-` unless `RequireHttpsCookie` is true) and the SEC-04 production guard for `DisableLogin`; validate `AutoLoginUser` semantics. Co-design with SEC-03 (which introduces the `__Host-` default) and SEC-04.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/GatewayOptionsValidator.cs` `ValidateDashboard`: add the checks.
|
||||||
|
- Tests: `GatewayOptionsValidatorTests` — `__Host-` + `RequireHttpsCookie=false` → invalid.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the validator test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-24 — Effective-config view omits the riskiest dashboard flags `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review CON-5)* `Configuration/GatewayConfigurationProvider.cs:57-64` projects `EffectiveDashboardConfiguration` without `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie`, or `CookieName`; TLS and Alarms sections are absent from `EffectiveGatewayConfiguration` entirely.
|
||||||
|
|
||||||
|
**Impact.** The Settings page cannot show an operator that login is disabled — the one flag they most need to see.
|
||||||
|
|
||||||
|
**Design.** Add the missing (non-secret) fields to the effective-config projection so the dashboard surfaces them. Depends-on SEC-04 (same flag). Optionally add TLS/Alarms sections; keep secrets redacted as the provider already does for the pepper/LDAP password.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Configuration/GatewayConfigurationProvider.cs`: extend `EffectiveDashboardConfiguration` with the four flags; consider TLS/Alarms sections.
|
||||||
|
- Tests: `GatewayConfigurationProviderTests` — effective config includes `DisableLogin` etc.; secrets still redacted.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the provider test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-25 — Per-session EventsHub ACL is an acknowledged TODO `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-4)* `Dashboard/Hubs/EventsHub.cs:29-44` carries `TODO(per-session-acl)` — any Viewer (and anonymous localhost per SEC-02) can `SubscribeSession` to any session's raw event feed, bypassing the per-gRPC-subscriber filtering (`docs/GatewayDashboardDesign.md:170`).
|
||||||
|
|
||||||
|
**Impact.** Acceptable per the in-code v1 rationale, but it is the single seam where tag values reach the least-privileged principals; combined with `ShowTagValues=false` expectations it can surprise operators.
|
||||||
|
|
||||||
|
**Design.** Keep the TODO tied to the tracked per-session-ACL work item (P2 roadmap item 12). As a near-term hardening, redact event values in the dashboard mirror when `ShowTagValues` is false, so the seam cannot leak values regardless of the ACL. Depends-on SEC-02 (loopback reaching hubs). Full ACL: introduce a role/scope that scopes a Viewer to a session/tenant and enforce it at `SubscribeSession`.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/Hubs/EventsHub.cs` / `DashboardEventBroadcaster`: honor `ShowTagValues` by redacting values in the mirror now; add the session-access check when the scoping role lands.
|
||||||
|
- Tests: broadcaster test asserting values redacted when `ShowTagValues=false`.
|
||||||
|
- Docs: `docs/GatewayDashboardDesign.md` — clarify the current v1 posture.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the events-hub/broadcaster test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-26 — Audit trail has no retention or pruning `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-5)* `audit_event` is append-only with no cleanup path (`Security/Audit/SqliteCanonicalAuditStore.cs`); constraint denials append per denied bulk entry (`docs/Authorization.md:194-198`).
|
||||||
|
|
||||||
|
**Impact.** A misconfigured constrained client hammering denied reads grows the auth DB without bound.
|
||||||
|
|
||||||
|
**Design.** Add a retention sweep (age- or row-count-based, config-gated) run by a hosted service, or document the operational archive expectation. Recommend an age-based sweep with a generous default (e.g. 90 days) so the DB is self-limiting. Do this in the same audit-store pass as SEC-19.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Security/Audit/`: add a retention sweep (hosted service or piggyback the migration service) with `MxGateway:Security:AuditRetentionDays` config.
|
||||||
|
- Tests: sweep deletes rows older than the window; keeps recent.
|
||||||
|
- Docs: `docs/Authorization.md`/`docs/Diagnostics.md` retention note; `docs/GatewayConfiguration.md` new key.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the audit test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-27 — Dashboard `GatewayStatus` is hardcoded Healthy `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-6)* `Dashboard/DashboardSnapshotService.cs:17,96` sets `GatewayStatus` to a `HealthyStatus` constant.
|
||||||
|
|
||||||
|
**Impact.** The home page headline status never reflects registered health checks (e.g. `AuthStoreHealthCheck` unhealthy while the banner says Healthy).
|
||||||
|
|
||||||
|
**Design.** Project `HealthCheckService` results into the snapshot so the banner reflects real health. Inject `HealthCheckService`, run/observe the aggregated status on the publish tick (cheap — health checks are cached), map to the dashboard status enum.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `Dashboard/DashboardSnapshotService.cs`: inject `HealthCheckService`; set `GatewayStatus` from the aggregated `HealthReport`.
|
||||||
|
- Tests: `DashboardSnapshotServiceTests` — unhealthy check surfaces as non-Healthy status.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the snapshot-service test filter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-28 — Positive security observations (preserve under change) `Info` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review SEC-11)* Verified-good behaviors to preserve: fail-closed unrecognized-request → `admin` (`GatewayGrpcScopeResolver.cs:28`); opaque auth failures (`GatewayGrpcAuthorizationInterceptor.cs:66-78`); effective-config redacts pepper name + LDAP password (`GatewayConfigurationProvider.cs`); Galaxy connection string rebuilt field-by-field for display (`Dashboard/DashboardConnectionStringDisplay.cs`); global identity-property masking seam (`Diagnostics/GatewayLogRedactorSeam.cs`); self-signed PFX generation hardens permissions before writing and clears the buffer (`Security/Tls/SelfSignedCertificateProvider.cs:160-193`); scope strings validated against the canonical catalog on every creation path; dashboard key-id input constrained to a safe charset; single generic login-failure message; `SanitizeReturnUrl` blocks open redirects. No secret-logging call site found.
|
||||||
|
|
||||||
|
**Impact.** These are the load-bearing controls; regressions here are high-severity.
|
||||||
|
|
||||||
|
**Design.** No action. Guard against regression: the tests named in other entries (redactor, scope catalog, opaque-failure) already pin most of these — keep them.
|
||||||
|
|
||||||
|
**Implementation.** None. Reference in review sign-off.
|
||||||
|
|
||||||
|
**Verification.** N/A.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-29 — UI-stack rule verified compliant `Info` · `—`
|
||||||
|
|
||||||
|
**Finding.** *(review CON-3)* `wwwroot/lib/` contains only local Bootstrap CSS/JS; grep for MudBlazor/Radzen/Syncfusion/Telerik across `.csproj`/`.razor`/`.cs` returns nothing. The shared `ZB.MOM.WW.Theme` package supplies CSS only.
|
||||||
|
|
||||||
|
**Impact.** Confirms the CLAUDE.md "no Blazor UI component libraries" invariant holds.
|
||||||
|
|
||||||
|
**Design.** No action. Optionally add a build-time or test guard that fails if a banned package reference appears, to keep the invariant enforced rather than remembered.
|
||||||
|
|
||||||
|
**Implementation.** Optional repo-hygiene test asserting no MudBlazor/Radzen/FluentUI/Telerik/Syncfusion package references.
|
||||||
|
|
||||||
|
**Verification.** N/A (or run the hygiene test if added).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## SEC-30 — Value-logging feature is unwired end-to-end `Info` · `P2`
|
||||||
|
|
||||||
|
**Finding.** *(review UND-7)* `GatewayLogRedactor.RedactCommandValue`/`IsCredentialBearingCommand` have no call sites in the server (grep-verified); the opt-in value-logging flag described in `docs/Diagnostics.md:124-148` has no knob in `GatewayOptions`.
|
||||||
|
|
||||||
|
**Impact.** Currently the safest state (no values logged anywhere), but the doc implies a capability that doesn't exist end-to-end.
|
||||||
|
|
||||||
|
**Design.** Either wire the flag (add a `MxGateway:Diagnostics:LogCommandValues` option, thread it to the redactor call sites, default off) or trim the doc to match. Recommend trimming the doc for now and deferring wiring; if wiring proceeds, land SEC-13 (bulk-secured names) first so secured payloads are covered. Depends-on SEC-13.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- If trimming: `docs/Diagnostics.md` — mark value logging as not-yet-implemented.
|
||||||
|
- If wiring: add the option + validation, call `RedactCommandValue` at the command-log site with the flag, tests asserting credential-bearing commands stay redacted even with the flag on.
|
||||||
|
|
||||||
|
**Verification.** If wired: `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayLogRedactor"`. Otherwise doc-only.
|
||||||
@@ -0,0 +1,623 @@
|
|||||||
|
# Language Clients — Remediation Design & Implementation
|
||||||
|
|
||||||
|
Source review: [50-clients.md](../50-clients.md) · Generated: 2026-07-09
|
||||||
|
|
||||||
|
This document turns the Language Clients review into buildable remediation entries for all five clients (`clients/dotnet`, `clients/go`, `clients/java`, `clients/python`, `clients/rust`). Every finding carries the target language explicitly. The JDK 17 retarget is verified correct in build config (J1 in the source report is a no-defect verification and gets no remediation row); its only fallout is stale docs (CLI-12). Two operational constraints from prior work carry into these fixes: the Java client cannot build or test on the macOS tree (no local JRE — build/test on windev), and the Java Gradle build regenerates a tracked ~64k-line `MxaccessGateway.java` whose spurious protobuf-version churn must be reverted (`git checkout`) whenever no `.proto` changed. Python regen must pin `grpcio-tools` to the baseline (grpcio 1.80.0 / protobuf 6.31.1); the Rust `build.rs` proto path and `--no-verify` packaging are themselves a finding (CLI-02).
|
||||||
|
|
||||||
|
## Finding index
|
||||||
|
|
||||||
|
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||||
|
|----|-----|-------|---------|--------|-----------|-------|
|
||||||
|
| CLI-01 | High | Go `Session.Events()` silently closes stream on 16-slot overflow | P0 | M | — | clients/go/mxgateway/session.go |
|
||||||
|
| CLI-02 | High | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) | P1 | M | — | clients/rust/build.rs, clients/rust/Cargo.toml, scripts/pack-clients.ps1 |
|
||||||
|
| CLI-03 | High | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY | P0 | M | — | clients/rust/src/error.rs, clients/rust/src/client.rs |
|
||||||
|
| CLI-04 | High | Typed-command parity gap across all five clients (WriteSecured/AuthenticateUser/AdviseSupervisory/buffered) | P2 | L | CLI-15 | clients/{dotnet,go,java,python,rust} session surfaces |
|
||||||
|
| CLI-05 | Medium | .NET session cannot be re-attached to an existing session id | — | S | — | clients/dotnet/.../MxGatewaySession.cs, MxGatewayClient.cs |
|
||||||
|
| CLI-06 | Medium | .NET `DisposeAsync` blocks/throws on unreachable gateway | — | S | — | clients/dotnet/.../MxGatewaySession.cs |
|
||||||
|
| CLI-07 | Medium | .NET retry budget self-defeats on `DeadlineExceeded` | — | S | — | clients/dotnet/.../MxGatewayClient.cs, MxGatewayClientRetryPolicy.cs |
|
||||||
|
| CLI-08 | Medium | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) | — | S | CLI-03 | clients/dotnet/.../MxCommandReplyExtensions.cs, clients/go/mxgateway/errors.go, clients/java/.../MxGatewayErrors.java |
|
||||||
|
| CLI-09 | Medium | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) | — | M | — | clients/go/mxgateway/errors.go, client.go |
|
||||||
|
| CLI-10 | Medium | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` | — | M | — | clients/go/mxgateway/client.go |
|
||||||
|
| CLI-11 | Medium | Go CLI cannot opt into strict TLS validation | — | S | — | clients/go/cmd/mxgw-go/main.go |
|
||||||
|
| CLI-12 | Medium | Java docs still say "Java 21" after the JDK 17 retarget | P2 | S | — | clients/java/README.md, clients/java/JavaClientDesign.md, docs/ClientPackaging.md |
|
||||||
|
| CLI-13 | Medium | Java event-stream buffer hardcoded 16, cancel-on-overflow | — | M | — | clients/java/.../MxGatewayClient.java, MxEventStream.java |
|
||||||
|
| CLI-14 | Medium | Python default TLS is blocking TOFU pin with silent `localhost` SNI | — | S | — | clients/python/.../options.py, clients/python/README |
|
||||||
|
| CLI-15 | Medium | No client handles `ReplayGap` or offers a reconnect helper | P2 | M | — | clients/{dotnet,go,java,python,rust} + READMEs |
|
||||||
|
| CLI-16 | Medium | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` | P2 | S | CLI-12 | docs/ClientPackaging.md, docs/ClientLibrariesDesign.md |
|
||||||
|
| CLI-17 | Medium | TLS default posture inconsistent across the five clients | — | M | CLI-14 | clients/{dotnet,go,java,python,rust} TLS paths |
|
||||||
|
| CLI-18 | Low | .NET csproj records no `<Version>` | P2 | S | — | clients/dotnet/.../ZB.MOM.WW.MxGateway.Client.csproj |
|
||||||
|
| CLI-19 | Low | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) | — | S | — | clients/dotnet/.../Properties/AssemblyInfo.cs, csproj |
|
||||||
|
| CLI-20 | Low | .NET `--tls` without CA installs accept-all callback | — | S | CLI-17 | clients/dotnet/.../MxGatewayClient.cs |
|
||||||
|
| CLI-21 | Low | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases | P2 | S | — | clients/go/mxgateway/version.go, scripts/tag-go-module.ps1 |
|
||||||
|
| CLI-22 | Low | Go `newCorrelationID` swallows `crypto/rand` error → empty id | — | S | — | clients/go/mxgateway/session.go |
|
||||||
|
| CLI-23 | Low | Go nil-vs-empty bulk short-circuit asymmetry | — | S | — | clients/go/mxgateway/session.go |
|
||||||
|
| CLI-24 | Low | Java `MxEventStream` single-consumer constraint undocumented | — | S | — | clients/java/.../MxEventStream.java |
|
||||||
|
| CLI-25 | Low | Java `close()` does not await channel termination | — | S | — | clients/java/.../MxGatewayClient.java |
|
||||||
|
| CLI-26 | Low | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) | P2 | S | — | clients/python/.../version.py, pyproject.toml |
|
||||||
|
| CLI-27 | Low | Python `Session.close()` not concurrency-safe; synthesizes reply | — | S | — | clients/python/.../session.py |
|
||||||
|
| CLI-28 | Low | Python circular-import workaround at end of `session.py` | — | S | — | clients/python/.../session.py |
|
||||||
|
| CLI-29 | Low | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) | P2 | S | — | clients/rust/src/version.rs |
|
||||||
|
| CLI-30 | Low | Rust has no `unregister` typed helper | — | S | CLI-04 | clients/rust/src/session.rs |
|
||||||
|
| CLI-31 | Low | Rust CLI is a single 2,699-line `main.rs` | — | M | — | clients/rust/crates/mxgw-cli/src/main.rs |
|
||||||
|
| CLI-32 | Low | Client-side bulk caps differ (.NET/Java unbounded) | — | S | — | clients/dotnet, clients/java session surfaces |
|
||||||
|
| CLI-33 | Low | Per-language event backpressure semantics undocumented | — | S | CLI-01, CLI-13 | docs/ClientBehaviorFixtures.md |
|
||||||
|
| CLI-34 | Low | Python `build/`/`.pytest_cache/` present on disk (untracked) | — | S | — | clients/python/.gitignore |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-01 — Go `Session.Events()` silently closes stream on 16-slot overflow `High` · `P0`
|
||||||
|
|
||||||
|
**Finding.** In the Go client, `subscribeEventsAfter` (`clients/go/mxgateway/session.go:700-742`) spawns a goroutine that pushes each event through `sendEventResult` into a 16-slot channel. When the buffer is full and `cancelWhenBufferFull` is set (the `Events()`/`EventsAfter()` path), the `default:` arm cancels the stream and returns `false` with **no terminal error enqueued** (`session.go:751-768`); the goroutine then `close(results)` via `defer` (`:714`). The doc comment promises the stream runs "until … a terminal error is sent" (`:675-677`) but never mentions this drop.
|
||||||
|
|
||||||
|
**Impact.** A consumer that stalls for more than 16 queued events sees a closed channel indistinguishable from graceful server end. Events are lost with no signal — silent data loss, the single most serious runtime defect in the client tree. This is roadmap P0 item 6.
|
||||||
|
|
||||||
|
**Design.** Reserve one slot for a terminal `EventResult` so overflow is always observable. Add an exported sentinel `ErrSlowConsumer` and, on the overflow branch, cancel the stream and attempt a **non-blocking** send of `EventResult{Err: ...wrapping ErrSlowConsumer}`; the reserved capacity guarantees that final send succeeds even when the 16 data slots are full. Keep the blocking `SubscribeEvents` path (which rides gRPC flow control) unchanged. Rejected alternative: dropping the buffered-cancel variant entirely — it is public API and some callers depend on non-blocking semantics; a loud terminal error preserves the contract while removing the silent-loss failure mode.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/go/mxgateway/errors.go`: add `var ErrSlowConsumer = errors.New("mxgateway: event consumer fell behind; stream terminated")`.
|
||||||
|
- `clients/go/mxgateway/session.go`: size the channel to `16+1` (or track a reserved terminal slot); in `sendEventResult`'s `default:` arm, call `cancel()` then a non-blocking `select { case results <- EventResult{Err: &GatewayError{Op: "stream events", Err: ErrSlowConsumer}}: default: }` before returning `false`. Update the `Events`/`EventsAfter` doc comments to state the slow-consumer termination contract.
|
||||||
|
- Tests: extend the session event tests (`clients/go/mxgateway/*_test.go`, the fixture-driven event suite) with a slow-consumer case asserting the channel yields a final `EventResult` whose `errors.Is(res.Err, ErrSlowConsumer)` is true before close.
|
||||||
|
- Docs: record the Go slow-consumer terminal-error behavior in `docs/ClientBehaviorFixtures.md` (see CLI-33).
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `gofmt -l .`, `go build ./...`, `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-02 — Rust crate unbuildable outside the repository `High` · `P1`
|
||||||
|
|
||||||
|
**Finding.** The Rust `build.rs` resolves protos two directories above the crate — `repo_root.join("src/ZB.MOM.WW.MxGateway.Contracts/Protos")` with the hard assumption "clients/rust must live two levels below the repository root" (`clients/rust/build.rs:8-16`) — and `src/generated.rs` is only `tonic::include_proto!` of that build output. `Cargo.toml` does not vendor the `.proto` files. The packaging script hides the defect with `cargo package --no-verify` / `cargo publish --no-verify` (`scripts/pack-clients.ps1:190-211`). Any consumer of the published `zb-mom-ww-mxgateway-client 0.1.2` fails in `build.rs` on first `cargo build`.
|
||||||
|
|
||||||
|
**Impact.** The Gitea-published crate is dead on arrival for every external consumer — the most serious packaging finding. This is roadmap P1 item 11.
|
||||||
|
|
||||||
|
**Design.** Vendor the three protos into the crate and make `build.rs` prefer the in-repo source but fall back to the vendored copy, then drop `--no-verify` so `cargo package` proves standalone buildability. Copy `mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto` into `clients/rust/protos/` and list that dir in `Cargo.toml`'s `include`. In `build.rs`, probe the repo path first (keeps in-repo edits live) and fall back to `CARGO_MANIFEST_DIR/protos` when the repo path is absent. The vendored copies are build inputs, not the canonical source (which remains in Contracts) — add a note so they are refreshed on `.proto` change. Rejected alternative: publishing pre-generated `.rs` — tonic's generated code is tied to the tonic/prost version and would rot; regenerating from vendored protos at build time is the idiomatic Rust approach.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Add `clients/rust/protos/{mxaccess_gateway,mxaccess_worker,galaxy_repository}.proto` (copies).
|
||||||
|
- `clients/rust/build.rs`: `let proto_root = if repo_proto_root.exists() { repo_proto_root } else { manifest_dir.join("protos") };` and compile from whichever exists; keep the descriptor-set output.
|
||||||
|
- `clients/rust/Cargo.toml`: add `include = ["src/**/*.rs", "protos/*.proto", "build.rs", ...]` so the protos ship in the package.
|
||||||
|
- `scripts/pack-clients.ps1:190-211`: remove `--no-verify` from the Rust `cargo package`/`cargo publish` invocations so verification runs.
|
||||||
|
- Optional: a small `build.rs` staleness note or a repo script that copies Contracts protos → `clients/rust/protos/` to keep them in sync (tie into the codegen-freshness theme of report 30/60).
|
||||||
|
- Docs: update the Rust section of `docs/ClientPackaging.md` and `clients/rust/README.md` to describe the vendored-proto layout.
|
||||||
|
|
||||||
|
**Verification.** From `clients/rust`: `cargo fmt`, `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`, and crucially `cargo package` (no `--no-verify`) to prove the crate builds from the packaged tarball.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-03 — Rust `invoke` never validates HRESULT / MXSTATUS_PROXY `High` · `P0`
|
||||||
|
|
||||||
|
**Finding.** In the Rust client, `ensure_command_success` checks only `protocol_status.code == Ok` (`clients/rust/src/error.rs:214-226`) and is the sole reply check on the `invoke` path (`clients/rust/src/client.rs:177-179`). It never inspects `hresult` or the `MXSTATUS_PROXY` array. Every other client performs a second MXAccess-level check (`MxCommandReplyExtensions.cs:27-41`, `errors.go:117-130`, `MxGatewayErrors.java`, `errors.py:122-148`).
|
||||||
|
|
||||||
|
**Impact.** A reply with an OK protocol envelope but failing per-item MXAccess statuses reads as success in Rust — a `session.write(...)` can report success while MXAccess rejected the write. There is also no distinct MXAccess error variant to catch. Violates MXAccess-parity-is-the-contract. Roadmap P0 item 6.
|
||||||
|
|
||||||
|
**Design.** Add an `ensure_mxaccess_success` pass mirroring the other clients and an `Error::MxAccess` variant carrying the reply, and call it after `ensure_command_success` on the typed command paths. Use the **correct COM semantics — `hresult < 0`** (see CLI-08), matching Python, rather than `!= 0`; then any per-item status where success is false raises. Keep the raw `invoke` escape hatch un-validated (callers opting out of typed helpers accept raw replies), but apply the check in the typed `session.write`/command helpers. This is co-designed with CLI-08 (HRESULT semantics) — Rust should land on `< 0` from the start rather than repeat the `!= 0` mistake.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/rust/src/error.rs`: add `Error::MxAccess(Box<MxAccessError>)` (boxed, consistent with the existing boxed `tonic::Status`/`CommandError` pattern) with a `thiserror` message summarizing hresult + status entries and credential-safe formatting; add `pub fn ensure_mxaccess_success(reply: MxCommandReply) -> Result<MxCommandReply, Error>` checking `hresult < 0` and each `MxStatusProxy.success`.
|
||||||
|
- `clients/rust/src/client.rs` / `session.rs`: invoke `ensure_mxaccess_success` after `ensure_command_success` in the typed write/command wrappers.
|
||||||
|
- Tests: add a Rust unit/fixture test asserting a reply with `protocol_status = Ok` but a failing status entry (and one with a negative hresult) yields `Error::MxAccess`; reuse the shared behavior fixtures if they carry such a case.
|
||||||
|
- Docs: note the new `Error::MxAccess` variant in `clients/rust/README.md`.
|
||||||
|
|
||||||
|
**Verification.** From `clients/rust`: `cargo fmt`, `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-04 — Typed-command parity gap across all five clients `High` · `P2`
|
||||||
|
|
||||||
|
**Finding.** No client (`.NET`, `Go`, `Java`, `Python`, `Rust`) exposes typed session helpers for single-item `WriteSecured`/`WriteSecured2`, `AuthenticateUser`, `ArchestrAUserToId`, `AdviseSupervisory`, `AddBufferedItem`, `SetBufferedUpdateInterval`, `Suspend`, or `Activate`, though the wire contract defines all of them (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto:150-160`) and `gateway.md` documents `AdviseSupervisory` as a precondition for user-attributed plain writes. Only the .NET, Go, and Python CLIs offer `advise-supervisory` via hand-built raw commands (`clients/go/cmd/mxgw-go/main.go:364-391`, `clients/dotnet/.../MxGatewayClientCli.cs:456-470`, `clients/python/.../commands.py:287-291`); Java and Rust CLIs lack even that.
|
||||||
|
|
||||||
|
**Impact.** The secured-write parity path (AuthenticateUser → AdviseSupervisory → WriteSecured) and the buffered-event family (`OnBufferedDataChange`) are reachable only through raw `Invoke` — the single most important MXAccess parity surface is untyped in every language. Roadmap P2 item 13.
|
||||||
|
|
||||||
|
**Design.** Add the missing typed session helpers to all five clients, phased by parity value: (1) `adviseSupervisory`, single-item `writeSecured`/`writeSecured2`, `authenticateUser`, `archestrAUserToId`; (2) buffered family `addBufferedItem`/`setBufferedUpdateInterval` and `suspend`/`activate`. Each helper wraps the existing raw-`Invoke` machinery already used by the bulk variants, so no new wire surface is needed — the contract already carries the command kinds (`mxaccess_gateway.proto:150-160`). Follow each language's existing typed-helper idiom (e.g. `.NET` `InvokeCommandAsync` in `MxGatewaySession.cs`, Go `invokeCommand`, Rust `session.rs` wrappers, Java stub methods, Python async methods). Preserve MXAccess parity semantics exactly — do not "fix" `WriteSecured` requiring a prior authenticate/advise; surface the native failure. Credentials passed to `authenticateUser`/`writeSecured` must route through each client's existing secret-redaction seam (`MxGatewaySecrets.java`, Rust `error.rs` scrubbing, Go/Python redaction) so they never reach logs or error messages. Depends on CLI-15 (both touch the same session surfaces and READMEs — land as a coordinated session-surface pass).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `.NET`: add `WriteSecuredAsync`/`AuthenticateUserAsync`/`AdviseSupervisoryAsync`/etc. to `MxGatewaySession.cs`, mirroring the bulk-helper pattern; extend `MxGatewayClientCli.cs`.
|
||||||
|
- `Go`: add methods to `session.go` alongside the bulk helpers; promote the CLI's hand-built `advise-supervisory` command into a typed method.
|
||||||
|
- `Java`: add methods to `MxGatewaySession.java` and a CLI subcommand (`zb-mom-ww-mxgateway-cli`).
|
||||||
|
- `Python`: add async methods to `session.py` and CLI commands in `commands.py`.
|
||||||
|
- `Rust`: add methods to `session.rs` (see CLI-30 for `unregister` symmetry) and subcommands to `mxgw-cli`.
|
||||||
|
- Tests: add per-client fixture/unit coverage for `writeSecured` (parity: fails without prior authenticate), `authenticateUser`, and `adviseSupervisory`; add these to the cross-language smoke matrix.
|
||||||
|
- Docs: update each client README's capability list, the parity matrix in `docs/ClientLibrariesDesign.md`, and `docs/CrossLanguageSmokeMatrix.md`.
|
||||||
|
|
||||||
|
**Verification.** Per language, the CLAUDE.md per-area commands: `.NET` `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` + tests; `Go` `go build/test ./...`; `Rust` `cargo check/test/clippy`; `Python` `python -m pytest`; `Java` `gradle test` **on windev** (no local JRE on the Mac — revert the regenerated `MxaccessGateway.java` churn afterward, since no `.proto` changed).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-05 — .NET session cannot be re-attached to an existing session id `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the .NET client, `MxGatewaySession`'s constructor is `internal` (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewaySession.cs:19`) and no public factory exists, unlike Go's `NewSessionForID` (`clients/go/mxgateway/session.go:70`), Java's `forSessionId`, Python's ctor, or Rust's `client.session()`.
|
||||||
|
|
||||||
|
**Impact.** After a client restart, the gateway's `DetachGraceSeconds`/replay features are usable from .NET only through the raw stub — all typed helpers are lost on reconnect. Parity gap; .NET is the only client that cannot re-attach.
|
||||||
|
|
||||||
|
**Design.** Add a public factory `MxGatewayClient.AttachSession(string sessionId)` returning an `MxGatewaySession` whose `OpenSessionReply` is synthesized to carry the given `SessionId` (no server round-trip; the session already exists). Keep the `internal` ctor for the open path. This aligns .NET with the other four clients and unblocks CLI-15's reconnect helper. Alternative rejected: making the ctor public — the factory name documents intent (re-attach vs open) and keeps `OpenSessionReply` construction internal.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxGatewayClient.cs`: add `public MxGatewaySession AttachSession(string sessionId)` building a minimal `OpenSessionReply { SessionId = sessionId }` and calling the internal ctor.
|
||||||
|
- Tests: add a case in the .NET session test project asserting `AttachSession(id).SessionId == id` and that a subsequent `StreamEventsAsync(afterWorkerSequence)` targets that id.
|
||||||
|
- Docs: note re-attach in `clients/dotnet/README.md` and flip the parity-matrix cell in `docs/ClientLibrariesDesign.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` and `dotnet test` the client test project.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-06 — .NET `DisposeAsync` blocks/throws on unreachable gateway `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the .NET client, `MxGatewaySession.DisposeAsync` calls `CloseAsync()` with no cancellation token or timeout (`clients/dotnet/.../MxGatewaySession.cs:881-885`). When the gateway is unreachable, `await using` disposal blocks for the full retry-pipeline budget and then throws out of disposal, masking the original exception.
|
||||||
|
|
||||||
|
**Impact.** `await using` on a session whose gateway just died hangs and then surfaces a disposal exception instead of the caller's real error — a poor failure mode on the common teardown path.
|
||||||
|
|
||||||
|
**Design.** Time-bound and swallow close failures in the disposal path only. Wrap the `CloseAsync` call in a short linked-timeout CTS and catch (`OperationCanceledException`/`RpcException`/`MxGatewayException`), since throwing from `DisposeAsync` is an anti-pattern that masks in-flight exceptions. Explicit `CloseAsync()` remains fully surfacing for callers who want to observe close failures. Keep the `_closeLock.Dispose()` unconditional.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxGatewaySession.cs` `DisposeAsync`: `using var cts = new CancellationTokenSource(disposeCloseTimeout);` (a small fixed bound, e.g. 2 s, or `Options`-derived), `try { await CloseAsync(cts.Token); } catch (Exception ex) when (ex is OperationCanceledException or RpcException or MxGatewayException) { /* best-effort */ }`.
|
||||||
|
- Tests: add a test using a transport that never responds, asserting `DisposeAsync` completes within the bound and does not throw.
|
||||||
|
- Docs: note best-effort dispose semantics in `clients/dotnet/README.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` + client tests.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-07 — .NET retry budget self-defeats on `DeadlineExceeded` `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the .NET client, `ExecuteSafeUnaryAsync` caps the whole retry pipeline with `CancelAfter(Options.DefaultCallTimeout)` (`clients/dotnet/.../MxGatewayClient.cs:305-316`) while each attempt's `CallOptions` also gets a `DefaultCallTimeout` deadline (`:290-303`). Because `DeadlineExceeded` is in the retryable set (`MxGatewayClientRetryPolicy.cs:62-67`), a first attempt that ends in `DeadlineExceeded` consumes the entire outer budget, so the configured retries never run.
|
||||||
|
|
||||||
|
**Impact.** Retries are silently inert for the exact transient class they were configured for; `Unavailable`/`ResourceExhausted` bursts shorter than one full timeout still retry, but any deadline-driven transient does not.
|
||||||
|
|
||||||
|
**Design.** Give the outer budget headroom so retries have room to execute. Recommended: size the outer CTS to cover the retry math — roughly `MaxAttempts × (DefaultCallTimeout + MaxDelay)` — rather than a single `DefaultCallTimeout`. Alternative (simpler, less faithful to intent): drop `DeadlineExceeded` from `IsTransientStatus`. Prefer the headroom fix because per-attempt deadlines remain meaningful and deadline transients still retry. Open question the review couldn't settle: whether an absolute wall-clock cap should exist at all — recommend keeping one but computed from the retry parameters.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxGatewayClient.cs` `ExecuteSafeUnaryAsync`: compute the outer `CancelAfter` from `Options.Retry` (attempts, delay, max-delay) plus per-attempt `DefaultCallTimeout`, instead of a bare `DefaultCallTimeout`.
|
||||||
|
- Tests: add a retry test whose first attempt returns `DeadlineExceeded` and asserts a second attempt occurs (the existing retry-policy test project already exercises the pipeline).
|
||||||
|
- Docs: clarify the timeout/retry interaction in `clients/dotnet/README.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` + retry test.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-08 — .NET/Go/Java treat any nonzero HRESULT as failure `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `EnsureMxAccessSuccess` in .NET tests `reply.HasHresult && reply.Hresult != 0` (`clients/dotnet/.../MxCommandReplyExtensions.cs:32`); Go tests `reply.GetHresult() != 0` (`clients/go/mxgateway/errors.go:121`); Java's `MxGatewayErrors` does the same. This misclassifies positive COM success codes (e.g. `S_FALSE = 1`). Python uses the correct `reply.hresult < 0` (`clients/python/.../errors.py:133`).
|
||||||
|
|
||||||
|
**Impact.** A parity-preserving gateway reply carrying a positive success HRESULT throws in three clients and passes in one — a cross-client inconsistency that can turn a legitimate MXAccess success into a raised error. Co-designed with CLI-03 (Rust must adopt `< 0` from the start).
|
||||||
|
|
||||||
|
**Design.** Align .NET, Go, and Java on the COM-correct `hresult < 0` check (negative = failure), matching Python and MXAccess semantics. Low-risk one-line change per client; keep the status-array check unchanged. Not individually enumerated in the roadmap, but shares the correctness theme of P0 item 6 (see CLI-03) — land alongside the Rust MXSTATUS work.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `.NET` `MxCommandReplyExtensions.cs:32`: `bool hResultFailure = reply.HasHresult && reply.Hresult < 0;`
|
||||||
|
- `Go` `errors.go:121`: `if reply.Hresult != nil && reply.GetHresult() < 0 {`
|
||||||
|
- `Java` `MxGatewayErrors.java` (~:50): change the `!= 0` comparison to `< 0`.
|
||||||
|
- Tests: add a case in each client asserting a reply with `hresult = 1` (S_FALSE) passes and `hresult = -2147...` fails; Python's existing regression suite already covers `< 0`.
|
||||||
|
- Docs: note the corrected semantics in each client README's error section.
|
||||||
|
|
||||||
|
**Verification.** `.NET` build+test; `Go` `go test ./...`; `Java` `gradle test` on windev (revert generated-file churn afterward).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-09 — Go has no typed auth-error mapping `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Go client, all RPC failures wrap into the generic `GatewayError` (`clients/go/mxgateway/errors.go:9-34`; call sites e.g. `client.go`), so distinguishing `Unauthenticated` from `PermissionDenied` requires `status.Code(errors.Unwrap(err))`. `docs/ClientLibrariesDesign.md:153` requires the two be treated distinctly, and the other four clients expose typed auth errors.
|
||||||
|
|
||||||
|
**Impact.** Go consumers cannot idiomatically branch on auth failure vs authorization failure — the documented contract is unmet only in Go.
|
||||||
|
|
||||||
|
**Design.** Add sentinel-backed typed errors `AuthenticationError` (gRPC `Unauthenticated`) and `AuthorizationError` (`PermissionDenied`) in `errors.go`, and map them at the transport boundary where `GatewayError` is currently constructed. Use `errors.Is`-friendly sentinels (`ErrUnauthenticated`, `ErrPermissionDenied`) plus wrapper structs preserving `Op` and the underlying `status.Status`. This mirrors the existing `MxAccessError`/`CommandError` layering already in the file.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `errors.go`: add the two error types + sentinels and a helper `classifyRPCError(op string, err error) error` that inspects `status.Code(err)`.
|
||||||
|
- `client.go` (and other RPC wrappers): route transport errors through `classifyRPCError` instead of always constructing `GatewayError`.
|
||||||
|
- Tests: add cases asserting `errors.Is(err, ErrUnauthenticated)` / `ErrPermissionDenied` for the corresponding gRPC codes.
|
||||||
|
- Docs: update `clients/go/README.md` and flip the Go cell in the parity matrix (`docs/ClientLibrariesDesign.md`).
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `gofmt -l .`, `go build ./...`, `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-10 — Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Go client, `Dial` builds options including `grpc.WithBlock()` and calls `grpc.DialContext` (`clients/go/mxgateway/client.go:60-68`). grpc-go ≥1.63 deprecates both in favor of `grpc.NewClient` with lazy connection.
|
||||||
|
|
||||||
|
**Impact.** Future grpc-go upgrades and `go vet`/staticcheck flag the deprecations; blocking dial also hides the per-RPC connection-error semantics the ecosystem now expects.
|
||||||
|
|
||||||
|
**Design.** Migrate to `grpc.NewClient` (lazy, non-blocking) and drop `grpc.WithBlock()`. Because `NewClient` no longer surfaces connect failures at dial time, expose readiness via a first `Ping` in the connect helpers that currently rely on blocking dial (or document that the first RPC surfaces connection errors). Preserve the existing transport-credential and auth-interceptor wiring unchanged.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `client.go`: replace `grpc.DialContext(dialCtx, endpoint, dialOptions...)` with `grpc.NewClient(endpoint, dialOptions...)`, remove `grpc.WithBlock()`; if a caller-facing "connected" guarantee is desired, add an optional `Ping` in the `Dial`/connect wrapper.
|
||||||
|
- Tests: existing dial/TLS tests should pass; add a case asserting a bad endpoint surfaces on first RPC rather than at construction.
|
||||||
|
- Docs: note the lazy-connect behavior change in `clients/go/README.md`.
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `go build ./...`, `go vet ./...`, `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-11 — Go CLI cannot opt into strict TLS validation `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Go client CLI, `dialForCommand` never sets `Options.RequireCertificateValidation` and exposes no flag for it (`clients/go/cmd/mxgw-go/main.go:1151-1158`), so every non-CA-pinned TLS run of `mxgw-go` is skip-verify even though the library supports strictness (`clients/go/mxgateway/client.go:231-241`).
|
||||||
|
|
||||||
|
**Impact.** `mxgw-go --tls` without a CA cannot authenticate the server at all, with no operator opt-in — a security-posture gap unique to the Go CLI. Ties into CLI-17 (cross-client TLS convergence).
|
||||||
|
|
||||||
|
**Design.** Add a `-require-certificate-validation` boolean flag that sets `Options.RequireCertificateValidation`, mirroring the library capability. Coordinate the default with CLI-17's convergence decision; at minimum emit a one-line stderr warning when TLS runs skip-verify (see CLI-17/CLI-20).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/go/cmd/mxgw-go/main.go`: register the flag and thread it into `dialForCommand`'s `Options`.
|
||||||
|
- Tests: extend the CLI/TLS tests to assert the flag sets the option.
|
||||||
|
- Docs: document the flag in `clients/go/README.md` and the CLI usage.
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `go build ./...`, `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-12 — Java docs still say "Java 21" after the JDK 17 retarget `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** After the verified-correct JDK 17 retarget (build.gradle `toolchain 17` + `options.release = 17`, `build.gradle:20-31`), three docs still claim Java 21: `clients/java/README.md:354`, `clients/java/JavaClientDesign.md:34-35`, and `docs/ClientPackaging.md:193`. This violates CLAUDE.md's docs-in-same-commit rule on the retarget's own branch.
|
||||||
|
|
||||||
|
**Impact.** Consumers targeting Ignition 8.3 (JDK 17) read contradictory prerequisites. Roadmap P2 item 15 (doc-drift sweep).
|
||||||
|
|
||||||
|
**Design.** Sweep all three references from "Java 21" to "Java 17" (noting a 17-targeted build still runs on 21+, as `build.gradle:20-21` already comments). Pure documentation change; no code. Land on the `feat/jdk17-client-retarget` branch before merge.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/java/README.md:354`, `clients/java/JavaClientDesign.md:34-35`, `docs/ClientPackaging.md:193`: change "Java 21" → "Java 17" (and correct the Gradle-toolchain wording). Note the `docs/ClientPackaging.md:193` line also carries the stale Java package/naming addressed in CLI-16 — fix together.
|
||||||
|
- Docs only; no tests.
|
||||||
|
|
||||||
|
**Verification.** Grep confirms zero remaining "Java 21" under `clients/java` and `docs/`: `grep -rn "Java 21" clients/java docs/`. No build needed for the doc change.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-13 — Java event-stream buffer hardcoded 16, cancel-on-overflow `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Java client, `MxGatewayClient.streamEvents` constructs `new MxEventStream(16)` (`clients/java/.../MxGatewayClient.java:248`) with no configuration, and overflow cancels the RPC (`MxEventStream.java:124-132`). Unlike Go (CLI-01), the failure is at least surfaced as `MxGatewayException("…queue overflowed")`, but a consumer stalling for 16 events is disconnected even though gRPC has native flow control.
|
||||||
|
|
||||||
|
**Impact.** Slow Java consumers are force-disconnected at an arbitrary 16-event threshold; capacity is not tunable. Backpressure semantics diverge from the unbuffered .NET/Python/Rust clients (CLI-33).
|
||||||
|
|
||||||
|
**Design.** Make the buffer capacity an option on the stream-events call (default 16 for compatibility) and/or adopt manual flow control (`disableAutoRequestWithInitial` on the stub) so backpressure rides gRPC instead of cancel-on-overflow. Recommended minimal change: parameterize capacity via an overload / options object; the manual-flow-control migration is a larger follow-up. Keep the loud `MxGatewayException` on overflow (Java's surfaced-error behavior is already correct relative to Go).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxGatewayClient.java`: add an overload `streamEvents(StreamEventsRequest, int bufferCapacity)` (or an options field) feeding `new MxEventStream(capacity)`.
|
||||||
|
- `MxEventStream.java`: no behavior change required for the capacity option; document the overflow contract (see CLI-24).
|
||||||
|
- Tests: extend the Java event-stream tests with a custom-capacity case and an overflow case asserting the `MxGatewayException` message.
|
||||||
|
- Docs: record Java's buffered/error-on-overflow behavior in `docs/ClientBehaviorFixtures.md` (CLI-33) and `clients/java/README.md`.
|
||||||
|
|
||||||
|
**Verification.** `gradle test` from `clients/java` **on windev** (no local JRE); revert the regenerated `MxaccessGateway.java` churn afterward since no `.proto` changed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-14 — Python default TLS is blocking TOFU pin with silent `localhost` SNI `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Python client, `create_channel`'s default TLS path fetches the server certificate unverified and pins it trust-on-first-use (`clients/python/.../options.py:128-160`), and silently defaults the SNI/target-name override to `localhost` when none is supplied (`options.py:153-154`). It is documented and bounded (probe timeout; `asyncio.to_thread`), but it is the only client that opens a second out-of-band TCP+TLS connection per channel, and TOFU is exposed to first-contact interception.
|
||||||
|
|
||||||
|
**Impact.** A first-contact MITM can pin a rogue certificate; the silent `localhost` SNI can mask a hostname mismatch. Bounded and intentional for the internal-tool posture, but the threat window is undocumented in the client's own README. Ties into CLI-17.
|
||||||
|
|
||||||
|
**Design.** Keep the TOFU default (removing it would break the internal-tool ergonomics the other clients also target) but document the MITM window in the README threat model, prefer `ca_file` in examples, and make the `localhost` SNI default explicit in a log/warning rather than silent. No behavioral break; a warning + doc change. Convergence across languages is CLI-17.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/python/README` (or `docs/`): add a threat-model note describing the first-contact TOFU window and recommending `ca_file`; make `ca_file` the primary TLS example.
|
||||||
|
- `options.py`: emit a one-line warning (via `logging`/`warnings`) when the TOFU path pins an unverified cert and when it auto-applies the `localhost` SNI override.
|
||||||
|
- Tests: extend the Python TLS tests to assert the warning fires on the TOFU path.
|
||||||
|
- Docs: cross-reference `docs/CrossLanguageSmokeMatrix.md`'s TLS divergence table.
|
||||||
|
|
||||||
|
**Verification.** `python -m pytest` from `clients/python`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-15 — No client handles `ReplayGap` or offers a reconnect helper `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** All five clients expose the `after_worker_sequence` resume cursor (`MxGatewaySession.cs:865-876`, `session.go:682-688`, `MxGatewaySession.java:718-722`, `session.py:571-582`, `session.rs:648`), but none handle the `ReplayGap` sentinel a resuming consumer must expect — `grep ReplayGap clients/` returns only build output (contracts XML, javadoc), no handwritten source. The gateway shipped `DetachGraceSeconds` + replay (on by default), so a resuming stream can legitimately emit a gap event that no client types or documents.
|
||||||
|
|
||||||
|
**Impact.** A client that reconnects with `after_worker_sequence` after the replay buffer overran receives a `ReplayGap` it does not recognize, silently mis-treating a lossy resume as continuous. Roadmap P2 item 12 (finish the session-resilience epic). Coordinates with CLI-04 (session-surface pass) and cross-domain with the gateway/testing reports.
|
||||||
|
|
||||||
|
**Design.** Two-stage: (1) document `ReplayGap` gap-detection per client README so consumers can branch on it via the raw event; (2) add a typed helper/observer that raises or signals gap detection during resume. Stage 1 is the P2-minimum. Recommended stage-2 shape: a per-client `on_replay_gap`/`ReplayGapError` hook surfaced from the event iterator when a `ReplayGap` event arrives, letting the consumer decide whether to re-snapshot. Do not synthesize or swallow the gap — forward the gateway's event faithfully (per the no-synthesized-events invariant); the client only makes it observable and typed.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Each client README: document that resuming with a stale `after_worker_sequence` can yield a `ReplayGap` event and what it means.
|
||||||
|
- `.NET`/`Go`/`Java`/`Python`/`Rust` event surfaces: recognize the `ReplayGap` event kind and expose it as a distinct typed result/error alongside normal events.
|
||||||
|
- Tests: add per-client fixture cases feeding a `ReplayGap` event and asserting the typed surface; add a resume-with-gap case to the cross-language smoke matrix.
|
||||||
|
- Docs: `docs/ClientLibrariesDesign.md` (update the "no client-side reconnect" v1 non-goal, `:64-70`) and `docs/CrossLanguageSmokeMatrix.md`.
|
||||||
|
|
||||||
|
**Verification.** Per-language build/test as in CLI-04 (Java on windev, revert generated churn).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-16 — `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `docs/ClientPackaging.md` names the Python package `mxaccess-gateway-client` and its generated dir `src/mxgateway/generated` (`:159-160`) vs the actual `zb-mom-ww-mxaccess-gateway-client` (`clients/python/pyproject.toml:8`) and `src/zb_mom_ww_mxgateway/generated`; the CLI module `python -m mxgateway_cli` (`:187`) vs actual `zb_mom_ww_mxgateway_cli`; the .NET solution `ZB.MOM.WW.MxGateway.Client.sln` (`:51-52`) vs actual `.slnx`; and Java 21 (`:193`, CLI-12). `docs/ClientLibrariesDesign.md:410` repeats the stale Python generated path.
|
||||||
|
|
||||||
|
**Impact.** Copy-paste build/run commands from the packaging doc fail — wrong package name, wrong module path, wrong solution extension. Roadmap P2 item 15.
|
||||||
|
|
||||||
|
**Design.** One documentation sweep aligning every reference to reality. Pure docs; no code. Land with CLI-12 (same file, `:193`).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/ClientPackaging.md`: `:51-52` `.sln` → `.slnx`; `:159-160` package → `zb-mom-ww-mxaccess-gateway-client`, generated dir → `src/zb_mom_ww_mxgateway/generated`; `:187` CLI module → `python -m zb_mom_ww_mxgateway_cli`; `:193` Java 21 → 17 and package naming.
|
||||||
|
- `docs/ClientLibrariesDesign.md:410`: generated path → `clients/python/src/zb_mom_ww_mxgateway/generated`.
|
||||||
|
- Verify against `clients/python/pyproject.toml` and `clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx`.
|
||||||
|
|
||||||
|
**Verification.** Grep the corrected tokens back against the source: `grep -n "slnx\|zb_mom_ww_mxgateway" docs/ClientPackaging.md` and confirm no stale `mxgateway_cli`/`src/mxgateway/generated` remain.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-17 — TLS default posture inconsistent across the five clients `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** The five clients diverge on default TLS: accept-any-cert in .NET (`MxGatewayClient.cs:361-364`), Go (`client.go:236-240`, `InsecureSkipVerify`), and Java (`InsecureTrustManagerFactory`); TOFU pinning in Python (`options.py:133-154`); strict pin-only in Rust. `docs/CrossLanguageSmokeMatrix.md:58-66` documents the divergence, but the same `--tls`-without-CA invocation authenticates the server in one language, half in another, and not at all in three.
|
||||||
|
|
||||||
|
**Impact.** Inconsistent, surprising security posture per language for identical invocations. Depends on CLI-14 (Python) and interacts with CLI-11/CLI-20.
|
||||||
|
|
||||||
|
**Design.** Converge, or at minimum make the weak default loud. Recommended target: the Python TOFU model as the shared default (pins on first contact, better than accept-all), and where a client keeps accept-all as an explicit opt-in, emit a one-line stderr/log warning whenever certificate verification is disabled. Full convergence is a larger cross-client effort; the review's minimum is the warning. Open question the review left: whether to make strict validation the default and require an explicit `--insecure`-style opt-out — recommended long-term but a breaking change for existing internal-tool callers, so stage it behind a major version.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `.NET` `MxGatewayClient.cs:361-364`, `Go` `client.go`, `Java` TLS path: emit a single warning when the accept-all callback / `InsecureSkipVerify` / `InsecureTrustManagerFactory` is installed (see CLI-20 for the .NET-specific note).
|
||||||
|
- Optionally add TOFU pinning to .NET/Go/Java to match Python (larger; stage separately).
|
||||||
|
- Tests: assert the warning fires when TLS runs without a CA and without strict validation.
|
||||||
|
- Docs: reconcile `docs/CrossLanguageSmokeMatrix.md:58-66` with the chosen convergence and note the warning behavior in each README.
|
||||||
|
|
||||||
|
**Verification.** Per-language build/test (Java on windev).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-18 — .NET csproj records no `<Version>` `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** `clients/dotnet/.../ZB.MOM.WW.MxGateway.Client.csproj` (packaging block lines 19-28) declares no `<Version>`; the published NuGet version (0.1.2) is supplied out-of-band at pack time, so the source tree does not record what ships.
|
||||||
|
|
||||||
|
**Impact.** The repo cannot be inspected to learn the shipped version; drift risk. Roadmap P2 item 16 (version alignment).
|
||||||
|
|
||||||
|
**Design.** Add `<Version>` to the csproj `PropertyGroup` so the source records the shipped version, keeping the pack-time override available for CI. Trivial.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `ZB.MOM.WW.MxGateway.Client.csproj`: add `<Version>0.1.2</Version>` (or the current release) to the packaging `PropertyGroup` (lines 19-28).
|
||||||
|
- Docs: none beyond the version-alignment note in the packaging doc.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-19 — .NET duplicate `InternalsVisibleTo` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `InternalsVisibleTo("ZB.MOM.WW.MxGateway.Client.Tests")` is declared both in `clients/dotnet/.../Properties/AssemblyInfo.cs:3` and as a csproj `AssemblyAttribute` (`ZB.MOM.WW.MxGateway.Client.csproj:35-39`). Harmless but redundant.
|
||||||
|
|
||||||
|
**Impact.** Cosmetic; two sources of truth for the same attribute.
|
||||||
|
|
||||||
|
**Design.** Keep one. Recommend the csproj `AssemblyAttribute` block (co-located with packaging config) and delete `Properties/AssemblyInfo.cs`, or vice versa.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Delete `clients/dotnet/.../Properties/AssemblyInfo.cs` (or remove the csproj block) — keep exactly one declaration.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` (confirms tests still see internals).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-20 — .NET `--tls` without CA installs accept-all callback `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the .NET client, with `UseTls` and no CA file, the default `RequireCertificateValidation = false` installs an accept-all certificate callback (`clients/dotnet/.../MxGatewayClient.cs:361-364`). Documented and intentional (`MxGatewayClientOptions.cs:31-36`), but `--tls` without a CA gives no server authentication.
|
||||||
|
|
||||||
|
**Impact.** Same class as CLI-17; called out separately for the .NET default. Low because documented/intentional.
|
||||||
|
|
||||||
|
**Design.** Emit a one-line warning when the accept-all callback is installed (co-designed with CLI-17). No behavior change to the default.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxGatewayClient.cs`: log a warning (via the client's `ILogger`) when the accept-all `RemoteCertificateValidationCallback` is installed (line 361-364 branch).
|
||||||
|
- Docs: cross-reference CLI-17 in `clients/dotnet/README.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build` + client TLS-handler test asserting the warning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-21 — Go `ClientVersion` stale vs tagged releases `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** In the Go client, `ClientVersion = "0.1.0-dev"` (`clients/go/mxgateway/version.go:6`) is stale relative to the module releases published via `scripts/tag-go-module.ps1`.
|
||||||
|
|
||||||
|
**Impact.** `mxgw-go` and callers report a dev version regardless of the tagged release. Roadmap P2 item 16.
|
||||||
|
|
||||||
|
**Design.** Bump `ClientVersion` to the released value and update it as part of the tagging script so the two cannot drift.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/go/mxgateway/version.go:6`: set to the current release (e.g. `0.1.2`).
|
||||||
|
- `scripts/tag-go-module.ps1`: update the constant (or fail the tag) when the version constant does not match the tag.
|
||||||
|
- Docs: version-alignment note in the packaging doc.
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `go build ./...`, `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-22 — Go `newCorrelationID` swallows `crypto/rand` error `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Go client, `newCorrelationID` returns an empty string on `rand.Read` error (`clients/go/mxgateway/session.go:786-792`), silently dropping traceability.
|
||||||
|
|
||||||
|
**Impact.** A rare `crypto/rand` failure yields empty correlation ids, breaking request tracing with no signal.
|
||||||
|
|
||||||
|
**Design.** Fall back to a monotonic timestamp+counter id rather than empty, preserving uniqueness/traceability without introducing an error-return on a hot helper.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `session.go:786-792`: on `rand.Read` error, build a fallback id from `time.Now().UnixNano()` plus an atomic counter.
|
||||||
|
- Tests: add a case (via an injectable rand source or by asserting non-empty output) that the id is never empty.
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-23 — Go nil-vs-empty bulk short-circuit asymmetry `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Go client, `WriteBulk`/`ReadBulk` short-circuit an empty (non-nil) slice locally (`session.go:399-407`, `:524-532`) while `AddItemBulk`/`SubscribeBulk` send an empty command to the wire (`session.go:255-274`). Harmless but inconsistent within one file.
|
||||||
|
|
||||||
|
**Impact.** Cosmetic; identical inputs produce a local no-op in some helpers and a wire round-trip in others.
|
||||||
|
|
||||||
|
**Design.** Pick one convention (recommend short-circuit empty locally, avoiding a pointless round-trip) and apply it uniformly across the bulk helpers.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `session.go`: make `AddItemBulk`/`SubscribeBulk` (and siblings) short-circuit empty input like `WriteBulk`/`ReadBulk`, or document the intentional difference.
|
||||||
|
- Tests: assert empty-input behavior is consistent.
|
||||||
|
|
||||||
|
**Verification.** From `clients/go`: `go test ./...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-24 — Java `MxEventStream` single-consumer constraint undocumented `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Java client, `MxEventStream` is a single-consumer iterator with unsynchronized `next` state (`MxEventStream.java:31`, `:65-92`); the single-consumer constraint is not documented.
|
||||||
|
|
||||||
|
**Impact.** A caller iterating from two threads corrupts `next` state with no warning.
|
||||||
|
|
||||||
|
**Design.** Add a Javadoc note stating the iterator is single-consumer / not thread-safe. Documentation only (matching CLI-13's overflow doc).
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxEventStream.java`: Javadoc on the class and `next` noting single-consumer usage.
|
||||||
|
|
||||||
|
**Verification.** `gradle test` on windev (revert generated churn); Javadoc-only change.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-25 — Java `close()` does not await channel termination `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Java client, `close()` initiates `shutdown()` without awaiting termination (`MxGatewayClient.java:346-351`); `closeAndAwaitTermination()` exists (`:360-367`), but try-with-resources users can leak a channel briefly at JVM exit.
|
||||||
|
|
||||||
|
**Impact.** Minor resource leak window on `try (var client = ...)` teardown.
|
||||||
|
|
||||||
|
**Design.** Make `close()` await a short bounded termination (e.g. a few seconds) so try-with-resources users get clean teardown, keeping `closeAndAwaitTermination()` for explicit longer bounds.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `MxGatewayClient.java` `close()`: after `shutdown()`, `awaitTermination(shortBound, SECONDS)` (swallow/timeout gracefully).
|
||||||
|
- Tests: assert `close()` completes and the channel is terminated.
|
||||||
|
- Docs: note the bounded-await in `clients/java/README.md`.
|
||||||
|
|
||||||
|
**Verification.** `gradle test` on windev; revert generated churn.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-26 — Python `version.py` ≠ `pyproject.toml` `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** In the Python client, `pyproject.toml:9` says `0.1.2` while `version.py:3` says `__version__ = "0.1.0"`, so `mxgw-py version` reports the wrong value.
|
||||||
|
|
||||||
|
**Impact.** CLI and callers report a stale version. Roadmap P2 item 16.
|
||||||
|
|
||||||
|
**Design.** Derive `__version__` from installed metadata so the two cannot drift: `importlib.metadata.version("zb-mom-ww-mxaccess-gateway-client")` with a fallback for editable/source runs. Aligns with the single-source-of-truth intent.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/python/src/zb_mom_ww_mxgateway/version.py`: replace the literal with `importlib.metadata.version(...)` guarded by `try/except PackageNotFoundError` falling back to a literal.
|
||||||
|
- Tests: assert `version.__version__` matches `pyproject` (or the installed dist) in the packaging/regression suite.
|
||||||
|
- Note: any Python regen for this work must pin `grpcio-tools` to the baseline (grpcio 1.80.0 / protobuf 6.31.1) — but this change touches no protos, so no regen is needed.
|
||||||
|
|
||||||
|
**Verification.** `python -m pytest` from `clients/python`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-27 — Python `Session.close()` not concurrency-safe `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Python client, `Session.close()` has no lock around `_closed` (`session.py:38-55`) and repeated closes return a locally synthesized `CloseSessionReply` rather than the cached server reply — divergent from .NET/Go which cache the real reply.
|
||||||
|
|
||||||
|
**Impact.** Concurrent closes can double-invoke; repeated closes return a synthetic reply instead of the server's. Minor.
|
||||||
|
|
||||||
|
**Design.** Add an `asyncio.Lock` around the close transition and cache the real server `CloseSessionReply` to return on repeat, matching the .NET/Go cached-reply pattern.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `session.py`: guard `_closed` with an `asyncio.Lock`, store the first server reply, return it on subsequent calls.
|
||||||
|
- Tests: assert concurrent `close()` calls invoke the server once and return the cached reply.
|
||||||
|
|
||||||
|
**Verification.** `python -m pytest` from `clients/python`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-28 — Python circular-import workaround at end of `session.py` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Python client, `from .client import GatewayClient` sits at the bottom of `session.py` with `# noqa: E402` (~`session.py:590`) to break a runtime import cycle.
|
||||||
|
|
||||||
|
**Impact.** Works, but the runtime cycle is fragile and lints noisily.
|
||||||
|
|
||||||
|
**Design.** Replace with a `TYPE_CHECKING`-guarded import plus a string annotation, removing the runtime cycle entirely.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `session.py`: `if TYPE_CHECKING: from .client import GatewayClient` at the top and annotate the reference as `"GatewayClient"`; delete the bottom-of-file import and `# noqa`.
|
||||||
|
- Tests: existing import/session tests confirm no runtime cycle.
|
||||||
|
|
||||||
|
**Verification.** `python -m pytest` from `clients/python`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-29 — Rust `CLIENT_VERSION` ≠ `Cargo.toml` `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** In the Rust client, `CLIENT_VERSION = "0.1.0-dev"` with a doc comment claiming it "Mirrors `Cargo.toml`" (`clients/rust/src/version.rs:6-7`) while `Cargo.toml:3` says `0.1.2`.
|
||||||
|
|
||||||
|
**Impact.** The advertised client version is wrong and contradicts its own doc. Roadmap P2 item 16.
|
||||||
|
|
||||||
|
**Design.** Source the constant from Cargo at compile time so it cannot drift: `env!("CARGO_PKG_VERSION")`.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/rust/src/version.rs:7`: `pub const CLIENT_VERSION: &str = env!("CARGO_PKG_VERSION");`.
|
||||||
|
- Tests: add a trivial assertion that `CLIENT_VERSION` equals the expected release, or that it is non-empty and matches `Cargo.toml` via a build script check.
|
||||||
|
|
||||||
|
**Verification.** From `clients/rust`: `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-30 — Rust has no `unregister` typed helper `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Rust client, the session surface (`clients/rust/src/session.rs:119-231`) covers register/add/advise/remove but not `Unregister`; Go/Java/Python expose it (and .NET also lacks it per the parity matrix).
|
||||||
|
|
||||||
|
**Impact.** Rust (and .NET) users must drop to raw `Invoke` to unregister — a small parity gap.
|
||||||
|
|
||||||
|
**Design.** Add an `unregister` typed helper wrapping the existing raw-command machinery, for symmetry with the other clients. Fold into the CLI-04 session-surface pass.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/rust/src/session.rs`: add `pub async fn unregister(...)` mirroring `register`.
|
||||||
|
- (Optionally add the .NET `UnregisterAsync` too, closing the matrix cell.)
|
||||||
|
- Tests: fixture/unit coverage for `unregister`.
|
||||||
|
- Docs: flip the parity-matrix cell in `docs/ClientLibrariesDesign.md`.
|
||||||
|
|
||||||
|
**Verification.** From `clients/rust`: `cargo check/test/clippy`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-31 — Rust CLI is a single 2,699-line `main.rs` `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Rust client, the CLI is one 2,699-line `main.rs` (`clients/rust/crates/mxgw-cli/src/main.rs`), the largest single file in the client tree; the Windows stack-size workaround it forced (`clients/rust/.cargo/config.toml:1-19`) is evidence the command enum has outgrown one module.
|
||||||
|
|
||||||
|
**Impact.** Maintainability; large compile unit; the stack-size workaround is a symptom.
|
||||||
|
|
||||||
|
**Design.** Split subcommands into per-command modules (`mxgw-cli/src/commands/*.rs`) with `main.rs` reduced to arg parsing and dispatch. Mechanical refactor; behavior-preserving. Re-evaluate whether the `.cargo/config.toml` stack workaround can be relaxed after the split.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `clients/rust/crates/mxgw-cli/src/`: extract subcommand handlers into modules; keep the clap/enum wiring in `main.rs`.
|
||||||
|
- Tests: existing CLI tests must pass unchanged.
|
||||||
|
|
||||||
|
**Verification.** From `clients/rust`: `cargo build`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-32 — Client-side bulk caps differ `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Go/Python/Rust enforce a 1,000-item client-side bulk cap (`session.go:19`, `session.py:11`, `session.rs:29`) while .NET and Java send unbounded lists and rely on the gateway. Same oversized call produces different error types per language.
|
||||||
|
|
||||||
|
**Impact.** Inconsistent error surface for oversized bulk calls across languages. Harmless functionally.
|
||||||
|
|
||||||
|
**Design.** Align: either all clients enforce the 1,000 cap locally (recommended — a fast, uniform client-side error) or none do. Add the cap to .NET and Java.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `.NET` (`MxGatewaySession.cs`) and `Java` (`MxGatewaySession.java`): add a client-side 1,000-item check on the bulk helpers, throwing the client's standard argument/validation error.
|
||||||
|
- Tests: assert an oversized bulk call fails locally with the expected error type in .NET and Java.
|
||||||
|
- Docs: note the uniform cap in the client READMEs.
|
||||||
|
|
||||||
|
**Verification.** `.NET` build+test; `Java` `gradle test` on windev (revert generated churn).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-33 — Per-language event backpressure semantics undocumented `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Event-stream backpressure differs by language: Go buffered-16 / silent-cancel (CLI-01), Java buffered-16 / error-cancel (CLI-13), .NET/Python/Rust unbuffered (native gRPC flow control). This per-language slow-consumer behavior is a parity-relevant observable absent from `docs/ClientBehaviorFixtures.md`.
|
||||||
|
|
||||||
|
**Impact.** Consumers cannot predict slow-consumer behavior from the docs. Depends on CLI-01 and CLI-13 landing first (they change the Go/Java behavior being documented).
|
||||||
|
|
||||||
|
**Design.** After CLI-01 (Go now emits a terminal `ErrSlowConsumer`) and CLI-13 (Java capacity option), document the per-language slow-consumer contract in `docs/ClientBehaviorFixtures.md` as a parity fixture row. Documentation only.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `docs/ClientBehaviorFixtures.md`: add a backpressure/slow-consumer section stating each client's behavior (Go: terminal `ErrSlowConsumer`; Java: `MxGatewayException` on overflow, configurable capacity; .NET/Python/Rust: native gRPC flow control).
|
||||||
|
- Optionally add a shared behavior-fixture case exercised by each client's fixture-driven tests.
|
||||||
|
|
||||||
|
**Verification.** Doc change; validated by the fixture-driven test suites already run per client.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## CLI-34 — Python `build/`/`.pytest_cache/` present on disk (untracked) `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** In the Python client, `build/` and `.pytest_cache/` exist on disk but are not git-tracked (report X7; generated-code hygiene is otherwise good across all clients — tracked generated dirs match the manifest, Rust uses `OUT_DIR` + `.gitkeep`, .NET references Contracts directly).
|
||||||
|
|
||||||
|
**Impact.** None functionally; risk of an accidental future commit of build artifacts.
|
||||||
|
|
||||||
|
**Design.** Confirm `clients/python/.gitignore` (or the repo root `.gitignore`) covers `build/` and `.pytest_cache/`; add entries if absent. No code change.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Verify/extend `.gitignore` to cover `clients/python/build/` and `clients/python/.pytest_cache/`.
|
||||||
|
|
||||||
|
**Verification.** `git status --ignored clients/python` shows the directories ignored, not untracked.
|
||||||
@@ -0,0 +1,460 @@
|
|||||||
|
# Testing, Documentation & Underdeveloped Areas — Remediation Design & Implementation
|
||||||
|
|
||||||
|
Source review: [60-testing-docs-gaps.md](../60-testing-docs-gaps.md) · Generated: 2026-07-09
|
||||||
|
|
||||||
|
This domain has no source-code defects to fix — its findings are process, coverage, documentation-currency, and half-shipped-feature gaps. The remediation therefore centres on standing up CI, filling the highest-value test gaps, finishing (or explicitly re-scoping) the session-resilience epic, and a focused doc-drift + repo-root triage pass. Every citation below was re-verified against the working tree on 2026-07-09; corrections are noted inline.
|
||||||
|
|
||||||
|
## Finding index
|
||||||
|
|
||||||
|
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||||
|
|----|-----|-------|---------|--------|-----------|-------|
|
||||||
|
| TST-01 | High | Reconnect/replay has no e2e test and no client consumer | P2 | L | TST-04 | oldtasks.md, clients/*, EventStreamServiceTests.cs |
|
||||||
|
| TST-02 | High | Reconnect owner re-validation not implemented | P0 | M | TST-04 | Sessions/SessionManager.cs, Grpc/EventStreamService.cs |
|
||||||
|
| TST-03 | High | No CI exists | P1 | M | — | (new) .gitea/workflows/ or .github/workflows/ |
|
||||||
|
| TST-04 | High | Session-resilience epic 16/28 tasks unfinished | P2 | L | — | oldtasks.md, docs/plans/2026-06-15-session-resilience.md.tasks.json |
|
||||||
|
| TST-05 | Medium | Real-worker control/COM paths verified opt-in only | P1 | S | TST-03 | WorkerLiveMxAccessSmokeTests.cs |
|
||||||
|
| TST-06 | Medium | Dashboard live-data path untested | — | M | — | Dashboard/DashboardLiveDataService.cs |
|
||||||
|
| TST-07 | Medium | Real-clock sleeps with negative assertions are latent flakes | — | S | — | WorkerClientTests.cs:431, SessionManagerTests.cs:400 |
|
||||||
|
| TST-08 | Medium | Full-suite orphaned testhost processes | P1 | M | — | Tests fixtures (pipe/hosted-service disposal) |
|
||||||
|
| TST-09 | Medium | Health checks cover only the auth store | — | M | — | GatewayApplication.cs:71 |
|
||||||
|
| TST-10 | Medium | Deployment/upgrade undocumented and hand-rolled | — | M | — | (new) docs/Deployment.md, scripts/ |
|
||||||
|
| TST-11 | Medium | No version discipline on server; client versions drift | P2 | M | — | src/Directory.Build.props, clients/* |
|
||||||
|
| TST-12 | Medium | CLAUDE.md misstates default retention behaviour | P0 | S | — | CLAUDE.md:79 |
|
||||||
|
| TST-13 | Medium | gateway.md carries stale design-era sketches | P2 | S | — | gateway.md:291,738,330 |
|
||||||
|
| TST-14 | Medium | Repo-root working artifacts need triage | P2 | S | — | *-docs-*.md, oldtasks.md, stillpending.md |
|
||||||
|
| TST-15 | Medium | Dashboard EventsHub has no per-session ACL | P2 | M | TST-04 | Dashboard/Hubs/EventsHub.cs:39 |
|
||||||
|
| TST-16 | Medium | `Dashboard:ShowTagValues` is a dead flag | — | S | — | Configuration/DashboardOptions.cs:62 |
|
||||||
|
| TST-17 | Medium | Vendor-gated alarm parity residuals silently lossy | — | S | — | Worker/MxAccess/WnWrapAlarmConsumer.cs:261 |
|
||||||
|
| TST-18 | Low | Hosted-service wrappers untested | — | S | — | SessionLeaseMonitorHostedService.cs, OrphanWorkerCleanupHostedService.cs |
|
||||||
|
| TST-19 | Low | Keep FakeWorkerHarness canned replies in lockstep | — | S | — | Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs |
|
||||||
|
| TST-20 | Low | E2E script papers over a real advise-without-consumer sharp edge | — | S | — | scripts/run-client-e2e-tests.ps1 |
|
||||||
|
| TST-21 | Low | Log rotation configured but minimal | — | S | — | Server/appsettings.json:8 |
|
||||||
|
| TST-22 | Low | Config-shape JSON block omits documented keys | — | S | — | docs/GatewayConfiguration.md:12 |
|
||||||
|
| TST-23 | Low | Bidirectional `Session` RPC never built | P2 | S | — | gateway.md:330 |
|
||||||
|
| TST-24 | Low | Client wire behaviour has no automated verification | — | M | TST-03 | clients/*, (new) in-process fake gateway |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-01 — Reconnect/replay has no e2e test and no client consumer `High` · `P2`
|
||||||
|
|
||||||
|
**Finding.** The server emits the `ReplayGap` sentinel and replays the ring on reconnect (unit-covered in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/EventStreamServiceTests.cs` and `MxAccessGatewayServiceTests.cs`), but epic Task 15 (fake-worker reconnect integration test) and Task 14 (client `ReplayGap` handling in all five clients) remain pending — verified `oldtasks.md:41-44` (`⏳ Task 14 (#121)`, `⏳ Task 15 (#122)`). No client parses the `ReplayGap` message.
|
||||||
|
|
||||||
|
**Impact.** The default-on reconnect protocol (`DetachGraceSeconds = 30`, `ReplayBufferCapacity = 1024` — both verified in code, see TST-12) is unproven end-to-end and unusable by every official client: a client that reconnects silently drops the gap marker and cannot tell replayed events from live ones, or that a gap was truncated.
|
||||||
|
|
||||||
|
**Design.** Land Tasks 14 and 15 as the completion of Phase 3 (co-designed with TST-02, TST-04). Two parts:
|
||||||
|
1. **Integration test (Task 15):** drive `FakeWorkerHarness` through advise → detach subscriber (keep session in grace) → emit events into the replay ring → reconnect a new `StreamEvents` with a `last_sequence` cursor → assert the exact replayed set plus a `ReplayGap` when the requested cursor predates the ring. This lives entirely in fakes; no live COM needed.
|
||||||
|
2. **Client consumers (Task 14):** each client's event-stream reader must surface `ReplayGap` as a typed, observable signal (not a dropped frame). Carry the per-language `optional`/presence-check idiom note from `oldtasks.md:44` — proto3 `optional` message fields need a `HasField`/`Option`/`is not None` presence check per language.
|
||||||
|
|
||||||
|
Rejected alternative: advertise reconnect as shipped and document the gap. Rejected because the roadmap already lists it as shipped and CLAUDE.md documents it as a feature (TST-12); leaving clients unable to consume it is a latent correctness trap, not a documentation nuance.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Test: add `GatewayEndToEndReconnectReplayTests` under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/` reusing `FakeWorkerHarness` and `ManualTimeProvider` to control the detach-grace clock.
|
||||||
|
- Clients: `clients/dotnet`, `clients/go`, `clients/rust`, `clients/python`, `clients/java` event readers — expose `ReplayGap` on the stream item type. Cross-reference the clients-domain report (`CLI`) for the shared stream-surface shape.
|
||||||
|
- Docs: `docs/Sessions.md` "Reconnect and replay" — change "server-side" caveat to "supported end-to-end"; each client README documents the gap signal.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~GatewayEndToEndReconnectReplayTests`; per-client build/test per CLAUDE.md's client table (`gradle test`, `cargo test --workspace`, `python -m pytest`, `go test ./...`, `dotnet build clients/dotnet/...`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-02 — Reconnect owner re-validation not implemented `High` · `P0`
|
||||||
|
|
||||||
|
**Finding.** Epic Task 13 ("Owner re-validation on reconnect") is pending — verified `oldtasks.md:40` (`⏳ Task 13 (#120): Owner re-validation on reconnect — blockedBy 12, 1`). Nothing ties a resuming `StreamEvents` call to the API key that opened the session beyond possessing the `event` scope and knowing the session id.
|
||||||
|
|
||||||
|
**Impact.** With fan-out (`AllowMultipleEventSubscribers`) or detach-grace enabled, any `event`-scoped key that learns a session id can attach to another key's session and receive its replayed and live data. This is a trust-boundary hole in a window that is **on by default** (`DetachGraceSeconds = 30`). It is a security control, not a resilience nicety.
|
||||||
|
|
||||||
|
**Design.** Persist the opening key's identity on the session and enforce it on every `StreamEvents` attach/reattach. Options:
|
||||||
|
- **(recommended)** Store the authenticated key id (already available on the auth context from `GatewayGrpcAuthorizationInterceptor`) on the session record at `OpenSession`; in `EventStreamService`/`MxAccessGatewayService.StreamEvents`, reject an attach whose caller key id ≠ the session owner with `PermissionDenied`. Admin-scope override deferred to Phase 4 (TST-15).
|
||||||
|
- Rejected: gate solely on the `event` scope (status quo) — that is exactly the hole.
|
||||||
|
|
||||||
|
This is a P0 item in the roadmap ("Close the reconnect owner re-validation gap or flip retention defaults off until it lands"). If Task 13 cannot land immediately, the safe interim is to flip `DetachGraceSeconds`/fan-out defaults off and correct CLAUDE.md (TST-12) to match — but the recommended path is to implement re-validation because the feature is already documented as shipped.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs` (session record gains an owner key id) and `Sessions/GatewaySession.cs`.
|
||||||
|
- `src/ZB.MOM.WW.MxGateway.Server/Gateway/Grpc/EventStreamService.cs` / `MxAccessGatewayService.cs` — owner check on attach.
|
||||||
|
- Tests: `EventStreamServiceTests` (owner match / mismatch / admin override), plus an attach-by-foreign-key rejection case in the reconnect integration test (TST-01).
|
||||||
|
- Docs: `docs/Sessions.md` and `docs/DesignDecisions.md` — document the owner-binding rule; CLAUDE.md Authentication section note that session attach is owner-scoped.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ... --filter FullyQualifiedName~EventStreamServiceTests` and the TST-01 integration filter; `dotnet build src/ZB.MOM.WW.MxGateway.Server`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-03 — No CI exists `High` · `P1`
|
||||||
|
|
||||||
|
**Finding.** No pipeline files anywhere — verified: `.github/` and `.gitea/` absent, no `*.yml`/`Jenkinsfile`/`azure-pipelines.yml` at repo root. Every documented verification step is manual.
|
||||||
|
|
||||||
|
**Impact.** The five-language, dual-runtime (net10 / net48-x86) matrix is precisely the shape that breaks silently. Repo history shows the failure class is routine: net48 CS0246 on unregenerated protos, Java tracked-generated-file churn, seven-week-stale client descriptors (cross-domain `IPC`/`CLI`).
|
||||||
|
|
||||||
|
**Design.** Stand up a minimal pipeline. Because the x86 Worker only builds on a Windows host with MXAccess installed, split into a portable job (any runner) and a Windows job (self-hosted runner on windev, `10.100.0.48`). Live-MXAccess runs stay opt-in and scheduled, never gating a push.
|
||||||
|
|
||||||
|
Minimal pipeline (Gitea Actions syntax; `.github/workflows/` is drop-in equivalent if the mirror is GitHub):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# .gitea/workflows/ci.yml
|
||||||
|
name: ci
|
||||||
|
on: [push, pull_request]
|
||||||
|
jobs:
|
||||||
|
portable: # any linux/mac runner — no MXAccess, no x86
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Build NonWindows solution
|
||||||
|
run: dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx -c Release
|
||||||
|
- name: Gateway fake-worker tests
|
||||||
|
run: dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj -c Release
|
||||||
|
- name: Descriptor / codegen freshness
|
||||||
|
run: pwsh scripts/check-descriptors.ps1 # see below; fail if regen would change tracked output
|
||||||
|
- name: Client builds/tests
|
||||||
|
run: |
|
||||||
|
dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx -c Release
|
||||||
|
(cd clients/go && gofmt -l . && go build ./... && go test ./...)
|
||||||
|
(cd clients/rust && cargo fmt --check && cargo test --workspace && cargo clippy --workspace --all-targets -- -D warnings)
|
||||||
|
(cd clients/python && python -m pip install -e ".[dev]" && python -m pytest)
|
||||||
|
- name: Java client
|
||||||
|
run: (cd clients/java && gradle test) # needs a JDK 17 runner; Mac dev box has none
|
||||||
|
windows: # self-hosted windev runner — x86 worker + net48 tests
|
||||||
|
runs-on: [self-hosted, windows]
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- run: dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
|
||||||
|
- run: dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86
|
||||||
|
live-mxaccess: # scheduled only, never gates a push
|
||||||
|
if: github.event_name == 'schedule'
|
||||||
|
runs-on: [self-hosted, windows, mxaccess]
|
||||||
|
env: { MXGATEWAY_RUN_LIVE_MXACCESS_TESTS: "1" }
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- run: dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj
|
||||||
|
```
|
||||||
|
|
||||||
|
**Freshness check.** The `descriptors/codegen freshness` step is the highest-leverage single guard (it protects the Generated/-commit rule and the stale client descriptor set). Implement `scripts/check-descriptors.ps1` to: rebuild `ZB.MOM.WW.MxGateway.Contracts`, regenerate the published client descriptor set, then `git diff --exit-code` the tracked `Generated/` and `clients/proto/descriptors/` paths — non-empty diff fails the job. This addresses the `IPC` domain's stale-descriptor finding directly; coordinate the exact regen command with that report.
|
||||||
|
|
||||||
|
Note the two known-noisy interactions to encode as CI hygiene: net48 requires the regenerated `Generated/*.cs` to be **committed** (regen-then-commit), while the Java build **regenerates and dirties** `clients/java/src/main/generated/.../MxaccessGateway.java` with spurious version churn that must be reverted when no `.proto` changed. The Java job must `git checkout` that file after `gradle test` (or the freshness check must exclude it) so the build stays green.
|
||||||
|
|
||||||
|
**Implementation.** New `.gitea/workflows/ci.yml` (+ `.github/workflows/ci.yml` if mirrored); new `scripts/check-descriptors.ps1`. Register a self-hosted runner on windev with `windows`/`mxaccess` labels. Docs: add a "Continuous Integration" section to `docs/GatewayTesting.md` describing the jobs and the scheduled live cadence; note the Windows-runner requirement for the x86 worker.
|
||||||
|
|
||||||
|
**Verification.** Push a branch and confirm the `portable` and `windows` jobs go green; deliberately introduce a proto change without regen and confirm the freshness step fails.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-04 — Session-resilience epic 16/28 tasks unfinished `High` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `oldtasks.md:39-63`: pending are Task 13 (owner re-validation, TST-02), 14–15 (client ReplayGap + reconnect test, TST-01), 16–19 (per-session dashboard ACL, TST-15), and 20–28 (orphan-worker reattach, incl. the `EnableOrphanReattach` flag at Task 26 that does not yet exist). The repo advertises reconnect and fan-out as shipped while their trust boundary and client support are absent.
|
||||||
|
|
||||||
|
**Impact.** The default-on retention window has no owner check (security, TST-02); no client can consume the protocol (TST-01); Phase 5 reattach would reverse a documented CLAUDE.md invariant ("Gateway restart does not reattach orphan workers"). The epic is in a state where "shipped" and "planned" are entangled in the same docs.
|
||||||
|
|
||||||
|
**Design.** Treat the epic as three decisions, not one:
|
||||||
|
- **Phase 3 (Tasks 13–15): finish now, as one unit.** These make the already-shipped server behaviour real and safe — TST-01 + TST-02 are that work.
|
||||||
|
- **Phase 4 (Tasks 16–19): scope explicitly.** Per-session dashboard ACL is TST-15; decide the Viewer default (admin-sees-all vs strict per-session, the open decision at `oldtasks.md:52`) before implementing.
|
||||||
|
- **Phase 5 (Tasks 20–28): re-scope or defer explicitly.** Orphan-worker reattach reverses a CLAUDE.md invariant and adds a manifest store + phone-home protocol. Recommendation: mark Phase 5 as **deferred, not planned** unless a concrete requirement exists; do not leave it as implied backlog. The `EnableOrphanReattach` flag must not be referenced anywhere as if it exists until Task 26 lands.
|
||||||
|
|
||||||
|
This finding is the umbrella; TST-01/02/15 are its actionable slices. The remediation here is the governance action: update the epic's tracking source and the roadmap so "shipped" claims match reality.
|
||||||
|
|
||||||
|
**Implementation.**
|
||||||
|
- Update `docs/plans/2026-06-15-session-resilience.md.tasks.json` and its mirror `oldtasks.md` to reflect the Phase-3-finish / Phase-4-scope / Phase-5-defer decision (retire the mirror once the epic resumes or is closed, per TST-14).
|
||||||
|
- CLAUDE.md "Repository-Specific Conventions" — the reconnect/fan-out paragraph must state which parts are shipped-and-consumable vs server-only-pending-clients until TST-01 lands.
|
||||||
|
|
||||||
|
**Verification.** Documentation-only for the umbrella; the slices carry their own build/test verification (TST-01, TST-02, TST-15).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-05 — Real-worker control/COM paths verified opt-in only `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (`stillpending.md` §1.1), but the default suite exercises `Ping`/`GetWorkerInfo`/`DrainEvents`/`ShutdownWorker` only through `FakeWorkerHarness.RespondToControlCommandAsync` (verify current line range in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs`), which returns canned replies.
|
||||||
|
|
||||||
|
**Impact.** A worker-side regression in these paths is invisible until someone sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`.
|
||||||
|
|
||||||
|
**Design.** Co-designed with TST-03: schedule `WorkerLiveMxAccessSmokeTests` on the windev runner on a cadence (the `live-mxaccess` job above). No new test code needed beyond ensuring the existing live suite covers the eleven command kinds; if any are absent, add live cases. This converts "opt-in, run by memory" into "runs nightly, reports failures."
|
||||||
|
|
||||||
|
**Implementation.** Covered by the `live-mxaccess` scheduled job in TST-03. Audit `WorkerLiveMxAccessSmokeTests.cs` for coverage of each command kind; add missing `[LiveMxAccessFact]` cases. Docs: `docs/GatewayTesting.md` opt-in matrix notes the scheduled cadence.
|
||||||
|
|
||||||
|
**Verification.** On windev: `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1 dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/...`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-06 — Dashboard live-data path untested `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** `Dashboard/DashboardLiveDataService.cs` (verified present; owns the shared lazily-opened gateway session backing `/browse` live values) has no test class; `EventsHub`/`AlarmsHubPublisher` hub methods are likewise untested.
|
||||||
|
|
||||||
|
**Impact.** The one dashboard component that holds a real worker session and can fault it has no regression net; a fault-recovery or double-open bug ships silently.
|
||||||
|
|
||||||
|
**Design.** Add a fake-worker-backed test for the three behaviours that matter: (1) session reuse across concurrent `/browse` requests (one shared session, not one per request), (2) fault recovery — after the backing session faults, the next request re-opens rather than throwing, (3) disposal releases the session. `FakeWorkerHarness` already models faults, so no live COM needed.
|
||||||
|
|
||||||
|
**Implementation.** New `DashboardLiveDataServiceTests` under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/`. Optionally add hub-method tests for `EventsHub`/`AlarmsHubPublisher` group join/leave. Docs: none required (internal test coverage).
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ... --filter FullyQualifiedName~DashboardLiveDataServiceTests`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-07 — Real-clock sleeps with negative assertions are latent flakes `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs:431` sleeps `Task.Delay(150ms)` ("give the heartbeat monitor a few real check-intervals") then asserts `client.State == Ready` (state did *not* change); `SessionManagerTests.cs:400` races a 50 ms delayed state flip against a bounded wait. (Report cited :433 and :402; the sleep is at :431 and the race setup at :400 — minor drift.) 47 `Thread.Sleep`/`Task.Delay` occurrences across 22 test files, most benign.
|
||||||
|
|
||||||
|
**Impact.** Fixed-real-time negative assertions can pass spuriously (the awaited thing hadn't happened yet anyway) or fail under CI load — corrosive in an already-manual regime, and doubly so once TST-03 runs them on shared runners.
|
||||||
|
|
||||||
|
**Design.** Convert the negative-assertion sleeps to `ManualTimeProvider`-driven check-interval pumping (advance the fake clock across the monitor's interval deterministically) or poll-until-stable with a short quiescence window. The heartbeat monitor already accepts an injected clock in production, so `WorkerClientTests` can advance it instead of sleeping. Leave the benign cancellation-bounded polls and `ManualTimeProvider` clock uses alone.
|
||||||
|
|
||||||
|
**Implementation.** `WorkerClientTests.cs` (heartbeat within-ceiling case) and `SessionManagerTests.cs` (state-flip race) — drive via the injected `TimeProvider`. Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ... --filter FullyQualifiedName~WorkerClientTests` and `~SessionManagerTests`; run each 10× to confirm no spurious pass/fail.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-08 — Full-suite orphaned testhost processes `Medium` · `P1`
|
||||||
|
|
||||||
|
**Finding.** CLAUDE.md ("Source Update Workflow") documents that the full gateway suite "leaves orphaned testhost processes" and mandates filtered runs as a workaround. The workaround is procedural and is now baked into the repo's memory (`feedback_targeted_tests_only.md`).
|
||||||
|
|
||||||
|
**Impact.** Any future CI (TST-03) or an unaware contributor inherits zombie processes; the leak points at an undisposed `NamedPipeServerStream` or a hosted-service test fixture that never stops.
|
||||||
|
|
||||||
|
**Design.** Find and fix the leaking fixture rather than institutionalizing the workaround. Method: run the full suite locally, enumerate surviving `testhost`/pipe handles, bisect by test collection to the leaking fixture (prime suspects: fake-worker harness pipe servers, `SessionLeaseMonitorHostedService`/`OrphanWorkerCleanupHostedService` test hosts that start a timer but are never `StopAsync`/`Dispose`d). Ensure every `IAsyncDisposable`/`IDisposable` fixture is disposed and every started hosted service is stopped in teardown. This is prerequisite to a reliable CI full-suite run; keep the filtered-run guidance until the leak is closed.
|
||||||
|
|
||||||
|
**Implementation.** Likely `src/ZB.MOM.WW.MxGateway.Tests` fixtures under `Gateway/Workers/Fakes/` and any hosted-service test. Docs: once fixed, update CLAUDE.md "Source Update Workflow" to drop the "leaves orphaned testhost processes" rationale (keep filtered-run guidance for speed, not correctness).
|
||||||
|
|
||||||
|
**Verification.** Run the full `dotnet test src/ZB.MOM.WW.MxGateway.Tests/...` once and confirm no surviving `testhost` processes (`pgrep -f testhost` empty after exit). Note the pre-existing macOS `SessionWorkerClientFactoryFakeWorkerTests` timeout-message failures are environmental and out of scope (`project_macos_pipe_timeout_test_failures.md`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-09 — Health checks cover only the auth store `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:71-75` registers exactly one `AuthStoreHealthCheck` tagged `Ready`. No readiness check for Galaxy SQL reachability, LDAP bind, alarm-monitor session health, or worker-executable presence/launchability — each a documented startup dependency.
|
||||||
|
|
||||||
|
**Impact.** `/health` reports Ready while the gateway cannot open a session (missing/invalid worker exe) or browse Galaxy (SQL down). Deploys look healthy and fail on first real request.
|
||||||
|
|
||||||
|
**Design.** Add tagged health checks next to the existing one, cheapest-first:
|
||||||
|
- **Worker-executable check** — reuse `WorkerExecutableValidator` (already unit-tested); pure filesystem/PE check, no side effects. Tag `Ready`.
|
||||||
|
- **Galaxy SQL** and **LDAP bind** — network checks; cache the result (e.g. 30 s) so `/health` polling doesn't hammer SQL Server / GLAuth. Tag `Ready` (or a separate `Dependencies` tag if operators want liveness vs readiness split).
|
||||||
|
- Alarm-monitor session health — expose the monitor's current state as a check if the alarm feature is enabled.
|
||||||
|
|
||||||
|
Gate the network checks behind config so a gateway with dashboard/alarms disabled doesn't fail readiness on an unconfigured dependency.
|
||||||
|
|
||||||
|
**Implementation.** `GatewayApplication.cs` health-check registration; new check classes under `src/ZB.MOM.WW.MxGateway.Server/` (e.g. `Health/WorkerExecutableHealthCheck.cs`, `Health/GalaxySqlHealthCheck.cs`, `Health/LdapHealthCheck.cs`). Tests: `...Tests/Gateway/Health/`. Docs: `docs/GatewayConfiguration.md` and `gateway.md` health section — list the checks and their tags; note the Wonder deployment exposes only `/health/live` (`project_wonder_deployment.md`) so keep a liveness-only endpoint working.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; unit tests for each check; manual `curl /health/ready` with a missing worker exe returns Unhealthy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-10 — Deployment/upgrade undocumented and hand-rolled `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** Deployments are NSSM-wrapped services with configuration in NSSM environment variables (acknowledged in `docs/GatewayConfiguration.md` and the project-memory deploy notes). `scripts/` has packaging and e2e scripts but no deploy/upgrade script, and no runbook for the `MxAccessGw`→`OtOpcUa` service-dependency dance.
|
||||||
|
|
||||||
|
**Impact.** Deploys are tribal knowledge (captured only in the operator's private memory files, `project_deploy_mechanics.md`); a second operator cannot reproduce a deploy or upgrade from the repo.
|
||||||
|
|
||||||
|
**Design.** Commit `docs/Deployment.md` (runbook) plus a publish/deploy script that captures the NSSM env-var configuration as code. The runbook records: framework-dependent publish command, exe layout under `C:\publish\mxaccessgw\{Server,Worker}`, the ports (5120 gRPC / 5130 dashboard), the `Stop-Service -Force` + manual `OtOpcUa` restart sequence, and the Wonder-host HTTPS/self-signed specifics. The script parameterizes host + env-var set. This is documentation/scripting only — no source change, and it must not embed secrets (LDAP service password, API keys) per CLAUDE.md.
|
||||||
|
|
||||||
|
**Implementation.** New `docs/Deployment.md`; new `scripts/deploy-gateway.ps1` (idempotent: stop, copy, apply NSSM env vars from a config file, restart, restart dependents). Reference — do not copy secrets from — the operator memory notes. Docs cross-link from `gateway.md`.
|
||||||
|
|
||||||
|
**Verification.** Dry-run the script against windev (`10.100.0.48`); confirm the service restarts and `OtOpcUa` comes back. No unit tests (ops script).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-11 — No version discipline on server; client versions drift `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified: Server/Worker csproj carry no `Version` (assemblies stamp 1.0.0); Contracts is `0.1.2` (`ZB.MOM.WW.MxGateway.Contracts.csproj:10`); Python `0.1.2` (`pyproject.toml:9`), Rust `0.1.2` (`Cargo.toml:3`), Java `0.2.0` (`build.gradle:16`) — and the Java bump required a hand-maintained `CLIENT_VERSION` constant that drifted once (commit `b6a0d90`).
|
||||||
|
|
||||||
|
**Impact.** Support cannot correlate a deployed gateway or a client artifact to a commit; duplicated version constants have already drifted.
|
||||||
|
|
||||||
|
**Design.** Single-source versions:
|
||||||
|
- **.NET side:** define `<Version>`/`<InformationalVersion>` in `src/Directory.Build.props` (which already exists and enforces build policy), stamping the git SHA into the informational version. This covers Server, Worker, Contracts uniformly.
|
||||||
|
- **Clients:** generate the client version constant at build from the package manifest rather than hand-maintaining it (Java's `CLIENT_VERSION` is the proven failure). Where a build-time generation step is impractical, add the descriptor-freshness-style check (TST-03) that fails if the constant ≠ the manifest version.
|
||||||
|
- Decide one version cadence: either lockstep all five clients + contracts, or document why Java leads (JDK-17 retarget shipped as 0.2.0). Recommendation: bring Python/Rust/dotnet/contracts to 0.2.0 to converge, then move in lockstep.
|
||||||
|
|
||||||
|
**Implementation.** `src/Directory.Build.props`; `clients/java/build.gradle` (generate constant from `version`); analogous generation/check for other clients. Docs: `docs/ClientPackaging.md` and each client README note the single-sourcing. Coordinate with the `CLI` domain report on client-version surface.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.slnx` and confirm assembly `InformationalVersion` carries the SHA (`dotnet --info` / reflection); per-client build confirms the constant matches the manifest.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-12 — CLAUDE.md misstates default retention behaviour `Medium` · `P0`
|
||||||
|
|
||||||
|
**Finding.** Verified `CLAUDE.md:79` states "Default config preserves the original single-subscriber, no-retention behavior," but code defaults enable retention: `DetachGraceSeconds = 30` (`Configuration/SessionOptions.cs:46`), `ReplayBufferCapacity = 1024` (`Configuration/EventOptions.cs:21`), `ReplayRetentionSeconds = 300` (`EventOptions.cs:29`). Only `AllowMultipleEventSubscribers` defaults off.
|
||||||
|
|
||||||
|
**Impact.** An agent or operator following CLAUDE.md assumes detached sessions die immediately and nothing is buffered — the opposite of reality, and it obscures the TST-02 security window that is on by default.
|
||||||
|
|
||||||
|
**Design.** Straight documentation correction (the roadmap lists this as P0 alongside owner re-validation). Replace the sentence with an accurate one.
|
||||||
|
|
||||||
|
**Implementation.** `CLAUDE.md:79` — change:
|
||||||
|
|
||||||
|
> Default config preserves the original single-subscriber, no-retention behavior.
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
> Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect.
|
||||||
|
|
||||||
|
If TST-02's interim mitigation (flip retention off) is chosen instead of implementing owner re-validation, this sentence must instead document the flipped defaults — keep the two changes in the same commit. Docs-change-in-same-commit rule applies.
|
||||||
|
|
||||||
|
**Verification.** Read-back; no build. Cross-check the reworded claim against `SessionOptions.cs`/`EventOptions.cs` defaults.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-13 — gateway.md carries stale design-era sketches `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified in `gateway.md`:
|
||||||
|
- The `WorkerEnvelope` snippet (~`:291-309`) shows `uint64 correlation_id = 4` and body tags `worker_hello=10 … command=20 … fault=26` with **no** `WorkerShutdownAck`. The actual proto uses `string correlation_id = 4` and `gateway_hello = 10 … worker_command = 13 … worker_shutdown_ack = 17 … worker_fault = 20` (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:24,27,30,34,37`).
|
||||||
|
- `gateway.md:738` still states the gateway "allow[s] one active client event subscriber per session, reject[s] a second subscriber" as unconditional policy — contradicted by config-gated fan-out later in the same file and by `docs/Sessions.md`.
|
||||||
|
- `gateway.md:330` includes `rpc Session(stream ClientMessage) returns (stream ServerMessage)` which was never built; the real service is unary + server-streaming only (`mxaccess_gateway.proto:18-22`, no bidi RPC).
|
||||||
|
|
||||||
|
**Impact.** gateway.md is the mandated pre-change reading; internally inconsistent load-bearing sections cost every reader a reconciliation pass and can mislead a change.
|
||||||
|
|
||||||
|
**Design.** Don't re-inline the wire types (they drift). Replace the `WorkerEnvelope` sketch with a one-line pointer to `mxaccess_worker.proto` as the source of truth. Mark the single-subscriber paragraph as the *default* mode with a pointer to the fan-out section and `docs/Sessions.md`. Label the `Session` bidi RPC explicitly as unbuilt/deferred future work (see TST-23).
|
||||||
|
|
||||||
|
**Implementation.** `gateway.md` — three edits at the cited locations. Docs-only; no source change. This is part of the roadmap's P2 documentation-drift sweep and should land with the wider sweep (cookie name, GroupToRole sample, Java 17, Grpc.md RPC table) tracked in the `SEC`/`IPC`/`CLI` domains — cross-reference so the sweep is one commit where practical.
|
||||||
|
|
||||||
|
**Verification.** Read-back against the three proto/code citations above.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-14 — Repo-root working artifacts need triage `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified: `MxAccessGateway-docs-{issues,fixed,final}.md` and `MxGatewayClient-docs-{issues,fixed}.md` are **untracked** (`git status` shows them untracked; `*-docs-{issues,fixed,final}.md` are gitignored at `.gitignore:152-154`), largest 586 KB / ~15 k lines. `stillpending.md` (tracked) is a 2026-06-15 audit snapshot serving as the de-facto backlog; `oldtasks.md` (tracked) is a human-readable mirror of the epic tasks.json; `REVIEW-PROCESS.md` + `code-reviews/` is a completed review system; `A2-galaxyrepository-adoption-handoff.md` is a completed migration handoff referenced from CLAUDE.md.
|
||||||
|
|
||||||
|
**Impact.** The dead untracked `*-docs-*.md` files (15 k+ lines) trip greps and agents. The tracked snapshots blur "backlog" vs "historical record."
|
||||||
|
|
||||||
|
**Design.** Triage by evidence, not blanket deletion:
|
||||||
|
- **Delete** the five untracked `*-docs-*.md` working files — dead local outputs of a finished docs-review pass. They are gitignored, so deletion is local only.
|
||||||
|
- **`stillpending.md`:** either promote to a maintained register (drop the "Generated: … Commit:" snapshot framing) or migrate its open items (§1.3, §1.4, §3.x, epic remainder) to the Gitea tracker `docs/ImplementationPlanIndex.md` describes. Recommendation: migrate, since a tracker already exists in the plan.
|
||||||
|
- **`oldtasks.md`:** keep only until the epic resumes/closes (TST-04), then delete — it is a mirror of the tasks.json.
|
||||||
|
- **`REVIEW-PROCESS.md` + `code-reviews/`:** keep (living, self-consistent, zero open findings).
|
||||||
|
- **`A2-galaxyrepository-adoption-handoff.md`:** archive under `docs/plans/` once its cross-repo follow-ups land; update the CLAUDE.md reference if moved.
|
||||||
|
|
||||||
|
**Implementation.** `rm` the untracked files; move/rewrite `stillpending.md`; update CLAUDE.md if `A2-…` moves. No source change.
|
||||||
|
|
||||||
|
**Verification.** `git status` clean of the `*-docs-*.md` files; `grep -rn` for those filenames returns nothing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-15 — Dashboard EventsHub has no per-session ACL `Medium` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHub.cs:39` carries `TODO(per-session-acl)` — the only production-source TODO in the repo. Any authenticated dashboard Viewer may join any `session:{id}` group (group name built at `EventsHub.cs:21`) and observe that session's event stream.
|
||||||
|
|
||||||
|
**Impact.** Acceptable for a single-tenant dashboard; wrong the moment `GroupToRole` admits low-trust viewers. It is the dashboard-side twin of the gRPC owner-revalidation gap (TST-02).
|
||||||
|
|
||||||
|
**Design.** Covered by epic Phase 4 (Tasks 16–19, TST-04). Introduce a role/scope that scopes a hub connection to permitted sessions, plus a hub-token session tag (Task 18). The open design decision (`oldtasks.md:52`) — Viewer default of admin-sees-all vs strict per-session — must be settled first. Recommendation: admin-sees-all, Viewer strictly scoped to sessions they own/are granted, matching TST-02's gRPC owner binding for consistency. Until Phase 4 lands, keep the TODO (it correctly documents the accepted single-tenant assumption); do not silently remove it.
|
||||||
|
|
||||||
|
**Implementation.** `Dashboard/Hubs/EventsHub.cs` (ACL check on group join), hub-token minting to carry the session tag, `Configuration/DashboardOptions.cs` for any group-to-tag config (Task 17). Tests: `...Tests/Gateway/Dashboard/` hub ACL cases incl. live-LDAP users (Task 19). Docs: `docs/Sessions.md`/`gateway.md` dashboard section document the ACL model; CLAUDE.md dashboard-auth paragraph.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ... --filter FullyQualifiedName~EventsHub`; `dotnet build src/ZB.MOM.WW.MxGateway.Server`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-16 — `Dashboard:ShowTagValues` is a dead flag `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** Verified bound at `Configuration/DashboardOptions.cs:62`, projected into effective config (`EffectiveDashboardConfiguration.cs:9`, `GatewayConfigurationProvider.cs:63`), and displayed read-only on the settings page (`Dashboard/Components/Pages/SettingsPage.razor:65`) — but it gates no value-display behaviour anywhere. `docs/GatewayConfiguration.md:174` calls it "Reserved". (Note `docs/GatewayDashboardDesign.md:305` claims it "continues to govern" value display — that prose is aspirational/stale.)
|
||||||
|
|
||||||
|
**Impact.** An operator toggling a flag that renders on the settings page reasonably expects an effect; there is none.
|
||||||
|
|
||||||
|
**Design.** Decide implement-or-remove. Recommendation: **implement** — the flag has an obvious intended meaning (gate live tag-value display in `/browse`, consistent with the redaction philosophy in CLAUDE.md "Don't log secrets or full tag values by default"). Wire `ShowTagValues` into `DashboardLiveDataService`/the browse value-rendering path so `false` suppresses live values. If the team prefers not to implement, remove it from the options, the projector, the settings page, and downgrade the docs row — a settings-page-visible no-op is the worse outcome. Open question: default value (currently `false`) and whether it interacts with the per-session ACL (TST-15).
|
||||||
|
|
||||||
|
**Implementation.** `Dashboard/DashboardLiveDataService.cs` + the browse value component to honour the flag (implement path); or remove from `DashboardOptions.cs:62`, `EffectiveDashboardConfiguration.cs`, `GatewayConfigurationProvider.cs`, `SettingsPage.razor` (remove path). Tests: extend the TST-06 `DashboardLiveDataServiceTests` for the flag. Docs: `docs/GatewayConfiguration.md:174` and `docs/GatewayDashboardDesign.md:305` — replace "Reserved"/stale prose with the real behaviour.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; test toggling the flag suppresses/shows values; docs match.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-17 — Vendor-gated alarm parity residuals silently lossy `Medium` · `—`
|
||||||
|
|
||||||
|
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs:261-278`: the 8-arg `AlarmAckByName` `domain`/`full_name` fields are accepted on the wire then wrapped into the 6-arg call because the 8-arg form returns −55 on this AVEVA build (comments at `:119`, `:265-270`). `AlarmAckByGUID` is `E_NOTIMPL`. The `provider_switches` metric reason tagging has never been exercised live (`stillpending.md` §1.3–1.4, §3.4–3.5).
|
||||||
|
|
||||||
|
**Impact.** Contract fields that silently do nothing, invisible to callers — a parity gap dressed as a working feature.
|
||||||
|
|
||||||
|
**Design.** This is a genuine vendor limitation, not a bug to "fix" — per CLAUDE.md, MXAccess behaviour is the baseline and must not be papered over. The remediation is **observability, not behaviour change**: surface the drop so callers know. Add a diagnostic string to the ack reply (e.g. reply `message`/diagnostic field) noting that operator `domain`/`full_name` were not applied on this build (8-arg returns −55), and that `AlarmAckByGUID` is `E_NOTIMPL`. Do not synthesize success or invent a result — just report the degrade. Keep the TODO/comment referencing the AVEVA stub so a future runtime that implements it can drop the workaround.
|
||||||
|
|
||||||
|
**Implementation.** `Worker/MxAccess/WnWrapAlarmConsumer.cs` — populate a diagnostic in the ack reply when the 8-arg path is downgraded. Confirm the gateway forwards that diagnostic to the client reply (`Gateway/Grpc/...`). Tests: worker ack-consumer test asserting the diagnostic is present on the downgraded path (`src/ZB.MOM.WW.MxGateway.Worker.Tests`). Docs: `gateway.md`/`docs/` alarm section and the relevant client READMEs note the vendor-gated residual. Cross-reference the `WRK` domain report for the alarm-consumer specifics.
|
||||||
|
|
||||||
|
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86`; `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/... -p:Platform=x86` (Windows).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-18 — Hosted-service wrappers untested `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` delegate to well-tested cores (sweep logic covered via `SessionManagerTests.CloseExpiredLeasesAsync` cases), but their timer/startup wiring has no direct tests.
|
||||||
|
|
||||||
|
**Impact.** Low — failure mode (a timer that never fires) is obvious at startup and in integration.
|
||||||
|
|
||||||
|
**Design.** Add a thin `StartAsync` → tick-once (via injected `TimeProvider`) → assert the core method was invoked → `StopAsync` disposes cleanly test for each. This also directly exercises the disposal path relevant to TST-08 (verify the hosted service stops its timer on `StopAsync`), so it doubles as a leak-regression guard.
|
||||||
|
|
||||||
|
**Implementation.** New small test classes under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/` and `.../Workers/`. Docs: none.
|
||||||
|
|
||||||
|
**Verification.** `dotnet test ... --filter FullyQualifiedName~HostedService`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-19 — Keep FakeWorkerHarness canned replies in lockstep `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `FakeWorkerHarness` shares `WorkerFrameReader`/`WorkerFrameWriter`/`WorkerEnvelope` with production and scripts malformed/oversized frames, heartbeats, faults, shutdown acks, and control-command reply shapes. What it cannot represent (STA pumping, COM HRESULT semantics, process exit codes, event timing under load) is honestly delegated to the live suite. History (`stillpending.md` §1.1) shows the canned replies drifted once: green fakes masked an unimplemented real worker for months.
|
||||||
|
|
||||||
|
**Impact.** Low today; the risk is future drift where fake replies stop matching `WorkerPipeSession`, hiding a real-worker regression.
|
||||||
|
|
||||||
|
**Design.** No code fix — this is a process guard. The durable protection is TST-05's scheduled live suite (it catches fake/real drift automatically) plus a code-review checklist note. Add a comment/README note in the fakes directory pointing at `WorkerPipeSession` as the reply-shape source of truth, so any control-command change updates both.
|
||||||
|
|
||||||
|
**Implementation.** `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/` — add a short README or header comment. Rely on TST-05 (scheduled live) for the real guard. Docs: `docs/GatewayTesting.md` already states the fidelity charter; add the "keep in lockstep, live suite catches drift" note.
|
||||||
|
|
||||||
|
**Verification.** Covered by TST-05's scheduled live run; no standalone test.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-20 — E2E script papers over a real advise-without-consumer sharp edge `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** `scripts/run-client-e2e-tests.ps1` (~1,715 lines) interleaves `StreamEvents` drains every ~15 advised tags because advising without a consumer overflows the worker event channel and faults the session under `FailFast` (`docs/GatewayTesting.md` "Client E2E Scripts" phase 3).
|
||||||
|
|
||||||
|
**Impact.** Low as a test detail, but it is real product feedback: any client doing bulk-advise-then-stream will fault sessions.
|
||||||
|
|
||||||
|
**Design.** Treat as product feedback, not a script fix. The clean solution is a subscribe-time channel policy (buffer/backpressure the event channel until a subscriber attaches, or a bounded pre-subscription buffer) — but that is a cross-domain backpressure decision owned by the `IPC`/`GWC`/`WRK` reports' "coordinated size/backpressure pass" (roadmap P1 #8). This finding's action is to **record** the sharp edge as an input to that pass and, until then, document the advise-then-stream ordering requirement in the client READMEs so real clients don't fault sessions.
|
||||||
|
|
||||||
|
**Implementation.** No script change. Docs: client READMEs and `gateway.md` event section note "attach `StreamEvents` before bulk advise, or drain periodically." Cross-reference the backpressure-pass finding in the other domains.
|
||||||
|
|
||||||
|
**Verification.** Docs read-back; the behavioural fix is verified under the coordinated backpressure pass, not here.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-21 — Log rotation configured but minimal `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Server/appsettings.json:8-11` rolls the Serilog file sink daily (`logs/mxgateway-.log`, relative to the service working directory) with no `retainedFileCountLimit` or size cap (Serilog defaults: 31 files, unbounded size per day).
|
||||||
|
|
||||||
|
**Impact.** Acceptable, but a high-rate event day can produce an unbounded single file, and a relative path lands logs in the service CWD.
|
||||||
|
|
||||||
|
**Design.** Set `fileSizeLimitBytes` + `rollOnFileSizeLimit: true` and an absolute log path for the deployed service. Keep it in the deployment config (NSSM env-var overlay / deploy script from TST-10) rather than only in the committed `appsettings.json`, since the absolute path is host-specific.
|
||||||
|
|
||||||
|
**Implementation.** `src/ZB.MOM.WW.MxGateway.Server/appsettings.json` Serilog file sink (add size cap + retained-count) and the TST-10 deploy config for the absolute path. Docs: `docs/GatewayConfiguration.md` logging notes; `docs/Deployment.md`.
|
||||||
|
|
||||||
|
**Verification.** `dotnet run --project src/ZB.MOM.WW.MxGateway.Server/...` and confirm rollover on size; no unit test.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-22 — Config-shape JSON block omits documented keys `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** Verified the Configuration Shape JSON block (`docs/GatewayConfiguration.md:12-83`) omits `DisableLogin`/`AutoLoginUser`/`CookieName`, the `Tls` block, and the `Alarms:Fallback` block — all of which the tables below it document. (The rest of `docs/GatewayConfiguration.md` is current: table defaults match `Configuration/*.cs`.)
|
||||||
|
|
||||||
|
**Impact.** Low — the tables are authoritative and correct; the shape block is a convenience that is now incomplete, so a copy-paste starting config misses those keys.
|
||||||
|
|
||||||
|
**Design.** Add the missing keys to the JSON block so it round-trips with the tables. Pure documentation.
|
||||||
|
|
||||||
|
**Implementation.** `docs/GatewayConfiguration.md:12-83` — add `Dashboard:DisableLogin`, `Dashboard:AutoLoginUser`, `Dashboard:CookieName`, the `Tls` section, and `Alarms:Fallback`. Verify against `Configuration/DashboardOptions.cs`, `TlsOptions.cs`, and the alarms fallback options.
|
||||||
|
|
||||||
|
**Verification.** Read-back against the cited options classes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-23 — Bidirectional `Session` RPC never built `Low` · `P2`
|
||||||
|
|
||||||
|
**Finding.** Verified `gateway.md:330` presents `rpc Session(stream ClientMessage) returns (stream ServerMessage)` as the "best long-term shape"; the shipped service stopped at unary + server-streaming (`mxaccess_gateway.proto:18-22`, no bidi RPC).
|
||||||
|
|
||||||
|
**Impact.** None functional — purely a docs-currency/roadmap-clarity issue (the twin of TST-13).
|
||||||
|
|
||||||
|
**Design.** Fold into the TST-13 gateway.md edit: label the `Session` RPC explicitly as unbuilt future work (or remove it if it is no longer the intended direction). Decide whether bidi `Session` is still on the roadmap; if not, drop the "best long-term shape" framing so it doesn't read as an imminent deliverable.
|
||||||
|
|
||||||
|
**Implementation.** `gateway.md` around `:328-345` — one clarifying edit, made with TST-13. Docs-only.
|
||||||
|
|
||||||
|
**Verification.** Read-back; consistent with `mxaccess_gateway.proto`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TST-24 — Client wire behaviour has no automated verification `Low` · `—`
|
||||||
|
|
||||||
|
**Finding.** All five clients have unit tests (13/8/3/13/7 files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1`. `CrossLanguageSmokeMatrixTests` checks shapes only.
|
||||||
|
|
||||||
|
**Impact.** Low-to-moderate: a gateway contract change can pass every default suite and break all five clients (partly mitigated by shared-proto codegen). The Java CLI already proves the cheap pattern — `InProcessGatewayHarness` (`stillpending.md` §8).
|
||||||
|
|
||||||
|
**Design.** Replicate the in-process fake-gateway pattern per client so each client's happy path (open → invoke → stream) runs against a lightweight fake gateway in the client's own test suite, wired into CI (TST-03). Java has the template; port the idea to the other four. This is the cheapest way to catch contract breaks without a live rig. Full parity is out of scope — target the core session/invoke/stream shapes and the `ReplayGap` handling added in TST-01.
|
||||||
|
|
||||||
|
**Implementation.** Per-client test harness (e.g. an in-process gRPC server serving canned replies) under each `clients/<lang>/` test tree; add to the CI client jobs. Coordinate the fake-gateway contract with the `CLI` domain report. Docs: each client README notes the harness.
|
||||||
|
|
||||||
|
**Verification.** Per-client test command (CLAUDE.md client table); the harness tests run inside CI (TST-03).
|
||||||
Reference in New Issue
Block a user