docs(archreview): add architecture review + per-domain remediation designs and tracker
Adds the 2026-07-08 architecture review (00-overall + six domain reports) and a remediation/ tree: one design+implementation doc per domain covering every finding, plus 00-tracking.md as the master progress tracker. - 153 findings with stable IDs (GWC/WRK/IPC/SEC/CLI/TST), each with design rationale, implementation steps, tests, docs, and verification. - Tracker rolls findings up by severity and P0/P1/P2 roadmap tier, records cross-cutting clusters and per-finding status (all Not started). - Planning docs only; no source changes.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,256 @@
|
||||
# MxAccessGateway — Remediation Tracking
|
||||
|
||||
Master progress tracker for the 2026-07-08 architecture review. Generated 2026-07-09.
|
||||
|
||||
Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation designs are linked below and hold the full **Finding / Impact / Design / Implementation / Verification** for every entry here.
|
||||
|
||||
## How to use this document
|
||||
|
||||
- Each finding has a stable ID (`GWC-01`, `WRK-01`, …) that never changes. Cite it in commits, branches, and PRs (e.g. `fix(GWC-01): split alarm feed`).
|
||||
- The **Status** column is the single source of truth for progress. Update it in the same change that lands the fix.
|
||||
- Do the work in **roadmap-tier order** (P0 → P1 → P2), respecting the `Dep` (depends-on) column within a tier.
|
||||
- When a fix lands: flip Status to `Done`, and per the repo rule, update the affected docs in the same commit.
|
||||
|
||||
**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / positive observation, no action).
|
||||
|
||||
## Severity roll-up
|
||||
|
||||
Every finding from all six domain reports, including Lows and informational/positive notes (which exceed the headline "106" because the domain docs cover each sub-finding individually).
|
||||
|
||||
| Domain | Doc | Critical | High | Medium | Low | Info | Total |
|
||||
|---|---|:-:|:-:|:-:|:-:|:-:|:-:|
|
||||
| Gateway core | [10-gateway-core.md](10-gateway-core.md) | 1 | 2 | 7 | 12 | 1 | 23 |
|
||||
| Worker | [20-worker.md](20-worker.md) | — | 1 | 6 | 13 | — | 20 |
|
||||
| Contracts & IPC | [30-contracts-ipc.md](30-contracts-ipc.md) | — | 1 | 8 | 13 | — | 22 |
|
||||
| Security & dashboard | [40-security-dashboard.md](40-security-dashboard.md) | — | — | 12 | 15 | 3 | 30 |
|
||||
| Clients | [50-clients.md](50-clients.md) | — | 4 | 13 | 17 | — | 34 |
|
||||
| Testing, docs & gaps | [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | — | 4 | 13 | 7 | — | 24 |
|
||||
| **Total** | | **1** | **12** | **59** | **77** | **4** | **153** |
|
||||
|
||||
## Roadmap-tier roll-up
|
||||
|
||||
Tiers follow the roadmap in `00-overall.md`. Work top-down; within a tier, `Dep` gives ordering.
|
||||
|
||||
| Tier | Meaning | Count | Findings |
|
||||
|---|---|:-:|---|
|
||||
| **P0** | Correctness & safety — before wider rollout | 8 | GWC-01, GWC-02, GWC-03, WRK-01, CLI-01, CLI-03, TST-02, TST-12 |
|
||||
| **P1** | Process & hardening — next few weeks | 26 | GWC-04, WRK-07, IPC-01, IPC-02, IPC-03, IPC-04, IPC-09, IPC-19, IPC-20, SEC-01, SEC-02, SEC-04, SEC-05, SEC-06, SEC-07, SEC-08, SEC-10, SEC-11, SEC-12, SEC-20, CLI-02, TST-03, TST-05, TST-08 |
|
||||
| **P2** | Completeness & polish | 39 | GWC-06/07/08/14/15, WRK-06/11/12/15, IPC-05/06/07/13/14/15/17/21, SEC-03/09/22/25/30, CLI-04/12/15/16/18/21/26/29, TST-01/04/11/13/14/15/23/24 |
|
||||
| **—** | Not individually in the roadmap (mostly Low/Info) | 80 | see per-domain registers below |
|
||||
|
||||
### P0 — do first (8)
|
||||
|
||||
| ID | Sev | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|---|---|---|
|
||||
| GWC-01 | Critical | M | — | Not started | Alarm monitor and distributor pump both drain the single worker event channel |
|
||||
| GWC-02 | High | M | — | Not started | Faulted sessions are never swept (worker + slot pinned up to 30 min) |
|
||||
| GWC-03 | High | S | — | Not started | Documented sparse-array max-length bound is unimplemented |
|
||||
| WRK-01 | High | M | — | Not started | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
|
||||
| CLI-01 | High | M | — | Not started | Go `Session.Events()` silently closes stream on 16-slot overflow |
|
||||
| CLI-03 | High | M | — | Not started | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
|
||||
| TST-02 | High | M | TST-04 | Not started | Reconnect owner re-validation not implemented (security control) |
|
||||
| TST-12 | Medium | S | — | Not started | CLAUDE.md misstates default retention behaviour (defaults are on, not off) |
|
||||
|
||||
## Finding registers by domain
|
||||
|
||||
Full design + implementation for each row lives in the linked domain doc under its ID heading.
|
||||
|
||||
### Gateway core — [10-gateway-core.md](10-gateway-core.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| GWC-01 | Critical | P0 | M | — | Not started | Alarm monitor and distributor pump both drain the single worker event channel |
|
||||
| GWC-02 | High | P0 | M | — | Not started | Faulted sessions are never swept |
|
||||
| GWC-03 | High | P0 | S | — | Not started | Documented sparse-array max-length bound is unimplemented |
|
||||
| GWC-04 | Medium | P1 | M | — | Not started | Full event channel stalls the worker read loop behind command replies |
|
||||
| GWC-05 | Medium | — | S | — | Not started | Worker pipe created with no ACL / no CurrentUserOnly |
|
||||
| GWC-06 | Medium | P2 | S | GWC-07,08 | Not started | Stopwatch allocated per streamed event |
|
||||
| GWC-07 | Medium | P2 | S | GWC-06,08 | Not started | Every mapped event is deep-cloned |
|
||||
| GWC-08 | Medium | P2 | M | GWC-06,07 | Not started | Pipe framing allocates per frame and writes twice |
|
||||
| GWC-09 | Medium | — | M | — | Not started | Startup probe is a no-op; its retry pipeline can never retry |
|
||||
| GWC-10 | Medium | — | S | — | Not started | Envelope `sequence` monotonicity specified but not enforced |
|
||||
| GWC-11 | Low | — | S | — | Not started | `_workerClient` written under lock but read lock-free |
|
||||
| GWC-12 | Low | — | S | — | Not started | `WorkerClient.DisposeAsync` not safe against double-dispose |
|
||||
| GWC-13 | Low | — | M | — | Not started | Worker-ready wait is a 25 ms poll loop |
|
||||
| GWC-14 | Low | P2 | M | — | Not started | Replay ring is a `LinkedList` with a node alloc per event |
|
||||
| GWC-15 | Low | P2 | S | — | Not started | Per-event gauge reads `ChannelReader.Count` every event |
|
||||
| GWC-16 | Low | — | S | — | Not started | `Invoke` resolves the session twice per command |
|
||||
| GWC-17 | Low | — | M | — | Not started | Sessions layer throws `Grpc.Core.RpcException` (layering leak) |
|
||||
| GWC-18 | Low | — | S | — | Not started | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` |
|
||||
| GWC-19 | Low | — | S | — | Not started | Dead `GatewaySession.KillWorker(string)` + stale doc |
|
||||
| GWC-20 | Low | — | S | — | Not started | Heartbeat config semantics conflated (send vs check interval) |
|
||||
| GWC-21 | Low | — | S | GWC-04 | Not started | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable |
|
||||
| GWC-22 | Low | — | S | — | Not started | `StreamDisconnected` always labeled `"Detached"` |
|
||||
| GWC-23 | Info | — | — | — | N/A | `MaxEventSubscribersPerSession` dead in default single-subscriber mode |
|
||||
|
||||
### Worker — [20-worker.md](20-worker.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| WRK-01 | High | P0 | M | — | Not started | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped |
|
||||
| WRK-02 | Medium | — | M | — | Not started | STA thread death after startup is silent; future work hangs forever |
|
||||
| WRK-03 | Medium | — | S | WRK-02 | Not started | Commands after shutdown starts are dropped with no reply |
|
||||
| WRK-04 | Medium | — | M | — | Not started | Envelope `sequence` can appear out of order on the wire |
|
||||
| WRK-05 | Medium | — | M | — | Not started | One transient alarm-poll failure kills the whole session |
|
||||
| WRK-06 | Medium | P2 | S | — | Not started | `MXSTATUS_PROXY` conversion reflects per field, per event |
|
||||
| WRK-07 | Medium | P1 | M | WRK-04 | Not started | Documented outbound write priority not implemented |
|
||||
| WRK-08 | Low | — | S | — | Not started | Residual event queue discarded at graceful shutdown, undocumented |
|
||||
| WRK-09 | Low | — | S | — | Not started | No `AppDomain.UnhandledException` hook |
|
||||
| WRK-10 | Low | — | S | — | Not started | Top-level catch logs exception type but never message |
|
||||
| WRK-11 | Low | P2 | S | — | Not started | Every accepted event is defensively cloned on enqueue |
|
||||
| WRK-12 | Low | P2 | M | WRK-04 | Not started | No event batching per envelope; one flush per event; 25 ms poll |
|
||||
| WRK-13 | Low | — | S | — | Not started | Frame writer allocates a fresh buffer per frame; reader pools |
|
||||
| WRK-14 | Low | — | M | — | Not started | Private-field naming split `_camelCase` vs `camelCase` |
|
||||
| WRK-15 | Low | P2 | S | — | Not started | Doc drift: STA thread name and heartbeat-counter note |
|
||||
| WRK-16 | Low | — | S | — | Not started | Boilerplate duplication in IPC envelope/ctor overloads |
|
||||
| WRK-17 | Low | — | S | — | Not started | Gateway death exits with wrong exit code (6 not 5) |
|
||||
| WRK-18 | Low | — | S | — | Not started | Event-queue overflow exits as generic `UnexpectedFailure` |
|
||||
| WRK-19 | Low | — | S | — | Not started | Command start/end logging with correlation id absent |
|
||||
| WRK-20 | Low | — | M | WRK-01..04 | Not started | Test-coverage gaps for the failure modes above |
|
||||
|
||||
### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| IPC-01 | High | P1 | M | — | Not started | Published client descriptor set 7 weeks stale; nothing enforces freshness |
|
||||
| IPC-02 | Medium | P1 | M | IPC-03 | Not started | Worker max frame size hard-coded, cannot follow gateway config |
|
||||
| IPC-03 | Medium | P1 | M | IPC-02 | Not started | gRPC max == pipe max with zero headroom; oversized write faults whole session |
|
||||
| IPC-04 | Medium | P1 | S | IPC-03 | Not started | `DrainEvents max_events=0` packs entire queue into one reply frame |
|
||||
| IPC-05 | Medium | P2 | S | — | Not started | Redundant deep copies on command and event hot paths |
|
||||
| IPC-06 | Medium | P2 | S | — | Not started | `gateway.md` Worker Envelope sketch no longer matches the contract |
|
||||
| IPC-07 | Medium | P2 | S | — | Not started | `docs/Grpc.md` says six RPCs; there are seven |
|
||||
| IPC-08 | Medium | — | M | — | Not started | `WorkerCancel` defined and handled but never sent |
|
||||
| IPC-09 | Medium | P1 | M | IPC-01 | Not started | Known codegen fragilities have no in-repo guards |
|
||||
| IPC-10 | Low | — | S | — | Not started | Envelope `sequence` is write-only; monotonicity never validated |
|
||||
| IPC-11 | Low | — | S | — | Not started | No protocol version negotiation despite `supported_protocol_version` name |
|
||||
| IPC-12 | Low | — | S | — | Not started | Gateway frame writer has no write lock; integrity rests on undocumented invariant |
|
||||
| IPC-13 | Low | P2 | S | IPC-05 | Not started | Gateway writer serializes twice and issues two stream writes |
|
||||
| IPC-14 | Low | P2 | S | IPC-05 | Not started | Gateway reader allocates fresh array per frame; worker rents from `ArrayPool` |
|
||||
| IPC-15 | Low | P2 | M | — | Not started | Worker event delivery 25 ms poll with per-event write+flush, no batching |
|
||||
| IPC-16 | Low | — | S | — | N/A | Correlation id carried twice per reply (Info) |
|
||||
| IPC-17 | Low | P2 | S | — | Not started | Two docs name the wrong Python generated-output directory |
|
||||
| IPC-18 | Low | — | S | IPC-01 | N/A | Generated-code hygiene verified good — preserve under change (positive) |
|
||||
| IPC-19 | Low | P1 | S | IPC-01 | Not started | Generated/-must-be-committed rule for net48 undocumented and unguarded |
|
||||
| IPC-20 | Low | P1 | S | IPC-01 | Not started | Descriptor `-Check` byte-compares source-info bytes; protoc-version-sensitive |
|
||||
| IPC-21 | Low | P2 | S | IPC-06 | Not started | `gateway.md` presents unimplemented `Session` RPC as live |
|
||||
| IPC-22 | Low | — | S | — | N/A | Public error-detail model stops at status codes plus prose (Info) |
|
||||
|
||||
### Security & dashboard — [40-security-dashboard.md](40-security-dashboard.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| SEC-01 | Medium | P1 | M | — | Not started | Windows-absolute default paths become relative files off-Windows |
|
||||
| SEC-02 | Medium | P1 | S | — | Not started | `AllowAnonymousLocalhost` satisfies AdminOnly, not just Viewer |
|
||||
| SEC-03 | Medium | P2 | S | — | Not started | Dashboard cookie lost its `__Host-` prefix; four docs still promise it |
|
||||
| SEC-04 | Medium | P1 | S | SEC-03 | Not started | `DisableLogin` has no production guard |
|
||||
| SEC-05 | Medium | P1 | M | — | Not started | Hub bearer tokens irrevocable for 30 min, carried in query string |
|
||||
| SEC-06 | Medium | P1 | M | — | Not started | LDAP plaintext-by-default with a committed service password |
|
||||
| SEC-07 | Medium | P1 | S | — | Not started | `QueryActiveAlarmsRequest` missing from scope resolver; tests mislabeled |
|
||||
| SEC-08 | Medium | P1 | L | — | Not started | Per-RPC SQLite read + `last_used_utc` write; no verification cache |
|
||||
| SEC-09 | Medium | P2 | S | — | Not started | Dashboard design-doc `GroupToRole` sample now fails startup validation |
|
||||
| SEC-10 | Medium | P1 | L | — | Not started | No API-key expiry |
|
||||
| SEC-11 | Medium | P1 | M | SEC-08 | Not started | No rate limiting or lockout on either auth surface |
|
||||
| SEC-12 | Medium | P1 | M | — | Not started | Dashboard Close/Kill bypass the canonical audit store |
|
||||
| SEC-13 | Low | — | S | SEC-30 | Not started | Redactor's credential-command list omits secured-bulk variants |
|
||||
| SEC-14 | Low | — | S | SEC-02,20 | Not started | `/metrics` + `/health` unauthenticated; a metric leaks session ids |
|
||||
| SEC-15 | Low | — | S | — | Not started | GET `/logout` skips antiforgery |
|
||||
| SEC-16 | Low | — | S | — | Not started | CLI accepts the pepper as a command-line argument |
|
||||
| SEC-17 | Low | — | S | — | Not started | Interceptor does not override client-streaming/duplex handlers |
|
||||
| SEC-18 | Low | — | S | — | Not started | Pepper-unavailable detection matches library message text |
|
||||
| SEC-19 | Low | — | S | — | Not started | Canonical audit store re-issues `CREATE TABLE` per write/read |
|
||||
| SEC-20 | Low | P1 | S | — | Not started | `session_id` metric tag is unbounded cardinality |
|
||||
| SEC-21 | Low | — | M | — | Not started | Snapshot publisher works every second regardless of audience |
|
||||
| SEC-22 | Low | P2 | M | — | Not started | `docs/Authentication.md` documents types no longer in this repo |
|
||||
| SEC-23 | Low | — | S | SEC-03,04 | Not started | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` |
|
||||
| SEC-24 | Low | — | S | SEC-04 | Not started | Effective-config view omits the riskiest dashboard flags |
|
||||
| SEC-25 | Low | P2 | M | SEC-02 | Not started | Per-session EventsHub ACL is an acknowledged TODO |
|
||||
| SEC-26 | Low | — | M | — | Not started | Audit trail has no retention or pruning |
|
||||
| SEC-27 | Low | — | S | — | Not started | Dashboard `GatewayStatus` is hardcoded Healthy |
|
||||
| SEC-28 | Info | — | S | — | N/A | Positive security observations (preserve under change) |
|
||||
| SEC-29 | Info | — | S | — | N/A | UI-stack rule verified compliant |
|
||||
| SEC-30 | Info | P2 | S | SEC-13 | Not started | Value-logging feature is unwired end-to-end |
|
||||
|
||||
### Clients — [50-clients.md](50-clients.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| CLI-01 | High | P0 | M | — | Not started | Go `Session.Events()` silently closes stream on 16-slot overflow |
|
||||
| CLI-02 | High | P1 | M | — | Not started | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) |
|
||||
| CLI-03 | High | P0 | M | — | Not started | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY |
|
||||
| CLI-04 | High | P2 | L | CLI-15 | Not started | Typed-command parity gap across all five clients |
|
||||
| CLI-05 | Medium | — | S | — | Not started | .NET session cannot be re-attached to an existing session id |
|
||||
| CLI-06 | Medium | — | S | — | Not started | .NET `DisposeAsync` blocks/throws on unreachable gateway |
|
||||
| CLI-07 | Medium | — | S | — | Not started | .NET retry budget self-defeats on `DeadlineExceeded` |
|
||||
| CLI-08 | Medium | — | S | CLI-03 | Not started | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) |
|
||||
| CLI-09 | Medium | — | M | — | Not started | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) |
|
||||
| CLI-10 | Medium | — | M | — | Not started | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` |
|
||||
| CLI-11 | Medium | — | S | — | Not started | Go CLI cannot opt into strict TLS validation |
|
||||
| CLI-12 | Medium | P2 | S | — | Not started | Java docs still say "Java 21" after the JDK 17 retarget |
|
||||
| CLI-13 | Medium | — | M | — | Not started | Java event-stream buffer hardcoded 16, cancel-on-overflow |
|
||||
| CLI-14 | Medium | — | S | — | Not started | Python default TLS is blocking TOFU pin with silent `localhost` SNI |
|
||||
| CLI-15 | Medium | P2 | M | — | Not started | No client handles `ReplayGap` or offers a reconnect helper |
|
||||
| CLI-16 | Medium | P2 | S | CLI-12 | Not started | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` |
|
||||
| CLI-17 | Medium | — | M | CLI-14 | Not started | TLS default posture inconsistent across the five clients |
|
||||
| CLI-18 | Low | P2 | S | — | Not started | .NET csproj records no `<Version>` |
|
||||
| CLI-19 | Low | — | S | — | Not started | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) |
|
||||
| CLI-20 | Low | — | S | CLI-17 | Not started | .NET `--tls` without CA installs accept-all callback |
|
||||
| CLI-21 | Low | P2 | S | — | Not started | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases |
|
||||
| CLI-22 | Low | — | S | — | Not started | Go `newCorrelationID` swallows `crypto/rand` error → empty id |
|
||||
| CLI-23 | Low | — | S | — | Not started | Go nil-vs-empty bulk short-circuit asymmetry |
|
||||
| CLI-24 | Low | — | S | — | Not started | Java `MxEventStream` single-consumer constraint undocumented |
|
||||
| CLI-25 | Low | — | S | — | Not started | Java `close()` does not await channel termination |
|
||||
| CLI-26 | Low | P2 | S | — | Not started | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) |
|
||||
| CLI-27 | Low | — | S | — | Not started | Python `Session.close()` not concurrency-safe; synthesizes reply |
|
||||
| CLI-28 | Low | — | S | — | Not started | Python circular-import workaround at end of `session.py` |
|
||||
| CLI-29 | Low | P2 | S | — | Not started | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) |
|
||||
| CLI-30 | Low | — | S | CLI-04 | Not started | Rust has no `unregister` typed helper |
|
||||
| CLI-31 | Low | — | M | — | Not started | Rust CLI is a single 2,699-line `main.rs` |
|
||||
| CLI-32 | Low | — | S | — | Not started | Client-side bulk caps differ (.NET/Java unbounded) |
|
||||
| CLI-33 | Low | — | S | CLI-01,13 | Not started | Per-language event backpressure semantics undocumented |
|
||||
| CLI-34 | Low | — | S | — | Not started | Python `build/`/`.pytest_cache/` present on disk (untracked) |
|
||||
|
||||
### Testing, docs & gaps — [60-testing-docs-gaps.md](60-testing-docs-gaps.md)
|
||||
|
||||
| ID | Sev | Tier | Eff | Dep | Status | Title |
|
||||
|---|---|:-:|:-:|---|---|---|
|
||||
| TST-01 | High | P2 | L | TST-04 | Not started | Reconnect/replay has no e2e test and no client consumer |
|
||||
| TST-02 | High | P0 | M | TST-04 | Not started | Reconnect owner re-validation not implemented |
|
||||
| TST-03 | High | P1 | M | — | Not started | No CI exists |
|
||||
| TST-04 | High | P2 | L | — | Not started | Session-resilience epic 16/28 tasks unfinished |
|
||||
| TST-05 | Medium | P1 | S | TST-03 | Not started | Real-worker control/COM paths verified opt-in only |
|
||||
| TST-06 | Medium | — | M | — | Not started | Dashboard live-data path untested |
|
||||
| TST-07 | Medium | — | S | — | Not started | Real-clock sleeps with negative assertions are latent flakes |
|
||||
| TST-08 | Medium | P1 | M | — | Not started | Full-suite orphaned testhost processes |
|
||||
| TST-09 | Medium | — | M | — | Not started | Health checks cover only the auth store |
|
||||
| TST-10 | Medium | — | M | — | Not started | Deployment/upgrade undocumented and hand-rolled |
|
||||
| TST-11 | Medium | P2 | M | — | Not started | No version discipline on server; client versions drift |
|
||||
| TST-12 | Medium | P0 | S | — | Not started | CLAUDE.md misstates default retention behaviour |
|
||||
| TST-13 | Medium | P2 | S | — | Not started | gateway.md carries stale design-era sketches |
|
||||
| TST-14 | Medium | P2 | S | — | Not started | Repo-root working artifacts need triage |
|
||||
| TST-15 | Medium | P2 | M | TST-04 | Not started | Dashboard EventsHub has no per-session ACL |
|
||||
| TST-16 | Medium | — | S | — | Not started | `Dashboard:ShowTagValues` is a dead flag |
|
||||
| TST-17 | Medium | — | S | — | Not started | Vendor-gated alarm parity residuals silently lossy |
|
||||
| TST-18 | Low | — | S | — | Not started | Hosted-service wrappers untested |
|
||||
| TST-19 | Low | — | S | — | Not started | Keep FakeWorkerHarness canned replies in lockstep |
|
||||
| TST-20 | Low | — | S | — | Not started | E2E script papers over a real advise-without-consumer sharp edge |
|
||||
| TST-21 | Low | — | S | — | Not started | Log rotation configured but minimal |
|
||||
| TST-22 | Low | — | S | — | Not started | Config-shape JSON block omits documented keys |
|
||||
| TST-23 | Low | P2 | S | — | Not started | Bidirectional `Session` RPC never built |
|
||||
| TST-24 | Low | P2 | M | TST-03 | Not started | Client wire behaviour has no automated verification |
|
||||
|
||||
## Cross-cutting clusters
|
||||
|
||||
Findings the review flagged as one coordinated design pass — sequence them together rather than piecemeal.
|
||||
|
||||
- **Size / backpressure topology:** IPC-02, IPC-03, IPC-04 + GWC-04 + WRK-12/WRK-07 + CLI-13/CLI-32/CLI-33. Convey `MaxMessageBytes` to the worker, bound `DrainEvents`, decouple event backlog from command replies, make client buffer-overflow behavior uniform and loud.
|
||||
- **Event hot-path performance:** GWC-06/07/08/14/15 + WRK-06/11/12/13 + IPC-05/13/14/15. Kill double clones, cache the status converter, batch event frames, pool gateway frame buffers.
|
||||
- **Silent-failure hardening:** GWC-02, WRK-01/02/03, CLI-01, CLI-03. Every death or drop must produce an observable error or fault frame.
|
||||
- **CI + codegen freshness (unlocks the rest):** TST-03 + IPC-01/09/19/20 + CLI-34. Stand up minimal CI so the descriptor/codegen/version guards actually run.
|
||||
- **Doc-drift sweep:** IPC-06/07/17/21 + SEC-03/09/22 + CLI-12/16 + TST-12/13/14 + GWC-19 + WRK-15. Correct load-bearing docs to match shipped behavior.
|
||||
- **Session-resilience epic:** TST-01/02/04/15 + SEC-25 + CLI-15. Finish owner re-validation, client `ReplayGap` handling, per-session event ACL, and the replay e2e test.
|
||||
|
||||
## Change log
|
||||
|
||||
| Date | Change |
|
||||
|---|---|
|
||||
| 2026-07-09 | Initial tracking doc generated from the six domain remediation designs. All 153 findings `Not started`. |
|
||||
@@ -0,0 +1,447 @@
|
||||
# Gateway Server Core — Remediation Design & Implementation
|
||||
|
||||
Source review: [10-gateway-core.md](../10-gateway-core.md) · Generated: 2026-07-09
|
||||
|
||||
This document turns the Gateway Server Core review into buildable remediation entries. Every cited `path:line` was re-verified against the working tree; all citations still match at the time of writing. Findings are grouped most-severe first, then in the source report's own order (Stability → Performance → Conventions → Underdeveloped). The one Critical (alarm-feed split), both Highs (faulted-session reaping, sparse-array bound), and one Medium (event-backlog stall) carry roadmap tiers from `00-overall.md`; the performance cluster is the P2 hot-path pass.
|
||||
|
||||
## Finding index
|
||||
|
||||
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||
|----|-----|-------|---------|--------|-----------|-------|
|
||||
| GWC-01 | Critical | Alarm monitor and distributor pump both drain the single worker event channel | P0 | M | — | Alarms/GatewayAlarmMonitor.cs, Sessions/GatewaySession.cs, Workers/WorkerClient.cs |
|
||||
| GWC-02 | High | Faulted sessions are never swept | P0 | M | — | Sessions/GatewaySession.cs, Sessions/SessionManager.cs |
|
||||
| GWC-03 | High | Documented sparse-array max-length bound is unimplemented | P0 | S | — | Sessions/SparseArrayExpander.cs, Configuration/EventOptions.cs, Configuration/GatewayOptionsValidator.cs |
|
||||
| GWC-04 | Medium | Full event channel stalls the worker read loop behind command replies | P1 | M | — | Workers/WorkerClient.cs |
|
||||
| GWC-05 | Medium | Worker pipe created with no ACL / no CurrentUserOnly | — | S | — | Sessions/SessionWorkerClientFactory.cs |
|
||||
| GWC-06 | Medium | Stopwatch allocated per streamed event | P2 | S | GWC-07, GWC-08 | Grpc/MxAccessGatewayService.cs |
|
||||
| GWC-07 | Medium | Every mapped event is deep-cloned | P2 | S | GWC-06, GWC-08 | Grpc/MxAccessGrpcMapper.cs, Sessions/GatewaySession.cs |
|
||||
| GWC-08 | Medium | Pipe framing allocates per frame and writes twice | P2 | M | GWC-06, GWC-07 | Workers/WorkerFrameReader.cs, Workers/WorkerFrameWriter.cs |
|
||||
| GWC-09 | Medium | Startup probe is a no-op; its retry pipeline can never retry | — | M | — | Workers/WorkerProcessStartedProbe.cs, Workers/WorkerProcessLauncher.cs, Configuration/WorkerOptions.cs |
|
||||
| GWC-10 | Medium | Envelope `sequence` monotonicity is specified but not enforced | — | S | — | Workers/WorkerEnvelopeValidator.cs, Workers/WorkerClient.cs |
|
||||
| GWC-11 | Low | `_workerClient` written under lock but read lock-free | — | S | — | Sessions/GatewaySession.cs |
|
||||
| GWC-12 | Low | `WorkerClient.DisposeAsync` not safe against double-dispose | — | S | — | Workers/WorkerClient.cs |
|
||||
| GWC-13 | Low | Worker-ready wait is a 25 ms poll loop | — | M | — | Sessions/GatewaySession.cs |
|
||||
| GWC-14 | Low | Replay ring is a `LinkedList` with a node alloc per event | P2 | M | — | Sessions/SessionEventDistributor.cs |
|
||||
| GWC-15 | Low | Per-event gauge reads `ChannelReader.Count` every event | P2 | S | — | Grpc/EventStreamService.cs |
|
||||
| GWC-16 | Low | `Invoke` resolves the session twice per command | — | S | — | Grpc/MxAccessGatewayService.cs, Sessions/SessionManager.cs |
|
||||
| GWC-17 | Low | Sessions layer throws `Grpc.Core.RpcException` | — | M | — | Sessions/SparseArrayExpander.cs, Sessions/GatewaySession.cs, Grpc/MxAccessGatewayService.cs |
|
||||
| GWC-18 | Low | `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` | — | S | — | Sessions/GatewaySession.cs |
|
||||
| GWC-19 | Low | Dead `GatewaySession.KillWorker(string)` + stale doc | — | S | — | Sessions/GatewaySession.cs, docs/Sessions.md |
|
||||
| GWC-20 | Low | Heartbeat config semantics conflated (send vs check interval) | — | S | — | Configuration/WorkerOptions.cs, Sessions/SessionWorkerClientFactory.cs, Workers/WorkerClient.cs |
|
||||
| GWC-21 | Low | `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable | — | S | GWC-04 | Sessions/SessionWorkerClientFactory.cs, Configuration/WorkerOptions.cs |
|
||||
| GWC-22 | Low | `StreamDisconnected` always labeled `"Detached"` | — | S | — | Grpc/EventStreamService.cs |
|
||||
| GWC-23 | Info | `MaxEventSubscribersPerSession` dead in default single-subscriber mode | — | — | — | Configuration/GatewayOptionsValidator.cs |
|
||||
|
||||
---
|
||||
|
||||
## GWC-01 — Alarm monitor and distributor pump both drain the single worker event channel `Critical` · `P0`
|
||||
|
||||
**Finding.** `GatewayAlarmMonitor.RunMonitorAsync` drains worker events directly through `_sessionManager.ReadEventsAsync(session.SessionId, …)` (`Alarms/GatewayAlarmMonitor.cs:228-231`) → `GatewaySession.ReadEventsAsync` (`Sessions/GatewaySession.cs:1427-1440`) → `WorkerClient.ReadEventsAsync` (`Workers/WorkerClient.cs:227-236`). But `MarkReady` starts the dashboard mirror (`Sessions/GatewaySession.cs:433-437`), whose distributor pump source `MapWorkerEventsAsync` also calls `ReadEventsAsync` (`Sessions/GatewaySession.cs:701-710`). `WorkerClient._events` is a single channel created with `SingleReader = false` (`Workers/WorkerClient.cs:70-77`), so the two concurrent `ReadAllAsync` enumerators each receive a random subset of events.
|
||||
|
||||
**Impact.** In every production deployment `IDashboardEventBroadcaster` is registered and injected, so the pump starts at session-ready — including on the alarm monitor's own session. Roughly half of `OnAlarmTransition` events land in the dashboard mirror instead of `ApplyTransition`. `ReconcileLoopAsync` eventually repairs Raise/Clear presence, but `ApplyReconcile` broadcasts only presence deltas (`Alarms/GatewayAlarmMonitor.cs:511-550`), so a stolen Acknowledge transition never reaches `StreamAlarms` subscribers — clients show unacked alarms indefinitely. Provider-mode-change events can also be stolen, delaying degraded-state visibility. `docs/Sessions.md:196` documents the exact race single-subscriber mode exists to prevent; the monitor recreates it internally.
|
||||
|
||||
**Design.** Make the alarm monitor a *distributor subscriber* rather than a second raw channel drain, so the single worker channel keeps exactly one consumer (the distributor pump) and the pump fans every event to both the dashboard mirror and the alarm monitor. Add an internal (non-counted) registration path on the session mirroring the existing dashboard mirror lease (`isInternal: true`, `Sessions/GatewaySession.cs:582`), so the alarm subscriber does not consume one of `MaxEventSubscribersPerSession` and a slow reconcile never faults the session.
|
||||
|
||||
To make any *future* dual-consumer regression fail loudly rather than silently split events, additionally flip `WorkerClient._events` to `SingleReader = true` and add a claimed-once guard in `WorkerClient.ReadEventsAsync` (throw if a second enumerator is created). After this fix the distributor pump is the only direct caller, so `SingleReader = true` is correct and the guard converts the bug class into an immediate exception.
|
||||
|
||||
Alternative rejected: giving the alarm monitor its own second worker channel (fan the worker frame to two channels in `EnqueueWorkerEventAsync`) — doubles per-event copy cost on the hot path and duplicates the backpressure/overflow accounting the distributor already owns.
|
||||
|
||||
Parity note: no synthesized events and no MXAccess behavior change — this only fixes *delivery* of events the worker already emits.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/GatewaySession.cs`: add an internal-subscriber accessor the alarm monitor can use, e.g. `AttachInternalEventSubscriber()` that runs `EnsureDistributorCreated` / `distributor.Register(isInternal: true)` / `StartPumpIfRequested` (the same sequence `StartDashboardMirror` uses at `:581-583`) and returns an `IAsyncEnumerable<MxEvent>` (or the lease + its channel reader). Reuse `MapWorkerEventsAsync`'s public `MxEvent` shape so the monitor's existing `MxEvent.BodyCase` switch (`Alarms/GatewayAlarmMonitor.cs:232-246`) is unchanged.
|
||||
- `Sessions/SessionManager.cs`: expose the new subscriber path (e.g. `IAsyncEnumerable<MxEvent> ReadAlarmEventsAsync(sessionId, ct)`) alongside the existing `ReadEventsAsync`, or repoint the monitor at the distributor via the session directly.
|
||||
- `Alarms/GatewayAlarmMonitor.cs:228-231`: consume the distributor subscriber stream (mapped `MxEvent`) instead of `_sessionManager.ReadEventsAsync` (raw `WorkerEvent`). `workerEvent.Event` reads become direct `MxEvent` reads.
|
||||
- `Workers/WorkerClient.cs:70-77`: set `SingleReader = true`; in `ReadEventsAsync` (`:227`) add an `Interlocked.CompareExchange` claimed-once guard that throws `InvalidOperationException` on a second enumerator.
|
||||
- Tests: extend `GatewayAlarmMonitorTests` (or add `GatewayAlarmMonitorDistributorTests`) driving the `FakeWorkerHarness` with a session that has the dashboard mirror active, asserting every `OnAlarmTransition` (including Acknowledge) reaches `StreamAlarms`; add a `WorkerClientTests` case asserting a second `ReadEventsAsync` enumerator throws.
|
||||
- Docs: note in `docs/Sessions.md` that the alarm monitor attaches as an internal distributor subscriber; update `gateway.md` alarm section if it describes the direct drain.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewayAlarmMonitor"` and `--filter "FullyQualifiedName~WorkerClient"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-02 — Faulted sessions are never swept `High` · `P0`
|
||||
|
||||
**Finding.** `MarkFaulted` only flips state and records the reason (`Sessions/GatewaySession.cs:716-728`); it does not kill the worker, stop the distributor, or notify the registry. `CloseExpiredLeasesAsync` sweeps only lease-expired or detach-grace-expired sessions (`Sessions/SessionManager.cs:273-277`); there is no `State == Faulted` branch. Detach-grace deliberately skips faulted sessions (`Sessions/GatewaySession.cs:2022-2028`), so the only exit is lease expiry at `DefaultLeaseSeconds` = 1800 s. In the FailFast single-subscriber overflow case (`Sessions/GatewaySession.cs:690-694`) the worker stays healthy and keeps pumping events while the session is permanently unusable (`EvaluateReadyUnderLock` fails every command).
|
||||
|
||||
**Impact.** A burst-slow client that overflows its queue faults the session; for up to 30 minutes the gateway pins one of `MaxSessions` (default 64) slots and a running x86 MXAccess worker with live COM subscriptions no client can use or close (clients rarely call `CloseSession` on a faulted session). A handful of such faults exhausts session capacity.
|
||||
|
||||
**Design.** Sweep `Faulted` sessions in `CloseExpiredLeasesAsync` so the existing serialized teardown (`TryBeginCloseIfExpired` → `CloseSessionCoreAsync`) reaps them promptly, reusing the proven TOCTOU-safe close gate rather than adding a second teardown path. Add a distinct close reason (`FaultedReason`) so operators can distinguish fault reaps from lease/detach reaps in logs and metrics. Optionally add a short `FaultedGraceSeconds` (default 0) so a monitoring client can still observe the fault via `GetSessionStatus` before the slot is reclaimed; recommend default-immediate to bound blast radius, with the grace knob as the tunable.
|
||||
|
||||
Alternative considered: have `MarkFaulted` synchronously kill the worker and schedule teardown (as `WorkerClient.SetFaulted` does on its side). Rejected as the primary fix because `MarkFaulted` is called from inside the distributor overflow handler and the gRPC pump (`Grpc/EventStreamService.cs:155`) under async contexts where a synchronous kill + registry mutation risks re-entrancy against the close gate; the sweeper path already centralizes teardown ordering. `MarkFaulted` can still *nudge* the sweeper (e.g. set an eligibility timestamp), but the reap stays in `CloseExpiredLeasesAsync`.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/GatewaySession.cs`: add `IsFaultedReapable(now)` (true when `_state == Faulted`, gated by optional `FaultedGraceSeconds` measured from a fault timestamp stamped in `MarkFaulted`).
|
||||
- `Sessions/SessionManager.cs:273-277`: extend the reason ladder — `session.IsLeaseExpired(now) ? LeaseExpiredReason : session.IsFaultedReapable(now) ? FaultedReason : session.IsDetachGraceExpired(now) ? DetachGraceExpiredReason : null`. Ensure `TryBeginCloseIfExpired` treats a faulted session as eligible.
|
||||
- `Configuration/SessionOptions.cs`: add `FaultedGraceSeconds` (default 0); validate `>= 0` in `Configuration/GatewayOptionsValidator.cs`.
|
||||
- Tests: `SessionManagerTests`/`SessionLeaseSweeperTests` — fault a session, advance `TimeProvider`, assert the sweeper closes it and kills the worker; assert a non-faulted leased session is untouched.
|
||||
- Docs: `docs/Sessions.md` (sweeper reasons) and `docs/GatewayConfiguration.md` (`FaultedGraceSeconds`) in the same commit.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionManager"` and `--filter "FullyQualifiedName~Sweeper"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-03 — Documented sparse-array max-length bound is unimplemented `High` · `P0`
|
||||
|
||||
**Finding.** `gateway.md:532` lists "`total_length` exceeds the gateway-configured maximum array length" as an `InvalidArgument` rejection, but `SparseArrayExpander.Expand` validates only `totalLength > (uint)Array.MaxLength` (`Sessions/SparseArrayExpander.cs:65-69`) — ~2.1 billion elements. No configuration option for the bound exists in `Configuration/`.
|
||||
|
||||
**Impact.** A single authorized `Write` carrying `total_length = 500_000_000` forces materialization of a multi-GB `MxArray` (`Sessions/SparseArrayExpander.cs:98-232`) before the 16 MB `MaxMessageBytes` frame check finally rejects the serialized result in `WorkerFrameWriter` (`Workers/WorkerFrameWriter.cs:49-54`) — a memory-exhaustion vector reachable through the normal command path.
|
||||
|
||||
**Design.** Add the documented configurable cap under `MxGateway:Events` (or a new `MxGateway:Sparse` section) and enforce it in `SparseArrayExpander.Expand` *before* allocation, throwing the same `InvalidArgument` the surrounding validators already use. Validate the bound at startup in `GatewayOptionsValidator`. A conservative default (e.g. 1_000_000 elements — comfortably above realistic MXAccess array writes yet far below the frame-size ceiling) matches the parity contract: MXAccess itself has no multi-hundred-million-element array use case, so the cap rejects only abusive inputs and does not alter valid behavior.
|
||||
|
||||
The expander is constructed per invocation from `NormalizeOutboundCommand` (`Sessions/GatewaySession.cs:984-1063`); thread the configured max in through its constructor or an `Expand(..., int maxTotalLength)` parameter sourced from `EventOptions`/a new option object the session already holds.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/EventOptions.cs` (or new `SparseArrayOptions`): add `MaxSparseArrayLength` (default 1_000_000).
|
||||
- `Configuration/GatewayOptionsValidator.cs`: validate `MaxSparseArrayLength >= 1` and `<= Array.MaxLength`.
|
||||
- `Sessions/SparseArrayExpander.cs:65-69`: check `totalLength > maxSparseArrayLength` first, with a message naming the configured cap; keep the `Array.MaxLength` guard as a backstop.
|
||||
- Wire the value from options through `GatewaySession.NormalizeOutboundCommand` into the expander.
|
||||
- Tests: `SparseArrayExpanderTests` — assert `total_length` above the cap throws `InvalidArgument` before allocation; assert a value below the cap still expands. Add a `GatewayOptionsValidatorTests` case for the new bound.
|
||||
- Docs: confirm `gateway.md:532` wording matches the option name; add the key to `docs/GatewayConfiguration.md`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SparseArrayExpander"` and `--filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-04 — Full event channel stalls the worker read loop behind command replies `Medium` · `P1`
|
||||
|
||||
**Finding.** `ReadLoopAsync` awaits `DispatchEnvelopeAsync` inline (`Workers/WorkerClient.cs:358-362`), and the `WorkerEvent` branch awaits `EnqueueWorkerEventAsync`, which blocks in `WriteAsync` for up to `EventChannelFullModeTimeout` (default 5 s) when `_events` is full (`Workers/WorkerClient.cs:511-553`, `Workers/WorkerClientOptions.cs:13`).
|
||||
|
||||
**Impact.** With no event consumer attached or a stalled distributor, each incoming event costs up to 5 s of read-loop stall before the fault fires. A `WorkerCommandReply` queued behind an event frame is not dispatched, so an in-flight `InvokeAsync` can hit `CommandTimeout` (`Workers/WorkerClient.cs:187-213`) even though the worker replied in time; heartbeats behind the stall feed the very watchdog interplay the code tries to compensate for (`Workers/WorkerClient.cs:394-424`).
|
||||
|
||||
**Design.** Decouple event enqueue from the read loop so replies and heartbeats are never blocked by an event backlog. The class already has a dedicated `WriteLoopAsync` for outbound envelopes (`Workers/WorkerClient.cs:332-338`); mirror that for events. In `DispatchEnvelopeAsync`, the `WorkerEvent` branch should hand the event to `_events` via a **non-blocking** `TryWrite`; when the channel is full, start (or continue) a short bounded backpressure window on a *separate* task rather than blocking the read loop. Command-reply, heartbeat, fault, and shutdown-ack branches dispatch synchronously and immediately, so they can never queue behind events.
|
||||
|
||||
Concretely: keep the `EventChannelFullModeTimeout` semantics but move the waiting `WriteAsync` off the read loop — either (a) route events through a small internal event channel drained by a dedicated writer task that performs the timed `WriteAsync` and faults on timeout, or (b) on a `TryWrite` miss, record a monotonic "backlog since" timestamp and fault only once the backlog persists past the timeout, all without awaiting inside `ReadLoopAsync`. Option (a) is cleaner and symmetrical with the existing write loop; recommend it.
|
||||
|
||||
This is the gateway half of the coordinated backpressure/size pass in `00-overall.md` P1.8 (co-designed with the worker and contracts backlog findings); the fault model (`ProtocolViolation` on sustained overflow) is unchanged.
|
||||
|
||||
**Implementation.**
|
||||
- `Workers/WorkerClient.cs`: add an internal unbounded-or-small event staging channel + `EventWriteLoopAsync` (parallel to `WriteLoopAsync`); `DispatchEnvelopeAsync`'s `WorkerEvent` branch does a non-awaiting hand-off; the writer task owns the timed `WriteAsync`/fault. Register the new task in `WaitForBackgroundTasksAsync` and complete it in `DisposeAsync`.
|
||||
- Tests: `WorkerClientTests` — with `_events` full and no consumer, assert a `WorkerCommandReply` arriving after an event is still dispatched promptly (command does not time out); assert sustained overflow still faults with the existing `ProtocolViolation` message. Reuse the fake pipe harness.
|
||||
- Docs: note the event-vs-reply decoupling in `docs/WorkerFrameProtocol.md` / `docs/MxAccessWorkerInstanceDesign.md` if they describe the read loop.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-05 — Worker pipe created with no ACL / no CurrentUserOnly `Medium` · `—`
|
||||
|
||||
**Finding.** `SessionWorkerClientFactory.CreatePipe` uses the plain `NamedPipeServerStream` constructor with `PipeOptions.Asynchronous` only (`Sessions/SessionWorkerClientFactory.cs:158-166`). `gateway.md:282-283` requires the pipe be "ACL restricted to the gateway identity and the launched worker identity, no anonymous access".
|
||||
|
||||
**Impact.** Any local process can connect to `mxaccess-gateway-{pid}-{sessionId}` before the real worker; with `maxNumberOfServerInstances: 1` the legitimate worker then never connects, so `OpenSession` fails on startup timeout — a trivially repeatable local denial of service. The startup nonce (`Workers/WorkerClient.cs:632-637`) prevents impersonation but not connection stealing.
|
||||
|
||||
**Design.** Create the pipe with an explicit DACL limited to the service identity via `NamedPipeServerStreamAcl.Create` (from `System.IO.Pipes.AccessControl`). Because workers run as the gateway identity (`docs/DesignDecisions.md`), `PipeOptions.CurrentUserOnly` is the simpler equivalent and is the recommended first choice — it restricts the pipe to the creating user's SID with no manual `PipeSecurity` construction. Keep `PipeOptions.Asynchronous`. This is Windows-only surface; the pipe factory is not exercised on the macOS `NonWindows.slnx`, so guard behind the existing Windows compilation/runtime path.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/SessionWorkerClientFactory.cs:158-166`: `return NamedPipeServerStreamAcl.Create(pipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Byte, PipeOptions.Asynchronous, 0, 0, pipeSecurity);` with a `PipeSecurity` granting only the current user full control and denying everyone else — or, if acceptable, `new NamedPipeServerStream(name, …, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly)`.
|
||||
- Tests: worker-pipe tests run on Windows only; add a `SessionWorkerClientFactory` test asserting a second connector as a different principal is rejected (or, minimally, that the pipe is created with restricted ACL). Document why it is skipped on non-Windows.
|
||||
- Docs: none needed beyond confirming `gateway.md:282-283` now matches code.
|
||||
|
||||
**Verification.** On the Windows host: `dotnet build src/ZB.MOM.WW.MxGateway.Server`; run the factory/pipe tests. On macOS the change is not compiled into `NonWindows.slnx`; document the skip.
|
||||
|
||||
---
|
||||
|
||||
## GWC-06 — Stopwatch allocated per streamed event `Medium` · `P2`
|
||||
|
||||
**Finding.** `MxAccessGatewayService.StreamEvents` runs `Stopwatch stopwatch = Stopwatch.StartNew()` inside the per-event loop (`Grpc/MxAccessGatewayService.cs:155-157`), one heap allocation per event per subscriber on the highest-volume gateway path.
|
||||
|
||||
**Impact.** Avoidable GC pressure under alarm bursts — the exact load the system exists to handle.
|
||||
|
||||
**Design.** Replace the `Stopwatch` object with the allocation-free timestamp API: `long ts = Stopwatch.GetTimestamp();` before the write and `metrics.RecordEventStreamSend(family, Stopwatch.GetElapsedTime(ts))` after. Behavior and the recorded metric are identical. Co-designed with GWC-07 and GWC-08 as the P2 hot-path pass.
|
||||
|
||||
**Implementation.**
|
||||
- `Grpc/MxAccessGatewayService.cs:155-157`: swap to `Stopwatch.GetTimestamp()` / `Stopwatch.GetElapsedTime(ts)`.
|
||||
- Tests: existing `MxAccessGatewayService`/`EventStreamService` streaming tests cover behavior; no new assertion needed beyond a green run.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~StreamEvents"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-07 — Every mapped event is deep-cloned `Medium` · `P2`
|
||||
|
||||
**Finding.** `MxAccessGrpcMapper.MapEvent` returns `workerEvent.Event?.Clone()` (`Grpc/MxAccessGrpcMapper.cs:64-73`), invoked once per event by the distributor pump via `MapWorkerEventsAsync` (`Sessions/GatewaySession.cs:701-710`).
|
||||
|
||||
**Impact.** A full protobuf deep copy (including value arrays) per event, even though the enclosing `WorkerEvent` is discarded immediately after mapping and nothing else retains the inner message.
|
||||
|
||||
**Design.** Transfer ownership of `workerEvent.Event` instead of cloning — return the inner message directly. The `WorkerEvent` wrapper is created by the frame reader per received frame and is not retained after `MapEvent`, so no other consumer aliases the inner `MxEvent`. Keep a comment stating the ownership-transfer invariant so a future second consumer of the `WorkerEvent` restores the clone. This is safe today precisely because GWC-01 makes the distributor pump the single consumer of the worker channel.
|
||||
|
||||
Depends on GWC-01 (single-consumer guarantee) to be strictly safe; co-designed with GWC-06/GWC-08.
|
||||
|
||||
**Implementation.**
|
||||
- `Grpc/MxAccessGrpcMapper.cs:64-73`: `return workerEvent.Event ?? new MxEvent { Family = MxEventFamily.Unspecified, RawStatus = "…" };` (drop `.Clone()`), with an ownership-transfer comment.
|
||||
- Tests: `MxAccessGrpcMapperTests` — assert the returned reference is the same instance as `workerEvent.Event` (ownership transferred) and the unspecified-family fallback path is unchanged.
|
||||
- Docs: none.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~MxAccessGrpcMapper"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-08 — Pipe framing allocates per frame and writes twice `Medium` · `P2`
|
||||
|
||||
**Finding.** The reader allocates `new byte[sizeof(uint)]` and `new byte[payloadLength]` per frame (`Workers/WorkerFrameReader.cs:32,50`); the writer allocates `new byte[sizeof(uint)]` plus `envelope.ToByteArray()` and performs two `WriteAsync` calls (`Workers/WorkerFrameWriter.cs:56-60`).
|
||||
|
||||
**Impact.** Per-frame GC pressure proportional to event rate and two pipe syscalls per outbound frame. The worker side already uses `ArrayPool`; the gateway side does not.
|
||||
|
||||
**Design.** On the write path, serialize length + payload into a single pooled buffer and issue one `WriteAsync`: rent `4 + payloadLength` from `ArrayPool<byte>.Shared`, write the little-endian length prefix into the first four bytes, serialize the envelope into the remainder via `envelope.WriteTo(new CodedOutputStream(...))` or `envelope.WriteTo(span)`, write once, then return the buffer. On the read path, rent a buffer sized to the frame instead of `new byte[payloadLength]`, parse from the rented span, and return it. Preserve the existing length-validation-before-allocation order (`Workers/WorkerFrameReader.cs:43-48`) — the `MaxMessageBytes` check must still gate the rent size. Co-designed with GWC-06/GWC-07.
|
||||
|
||||
**Implementation.**
|
||||
- `Workers/WorkerFrameWriter.cs:56-60`: rent `4 + payloadLength`, fill prefix + serialized payload, single `WriteAsync(buffer, 0, total)`, `try/finally` return to pool.
|
||||
- `Workers/WorkerFrameReader.cs:32,50`: rent buffers (or reuse a per-reader scratch buffer for the 4-byte prefix), parse from the rented region, return in `finally`. Keep exact-read semantics (`ReadExactlyOrThrowAsync`).
|
||||
- Tests: `WorkerFrameReaderTests`/`WorkerFrameWriterTests` round-trip must stay green; add a large-payload round-trip and an oversize-rejection test to confirm the validation order is preserved. These also protect the `FakeWorkerHarness` which reuses this frame code.
|
||||
- Docs: `docs/WorkerFrameProtocol.md` if it describes the two-write layout.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerFrame"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-09 — Startup probe is a no-op; its retry pipeline can never retry `Medium` · `—`
|
||||
|
||||
**Finding.** `WorkerProcessStartedProbe.WaitUntilReadyAsync` does one instantaneous `HasExited` check and throws `WorkerProcessLaunchException` on failure (`Workers/WorkerProcessStartedProbe.cs:6-19`); `ShouldRetryStartupProbe` explicitly excludes `WorkerProcessLaunchException` and `OperationCanceledException` from retry (`Workers/WorkerProcessLauncher.cs:291-299`). The Polly pipeline with exponential backoff and jitter (`Workers/WorkerProcessLauncher.cs:264-289`) therefore executes exactly one attempt in every case, so `StartupProbeRetryAttempts` and `StartupProbeRetryDelayMilliseconds` (`Configuration/WorkerOptions.cs:19-22`, validated at `Configuration/GatewayOptionsValidator.cs:119-126`) have no observable effect.
|
||||
|
||||
**Impact.** Two configuration options and their validation are dead; operators may tune them expecting an effect. Actual worker readiness is instead established later by the pipe-connect + nonce handshake, so nothing is *broken*, but the surface is misleading.
|
||||
|
||||
**Design.** Two viable options; recommend **(B) remove the dead surface** unless a real readiness signal is planned, because the pipe handshake already gates readiness and a genuine probe would duplicate it.
|
||||
|
||||
- (A) Implement a real readiness probe with retryable transient failures: have the probe poll a real signal (e.g. pipe-created / worker "ready" marker) and throw a *retryable* exception type (not `WorkerProcessLaunchException`) on transient not-ready, so the Polly pipeline actually retries. Then the two options become live.
|
||||
- (B) Delete the retry pipeline and the two options: replace `CreateStartupProbePipeline` usage with a single direct `WaitUntilReadyAsync` call, remove `StartupProbeRetryAttempts` / `StartupProbeRetryDelayMilliseconds` from `WorkerOptions` and their validator branch, and update `docs/WorkerProcessLauncher.md` in the same commit.
|
||||
|
||||
Open question the review cannot settle: whether a future probe (COM-init readiness) is planned. If yes, do (A); if no, do (B). Recommended: (B) now, reintroduce a real probe if/when COM-readiness signaling lands.
|
||||
|
||||
**Implementation (B).**
|
||||
- `Workers/WorkerProcessLauncher.cs`: drop `CreateStartupProbePipeline` and call the probe directly; remove `ShouldRetryStartupProbe`.
|
||||
- `Configuration/WorkerOptions.cs:19-22` and `Configuration/GatewayOptionsValidator.cs:119-126`: remove the two options and their validation.
|
||||
- Tests: `WorkerProcessLauncherTests` — remove/adjust retry-count assertions; assert an exited-before-ready worker fails fast once.
|
||||
- Docs: `docs/WorkerProcessLauncher.md`, `docs/GatewayConfiguration.md` (drop the two keys) in the same commit.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerProcessLauncher"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-10 — Envelope `sequence` monotonicity specified but not enforced `Medium` · `—`
|
||||
|
||||
**Finding.** `gateway.md:314` states "`sequence` is monotonic per sender", but `WorkerEnvelopeValidator.Validate` checks only protocol version, session id, and body presence (`Workers/WorkerEnvelopeValidator.cs:15-39`). Nothing on the gateway side detects out-of-order, duplicated, or replayed frames from a misbehaving worker.
|
||||
|
||||
**Impact.** A worker bug that reorders or repeats frames is invisible; the event-ordering guarantee rests solely on the worker's writer discipline. Latent — depends on a worker defect — hence Medium.
|
||||
|
||||
**Design.** Track the last-received `sequence` per `WorkerClient` connection and fault the client with `ProtocolViolation` on a regression (non-increasing sequence), matching the existing fault model (`Workers/WorkerClient.cs:376-380`). Enforce in the read loop after `Validate`, not inside the static validator (which has no per-connection state). Allow the very first frame to establish the baseline. Whether strict `+1` monotonicity or merely strictly-increasing is required depends on whether the worker sends a continuous sequence across all envelope types — recommend strictly-increasing (regression → fault) as the safe superset, since gaps are legal if any frame type is unsequenced.
|
||||
|
||||
**Implementation.**
|
||||
- `Workers/WorkerClient.cs`: add `ulong _lastReceivedSequence` (per connection); in `ReadLoopAsync`/`DispatchEnvelopeAsync` after validation, if `envelope.Sequence <= _lastReceivedSequence` (and not the first frame) call `SetFaulted(ProtocolViolation, …)`; else advance. Keep the check off the hot allocation path.
|
||||
- Tests: `WorkerClientTests` — feed a decreasing/duplicate sequence via the fake pipe and assert the client faults with `ProtocolViolation`; assert normal increasing sequences pass.
|
||||
- Docs: confirm `gateway.md:314` matches; note enforcement point.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-11 — `_workerClient` written under lock but read lock-free `Low` · `—`
|
||||
|
||||
**Finding.** `_workerClient` is written under `_syncRoot` in `AttachWorkerClient` (`Sessions/GatewaySession.cs:368-376`) but read lock-free in `CloseAsync` (`:1470`), `DisposeAsync` (`:1762`), `KillWorker` (`:1617`), and `WorkerProcessId` (`:268`). Reference reads are atomic and the manager's call ordering makes a torn interleaving unlikely, but the discipline documented for `_state` is not applied to the field.
|
||||
|
||||
**Impact.** Latent consistency risk only; no observed failure.
|
||||
|
||||
**Design.** Apply the class's own locking contract: either read `_workerClient` under `_syncRoot` on those paths or mark the field `volatile`. Recommend `volatile` — the reads are single-reference snapshots and lock acquisition on the close/dispose paths is otherwise unnecessary; `volatile` documents the shared-field intent with minimal churn. (`KillWorker` itself is dead — see GWC-19 — so that read disappears if GWC-19 is applied.)
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/GatewaySession.cs`: mark `_workerClient` `volatile` (or wrap the four reads in `lock (_syncRoot)`).
|
||||
- Tests: covered by existing session lifecycle tests; no new assertion.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-12 — `WorkerClient.DisposeAsync` not safe against double-dispose `Low` · `—`
|
||||
|
||||
**Finding.** `DisposeAsync` uses a plain `if (_disposed) return; _disposed = true;` with no interlock (`Workers/WorkerClient.cs:296-303`). Two concurrent disposals would both run kill/complete/dispose, and the second `_stopCts.Cancel()` after `_stopCts.Dispose()` (`:305`, `:328`) would throw `ObjectDisposedException`.
|
||||
|
||||
**Impact.** Latent only — `SessionManager.RemoveSessionAsync`'s registry `TryRemove` gate makes the session's `DisposeAsync` single-shot in practice.
|
||||
|
||||
**Design.** Use `Interlocked.Exchange` to make disposal single-shot, matching the pattern the lease classes already use. Minimal, no behavior change on the single-dispose path.
|
||||
|
||||
**Implementation.**
|
||||
- `Workers/WorkerClient.cs:296-303`: `if (Interlocked.Exchange(ref _disposedFlag, 1) == 1) return;` (int field), keeping the remainder unchanged.
|
||||
- Tests: `WorkerClientTests` — call `DisposeAsync` concurrently twice and assert no throw.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-13 — Worker-ready wait is a 25 ms poll loop `Low` · `—`
|
||||
|
||||
**Finding.** `GetReadyWorkerClientAsync` polls with `Task.Delay(25 ms)` up to `WorkerReadyWaitTimeoutMs` (`Sessions/GatewaySession.cs:1841-1909`). It is default-off (`Configuration/SessionOptions.cs:69`), bounded, and testable via `TimeProvider`, so impact is minor; it is still a poll on the command hot path when enabled.
|
||||
|
||||
**Impact.** Minor latency/CPU when the option is enabled; none by default.
|
||||
|
||||
**Design.** If the option sees real use, replace the poll with a `TaskCompletionSource` pulsed on worker state transitions (the worker client already raises state changes internally). Register a waiter under `_syncRoot` in `GetReadyWorkerClientAsync` when the state is transient, and complete it when the worker becomes Ready or terminal, with the existing timeout as a `CancelAfter`. Keep the existing zero-timeout fail-fast path byte-for-byte. Because this is default-off and low-value, this is a defer-until-used item; the recommended action is to leave the poll but document it as a known trade-off unless the option is turned on in a real deployment.
|
||||
|
||||
**Implementation (if pursued).**
|
||||
- `Sessions/GatewaySession.cs`: add a state-change signal (`TaskCompletionSource`) set on worker Ready/terminal transitions; await it with the timeout instead of the poll loop.
|
||||
- Tests: `GatewaySessionTests` using `TimeProvider` — assert readiness is observed without the 25 ms granularity delay.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-14 — Replay ring is a `LinkedList` with a node alloc per event `Low` · `P2`
|
||||
|
||||
**Finding.** The distributor replay ring is a `LinkedList<ReplayEntry>` (`Sessions/SessionEventDistributor.cs:108`), appended per event under `_replayLock` (`:766-793`). Capacity is fixed (`ReplayBufferCapacity`, default 1024).
|
||||
|
||||
**Impact.** A node allocation and poor cache locality per event on the fan-out hot path — exactly the shape a circular array serves allocation-free.
|
||||
|
||||
**Design.** Replace the `LinkedList` with a fixed-size ring array (`ReplayEntry[]` + head/count) sized to `ReplayBufferCapacity`. Keep the `_replayLock` protocol and the ascending-`WorkerSequence` ordering invariant (`:100-108`) unchanged; append overwrites the oldest slot, and `TryGetReplayFrom` binary-searches or linear-scans from the front. Age-based eviction (`_ageEvictionEnabled`) maps to advancing the tail index. Part of the P2 hot-path pass; behavior (replay contents, gap detection) must be identical.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/SessionEventDistributor.cs`: swap the backing store to a ring array with head/tail/count; update append (`:766-793`) and `TryGetReplayFrom` accordingly; keep capacity-0-disables-retention and retention-eviction semantics.
|
||||
- Tests: `SessionEventDistributorTests` (replay) must stay green — replay-from-sequence, gap detection, capacity overflow eviction, and age eviction; add a wraparound test crossing the ring boundary.
|
||||
- Docs: none (internal).
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionEventDistributor"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-15 — Per-event gauge reads `ChannelReader.Count` every event `Low` · `P2`
|
||||
|
||||
**Finding.** `EventStreamService` reconciles the queue-depth gauge by reading `subscriber.Reader.Count` on every event (`Grpc/EventStreamService.cs:173-179`). Bounded-channel `Count` acquires the channel's internal lock; combined with the metric adjustment this adds measurable per-event overhead at high rates.
|
||||
|
||||
**Impact.** Minor per-event cost on the streaming hot path.
|
||||
|
||||
**Design.** Sample the backlog periodically instead of per event — e.g. every N events (a cheap counter modulo) or on a time interval — and reconcile the gauge then. The gauge is a diagnostic backlog indicator, not an exact-per-event quantity, so coarser sampling is acceptable. Keep the terminal reconcile in the `finally` (`:189-193`) so the gauge returns to zero on disconnect. Part of the P2 hot-path pass.
|
||||
|
||||
**Implementation.**
|
||||
- `Grpc/EventStreamService.cs:173-179`: gate the `Count`/`AdjustGrpcEventStreamQueueDepth` reconciliation behind `if ((++counter % SampleEvery) == 0)`; retain the `finally` zeroing.
|
||||
- Tests: existing streaming tests confirm the gauge zeroes on disconnect; add a case asserting the gauge is still reconciled at least at stream end.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~EventStream"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-16 — `Invoke` resolves the session twice per command `Low` · `—`
|
||||
|
||||
**Finding.** `Invoke` calls `ResolveSession(request.SessionId)` (`Grpc/MxAccessGatewayService.cs:104`) then `sessionManager.InvokeAsync(request.SessionId, …)` (`:122-124`) → `GetRequiredSession` (`Sessions/SessionManager.cs:161`) — a redundant `ConcurrentDictionary` lookup on every command.
|
||||
|
||||
**Impact.** Small but per-command overhead.
|
||||
|
||||
**Design.** Add an `ISessionManager.InvokeAsync(GatewaySession session, WorkerCommand, ct)` overload (or pass the already-resolved session through) so the service reuses the session it resolved for constraint application. Keep the existing `InvokeAsync(sessionId, …)` overload for other callers. No behavior change.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/SessionManager.cs`: add the session-typed overload; the existing one resolves then delegates to it.
|
||||
- `Grpc/MxAccessGatewayService.cs:122-124`: pass `session`.
|
||||
- Tests: `SessionManagerTests` — assert both overloads behave identically; existing `MxAccessGatewayService` invoke tests stay green.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SessionManager"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-17 — Sessions layer throws `Grpc.Core.RpcException` `Low` · `—`
|
||||
|
||||
**Finding.** `SparseArrayExpander.Invalid` creates `new RpcException(new Status(StatusCode.InvalidArgument, …))` (`Sessions/SparseArrayExpander.cs:283-284`), invoked from `GatewaySession.NormalizeOutboundCommand` (`Sessions/GatewaySession.cs:984-1063`), so `GatewaySession.InvokeAsync` — a transport-agnostic session API also used by the alarm monitor — throws a gRPC type. `gateway.md:1063-1065` places translation at the gRPC layer.
|
||||
|
||||
**Impact.** Layering leak: a non-gRPC caller (alarm monitor, tests) receives a gRPC exception type; convention deviation from the style guide's layer boundaries.
|
||||
|
||||
**Design.** Throw a domain exception from the expander/session — reuse `SessionManagerException` with a `SessionManagerErrorCode` that maps to `InvalidArgument` (the codebase already routes these through `MxAccessGatewayService.MapException`). Add the mapping so the gRPC surface still returns `InvalidArgument`. Coordinate with GWC-03, which also adds an `InvalidArgument` rejection through the same path — both should use the domain exception. Verify no caller currently catches `RpcException` specifically from this path (the `Invoke` catch already rethrows non-`RpcException` via `MapException` at `:135`, so domain exceptions are handled).
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/SparseArrayExpander.cs:283-284`: return a domain exception (e.g. `SessionManagerException(SessionManagerErrorCode.InvalidArgument, message)`); rename `Invalid` accordingly. Add the error code if absent.
|
||||
- `Grpc/MxAccessGatewayService.cs` `MapException`: ensure the new code maps to `StatusCode.InvalidArgument`.
|
||||
- Tests: `SparseArrayExpanderTests` assert the domain exception type; `MxAccessGatewayService`/mapping tests assert it surfaces as `InvalidArgument` over gRPC.
|
||||
- Docs: confirm `gateway.md:1063-1065` layering statement now holds.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~SparseArrayExpander"` and `--filter "FullyQualifiedName~MxAccessGatewayService"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-18 — `GatewaySession` implements `DisposeAsync` without `IAsyncDisposable` `Low` · `—`
|
||||
|
||||
**Finding.** `public sealed class GatewaySession` declares no interfaces (`Sessions/GatewaySession.cs:13`) but exposes a public `async ValueTask DisposeAsync()` (`:1672`). `await using` does not compile against the type; disposal is discoverable only by convention.
|
||||
|
||||
**Impact.** Convention deviation; latent maintainability risk.
|
||||
|
||||
**Design.** Declare `: IAsyncDisposable` on the class. The method already matches the interface signature, so this is a declaration-only change with no behavior impact.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/GatewaySession.cs:13`: `public sealed class GatewaySession : IAsyncDisposable`.
|
||||
- Tests: existing lifecycle tests compile/run unchanged; optionally add an `await using` usage in a test.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-19 — Dead `GatewaySession.KillWorker(string)` + stale doc `Low` · `—`
|
||||
|
||||
**Finding.** `GatewaySession.KillWorker(string)` (`Sessions/GatewaySession.cs:1615-1619`) has no callers (grep across `src/` and `clients/` finds none; the gated `KillWorkerWithCloseGateAsync` is what `SessionManager.KillWorkerAsync` uses). `docs/Sessions.md:57` still states `KillWorkerAsync` "calls `GatewaySession.KillWorker` directly".
|
||||
|
||||
**Impact.** Dead public method and a doc that describes a non-existent call path — violates the repo's docs-with-source rule.
|
||||
|
||||
**Design.** Delete the dead method and correct the doc in the same commit, per the repo convention. Confirm no reflection/DI reference exists (none found — it is a plain public method). This also removes one of the lock-free `_workerClient` reads flagged in GWC-11.
|
||||
|
||||
**Implementation.**
|
||||
- `Sessions/GatewaySession.cs:1615-1619`: delete `KillWorker`.
|
||||
- `docs/Sessions.md:57`: reword to describe `KillWorkerWithCloseGateAsync` (the actual forceful path).
|
||||
- Tests: none reference it; a full build confirms no dangling call site.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewaySession"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-20 — Heartbeat config semantics conflated (send vs check interval) `Low` · `—`
|
||||
|
||||
**Finding.** `WorkerOptions.HeartbeatIntervalSeconds` is documented as "the interval in seconds for worker heartbeats" (`Configuration/WorkerOptions.cs:30-31`, default 5) but is bound to the gateway-side `HeartbeatCheckInterval` (`Sessions/SessionWorkerClientFactory.cs:86`), whose own default is 1 s (`Workers/WorkerClientOptions.cs:10`). Production checks every 5 s while unit-constructed clients check every 1 s, and the option name does not describe what it controls. Separately, `HeartbeatLoopAsync` uses raw `Task.Delay` (`Workers/WorkerClient.cs:400`) rather than the injected `TimeProvider`, unlike the rest of the class.
|
||||
|
||||
**Impact.** Misleading configuration surface; the option name implies the worker's *send* cadence but controls the gateway's *check* cadence. The raw `Task.Delay` makes the heartbeat loop the one loop not virtualizable under `TimeProvider` for tests.
|
||||
|
||||
**Design.** Rename the option to describe what it controls — add `HeartbeatCheckIntervalSeconds` (and correct the XML doc), keeping the old key as a deprecated alias for one release if config compatibility matters, or rename outright since this is pre-1.0. Fix the doc comment to state it is the gateway-side check interval. Separately, route `HeartbeatLoopAsync`'s delay through `_timeProvider` (e.g. a `PeriodicTimer` from `TimeProvider.CreateTimer` or `Task.Delay(interval, _timeProvider, token)`) for consistency and testability.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/WorkerOptions.cs:30-31`: rename to `HeartbeatCheckIntervalSeconds`; fix the XML doc.
|
||||
- `Sessions/SessionWorkerClientFactory.cs:86`: bind the renamed option.
|
||||
- `Workers/WorkerClient.cs:400`: use the injected `TimeProvider` for the loop delay.
|
||||
- Tests: `WorkerClientTests` heartbeat-watchdog tests should drive the loop via `FakeTimeProvider` once the delay is virtualized; assert the check-interval semantics.
|
||||
- Docs: `docs/GatewayConfiguration.md` (key rename) and any heartbeat description in `docs/MxAccessWorkerInstanceDesign.md`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient"` and `--filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-21 — `EventChannelFullModeTimeout` / `HeartbeatStuckCeiling` not configurable `Low` · `—`
|
||||
|
||||
**Finding.** `SessionWorkerClientFactory` populates only four of the six `WorkerClientOptions` fields (`Sessions/SessionWorkerClientFactory.cs:83-89`); `EventChannelFullModeTimeout` and `HeartbeatStuckCeiling` always use the hardcoded defaults (`Workers/WorkerClientOptions.cs:13,23`) and have no `WorkerOptions` counterparts.
|
||||
|
||||
**Impact.** Two operationally significant tunables (event-backlog fault window and the stuck-command watchdog ceiling) cannot be adjusted without a rebuild.
|
||||
|
||||
**Design.** Expose both under `MxGateway:Worker:*` (`EventChannelFullModeTimeoutMilliseconds`, `HeartbeatStuckCeilingSeconds`) with defaults matching the current constants (5000 ms, 75 s), bind them in the factory, and validate ranges in `GatewayOptionsValidator`. Coordinate with GWC-04: once the event enqueue is decoupled from the read loop, `EventChannelFullModeTimeout` becomes the fault window for the dedicated event writer task, and making it configurable is the natural companion. Alternatively, if the values should stay fixed, document them as non-configurable in `docs/GatewayConfiguration.md` — but given GWC-04, exposing them is preferred.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/WorkerOptions.cs`: add the two options with defaults.
|
||||
- `Configuration/GatewayOptionsValidator.cs`: validate `> 0` (ceiling `>=` grace, matching the watchdog logic).
|
||||
- `Sessions/SessionWorkerClientFactory.cs:83-89`: bind both.
|
||||
- Tests: `GatewayOptionsValidatorTests` for the new bounds; `WorkerClientTests` confirming the configured timeout drives the overflow fault.
|
||||
- Docs: `docs/GatewayConfiguration.md`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~GatewayOptionsValidator"` and `--filter "FullyQualifiedName~WorkerClient"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-22 — `StreamDisconnected` always labeled `"Detached"` `Low` · `—`
|
||||
|
||||
**Finding.** The single `metrics.StreamDisconnected(...)` call site hardcodes `"Detached"` in the `finally` block (`Grpc/EventStreamService.cs:195`); the fault/overflow paths above it (`:148-158`) do not differentiate the label.
|
||||
|
||||
**Impact.** The disconnect-reason dimension the metric implies is uninformative for diagnosing overflow-vs-fault-vs-client-cancel.
|
||||
|
||||
**Design.** Record the actual terminal cause. Track a terminal-reason variable in the stream method, set it to `worker-fault` in the `WorkerClientException` catch (`:148-158`), `canceled` on `OperationCanceledException` tied to the caller's token, `overflow` if the subscriber channel completed via overflow, and default `detached` for a clean end; pass it to `StreamDisconnected` in the `finally`. No new metric — only the existing label dimension becomes meaningful.
|
||||
|
||||
**Implementation.**
|
||||
- `Grpc/EventStreamService.cs`: introduce a `string terminalReason = "detached";` updated on each terminal branch; use it at `:195`.
|
||||
- Tests: `EventStreamService` tests asserting the label reflects fault vs cancel vs clean end.
|
||||
- Docs: `docs/Metrics.md` (or wherever the metric's label values are enumerated) if the reason set is documented — note this is in the metrics domain; coordinate with that owner.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~EventStream"`.
|
||||
|
||||
---
|
||||
|
||||
## GWC-23 — `MaxEventSubscribersPerSession` dead in default single-subscriber mode `Info` · `—`
|
||||
|
||||
**Finding.** `MaxEventSubscribersPerSession` is a knowingly dead knob under the default single-subscriber configuration; this is acknowledged and justified in a comment (`Configuration/GatewayOptionsValidator.cs:189-196`).
|
||||
|
||||
**Impact / action.** None. Recorded so it is not re-reported. No change proposed — the option becomes live when `AllowMultipleEventSubscribers` is enabled, and the validator comment already explains the intent. Leave as-is.
|
||||
|
||||
**Verification.** N/A (no change).
|
||||
@@ -0,0 +1,335 @@
|
||||
# Worker Process — Remediation Design & Implementation
|
||||
|
||||
Source review: [20-worker.md](../20-worker.md) · Generated: 2026-07-09
|
||||
|
||||
The worker's STA pump and COM-teardown discipline are correct; the risk is concentrated at the IPC edges — a long `ReadBulk` that self-faults, silent thread/command deaths, and a wire-sequence race — plus a small event hot-path allocation cluster. Every fix below stays on the STA for COM work and respects net48 constraints (no init-only properties, no positional records). The worker builds and tests only on the Windows x86 host, so verification commands assume that host.
|
||||
|
||||
## Finding index
|
||||
|
||||
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||
|----|-----|-------|---------|--------|-----------|-------|
|
||||
| WRK-01 | High | Long `ReadBulk` self-faults as `StaHung`; all replies then dropped | P0 | M | — | MxAccess/MxAccessSession.cs, Sta/StaRuntime.cs, Ipc/WorkerPipeSession.cs |
|
||||
| WRK-02 | Medium | STA thread death after startup is silent; future work hangs forever | — | M | — | Sta/StaRuntime.cs |
|
||||
| WRK-03 | Medium | Commands after shutdown starts are dropped with no reply | — | S | WRK-02 | Ipc/WorkerPipeSession.cs |
|
||||
| WRK-04 | Medium | Envelope `sequence` can appear out of order on the wire | — | M | — | Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs |
|
||||
| WRK-05 | Medium | One transient alarm-poll failure kills the whole session | — | M | — | MxAccess/MxAccessStaSession.cs, MxAccess/AlarmCommandHandler.cs |
|
||||
| WRK-06 | Medium | `MXSTATUS_PROXY` conversion reflects per field, per event | P2 | S | — | Conversion/MxStatusProxyConverter.cs |
|
||||
| WRK-07 | Medium | Documented outbound write priority not implemented | P1 | M | WRK-04 | Ipc/WorkerFrameWriter.cs, Ipc/WorkerPipeSession.cs |
|
||||
| WRK-08 | Low | Residual event queue discarded at graceful shutdown, undocumented | — | S | — | Ipc/WorkerPipeSession.cs, docs/MxAccessWorkerInstanceDesign.md |
|
||||
| WRK-09 | Low | No `AppDomain.UnhandledException` hook | — | S | — | WorkerApplication.cs, Program.cs |
|
||||
| WRK-10 | Low | Top-level catch logs exception type but never message | — | S | — | WorkerApplication.cs, Conversion/HResultConverter.cs |
|
||||
| WRK-11 | Low | Every accepted event is defensively cloned on enqueue | P2 | S | — | MxAccess/MxAccessEventQueue.cs |
|
||||
| WRK-12 | Low | No event batching per envelope; one flush per event; 25 ms poll | P2 | M | WRK-04 | Ipc/WorkerPipeSession.cs, Ipc/WorkerFrameWriter.cs |
|
||||
| WRK-13 | Low | Frame writer allocates a fresh buffer per frame; reader pools | P2 | S | — | Ipc/WorkerFrameWriter.cs |
|
||||
| WRK-14 | Low | Private-field naming split `_camelCase` vs `camelCase` | — | M | — | Ipc/*, Bootstrap/*, Sta/*, MxAccess/* |
|
||||
| WRK-15 | Low | Doc drift: STA thread name and heartbeat-counter note | P2 | S | — | docs/WorkerSta.md, docs/MxAccessWorkerInstanceDesign.md |
|
||||
| WRK-16 | Low | Boilerplate duplication in IPC envelope/ctor overloads | — | S | — | Ipc/WorkerPipeSession.cs, Ipc/WorkerPipeClient.cs |
|
||||
| WRK-17 | Low | Gateway death exits with wrong exit code (6 not 5) | — | S | — | WorkerApplication.cs |
|
||||
| WRK-18 | Low | Event-queue overflow exits as generic `UnexpectedFailure` | — | S | — | Ipc/WorkerPipeSession.cs, WorkerApplication.cs, docs/MxAccessWorkerInstanceDesign.md |
|
||||
| WRK-19 | Low | Command start/end logging with correlation id absent | — | S | — | Sta/StaCommandDispatcher.cs |
|
||||
| WRK-20 | Low | Test-coverage gaps for the failure modes above | — | M | WRK-01..04 | Worker.Tests/Ipc, Worker.Tests/Sta |
|
||||
|
||||
---
|
||||
|
||||
## WRK-01 — Long `ReadBulk` self-faults as `StaHung`; all replies then dropped `High` · `P0`
|
||||
|
||||
**Finding.** `ReadOneTag` reads uncached tags one at a time, waiting up to `timeout` per tag via `valueCache.TryWaitForUpdate(..., pumpStep, ...)` (`MxAccess/MxAccessSession.cs:876-889`, `918-931`). The `pumpStep` handed down is `StaRuntime.PumpPendingMessages()`, which pumps Windows messages but never refreshes the activity timestamp (`Sta/StaRuntime.cs:83-90`; `MarkActivity` is private, `Sta/StaRuntime.cs:304-307`). The watchdog suppresses `StaHung` only while `staleFor <= HeartbeatStuckCeiling` (default 75 s); past the ceiling it faults even with a command in flight (`Ipc/WorkerPipeSession.cs:830-860`, `Ipc/WorkerPipeSessionOptions.cs`). Once faulted, every completed reply is dropped at the `_state != Ready` gate (`Ipc/WorkerPipeSession.cs:604-607`).
|
||||
|
||||
**Impact.** A legitimate `ReadBulk` with `timeout_ms=5000` over ~20 unreachable tags holds the STA ~100 s with `LastActivityUtc` frozen. At 75 s the worker emits `StaHung`, sets `_state = Faulted`, and thereafter silently drops every command reply, so the gateway kills a healthy session. This is the one path where a healthy worker declares itself hung. Also called out as P0 item 4 in the roadmap.
|
||||
|
||||
**Design.** The design doc's watchdog contract assumes "no legitimate STA command should run that long without periodically refreshing activity" (`docs/MxAccessWorkerInstanceDesign.md:688-690`) — but no refresh mechanism exists. The minimal, correct fix is to make the pump step used by long-running STA commands *also* refresh activity: have `StaRuntime.PumpPendingMessages()` call `MarkActivity()` after pumping. Because the pump step is invoked on every wait iteration inside `TryWaitForUpdate` while the STA legitimately holds the thread, activity stays fresh for the whole in-flight command; the moment the command stops pumping (a genuine hang) staleness accrues and the watchdog still fires correctly. This preserves the watchdog's purpose (detect a *stuck* STA) while removing the false positive (a *busy* STA).
|
||||
|
||||
Rejected alternatives: (a) clamping total `ReadBulk` duration below `HeartbeatStuckCeiling` — changes MXAccess parity (per-tag `timeout` is the contract) and still breaks for large tag counts; (b) exposing a public activity-refresh hook threaded through every executor — larger surface, easy to forget on new commands. Refreshing inside the shared `pumpStep` covers all current and future long-running STA commands in one place. Parity is unaffected: no MXAccess behavior changes, only the local liveness signal.
|
||||
|
||||
**Implementation.**
|
||||
- `Sta/StaRuntime.cs`: in `PumpPendingMessages()`, capture the pump count, call `MarkActivity()`, return the count. (`MarkActivity` is already private on the same class — no visibility change.)
|
||||
- Confirm every long-hold command routes its wait through this method: `ReadBulk` does (`MxAccessSession.cs:930` passes `pumpStep`, wired from `StaRuntime.PumpPendingMessages` at the executor). No new config.
|
||||
- Optional defense-in-depth: document in `docs/GatewayConfiguration.md` that operators running very long bulk ops may raise `HeartbeatStuckCeiling` (already the doc's stated escape hatch, `docs/MxAccessWorkerInstanceDesign.md:689-690`).
|
||||
- Tests: add `StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity`, and an integration-style test in `Worker.Tests/Ipc/WorkerPipeSessionTests` using the fake runtime to prove a >75 s simulated in-flight command that keeps pumping does not emit `StaHung` and its reply is delivered.
|
||||
- Docs: adjust `docs/MxAccessWorkerInstanceDesign.md:688-690` to state that the pump step refreshes activity, so the "no legitimate command runs that long" caveat is now enforced mechanically rather than assumed.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86` then `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests|FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-02 — STA thread death after startup is silent; future work hangs forever `Medium` · `—`
|
||||
|
||||
**Finding.** `ThreadMain` catches any loop exception into the write-only `startupException` field and sets `startedEvent` (`Sta/StaRuntime.cs:255-259`). After `Start()` has returned, nothing observes that field: the exception is never logged, never turned into a `WorkerFault`, and `shutdownRequested` stays false, so `InvokeAsync` keeps enqueuing work into a queue with no consumer and returns tasks that never complete (`Sta/StaRuntime.cs:165-179`). The only in-loop throw site is a pump-wait failure (`Sta/StaMessagePump.cs:38-42`).
|
||||
|
||||
**Impact.** If `MsgWaitForMultipleObjectsEx` returns `WAIT_FAILED` once, the STA exits; the dispatcher's drain wedges on the first stuck `InvokeAsync`; heartbeats keep flowing with a frozen `LastStaActivityUtc` and a pinned `CurrentCommandCorrelationId`; the worker is only declared hung after the 75 s ceiling, and the true root cause is permanently lost. Matches cross-cutting theme 1 (silent failure modes) in `00-overall.md`.
|
||||
|
||||
**Design.** Post-startup thread death must (1) fail all queued and future `InvokeAsync` calls deterministically and (2) surface the cause. Introduce a `terminalException` field set in the `ThreadMain` `catch`/`finally`, and a `Faulted` state distinct from graceful shutdown. In the `finally`, after `CancelQueuedCommands()`, transition to a terminal-faulted state so `InvokeAsync` returns `Task.FromException` (using the captured exception) for all subsequent calls, and `CancelQueuedCommands` completes already-queued items faulted rather than cancelled when the exit was abnormal. Add an optional `Action<Exception>? onTerminalFault` callback (constructor-injected, defaulted null) that `MxAccessStaSession` wires to record a `WorkerFault` on the event queue — reusing the existing fault→drain→IPC path (`MxAccessStaSession.RecordFault`, `Ipc/WorkerPipeSession.cs:344-353`) so the gateway sees a real fault frame instead of a slow watchdog timeout. Keep the callback optional to avoid disturbing the many unit tests that construct `StaRuntime` directly.
|
||||
|
||||
Rejected: polling `IsRunning` from the dispatcher — racy and still loses the exception. A push callback plus terminal task-completion is deterministic.
|
||||
|
||||
**Implementation.**
|
||||
- `Sta/StaRuntime.cs`: add `private Exception? terminalException;` and a `volatile bool terminated;` (net48 — plain field, no init-only). In `ThreadMain`'s `catch`, store the exception; in `finally`, set `terminated = true` before `stoppedEvent.Set()`. In `InvokeAsync<T>`, after the `shutdownRequested` check, if `terminated` return `Task.FromException<T>(terminalException ?? new StaRuntimeShutdownException())`. Make `CancelQueuedCommands` fault (not cancel) items when `terminalException is not null`.
|
||||
- Add constructor param `Action<Exception>? onTerminalFault = null`; invoke it once from `finally` when `terminalException is not null`.
|
||||
- `MxAccess/MxAccessStaSession.cs`: pass a callback that calls `eventQueue.RecordFault(...)` with category `StaHung` (or a new `StaTerminated` if the contract enum allows — otherwise reuse `StaHung` and set a descriptive `DiagnosticMessage`).
|
||||
- Tests: `Worker.Tests/Sta/StaRuntimeTests` — inject a message pump stub that throws on `WaitForWorkOrMessages`; assert the terminal callback fires, `InvokeAsync` after death returns a faulted task, and queued items fault.
|
||||
- Docs: note in `docs/WorkerSta.md` that abnormal STA exit faults pending/future work and emits a fault frame.
|
||||
|
||||
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` then `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntimeTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-03 — Commands after shutdown starts are dropped with no reply `Medium` · `—`
|
||||
|
||||
**Finding.** `TryStartCommandTask` returns silently when `_acceptingCommands` is false (`Ipc/WorkerPipeSession.cs:690-707`): no `WorkerUnavailable` reply, not even the `LogCommandResultDropped` diagnostic (which fires only for completed-then-dropped replies, `Ipc/WorkerPipeSession.cs:604-607`, `645-655`). The dispatcher layer *does* reply `WorkerUnavailable` when it is the one shutting down (`Sta/StaCommandDispatcher.cs:117-123`), but this earlier gate short-circuits before reaching it.
|
||||
|
||||
**Impact.** A command racing `WorkerShutdown` leaves the gateway's correlation wait to expire on its own timeout with no trace, exactly the silent-drop pattern `docs/MxAccessWorkerInstanceDesign.md:697-699` ("reject new commands") intends to avoid.
|
||||
|
||||
**Design.** Make the gate *loud*: when `_acceptingCommands` is false, write a `WorkerCommandReply` carrying `ProtocolStatusCode.WorkerUnavailable` for that correlation id (mirroring `StaCommandDispatcher.CreateRejectedReply`), then return. This matches the dispatcher-level rejection and gives the gateway an immediate, correlated failure. Guard the write with the same `_state`/`TryWriteFault`-style tolerance so a half-closed pipe during shutdown does not throw. Co-designed with WRK-02 (both close silent-death holes).
|
||||
|
||||
**Implementation.**
|
||||
- `Ipc/WorkerPipeSession.cs`: in the `!_acceptingCommands` branch of `TryStartCommandTask`, build a rejection reply (reuse a small helper `CreateRejectedCommandReply(correlationId, method, WorkerUnavailable, "Worker is shutting down.")`) and enqueue a best-effort `_writer.WriteAsync(CreateEnvelope(reply), ...)` via a fire-and-forget observed task (same pattern as `ObserveCommandTaskAsync`). At minimum, log a `WorkerCommandRefusedDuringShutdown` diagnostic even if the write is skipped.
|
||||
- Tests: `Worker.Tests/Ipc/WorkerPipeSessionTests` — after triggering shutdown, feed a command envelope and assert a `WorkerUnavailable` reply (or the diagnostic) is observed on the fake writer; covers the WRK-20 gap for S3.
|
||||
- Docs: none beyond the existing shutdown sequence, which already promises rejection.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-04 — Envelope `sequence` can appear out of order on the wire `Medium` · `—`
|
||||
|
||||
**Finding.** `NextSequence()` is called while *building* the envelope in `CreateBaseEnvelope()` (`Ipc/WorkerPipeSession.cs:1005-1018`), but the write lock is acquired later, inside `WorkerFrameWriter.WriteAsync` (`Ipc/WorkerFrameWriter.cs:68-77`). Command replies are written from independent per-command tasks (`Ipc/WorkerPipeSession.cs:690-706`) concurrently with the heartbeat and event-drain loops, so task B can take sequence n+1 yet win the write lock before task A's sequence n. (Per-event ordering is safe — `WorkerSequence` is stamped inside the queue lock, `MxAccess/MxAccessEventQueue.cs:135-143` — but the envelope-level guarantee is not.)
|
||||
|
||||
**Impact.** Violates the `gateway.md` envelope rule "`sequence` is monotonic per sender"; any gateway consumer trusting wire-order monotonicity mis-sorts frames.
|
||||
|
||||
**Design.** Assign the envelope sequence *inside* the writer's critical section so the number and the write are atomic. Change `WorkerFrameWriter.WriteAsync` to accept a sequence-stamping callback (`Action<WorkerEnvelope>` or a `Func<ulong>` that the writer invokes under `_writeLock` to set `envelope.Sequence`) rather than reading a pre-stamped value. `CreateBaseEnvelope` stops calling `NextSequence()`; the writer stamps `Sequence` immediately before serialization, under the lock. This keeps a single `_nextSequence` counter and removes the window entirely. Co-designed with WRK-07 (both restructure the write path) — sequence stamping and priority scheduling should land together to avoid two passes over the writer.
|
||||
|
||||
Rejected: making `NextSequence` `Interlocked` (already is) does not help — the reordering is between *assignment* and *write*, not in the increment. The atomic region must span both.
|
||||
|
||||
**Implementation.**
|
||||
- `Ipc/WorkerFrameWriter.cs`: add an overload `WriteAsync(WorkerEnvelope envelope, Func<ulong> sequenceProvider, CancellationToken)` that, after acquiring `_writeLock`, sets `envelope.Sequence = sequenceProvider()` *before* `Validate`/`CalculateSize`/`WriteTo`. (Serialize inside the lock now, since the payload depends on the stamped sequence.)
|
||||
- `Ipc/WorkerPipeSession.cs`: `CreateBaseEnvelope()` no longer sets `Sequence`; every `_writer.WriteAsync(CreateEnvelope(...), ...)` call site passes `NextSequence` as the provider. Keep `NextSequence()` on the session (single owner of `_nextSequence`).
|
||||
- Tests: `Worker.Tests/Ipc/WorkerPipeSessionTests` — a concurrent-writer test that fires N reply writes and M event writes in parallel through a capturing stream and asserts observed `Sequence` values are strictly monotonic in wire order (covers WRK-20 gap for S4). A `WorkerFrameWriterTests` unit test that the provider runs under the lock.
|
||||
- Docs: none — restores the documented invariant.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-05 — One transient alarm-poll failure kills the whole session `Medium` · `—`
|
||||
|
||||
**Finding.** Any exception from the alarm poll loop is recorded as a fault on the shared event queue and permanently stops the loop (`MxAccess/MxAccessStaSession.cs:278-291`); the drain loop turns that fault into full session termination (`Ipc/WorkerPipeSession.cs:344-353`). Failover only absorbs primary failures in composite mode (`MxAccess/FailoverAlarmConsumer.cs`); the default `WorkerPipeSession` builds the alarm handler with `standbyFactory: null` (`Ipc/WorkerPipeSession.cs:54`), so `AlarmCommandHandler.BuildConsumer` returns a bare consumer and a single `GetXmlCurrentAlarms2` COM error propagates unwrapped.
|
||||
|
||||
**Impact.** One transient `E_FAIL` from the AVEVA alarm subsystem terminates a client's healthy `OnDataChange` data stream even though the data path never failed — a whole-session death from an isolated alarm blip.
|
||||
|
||||
**Design.** Two viable scopes; recommend the smaller, parity-safe one:
|
||||
|
||||
1. **Recommended — bounded retry before declaring the alarm subscription dead.** Count *consecutive* poll failures against a threshold (mirror `FailoverSettings.Threshold`, default e.g. 3) with a short back-off; reset the counter on any successful poll. Only after the threshold record the fault. This tolerates transient COM errors without touching the data path and matches the existing failover semantics.
|
||||
2. Scope the fault to the alarm feature (stop alarm delivery, keep data subscriptions and the session alive). Larger change — requires a per-feature fault channel the drain loop does not currently model, and risks masking a genuinely dead provider. Defer.
|
||||
|
||||
Parity note: this does not synthesize or suppress alarm *events* — it only changes how many consecutive *poll infrastructure* failures constitute "the subscription is dead." A persistent failure still faults the session, preserving fail-fast.
|
||||
|
||||
**Implementation.**
|
||||
- `MxAccess/MxAccessStaSession.cs`: in the poll loop, maintain `int consecutiveFailures`; on catch, increment and only call `eventQueue.RecordFault(...)` + `return` when `consecutiveFailures >= threshold`; otherwise log a warning, `await Task.Delay(backoff)`, and continue. Reset on success. Keep the STA-affinity `InvalidOperationException` (from `EnsureOnAlarmConsumerThread`) as an *immediate* fault (it is a programming-error regression, not transient) — distinguish it before the counting branch.
|
||||
- Config: add `AlarmPollFailureThreshold` and `AlarmPollBackoff` to the alarm handler options (thread through `AlarmCommandHandler`); default threshold to match `FailoverSettings.Threshold`.
|
||||
- Tests: `Worker.Tests` alarm units — a consumer stub that throws N-1 times then succeeds keeps the session alive; N consecutive throws faults it; an affinity `InvalidOperationException` faults immediately.
|
||||
- Docs: document the threshold in `docs/GatewayConfiguration.md` and note the transient-tolerance behavior in `docs/MxAccessWorkerInstanceDesign.md` alarm section.
|
||||
|
||||
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` then `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~Alarm"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-06 — `MXSTATUS_PROXY` conversion reflects per field, per event `Medium` · `P2`
|
||||
|
||||
**Finding.** `MxStatusProxyConverter.Convert` calls `ReadInt32Field` four times per status (`Conversion/MxStatusProxyConverter.cs:22-26`); each does `Type.GetField` + `FieldInfo.GetValue` + `Convert.ToInt32` (`:83-103`). This runs on the STA event path for every status of every `OnDataChange`.
|
||||
|
||||
**Impact.** Eight reflection ops plus boxing per event under data-change/alarm bursts — the exact load the gateway exists to handle. The status type is always the interop `MXSTATUS_PROXY` struct, so the field lookups are fully cacheable. Roadmap P2 item 14 (event hot-path pass).
|
||||
|
||||
**Design.** Cache the four `FieldInfo` objects keyed by `Type` in a small static `ConcurrentDictionary<Type, (FieldInfo success, category, detectedBy, detail)>`, resolved once per type and reused. `GetValue` + `Convert.ToInt32` still run per event (unavoidable via reflection over a late-bound COM RCW), but the `GetField` metadata scan — the expensive part — is eliminated. A direct cast to the interop struct type is faster still but couples the converter to the interop assembly and its exact struct shape; the review notes the type is stable, but the cached-`FieldInfo` approach keeps the converter interop-agnostic and testable with the existing plain-CLR test doubles. Recommend cached `FieldInfo`.
|
||||
|
||||
**Implementation.**
|
||||
- `Conversion/MxStatusProxyConverter.cs`: add a static cache; a private `GetFields(Type)` that populates it once (throwing the same `MxStatusConversionException` if a field is missing) and `Convert` reads the four cached `FieldInfo`s. Behavior and exceptions unchanged.
|
||||
- Tests: existing `MxStatusProxyConverter` tests must still pass (same outputs); add one asserting two conversions of the same type reuse cached metadata (e.g. via a type whose `GetField` is instrumented, or simply a throughput/regression test).
|
||||
- Docs: none (internal perf).
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxStatusProxyConverter"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-07 — Documented outbound write priority not implemented `Medium` · `P1`
|
||||
|
||||
**Finding.** `docs/MxAccessWorkerInstanceDesign.md:600-613` specifies write priority faults > command replies > shutdown acks > heartbeats > events; the worker has no prioritized queue — all writers contend on a single FIFO `SemaphoreSlim` in `WorkerFrameWriter` (`Ipc/WorkerFrameWriter.cs:14`, `68-77`), and events are written inline by the drain loop (`Ipc/WorkerPipeSession.cs:362-367`).
|
||||
|
||||
**Impact.** With a deep event backlog draining (up to 128 per batch), a `WorkerFault` or command reply queues behind those event writes; on a slow pipe this delays the gateway's fault reaction. Relates to roadmap P1 item 8 (backpressure/size topology).
|
||||
|
||||
**Design.** Two acceptable outcomes; the review explicitly allows either. Recommend the *documented-decision* path first, with a small scheduler as the follow-up:
|
||||
|
||||
1. **Minimal now:** amend the design doc to state that FIFO write ordering was accepted for v1 (the single `_writeLock` serializes but does not prioritize), and that event batching (WRK-12) plus the sequence fix (WRK-04) bound the worst-case delay. This removes the doc/code contradiction immediately.
|
||||
2. **Full fix (recommended for P1):** introduce a priority write scheduler in `WorkerFrameWriter` — a small set of per-priority queues drained newest-priority-first under `_writeLock`, or a `Channel`-per-priority merged by a single writer task. Faults and replies jump ahead of queued events. Must preserve per-sender sequence monotonicity — co-designed with WRK-04, since sequence is now stamped inside the lock at actual write time, priority reordering before stamping keeps sequence consistent with wire order automatically.
|
||||
|
||||
The full scheduler is the correct end state; if effort is constrained, ship option 1 in the same commit that lands WRK-04/WRK-12 and file option 2.
|
||||
|
||||
**Implementation.**
|
||||
- `Ipc/WorkerFrameWriter.cs`: add an internal priority-ordered pending set drained by the lock holder; or expose `WriteAsync(envelope, priority, sequenceProvider, ct)`. `Ipc/WorkerPipeSession.cs` tags each write site with a priority (fault=0 … event=4).
|
||||
- Tests: `Worker.Tests/Ipc/WorkerFrameWriterTests` — enqueue a fault behind many event writes on a blocked stream, unblock, assert the fault frame emerges first while sequences remain monotonic.
|
||||
- Docs: update `docs/MxAccessWorkerInstanceDesign.md:600-613` to describe the implemented scheduler (or the accepted-FIFO decision if option 1).
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests|FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-08 — Residual event queue discarded at graceful shutdown, undocumented `Low` · `—`
|
||||
|
||||
**Finding.** `ShutdownAsync` writes the ack and returns; `RunMessageLoopAsync`'s `finally` cancels the event-drain loop (`Ipc/WorkerPipeSession.cs:287-292`) with whatever remains in `MxAccessEventQueue` unshipped; late replies on `_state != Ready` are dropped (`Ipc/WorkerPipeSession.cs:604-607`). An `OnWriteComplete` raised during cleanup never reaches the gateway. Acceptable for a closing session but not stated in `docs/MxAccessWorkerInstanceDesign.md`.
|
||||
|
||||
**Design.** Cheapest correct action: **document the discard** as intended v1 behavior — a session tearing down does not guarantee delivery of events queued after `WorkerShutdown`. Optionally (if a downstream wants last-gasp events) drain the queue once after `ShutdownGracefullyAsync` returns and before writing the ack, bounded by the grace period. Recommend documenting now; the final drain is a small enhancement to schedule only if a client needs it.
|
||||
|
||||
**Implementation.** Add a paragraph to `docs/MxAccessWorkerInstanceDesign.md` shutdown section. If implementing the drain: in `ShutdownAsync`, after `ShutdownGracefullyAsync`, call `DrainEvents` once and write each before the ack. Test: `WorkerPipeSessionTests` asserting either the documented discard or the final-drain delivery.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-09 — No `AppDomain.UnhandledException` hook `Low` · `—`
|
||||
|
||||
**Finding.** `Program.cs:1-4` and `WorkerApplication.Run` (`WorkerApplication.cs:46-141`) install no unhandled-exception or unobserved-task handlers (confirmed: no `AppDomain`/`UnhandledException` reference exists anywhere in the worker). An exception on an unobserved thread crashes the process with no `WorkerFault` and no log.
|
||||
|
||||
**Design.** Register `AppDomain.CurrentDomain.UnhandledException` and `TaskScheduler.UnobservedTaskException` at process start, logging the (redacted) exception through `IWorkerLogger` before exit. The gateway still detects death via process exit + pipe closure, so this is diagnostics-only; keep it minimal.
|
||||
|
||||
**Implementation.** In `WorkerApplication.Run` (or a tiny bootstrap in `Program.cs`), before parsing args, subscribe both handlers and log `WorkerUnhandledException` / `WorkerUnobservedTaskException` with `WorkerLogRedactor`-scrubbed message + type. Test: unit test the handler delegate logs and does not throw. Docs: mention in `docs/MxAccessWorkerInstanceDesign.md` logging list.
|
||||
|
||||
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86`; `dotnet test ...Worker.Tests... -p:Platform=x86`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-10 — Top-level catch logs exception type but never message `Low` · `—`
|
||||
|
||||
**Finding.** `WorkerApplication.cs:112-139` logs only `exception_type` for protocol, pipe, and unexpected failures; `HResultConverter.CreateSafeDiagnosticMessage` reduces every command exception to `Type: HRESULT 0x…` (`Conversion/HResultConverter.cs:46-49`).
|
||||
|
||||
**Design.** Log `exception.Message` (routed through `Bootstrap/WorkerLogRedactor` — which already scrubs nonces/credentials, `WorkerLogRedactor.cs:16-25`) alongside the type at the *process boundary*. Keep the credential-safe reply shape for IPC *replies* unchanged if the HRESULT-only stripping is intentional parity/secret policy — the fix is scoped to worker stderr/log, not the wire reply.
|
||||
|
||||
**Implementation.** In the three `WorkerApplication.Run` catch blocks, add `["exception_message"] = WorkerLogRedactor.Redact(exception.Message)`. Leave `HResultConverter` reply text as-is (document the intent in a code comment). Test: `WorkerApplicationTests` (or add) asserting the redacted message is present. Docs: none.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-11 — Every accepted event is defensively cloned on enqueue `Low` · `P2`
|
||||
|
||||
**Finding.** `MxAccessEventQueue.Enqueue` calls `mxEvent.Clone()` (`MxAccess/MxAccessEventQueue.cs:135`) for an event the mapper built exclusively for this call (`MxAccess/MxAccessBaseEventSink.cs:210-256`); only the value-cache post-publish shares the original.
|
||||
|
||||
**Design.** Take ownership of the passed event in the queue (stamp `WorkerSequence`/`WorkerTimestamp` on it directly) and let the value cache store the copy — the cache already snapshots only value/quality/timestamp/statuses (`MxAccess/MxAccessValueCache.cs:44-57`), so it does not need the full `MxEvent` alias. This halves protobuf allocation on the hottest path. Requires confirming no caller reuses the passed `MxEvent` after enqueue (the mapper builds a fresh one per event — safe). Roadmap P2 item 14.
|
||||
|
||||
**Implementation.** Remove the `Clone()`; mutate the incoming `mxEvent` in place under the queue lock. Audit `MxAccessBaseEventSink` call sites to confirm single-ownership. Tests: existing event-queue tests must still pass; add one asserting the enqueued instance is the same reference passed in and that the value cache's stored copy is independent. Docs: none.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~MxAccessEventQueue|FullyQualifiedName~EventSink"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-12 — No event batching per envelope; one flush per event; 25 ms poll `Low` · `P2`
|
||||
|
||||
**Finding.** The drain loop writes one `WriteAsync` per event (`Ipc/WorkerPipeSession.cs:362-367`), the writer flushes per frame (`Ipc/WorkerFrameWriter.cs:71-72`), and the poll interval is 25 ms (`Ipc/WorkerPipeSession.cs:17`). Each event costs a semaphore round-trip, a pipe write, and a flush; idle-to-active latency up to 25 ms. `gateway.md` lists event batching as the intended optimization.
|
||||
|
||||
**Design.** Acceptable for v1 parity (the review agrees). When throughput matters: either (a) add a repeated-event `WorkerEnvelope` body (a contracts/proto change — coordinate with domain 30 IPC, cross-references `IPC` findings; must preserve per-event order and sequence semantics), or (b) keep one event per envelope but coalesce *flushes* across a drained batch (flush once after writing the batch), which needs no proto change and captures most of the benefit. Recommend (b) now, (a) as a coordinated cross-domain change. Depends on WRK-04 (sequence stamping under the lock) so a coalesced batch keeps monotonic sequences.
|
||||
|
||||
**Implementation.** (Option b) `WorkerFrameWriter`: add a `WriteBatchAsync(IEnumerable<WorkerEnvelope>, ...)` that writes all frames then flushes once; drain loop calls it per drained batch. Tests: `WorkerFrameWriterTests` asserting one flush per batch and correct framing. Docs: note the flush-coalescing in `docs/WorkerFrameProtocol.md`; if option (a), update `.proto` and regenerate per CLAUDE.md contracts rule.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-13 — Frame writer allocates a fresh buffer per frame; reader pools `Low` · `P2`
|
||||
|
||||
**Finding.** `WorkerFrameWriter` does `new byte[frameLength]` per frame (`Ipc/WorkerFrameWriter.cs:63-66`) while `WorkerFrameReader` rents from `ArrayPool<byte>.Shared` (`Ipc/WorkerFrameReader.cs:55-77`).
|
||||
|
||||
**Design.** Rent the write buffer from `ArrayPool<byte>.Shared` for symmetry; return it in a `finally` after the write completes. Because `ArrayPool` may return an oversized buffer, pass explicit `(0, frameLength)` to `WriteAsync` (already does) and never leak the buffer's tail. Trivial, self-contained. Roadmap P2 item 14.
|
||||
|
||||
**Implementation.** `WorkerFrameWriter.WriteAsync`: `byte[] frame = ArrayPool<byte>.Shared.Rent(frameLength);` … `try { … } finally { ArrayPool<byte>.Shared.Return(frame); }`. Note the return must occur after the awaited write completes (it does — inside the same method). Tests: existing `WorkerFrameWriterTests` framing tests cover correctness; no behavior change. Docs: none.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerFrameWriterTests"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-14 — Private-field naming split `_camelCase` vs `camelCase` `Low` · `—`
|
||||
|
||||
**Finding.** `Ipc/` and `Bootstrap/` use `_camelCase` (`Ipc/WorkerPipeSession.cs:21-38`, `Ipc/WorkerFrameWriter.cs:13-15`, `Bootstrap/WorkerConsoleLogger.cs:10`); `Sta/` and `MxAccess/` use bare `camelCase` (`Sta/StaRuntime.cs:10-24`, `MxAccess/MxAccessStaSession.cs:16-27`). `docs/style-guides/CSharpStyleGuide.md:30-32` permits the underscore prefix "only when already established" — both are established, so the worker has no single convention.
|
||||
|
||||
**Design.** Pick one style project-wide and migrate opportunistically (not in one churn commit). The gateway server is the tie-breaker: adopt whatever the gateway uses so the whole solution converges. This is a mechanical rename with no behavior change; do it file-by-file as those files are touched for other findings to keep diffs reviewable.
|
||||
|
||||
**Implementation.** Decide the target (check `src/ZB.MOM.WW.MxGateway.Server` private-field style), record it in `docs/style-guides/CSharpStyleGuide.md`, and rename incrementally. Tests: build only. Docs: the style-guide note.
|
||||
|
||||
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86` (analyzers/`TreatWarningsAsErrors` must stay green).
|
||||
|
||||
---
|
||||
|
||||
## WRK-15 — Doc drift: STA thread name and heartbeat-counter note `Low` · `P2`
|
||||
|
||||
**Finding.** The STA thread is named `"MxGateway.Worker.STA"` (`Sta/StaRuntime.cs:61`) but `docs/WorkerSta.md:23,30` and `docs/MxAccessWorkerInstanceDesign.md:254` say `ZB.MOM.WW.MxGateway.Worker.STA`. And `docs/MxAccessWorkerInstanceDesign.md:653-654` says event-queue depth and sequence "are reported as zero until the event queue implementation owns those counters," but `CaptureHeartbeat` now populates both from the live queue (`MxAccess/MxAccessStaSession.cs:375-380`).
|
||||
|
||||
**Design.** Docs must match source (CLAUDE.md rule). Choose: either rename the thread to the documented `ZB.MOM.WW.MxGateway.Worker.STA` (operators grep thread dumps for it) or update the docs to the actual name — recommend renaming the thread to the fully-qualified documented name for operability, in the same commit that fixes the heartbeat-counter note. Roadmap P2 item 15 (doc-drift sweep).
|
||||
|
||||
**Implementation.** Either edit `Sta/StaRuntime.cs:61` thread `Name` to `ZB.MOM.WW.MxGateway.Worker.STA`, or edit the two docs. Fix `docs/MxAccessWorkerInstanceDesign.md:653-654` to state the counters are live. Tests: if renaming, update any test asserting the thread name. Docs: as above.
|
||||
|
||||
**Verification.** `dotnet build ...Worker.csproj -p:Platform=x86`; `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaRuntime"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-16 — Boilerplate duplication in IPC envelope/ctor overloads `Low` · `—`
|
||||
|
||||
**Finding.** Seven near-identical `CreateEnvelope`/`CreateBaseEnvelope` overload pairs (`Ipc/WorkerPipeSession.cs:920-1003`) and eight `WorkerPipeClient` constructor overloads (`Ipc/WorkerPipeClient.cs:36-140`). Maintenance noise — each new body means two more copy-paste methods.
|
||||
|
||||
**Design.** Collapse the envelope overloads to a single `CreateEnvelope(Action<WorkerEnvelope> setBody)` (or a switch on the body message type); keep the correlation-id special case for `WorkerCommandReply` inside the setter. Reduce the client constructors to one primary constructor with the rest chaining via defaulted parameters (net48 supports optional params — no init-only needed). Pure refactor, no behavior change. Fold into WRK-04/WRK-07 since those already rework `CreateBaseEnvelope`.
|
||||
|
||||
**Implementation.** Refactor `WorkerPipeSession` envelope factories and `WorkerPipeClient` ctors. Tests: existing IPC tests must pass unchanged; they are the regression guard. Docs: none.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSession|FullyQualifiedName~WorkerPipeClient"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-17 — Gateway death exits with wrong exit code (6 not 5) `Low` · `—`
|
||||
|
||||
**Finding.** Pipe EOF surfaces as `WorkerFrameProtocolException(EndOfStream)` (`Ipc/WorkerFrameReader.cs:104-109`), which `WorkerApplication.Run` catches first and maps to `ProtocolViolation` (6) (`WorkerApplication.cs:110-119`) even though `PipeConnectionFailed` (5) exists (`Bootstrap/WorkerExitCode.cs:10`) and the in-session fault mapping already distinguishes pipe disconnect.
|
||||
|
||||
**Design.** Special-case `WorkerFrameProtocolErrorCode.EndOfStream` in the `WorkerFrameProtocolException` catch to return `PipeConnectionFailed` (5) — EOF means the gateway went away, not that the worker misbehaved. Improves orphan-worker post-mortem triage.
|
||||
|
||||
**Implementation.** In `WorkerApplication.cs:110-119`, branch on `exception.ErrorCode == WorkerFrameProtocolErrorCode.EndOfStream` → log + return `PipeConnectionFailed`; else `ProtocolViolation`. Tests: `WorkerApplicationTests` — inject a pipe client throwing `EndOfStream` and assert exit code 5. Docs: `docs/WorkerFrameProtocol.md` / exit-code table if one exists.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerApplication"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-18 — Event-queue overflow exits as generic `UnexpectedFailure` `Low` · `—`
|
||||
|
||||
**Finding.** `docs/MxAccessWorkerInstanceDesign.md:615-624` says overflow should "stop accepting new commands" and "let the gateway close or kill the worker"; the implementation instead terminates immediately — the drain loop writes the fault then throws (`Ipc/WorkerPipeSession.cs:344-353`), unwinding into the generic handler and exit code 1 (`WorkerApplication.cs:131-139`). The designed fault path is indistinguishable from a crash by exit code.
|
||||
|
||||
**Design.** The implemented fail-fast is arguably stronger than the doc and acceptable; the defect is *observability*. Give overflow a dedicated exit code (e.g. add `EventQueueOverflow` to `WorkerExitCode`) and catch the drain-fault termination in `WorkerApplication` to return it, then update the doc to describe the implemented immediate-terminate policy. Recommend keeping fail-fast (do not weaken to "stop accepting commands") and aligning the doc + exit code to it.
|
||||
|
||||
**Implementation.** Add `WorkerExitCode.EventQueueOverflow`; wrap the drain-fault throw in a typed exception (`WorkerEventQueueOverflowTerminationException`) so `WorkerApplication.Run` can map it. Tests: `WorkerPipeSessionTests` overflow path asserts the dedicated code. Docs: rewrite `docs/MxAccessWorkerInstanceDesign.md:615-624` to match.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~WorkerPipeSessionTests|FullyQualifiedName~WorkerApplication"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-19 — Command start/end logging with correlation id absent `Low` · `—`
|
||||
|
||||
**Finding.** `docs/MxAccessWorkerInstanceDesign.md:790-791` lists "command start/end with correlation id" among required worker logs; the only per-command log is the dropped-reply diagnostic (`Ipc/WorkerPipeSession.cs:645-655`). `StaCommand.EnqueueTimestamp` is captured (`Sta/StaCommand.cs`) but never used for latency.
|
||||
|
||||
**Design.** Add optional, level-gated start/end logging in `StaCommandDispatcher.ExecuteQueuedCommandAsync`, which already brackets each command (`Sta/StaCommandDispatcher.cs:265-281`). Log correlation id + method at start; at end log outcome + latency (`now - EnqueueTimestamp`). Gate at a verbose/debug level so production noise is opt-in.
|
||||
|
||||
**Implementation.** Inject the optional `IWorkerLogger` into `StaCommandDispatcher` (or pass through the session); emit `WorkerCommandStarted`/`WorkerCommandCompleted` with correlation id, method, latency, outcome. Tests: `StaCommandDispatcherTests` asserting both logs fire with the correlation id. Docs: confirm the log names in `docs/MxAccessWorkerInstanceDesign.md` logging list.
|
||||
|
||||
**Verification.** `dotnet test ...Worker.Tests... -p:Platform=x86 --filter "FullyQualifiedName~StaCommandDispatcher"`.
|
||||
|
||||
---
|
||||
|
||||
## WRK-20 — Test-coverage gaps for the failure modes above `Low` · `—`
|
||||
|
||||
**Finding.** `Worker.Tests` covers the pump wake behavior, dispatcher ordering/cancellation/shutdown, handshake/heartbeat/watchdog (incl. the stuck ceiling), control commands, shutdown races, late-reply drops, frame protocol, conversion, event queue, and alarm units. Not covered: STA thread death mid-run (WRK-02), wire-level envelope sequence monotonicity under concurrent writers (WRK-04), the silent no-reply drop at the `_acceptingCommands` gate (WRK-03), and the `ReadBulk`-exceeds-ceiling false fault (WRK-01).
|
||||
|
||||
**Design.** Add the four tests *alongside* their fixes (named in each entry above) using the existing fake-runtime harness in `WorkerPipeSessionTests`, which already supports all four. This is not a separate work item so much as the acceptance criterion for WRK-01..04 — tracked here so it is not dropped.
|
||||
|
||||
**Implementation.** Tests to add: `StaRuntimeTests.ThreadDeath_FaultsPendingAndFutureWork` (WRK-02); `WorkerPipeSessionTests.ConcurrentWriters_SequenceIsMonotonic` (WRK-04); `WorkerPipeSessionTests.CommandAfterShutdown_RepliesWorkerUnavailable` (WRK-03); `WorkerPipeSessionTests.LongReadBulk_DoesNotFaultWhilePumping` + `StaRuntimeTests.PumpPendingMessages_RefreshesLastActivity` (WRK-01). Docs: none.
|
||||
|
||||
**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86` (full worker suite once, after the batch lands, per the targeted-tests-then-phase-suite rule).
|
||||
|
||||
---
|
||||
|
||||
## Cross-references
|
||||
|
||||
- WRK-01 is roadmap P0 item 4; WRK-06/11/12/13 are the worker slice of P2 item 14 (event hot-path); WRK-15 is part of P2 item 15 (doc-drift sweep); WRK-07 relates to P1 item 8 (backpressure/size topology).
|
||||
- WRK-04 and WRK-07 rework the write path together; WRK-16 folds into that refactor. WRK-12 depends on WRK-04's under-lock sequence stamping.
|
||||
- WRK-02 and WRK-03 both close silent-failure edges (cross-cutting theme 1 in `00-overall.md`); the same theme spans gateway and client findings — coordinate the "every death/drop is observable" pattern across domains.
|
||||
@@ -0,0 +1,409 @@
|
||||
# Contracts & IPC Protocol — Remediation Design & Implementation
|
||||
|
||||
Source review: [30-contracts-ipc.md](../30-contracts-ipc.md) · Generated: 2026-07-09
|
||||
|
||||
The frame protocol and proto evolution hygiene are sound; the remediable risk clusters in three seams: an unenforced codegen/descriptor freshness gate, an un-negotiated size/backpressure topology that converts legitimate large payloads into whole-session death, and contract-boundary documentation drift. All `path:line` citations below were re-verified against the working tree; corrections are noted inline where the review's lines had shifted.
|
||||
|
||||
## Finding index
|
||||
|
||||
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||
|----|-----|-------|---------|--------|-----------|-------|
|
||||
| IPC-01 | High | Published client descriptor set is 7 weeks stale, nothing enforces freshness | P1 | M | — | clients/proto/descriptors/mxaccessgw-client-v1.protoset, scripts/publish-client-proto-inputs.ps1, src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs |
|
||||
| IPC-02 | Medium | Worker max frame size hard-coded, cannot follow gateway config | P1 | M | IPC-03 | src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs, Contracts/Protos/mxaccess_worker.proto |
|
||||
| IPC-03 | Medium | gRPC max == pipe max with zero headroom; oversized write faults whole session | P1 | M | IPC-02 | src/ZB.MOM.WW.MxGateway.Server/Configuration/ProtocolOptions.cs, Workers/WorkerClient.cs, Workers/WorkerFrameWriter.cs |
|
||||
| IPC-04 | Medium | `DrainEvents max_events=0` packs entire queue into one reply frame | P1 | S | IPC-03 | src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs, Grpc/MxAccessGrpcRequestValidator.cs |
|
||||
| IPC-05 | Medium | Redundant deep copies on command and event hot paths | P2 | S | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs, Grpc/MxAccessGrpcMapper.cs |
|
||||
| IPC-06 | Medium | `gateway.md` Worker Envelope sketch no longer matches the contract | P2 | S | — | gateway.md |
|
||||
| IPC-07 | Medium | `docs/Grpc.md` says six RPCs; there are seven | P2 | S | — | docs/Grpc.md |
|
||||
| IPC-08 | Medium | `WorkerCancel` defined and handled but never sent | — | M | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs, Worker/Ipc/WorkerPipeSession.cs, Contracts/Protos/mxaccess_worker.proto |
|
||||
| IPC-09 | Medium | Known codegen fragilities have no in-repo guards | P1 | M | IPC-01 | clients/python/pyproject.toml, clients/java/.../build.gradle, clients/*/generate-proto.ps1, scripts/publish-client-proto-inputs.ps1 |
|
||||
| IPC-10 | Low | Envelope `sequence` is write-only; monotonicity never validated | — | S | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerEnvelopeValidator.cs, Worker/Ipc/WorkerEnvelopeValidator.cs, gateway.md |
|
||||
| IPC-11 | Low | No protocol version negotiation despite `supported_protocol_version` name | — | S | — | Contracts/Protos/mxaccess_worker.proto, Worker/Ipc/WorkerPipeSession.cs |
|
||||
| IPC-12 | Low | Gateway frame writer has no write lock; integrity rests on an undocumented invariant | — | S | — | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameWriter.cs |
|
||||
| IPC-13 | Low | Gateway writer serializes twice and issues two stream writes | P2 | S | IPC-05 | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameWriter.cs |
|
||||
| IPC-14 | Low | Gateway reader allocates a fresh array per frame; worker rents from `ArrayPool` | P2 | S | IPC-05 | src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerFrameReader.cs |
|
||||
| IPC-15 | Low | Worker event delivery is a 25 ms poll with per-event write+flush, no batching | P2 | M | — | src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs, Contracts/Protos/mxaccess_worker.proto |
|
||||
| IPC-16 | Low | Correlation id carried twice per reply (Info) | — | S | — | Contracts/Protos/mxaccess_worker.proto, mxaccess_gateway.proto |
|
||||
| IPC-17 | Low | Two docs name the wrong Python generated-output directory | P2 | S | — | docs/ClientProtoGeneration.md, CLAUDE.md |
|
||||
| IPC-18 | Low | Generated-code hygiene verified good — preserve under change (positive) | — | S | IPC-01 | src/ZB.MOM.WW.MxGateway.Contracts/Generated/ |
|
||||
| IPC-19 | Low | Generated/-must-be-committed rule for net48 is undocumented and unguarded | P1 | S | IPC-01 | docs/Contracts.md, ZB.MOM.WW.MxGateway.Contracts.csproj |
|
||||
| IPC-20 | Low | Descriptor `-Check` byte-compares source-info-bearing bytes; protoc-version-sensitive | P1 | S | IPC-01 | scripts/publish-client-proto-inputs.ps1 |
|
||||
| IPC-21 | Low | `gateway.md` presents unimplemented `Session` RPC inside the live service block | P2 | S | IPC-06 | gateway.md, Contracts/Protos/mxaccess_gateway.proto |
|
||||
| IPC-22 | Low | Public error-detail model stops at status codes plus prose (Info) | — | S | — | Contracts/Protos/mxaccess_gateway.proto, docs/Grpc.md |
|
||||
|
||||
---
|
||||
|
||||
## IPC-01 — Published client descriptor set is 7 weeks stale and nothing enforces freshness `High` · `P1`
|
||||
|
||||
**Finding.** `clients/proto/descriptors/mxaccessgw-client-v1.protoset` was last committed 2026-04-30 (`git log` verified: commit `0f88a95`), while `mxaccess_gateway.proto` changed through 2026-06-18 (`MxSparseArray`, commit `8ac9a33`), 2026-06-16 (`ReplayGap`), and 2026-06-15 (alarm provenance). `strings` over the protoset returns zero hits for `MxSparseArray`, `replay_gap`, or `provider_status` — confirmed stale. `docs/Contracts.md:127` mandates regenerating the descriptor after any proto change, `scripts/publish-client-proto-inputs.ps1` has a `-Check` mode (verified at line 3, 70, 88), but no CI workflow exists and `ClientProtoInputTests.cs` validates only manifest versions and path existence (verified lines 12-34: `Manifest_DeclaresCurrentProtocolVersionsAndExistingInputs` checks `schemaVersion`, protocol versions, `File.Exists`, `Directory.Exists` — never descriptor content).
|
||||
|
||||
**Impact.** The artifact documented as the "stable client input" (`docs/ClientProtoGeneration.md:48`) silently misrepresents the contract. Any consumer that prefers a descriptor input generates against a schema missing three shipped features; grpcurl users on the reflection-disabled deployments (per the deployment memory, gRPC reflection is off on both hosts) get a stale schema. This is the domain's single worst concrete defect and appears in the overall roadmap under P1 item 7.
|
||||
|
||||
**Design.** Two parts: (1) regenerate and commit the descriptor now; (2) add an automated freshness gate so the doc-mandated regeneration cannot be skipped silently again. The gate belongs in the gateway test project (`ClientProtoInputTests`) rather than only in the `-Check` script, because the test runs in the NonWindows build that the P1 CI will exercise, and it does not depend on `protoc` being on the runner. The most robust check compares the *descriptor's own descriptor set* against a set rebuilt from the current `.proto` sources at test time — but that reintroduces the protoc dependency and the byte-sensitivity of IPC-20. The lighter, sufficient check: reflect over the in-process `MxaccessGatewayReflection.Descriptor` / `MxaccessWorkerReflection.Descriptor` (already compiled into `Contracts`) and assert every message and field name present in the live descriptor also appears in the committed protoset's `FileDescriptorSet` (parsed with `FileDescriptorSet.Parser`). This catches "protoset missing a symbol the contract has" — exactly the stale-descriptor failure mode — without invoking protoc or being sensitive to source-info bytes. Co-designed with IPC-19 (same freshness-drift class for `Generated/`) and IPC-20 (protoc pinning for the script path).
|
||||
|
||||
**Implementation.**
|
||||
- Regenerate: run `scripts/publish-client-proto-inputs.ps1` (no `-Check`) on a box with the pinned protoc (IPC-20), commit the refreshed `.protoset`.
|
||||
- Extend `src/ZB.MOM.WW.MxGateway.Tests/Contracts/ClientProtoInputTests.cs`: add `Descriptor_ContainsEveryContractMessageAndField`. Load the protoset bytes, `FileDescriptorSet.Parser.ParseFrom`, build the set of `{file}.{message}.{field}` names; enumerate `MxaccessGatewayReflection.Descriptor.MessageTypes` (recursively for nested types) and `MxaccessWorkerReflection.Descriptor`; assert each contract symbol is present in the protoset set. Fail with the missing symbol name.
|
||||
- Wire the same assertion into the P1 CI job (see IPC-09) as a redundant guard.
|
||||
- Config/proto surface: none changes.
|
||||
- Docs: note in `docs/ClientProtoGeneration.md` that the freshness test guards the descriptor and that a red test means "regenerate and commit the protoset."
|
||||
|
||||
**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~ClientProtoInputTests` (runs on macOS NonWindows tree). After regenerating, confirm `strings clients/proto/descriptors/mxaccessgw-client-v1.protoset | grep MxSparseArray` returns hits.
|
||||
|
||||
---
|
||||
|
||||
## IPC-02 — Worker max frame size is hard-coded and cannot follow the gateway's configured limit `Medium` · `P1`
|
||||
|
||||
**Finding.** `WorkerFrameProtocolOptions`' `WorkerOptions` ctor always passes `DefaultMaxMessageBytes` = 16 MiB (verified `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameProtocolOptions.cs:15-21`). The gateway-side `MxGateway:Worker:MaxMessageBytes` is configurable (default 16 MiB, `Server/Configuration/WorkerOptions.cs:37`; validated 1 KiB–256 MiB per the review's `GatewayOptionsValidator` reference). Neither `GatewayHello` (`mxaccess_worker.proto:41-45`) nor the worker launch arguments carry the value, so the two limits are set independently and never reconciled.
|
||||
|
||||
**Impact.** An operator who raises the gateway limit above 16 MiB for large-array workloads gets a gateway that emits frames the worker's reader rejects with `MessageTooLarge` (`Worker/Ipc/WorkerFrameReader.cs:44-48`), faulting the session; the worker still cannot emit anything above 16 MiB in the return direction. The config appears to work until the first large frame. Part of the P1 backpressure-topology pass (roadmap item 8).
|
||||
|
||||
**Design.** Convey the negotiated max frame size in the handshake and fail fast on disagreement, rather than failing mid-traffic. Add `uint32 max_frame_bytes = 4;` to `GatewayHello` (additive, next free tag). The worker reads it during handshake and constructs its `WorkerFrameProtocolOptions.MaxMessageBytes` from the negotiated value instead of the hard-coded default; if the value is 0 (older gateway) it falls back to `DefaultMaxMessageBytes`. Because the worker and gateway are lockstep-deployed (one-worker-per-session invariant; gateway launches the worker), a launch-argument would also work, but the `GatewayHello` field keeps the value on the wire where an alternate-language worker (contemplated in `gateway.md`) can read it. Reject at handshake if the value exceeds a sane worker ceiling. This is a proto change, so it triggers the regen-and-commit rule (IPC-19) and the descriptor refresh (IPC-01). Co-designed with IPC-03; the negotiated value must sit *above* the public gRPC cap plus envelope headroom.
|
||||
|
||||
**Implementation.**
|
||||
- `Contracts/Protos/mxaccess_worker.proto`: add `uint32 max_frame_bytes = 4;` to `GatewayHello`. Regenerate `Contracts/Generated/` and commit (net48 rule).
|
||||
- Gateway: populate the field when building `GatewayHello` (in the worker-client handshake path) from `WorkerOptions.MaxMessageBytes`.
|
||||
- Worker: in `WorkerPipeSession` handshake (around the version check at `:215`), pass the received `max_frame_bytes` into the `WorkerFrameProtocolOptions` construction, replacing the `DefaultMaxMessageBytes` argument in the `WorkerOptions` ctor overload. Keep the `> 0` guard already in the all-parameters ctor.
|
||||
- Tests: `WorkerFrameProtocolTests` / handshake tests — assert the worker adopts the gateway's advertised limit and that a frame between the old (16 MiB) and new limit now succeeds; a fake-worker gateway test asserting handshake carries the value.
|
||||
- Docs: `docs/WorkerFrameProtocol.md` and `docs/GatewayConfiguration.md` — document that `MaxMessageBytes` is negotiated to the worker via `GatewayHello`.
|
||||
|
||||
**Verification.** Contracts changed → regenerate, then `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx`; `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86` and `dotnet test ...Worker.Tests -p:Platform=x86` (Windows host); `dotnet test ...Tests --filter FullyQualifiedName~Handshake`.
|
||||
|
||||
---
|
||||
|
||||
## IPC-03 — Public gRPC max message size equals the pipe max with zero headroom; oversized outbound frame faults the whole session `Medium` · `P1`
|
||||
|
||||
**Finding.** `MaxGrpcMessageBytes` defaults to 16 MiB (`Server/Configuration/ProtocolOptions.cs:16`) and the pipe `MaxMessageBytes` defaults to the same 16 MiB (`WorkerOptions.cs:37`). A gRPC-accepted `Invoke` payload near the limit gains `WorkerCommand` + `WorkerEnvelope` overhead (session id, sequence, correlation id, enqueue timestamp, oneof tags) before hitting `WorkerFrameWriter.cs:49-54`, whose `WorkerFrameProtocolException` escapes the write loop at `WorkerClient.cs:344-350` and calls `SetFaulted`, which kills the worker process (verified `WorkerClient.cs:722-749`: `SetFaulted` → `KillOwnedProcess` → `CompletePendingCommands`).
|
||||
|
||||
**Impact.** One legitimate, gateway-accepted oversized write tears down the session, all its subscriptions, and its event stream — instead of failing that single command. Roadmap item 8.
|
||||
|
||||
**Design.** Two coordinated changes. (1) Headroom: keep `MaxGrpcMessageBytes` strictly below the negotiated pipe max (IPC-02) by an explicit envelope-overhead margin. Rather than a second magic constant, derive the accepted public payload cap as `pipeMax − EnvelopeOverheadReserve` (a named constant, e.g. 64 KiB, sufficient for the fixed envelope fields plus timestamp/oneof framing), and validate at startup (`GatewayOptionsValidator`) that `MaxGrpcMessageBytes ≤ pipeMax − reserve`, failing config otherwise. (2) Per-command failure: at the enqueue/write boundary, catch `WorkerFrameProtocolException` with `ErrorCode == MessageTooLarge` and fail only the offending correlation id (complete its `PendingCommand` with a `WorkerClientErrorCode` → `ResourceExhausted`/`InvalidArgument` gRPC status) instead of letting it reach `SetFaulted`. Because all writes funnel through the single `WriteLoopAsync` (`WorkerClient.cs:332-339`), the cleanest seam is to pre-check `envelope.CalculateSize()` against `MaxMessageBytes` in `CreateCommandEnvelope`/enqueue and reject there synchronously in `InvokeAsync`, before the frame ever enters the outbound channel — this keeps the write loop's remaining exceptions genuinely fatal (a mid-frame `MessageTooLarge` there really is a desync). Co-designed with IPC-02 and IPC-04 (the three form the size/backpressure pass). Does not touch MXAccess parity — this is a transport cap, not a COM behavior.
|
||||
|
||||
**Implementation.**
|
||||
- `Server/Configuration/ProtocolOptions.cs` / `GatewayOptionsValidator`: add the headroom invariant; document `EnvelopeOverheadReserve`.
|
||||
- `Server/Workers/WorkerClient.cs`: in `InvokeAsync` before `EnqueueAsync`, size-check the built envelope; on overshoot, throw `WorkerClientException(PayloadTooLarge, ...)` mapped to gRPC `ResourceExhausted` in `MxAccessGatewayService` — do not enqueue, do not fault. Add `WorkerClientErrorCode.PayloadTooLarge`.
|
||||
- Keep the write-loop `MessageTooLarge` → `SetFaulted` path for genuine desync (now unreachable for legitimate command payloads).
|
||||
- Tests: `WorkerClientTests` — an over-cap `Invoke` fails that one command with the mapped status and leaves the session `Ready` and other pending commands intact (assert worker not killed). `GatewayOptionsValidatorTests` — headroom invariant rejects `MaxGrpcMessageBytes ≥ pipeMax`.
|
||||
- Docs: `docs/GatewayConfiguration.md` (headroom rule), `docs/Grpc.md` (Invoke now returns `ResourceExhausted` for oversized payloads).
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClientTests|FullyQualifiedName~GatewayOptionsValidatorTests"`.
|
||||
|
||||
---
|
||||
|
||||
## IPC-04 — `DrainEvents` with `max_events = 0` packs the entire event queue into one reply envelope `Medium` · `P1`
|
||||
|
||||
**Finding.** `MxAccessEventQueue` treats `0` as drain-all (review cite `Worker/MxAccess/MxAccessEventQueue.cs:172-192`); the reply is a single `DrainEventsReply` written as one frame (`Worker/Ipc/WorkerPipeSession.cs` control-reply path). `MxAccessGrpcRequestValidator` imposes no cap on `DrainEventsCommand.max_events`.
|
||||
|
||||
**Impact.** A deep queue of large array events produces a reply exceeding `MaxMessageBytes`; the control-reply write throws, the exception propagates out of the message loop, and the worker exits — a diagnostics command kills the session. Roadmap item 8.
|
||||
|
||||
**Design.** Cap the effective drain worker-side rather than chunking (chunking `DrainEvents` needs a streaming reply arm the control-command path doesn't have; a bounded single reply is simpler and sufficient for a diagnostics RPC). Introduce a `MaxDrainEventsPerReply` worker constant (e.g. `WorkerPipeSession` alongside the existing `EventDrainInterval`/batch-size constants at `:17-19`), and interpret `max_events = 0` as "up to `MaxDrainEventsPerReply`", never unbounded. Optionally surface a `has_more`/`remaining` field on `DrainEventsReply` (additive) so a caller can drain iteratively. Also add an upper-bound validation in `MxAccessGrpcRequestValidator` for the public `DrainEventsCommand.max_events`. This keeps parity untouched (drain is a gateway diagnostics feature, not an MXAccess behavior). Co-designed with IPC-03 (both prevent one accepted request from producing a session-killing frame).
|
||||
|
||||
**Implementation.**
|
||||
- `Worker/Ipc/WorkerPipeSession.cs`: clamp the requested count to `MaxDrainEventsPerReply`; treat 0 as the clamp value.
|
||||
- (Optional, additive) `Contracts/Protos/mxaccess_gateway.proto`: add `uint32 remaining = N;` to `DrainEventsReply` — regen + commit if taken.
|
||||
- `Server/Grpc/MxAccessGrpcRequestValidator`: reject `max_events` above the documented ceiling with `InvalidArgument`.
|
||||
- Tests: `MxAccessEventQueueTests` / worker session tests — a queue larger than the cap drains at most `MaxDrainEventsPerReply` per reply and never builds an over-`MaxMessageBytes` frame; validator test for the public cap.
|
||||
- Docs: `docs/Grpc.md` DrainEvents rules row; `docs/WorkerFrameProtocol.md` if the reply gains `remaining`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86` + `dotnet test ...Worker.Tests -p:Platform=x86 --filter FullyQualifiedName~MxAccessEventQueue`; gateway validator test on macOS.
|
||||
|
||||
---
|
||||
|
||||
## IPC-05 — Redundant deep copies on the command and event hot paths `Medium` · `P2`
|
||||
|
||||
**Finding.** `MxAccessGrpcMapper.MapCommand` clones the inbound `MxCommand`, then `WorkerClient.CreateCommandEnvelope` clones the whole `WorkerCommand` a second time (verified `WorkerClient.cs:902`, `envelope.WorkerCommand = command.Clone()`); `MapEvent` deep-clones every worker event (verified `MxAccessGrpcMapper.cs:68`, `workerEvent.Event?.Clone()`). So each `Invoke` materializes the command graph three times (gRPC parse, mapper clone, worker-client clone); each event is parsed once and cloned once.
|
||||
|
||||
**Impact.** Measurable allocation pressure on array-heavy `OnDataChange` streams — the exact load the gateway exists to handle (overall theme 7). Not a correctness bug.
|
||||
|
||||
**Design.** Drop the second `Clone()` in `CreateCommandEnvelope`: the mapper's clone already isolates the graph from the caller-owned gRPC message, and the envelope is built and owned entirely inside `WorkerClient`, so no aliasing hazard remains once the envelope is enqueued. For `MapEvent`, evaluate ownership transfer: the `WorkerEvent` is parsed fresh from the pipe frame in the read loop and is not retained after mapping, so the mapper can move `workerEvent.Event` into the outbound `MxEvent` graph rather than cloning — but confirm no code path re-reads `workerEvent` after `MapEvent` (the fan-out distributor may share one `MxEvent` across subscribers, which is read-only and safe). Recommend dropping the command clone unconditionally (clear win, no aliasing) and gating the event ownership-transfer on a quick audit of the fan-out path (coordinate with the gateway-core distributor findings). Parity-neutral. Co-designed with IPC-13/IPC-14 as the frame-I/O + allocation pass (roadmap item 14).
|
||||
|
||||
**Implementation.**
|
||||
- `Server/Workers/WorkerClient.cs:902`: `envelope.WorkerCommand = command;` (remove `.Clone()`), with a comment that the mapper already isolated the graph.
|
||||
- `Server/Grpc/MxAccessGrpcMapper.cs:68`: keep the clone only if the audit shows the event is aliased downstream; otherwise transfer ownership.
|
||||
- Tests: existing `WorkerClientTests`/mapper tests must still pass (no behavior change); add an assertion that the enqueued envelope's command is reference-equal to the mapper output where the clone was removed, to lock the intent.
|
||||
- Docs: `docs/Grpc.md:197-211` (the clone-count description) must be corrected to match the reduced copies.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter "FullyQualifiedName~WorkerClient|FullyQualifiedName~MxAccessGrpcMapper"`.
|
||||
|
||||
---
|
||||
|
||||
## IPC-06 — `gateway.md`'s Worker Envelope section no longer matches the shipped contract `Medium` · `P2`
|
||||
|
||||
**Finding.** Verified `gateway.md:291-309` shows `uint64 correlation_id = 4` and body cases `command = 20; command_reply = 21; event = 22; heartbeat = 23; cancel = 24; shutdown = 25; fault = 26`. The actual contract (`mxaccess_worker.proto:20-39`) is `string correlation_id = 4` and `worker_command = 13; worker_command_reply = 14; worker_cancel = 15; worker_shutdown = 16; worker_shutdown_ack = 17; worker_event = 18; worker_heartbeat = 19; worker_fault = 20`. `WorkerShutdownAck` is absent from the doc sketch entirely.
|
||||
|
||||
**Impact.** The top-level architecture doc's field numbers and types are wrong; anyone implementing an alternate worker (the doc contemplates a C++ worker) from this section produces an incompatible peer. Violates the repo's docs-change-with-source rule. Roadmap item 15.
|
||||
|
||||
**Design.** Replace the hand-maintained protobuf sketch with either the verbatim current message or, preferably, a short prose description plus a pointer to `mxaccess_worker.proto` as the single source of truth — hand-copied proto in docs is exactly what drifted. Keep the "Rules" list but reconcile it (`correlation_id` is a string; note `WorkerShutdownAck` and `WorkerReady`). Do this in the same sweep as IPC-07, IPC-17, IPC-21, IPC-22.
|
||||
|
||||
**Implementation.**
|
||||
- `gateway.md:288-320`: replace the code block with the real oneof (or a pointer), add `WorkerReady`/`WorkerShutdownAck`, fix `correlation_id` type.
|
||||
- No source/proto/config change.
|
||||
|
||||
**Verification.** Doc-only; no build. Cross-check the doc block against `mxaccess_worker.proto:20-39` after editing.
|
||||
|
||||
---
|
||||
|
||||
## IPC-07 — `docs/Grpc.md` says the service has six RPCs; it has seven `Medium` · `P2`
|
||||
|
||||
**Finding.** Verified `docs/Grpc.md:13` and `:32` say "six in total" and omit `QueryActiveAlarms`. The proto declares seven RPCs (`mxaccess_gateway.proto:18-37`: `OpenSession`, `CloseSession`, `Invoke`, `StreamEvents`, `AcknowledgeAlarm`, `StreamAlarms`, `QueryActiveAlarms`); `QueryActiveAlarms` is implemented in `MxAccessGatewayService`.
|
||||
|
||||
**Impact.** The authoritative gRPC-layer doc undercounts the public surface and documents no validation/handler behavior for the missing RPC. Note the cross-domain link: the security review found `QueryActiveAlarms` also missing from the scope resolver (SEC domain) — the doc omission masked that gap. Roadmap item 15.
|
||||
|
||||
**Design.** Update the count (six → seven) at both cites, add `QueryActiveAlarms` to the collaborators/RPC table, and add a handler section describing its validation, streaming behavior, and required scope. Flag to the security-domain remediation that the same RPC needs a scope-resolver entry.
|
||||
|
||||
**Implementation.**
|
||||
- `docs/Grpc.md:13,32`: fix count and RPC list; add `QueryActiveAlarms` row and handler subsection.
|
||||
- No source change in this domain (scope resolver fix is SEC).
|
||||
|
||||
**Verification.** Doc-only.
|
||||
|
||||
---
|
||||
|
||||
## IPC-08 — `WorkerCancel` is defined and handled but never sent `Medium` · `—`
|
||||
|
||||
**Finding.** `WorkerCancel` is defined (`mxaccess_worker.proto:71-73`) and dispatched by the worker to `_runtimeSession.CancelCommand` (verified `Worker/Ipc/WorkerPipeSession.cs:399-401`), but no gateway code sends it — `WorkerClient` handles timeout/cancel purely by abandoning the pending correlation (verified `WorkerClient.cs:195-213`: on timeout it calls `RemovePendingCommandAsFailed` and throws, never emitting a `WorkerCancel`).
|
||||
|
||||
**Impact.** The cancellation contract in `gateway.md:713-719` ("the worker should finish the COM call and discard or log the late reply if the correlation was canceled") is half-implemented: the worker never learns a correlation was canceled, so it always writes the late reply (which the gateway then drops at `WorkerClient.cs:565-571`), and the worker-side discard logic behind `CancelCommand` is unreachable. Not in the roadmap.
|
||||
|
||||
**Design.** Decide between two coherent end states; recommend **wiring it up** because the worker handling already exists and the late-reply-drop path is real work being wasted. On timeout or caller-cancel in `InvokeAsync`, enqueue a `WorkerCancel` envelope carrying the correlation id (add `string correlation_id`-based routing — the envelope already has `CorrelationId`, and the worker's `CancelCommand(envelope.CorrelationId)` already reads it, so no proto change is needed; `WorkerCancel.reason` is the human note). The worker then discards the late reply instead of writing it, saving a frame and a pointless gateway-side drop. Keep the abandon-the-pending-correlation behavior as the gateway's own timeout resolution — `WorkerCancel` is best-effort cleanup, not a synchronous ack. The alternative (mark the arm reserved-for-future and delete the worker handling) loses working behavior and still needs a `gateway.md` edit; reject it. This does not synthesize events and does not alter MXAccess parity — the COM call still completes; only the late reply is suppressed. Because the worker must keep pumping the STA, `CancelCommand` must remain a non-blocking mark-and-discard (confirm it is).
|
||||
|
||||
**Implementation.**
|
||||
- `Server/Workers/WorkerClient.cs`: add a `CreateCancelEnvelope(correlationId, reason)` (mirrors `CreateShutdownEnvelope`) and enqueue it in the timeout and cancellation-token branches of `InvokeAsync` (`:195-213`) after removing the pending command. Best-effort: swallow enqueue failures if the session is already terminal.
|
||||
- Worker: confirm `MxAccessSession.CancelCommand` marks the correlation for late-reply discard and that `WorkerPipeSession`'s reply-write path checks that mark (it already drops replies once state leaves `Ready` at `:604-608`; extend to "or correlation was canceled").
|
||||
- Tests: `WorkerClientTests` — a timed-out `Invoke` enqueues a `WorkerCancel` with the right correlation id; worker session test — a canceled correlation suppresses the late reply.
|
||||
- Docs: `gateway.md:713-719` — mark the cancellation path as implemented; `docs/WorkerFrameProtocol.md` — document `WorkerCancel` semantics.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` + `dotnet test ...Tests --filter FullyQualifiedName~WorkerClient`; worker side `-p:Platform=x86`.
|
||||
|
||||
---
|
||||
|
||||
## IPC-09 — Known codegen fragilities have no in-repo guards `Medium` · `P1`
|
||||
|
||||
**Finding.** (a) Python — `clients/python/pyproject.toml:15,41` allow `grpcio>=1.80,<2` and `grpcio-tools>=1.80,<2`; `clients/python/generate-proto.ps1` does no version check, so regenerating with any newer 1.x stamps a `GRPC_GENERATED_VERSION` above the pinned runtime and breaks pytest (a previously-hit failure per the memory). (b) Java — `clients/java/zb-mom-ww-mxgateway-client/build.gradle:50` points `generatedFilesBaseDir` at the tracked `src/main/generated`, so every `gradle build` rewrites the tracked output with protobuf-version churn and no task detects spurious diffs. (c) Portability — `clients/python/generate-proto.ps1:7` and `clients/go/generate-proto.ps1:8-9` hard-code `C:\Users\dohertj2\...` tool paths (verified), making generation single-machine and Windows-only.
|
||||
|
||||
**Impact.** Contract evolution safety depends on operator memory. A regeneration on a different machine or tool version silently produces incompatible or noisy output. Roadmap item 7 (CI) and the "everything guarded by operator memory" theme.
|
||||
|
||||
**Design.** Three targeted guards, each fail-fast:
|
||||
- Python: pin exact generator versions used for regeneration. Add a version assertion at the top of `generate-proto.ps1` that reads `grpcio-tools.__version__` and refuses to run if it differs from the pinned baseline (grpcio 1.80.0 / protobuf 6.31.1 per the memory). Keep the `pyproject.toml` runtime range but document that regeneration must use the exact pin.
|
||||
- Java: add a `checkGeneratedClean` Gradle task (git-diff the `src/main/generated` tree after generation and fail on non-empty diff when no `.proto` changed) so the ~64k-line spurious churn is caught in review rather than committed. Alternatively, and preferred long-term, move `generatedFilesBaseDir` to `$buildDir` and stop tracking generated Java — but that is a larger change; the check task is the minimal guard.
|
||||
- Portability: resolve `protoc`/`python`/plugins from `PATH` with a documented version assertion (the `publish-client-proto-inputs.ps1` `Resolve-Protoc` at lines 15-25 already does PATH-first with a documented fallback — mirror that pattern in the per-client scripts) instead of absolute user paths.
|
||||
|
||||
Coordinate with IPC-01/IPC-20 (descriptor freshness + protoc pin) so the CI job that runs the freshness gate also runs the codegen-clean checks.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/python/generate-proto.ps1`: PATH-resolve python; assert `grpcio-tools` version equals the pin, else throw.
|
||||
- `clients/go/generate-proto.ps1`: PATH-resolve `protoc`, `protoc-gen-go`, `protoc-gen-go-grpc`.
|
||||
- `clients/java/.../build.gradle`: add `checkGeneratedClean` task; wire into CI (not into every local build).
|
||||
- CI (new, per roadmap item 7): a job that regenerates and runs the clean/freshness checks.
|
||||
- Docs: `docs/ClientProtoGeneration.md` — record the exact pinned generator versions and the check tasks; `docs/Contracts.md` — cross-link.
|
||||
|
||||
**Verification.** Per-client: `python -m pytest` from `clients/python` after a pinned regen; `gradle checkGeneratedClean` from `clients/java` on a Windows/JDK-17 host; `go build ./...` from `clients/go`. Note the Java build must run on windev (Mac has no JRE per the memory) via an isolated `origin/<branch>` worktree.
|
||||
|
||||
---
|
||||
|
||||
## IPC-10 — Envelope `sequence` is write-only; monotonicity is never validated on receive `Low` · `—`
|
||||
|
||||
**Finding.** Sequences are assigned on send (verified `WorkerClient.cs:934`, `Interlocked.Increment(ref _nextSequence)`; worker side at `WorkerPipeSession.cs:1005-1018`), but neither validator checks them (verified `WorkerEnvelopeValidator.cs:15-39` checks only protocol version, session id, and non-empty body; the worker validator likewise). `gateway.md:312` states "`sequence` is monotonic per sender" as a protocol rule.
|
||||
|
||||
**Impact.** Gap/duplication detection promised by the design is diagnostics-only; ordering integrity rests entirely on pipe FIFO semantics (fine for a local named pipe, but the rule is unenforced). Low.
|
||||
|
||||
**Design.** Choose the cheap enforcement or the honest annotation. Recommend **annotate** rather than enforce: on a local named pipe FIFO is guaranteed, and adding per-sender monotonicity tracking to the validators introduces state and a new fault mode for no real safety gain (a reordering would already be a kernel-level pipe bug). Mark `sequence` as diagnostic-only in the proto comment and soften `gateway.md`'s "rule" to "diagnostic aid." If enforcement is later wanted, it belongs in the validators as a per-sender last-seen check, faulting on regression.
|
||||
|
||||
**Implementation.**
|
||||
- `Contracts/Protos/mxaccess_worker.proto:23`: comment that `sequence` is a monotonic diagnostic counter, not validated on receive (regen + commit — comment-only proto changes still regenerate).
|
||||
- `gateway.md:312`: reword the rule.
|
||||
|
||||
**Verification.** Regenerate contracts; `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` to confirm the comment-only regen is clean.
|
||||
|
||||
---
|
||||
|
||||
## IPC-11 — No protocol version negotiation despite a field name that implies it `Low` · `—`
|
||||
|
||||
**Finding.** `GatewayHello.supported_protocol_version` (`mxaccess_worker.proto:42`) is a single value compared for strict equality (`Worker/Ipc/WorkerPipeSession.cs:215-219`); the worker also hard-pins its version to `GatewayContractInfo.WorkerProtocolVersion` (=1, verified `GatewayContractInfo.cs:15`) at options construction (`WorkerFrameProtocolOptions.cs:65-70`), and every envelope re-checks equality in both validators.
|
||||
|
||||
**Impact.** Acceptable for a lockstep-deployed pair (`gateway.md:319` documents mismatch-fails-session), but the singular "supported" field cannot express a range; any future skewed upgrade needs new machinery. Low — no action required now.
|
||||
|
||||
**Design.** No change now. When the worker protocol first changes (bumps past 1), replace the single field with a `min`/`max` supported range in `GatewayHello`/`WorkerHello` and negotiate the highest common version rather than bumping the single constant. Record this as the intended evolution path in the proto comment so the next editor doesn't just increment the constant.
|
||||
|
||||
**Implementation.**
|
||||
- `Contracts/Protos/mxaccess_worker.proto`: add a comment on `supported_protocol_version` describing the min/max-range migration when a second version appears (regen + commit).
|
||||
- No code change.
|
||||
|
||||
**Verification.** Regenerate; NonWindows build clean.
|
||||
|
||||
---
|
||||
|
||||
## IPC-12 — Gateway frame writer has no write lock; integrity rests on an undocumented single-writer invariant `Low` · `—`
|
||||
|
||||
**Finding.** `Server/Workers/WorkerFrameWriter.cs` has no synchronization (verified: no lock/semaphore; `WriteAsync` at `:34-61` issues prefix and payload as two separate `_stream.WriteAsync` calls at `:59-60`). The worker's writer serializes with a `SemaphoreSlim` (verified `Worker/Ipc/WorkerFrameWriter.cs:14,68-77`) because its heartbeat, event-drain, and command tasks write concurrently. The gateway is safe only because all writes funnel through the single-reader outbound channel loop (`WorkerClient.cs:332-339`).
|
||||
|
||||
**Impact.** A future direct `_writer.WriteAsync` call outside the write loop interleaves the two stream writes and corrupts the frame stream unrecoverably. Latent, not live. Low.
|
||||
|
||||
**Design.** Defense in depth: either (a) collapse the two writes into one (adopt the worker's single-buffer approach — this is IPC-13, which also removes the interleaving surface entirely), or (b) add the same `SemaphoreSlim` write lock plus an XML-doc invariant note. Recommend doing IPC-13 (single write) *and* documenting the single-writer invariant on the class, which together make interleaving impossible even under a future stray caller. A bare assertion is weaker than a single atomic write.
|
||||
|
||||
**Implementation.**
|
||||
- Fold into IPC-13's single-buffer rewrite; add a class-level `<remarks>` stating "all frames must be written through `WorkerClient.WriteLoopAsync`; the writer is not internally synchronized for concurrent callers" (or, if a lock is added, document that it is).
|
||||
- Tests: covered by IPC-13's frame round-trip tests.
|
||||
|
||||
**Verification.** See IPC-13.
|
||||
|
||||
---
|
||||
|
||||
## IPC-13 — Gateway writer serializes each envelope twice and issues two stream writes `Low` · `P2`
|
||||
|
||||
**Finding.** Verified `Server/Workers/WorkerFrameWriter.cs:41,60`: `CalculateSize()` then `ToByteArray()` (which re-runs size calculation internally) and separate prefix/payload writes at `:59-60`. The worker writer already builds one prefixed buffer with a single `WriteTo(Span)` and one write (verified `Worker/Ipc/WorkerFrameWriter.cs:63-72`, with a comment explaining exactly this).
|
||||
|
||||
**Impact.** Extra CPU pass and extra pipe write per frame on the higher-volume side (the gateway writes every command). Low; roadmap item 14.
|
||||
|
||||
**Design.** Back-port the worker's single-buffer implementation verbatim: allocate `sizeof(uint) + payloadLength`, write the little-endian length prefix, `envelope.WriteTo(new Span<byte>(frame, 4, payloadLength))`, one `_stream.WriteAsync`. This also closes IPC-12's interleaving surface. Keep the existing size/`MessageTooLarge`/empty-payload guards (already present at `:41-54`). Consider adding the `FlushAsync` the worker does — verify the gateway's underlying pipe stream doesn't already auto-flush before adding it.
|
||||
|
||||
**Implementation.**
|
||||
- `Server/Workers/WorkerFrameWriter.cs:56-60`: replace the two-array/two-write block with the single-buffer/single-write block from `Worker/Ipc/WorkerFrameWriter.cs:63-72`.
|
||||
- Tests: `WorkerFrameWriterTests` / frame round-trip tests must still pass (byte-identical wire output); add a test asserting a single write path if a mock stream is available.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test src/ZB.MOM.WW.MxGateway.Tests --filter FullyQualifiedName~WorkerFrame`.
|
||||
|
||||
---
|
||||
|
||||
## IPC-14 — Gateway reader allocates a fresh array per frame; the worker rents from `ArrayPool` `Low` · `P2`
|
||||
|
||||
**Finding.** Verified `Server/Workers/WorkerFrameReader.cs:50`: `byte[] payload = new byte[payloadLength];` per frame. The worker rents/returns from `ArrayPool<byte>.Shared` with a comment that `ParseFrom` copies (verified `Worker/Ipc/WorkerFrameReader.cs:55-77`).
|
||||
|
||||
**Impact.** Large event frames (arrays near the cap) allocate LOH buffers per frame on the side that receives the entire event stream. Low; roadmap item 14.
|
||||
|
||||
**Design.** Mirror the pooled read: `int length = checked((int)payloadLength); byte[] payload = ArrayPool<byte>.Shared.Rent(length);` read exactly `length`, `WorkerEnvelope.Parser.ParseFrom(payload, 0, length)`, `Return` in `finally`. The gateway reader already reads-exactly via `ReadExactlyAsync`; keep that. Note `ParseFrom(payload, 0, length)` is required (the rented buffer may be larger than `length`).
|
||||
|
||||
**Implementation.**
|
||||
- `Server/Workers/WorkerFrameReader.cs:50-56`: adopt the rent/return pattern from the worker reader; use the length-bounded `ParseFrom` overload.
|
||||
- Tests: `WorkerFrameReaderTests` round-trip must pass; add a large-frame test to exercise the pool path.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; `dotnet test ...Tests --filter FullyQualifiedName~WorkerFrame`.
|
||||
|
||||
---
|
||||
|
||||
## IPC-15 — Worker event delivery is a 25 ms poll with per-event write+flush, no batching `Low` · `P2`
|
||||
|
||||
**Finding.** `Worker/Ipc/WorkerPipeSession.cs:17-19` sets `EventDrainInterval` 25 ms and batch size 128; the drain loop writes each event with its own `WriteAsync` (each acquiring the writer lock and `FlushAsync`, verified `Worker/Ipc/WorkerFrameWriter.cs:71-72`). `gateway.md:849-850` lists worker→gateway event batching as a planned optimization that doesn't exist — there is no multi-event envelope body.
|
||||
|
||||
**Impact.** Idle-to-active latency floor up to 25 ms per batch, plus per-event flush syscalls under burst. Acceptable for v1 parity; roadmap item 14.
|
||||
|
||||
**Design.** Do not tighten the poll (that trades latency for CPU wakeups). When event-rate targets firm up, add a batched event body as an *additive* `oneof` arm — a `WorkerEventBatch { repeated MxEvent events = 1; }` message and a `worker_event_batch = N` case in `WorkerEnvelope` — so the drain loop packs up to N events (bounded by `MaxMessageBytes`, coordinate with IPC-03/IPC-04) into one frame with one flush. The gateway read loop unpacks the batch and dispatches each event to the distributor. This preserves the "don't synthesize events" invariant (the worker still forwards only real events; batching is framing, not fabrication) and the one-worker-per-session invariant. Keep single-event `worker_event` for compatibility during rollout. This is deferred work, not a v1 fix — flagged so the batching claim in `gateway.md` is either implemented or re-marked as future.
|
||||
|
||||
**Implementation.**
|
||||
- (When scheduled) `Contracts/Protos/mxaccess_worker.proto`: add `WorkerEventBatch` + `worker_event_batch` oneof arm (regen + commit + descriptor refresh).
|
||||
- `Worker/Ipc/WorkerPipeSession.cs`: drain up to the batch size into one `WorkerEventBatch` frame under the frame-size cap.
|
||||
- `Server/Workers/WorkerClient.cs` read dispatch: handle the batch arm.
|
||||
- Now (doc-only, immediate): correct `gateway.md:849-850` to mark event batching as not-yet-implemented.
|
||||
- Tests: event round-trip under batch; ordering preserved.
|
||||
|
||||
**Verification.** Immediate: doc edit. When implemented: contracts regen + `dotnet build NonWindows.slnx`, worker `-p:Platform=x86` tests, gateway fake-worker event tests.
|
||||
|
||||
---
|
||||
|
||||
## IPC-16 — Correlation id is carried twice per reply `Low (Info)` · `—`
|
||||
|
||||
**Finding.** `WorkerEnvelope.correlation_id` (`mxaccess_worker.proto:24`) and `MxCommandReply.correlation_id` (review cite `mxaccess_gateway.proto:520`) both carry the value; `CompleteCommand` falls back from the envelope to the inner reply (verified `WorkerClient.cs:559-562`: uses `envelope.CorrelationId`, then `envelope.WorkerCommandReply.Reply?.CorrelationId`).
|
||||
|
||||
**Impact.** Two sources of truth for one value; harmless today, a divergence hazard for a future writer that sets one but not the other. Info.
|
||||
|
||||
**Design.** No wire change. Document that the **envelope** `correlation_id` is authoritative and the inner reply copy is the MXAccess-parity echo; the fallback in `CompleteCommand` exists only for defensive robustness. Add a proto comment on both fields naming the envelope as canonical.
|
||||
|
||||
**Implementation.**
|
||||
- `Contracts/Protos/mxaccess_worker.proto:24` and `mxaccess_gateway.proto` `MxCommandReply.correlation_id`: cross-referencing comments (regen + commit).
|
||||
|
||||
**Verification.** Regenerate; NonWindows build clean.
|
||||
|
||||
---
|
||||
|
||||
## IPC-17 — Two docs name the wrong Python generated-output directory `Low` · `P2`
|
||||
|
||||
**Finding.** Verified `docs/ClientProtoGeneration.md:80,145` say `clients/python/src/mxgateway/generated`; the manifest (`clients/proto/proto-inputs.json:28`) and the tree use `clients/python/src/zb_mom_ww_mxgateway/generated`, which is also what `clients/python/generate-proto.ps1` writes. CLAUDE.md's generated-code bullet repeats the wrong path.
|
||||
|
||||
**Impact.** A follow-the-doc regeneration writes to a dead directory. Low; roadmap item 15.
|
||||
|
||||
**Design.** Fix both docs (and the CLAUDE.md bullet) to the manifest path `clients/python/src/zb_mom_ww_mxgateway/generated`. Trivial, no source change.
|
||||
|
||||
**Implementation.**
|
||||
- `docs/ClientProtoGeneration.md:80,145`: correct the path.
|
||||
- `CLAUDE.md` generated-code bullet: correct `clients/python/src/mxgateway/generated` → `clients/python/src/zb_mom_ww_mxgateway/generated`.
|
||||
|
||||
**Verification.** Doc-only; grep to confirm no `src/mxgateway/generated` remains.
|
||||
|
||||
---
|
||||
|
||||
## IPC-18 — Generated-code hygiene verified good — preserve under change (positive) `Low` · `—`
|
||||
|
||||
**Finding.** No defect. Verified positive: `Contracts/Generated/*.cs` carry `<auto-generated>` headers and contain the newest symbols (`MxSparseArray`); all five client generated trees contain the sparse-array surface; `galaxy_repository.proto` is wire-identical to the GalaxyRepository package copy (only `csharp_namespace` differs), consistent with the retention comment in `ZB.MOM.WW.MxGateway.Contracts.csproj:29-33` that keeps it as the language-client codegen source.
|
||||
|
||||
**Impact.** None — this is a property to preserve, not a fix. Recorded so it is protected under change.
|
||||
|
||||
**Design.** No remediation. The freshness gates in IPC-01 (descriptor) and IPC-19 (`Generated/` net48 rule) are what keep this property true; the galaxy wire-identity should be asserted by a small test if the two copies ever diverge in practice (a byte-diff test comparing the two `.proto` files modulo `csharp_namespace`), but this is optional given the csproj comment already documents the invariant.
|
||||
|
||||
**Implementation.**
|
||||
- Optional: a `ContractsTests` case diffing `galaxy_repository.proto` against the package source path modulo `csharp_namespace`. Not required.
|
||||
|
||||
**Verification.** N/A (no change).
|
||||
|
||||
---
|
||||
|
||||
## IPC-19 — The `Generated/`-must-be-committed rule for net48 consumers is undocumented and unguarded `Low` · `P1`
|
||||
|
||||
**Finding.** Verified `ZB.MOM.WW.MxGateway.Contracts.csproj:26-35` does `Compile Remove="Generated\**\*.cs"` + `Protobuf ... OutputDir="Generated"`, regenerating tracked files on every build; the worker consumes contracts via `ProjectReference`. `docs/Contracts.md:97-98` says only "do not hand-edit." The operational rule — a proto edit requires regenerating and committing `Generated/`, or the net48 worker build fails CS0246 on new types (confirmed by the repo memory `project_proto_codegen_regen`) — appears nowhere, and no test compares `Generated/` to the protos.
|
||||
|
||||
**Impact.** Same silent-drift class as IPC-01: committed C# can lag the protos with nothing failing until a downstream net48 consumer breaks. Low; roadmap item 7.
|
||||
|
||||
**Design.** Document the rule and lean on the IPC-01 freshness test for enforcement. State in `docs/Contracts.md` (near the existing "do not hand-edit" line) that after any `.proto` edit you must regenerate `Contracts/Generated/` and commit it, and why (net48 worker has no first-class regen-on-restore; a stale `Generated/` fails CS0246). The IPC-01 reflection-based test (comparing `MxaccessGatewayReflection.Descriptor` against the committed protoset) already indirectly catches a proto edited without regenerating the descriptor; for `Generated/` specifically, the .NET build itself compiles against the freshly regenerated output on the net10 side, so the practical guard is the net48 worker build in CI (IPC-09/roadmap item 7) plus the documented rule.
|
||||
|
||||
**Implementation.**
|
||||
- `docs/Contracts.md:97`: add the regenerate-and-commit rule with the net48 rationale.
|
||||
- Optionally reference the memory-known `del Generated/*.cs` force-regen trick in the doc.
|
||||
- CI (roadmap item 7): the Windows net48 worker build job is the real enforcement.
|
||||
|
||||
**Verification.** Doc-only for the rule; the net48 build (`dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86` on Windows) is what proves a stale `Generated/` would fail.
|
||||
|
||||
---
|
||||
|
||||
## IPC-20 — Descriptor `-Check` byte-compares source-info-bearing bytes; protoc-version-sensitive `Low` · `P1`
|
||||
|
||||
**Finding.** Verified `scripts/publish-client-proto-inputs.ps1`: `-Check` (line 70) rebuilds the descriptor with `--include_source_info` (line 78) and `Compare-FileBytes` (line 36, 88) byte-compares. Source-info bytes differ across protoc releases even for identical schemas.
|
||||
|
||||
**Impact.** Once a check exists (IPC-01), a protoc upgrade produces a false "stale" failure or forces a churn commit unless protoc is pinned. Low; roadmap item 7.
|
||||
|
||||
**Design.** Two options, do both. (1) Pin protoc for descriptor generation: `Resolve-Protoc` (lines 15-25) already prefers PATH and documents a fallback; add a version assertion (`protoc --version` must equal the pinned toolchain in `docs/ToolchainLinks.md`, protoc 34.1 per CLAUDE.md) and fail fast on mismatch. (2) Make the in-process freshness test (IPC-01) the *primary* gate since it compares semantically (symbol presence) rather than byte-wise, sidestepping source-info sensitivity entirely; keep the script's byte-`-Check` as a stricter secondary that runs only with the pinned protoc. Alternatively drop `--include_source_info` and compare descriptors without source info for a version-tolerant byte compare — but source info is useful for grpcurl UX, so prefer pin + semantic test.
|
||||
|
||||
**Implementation.**
|
||||
- `scripts/publish-client-proto-inputs.ps1`: add a protoc-version assertion in `Resolve-Protoc`; document the pin.
|
||||
- Rely on IPC-01's semantic test as the CI gate.
|
||||
- Docs: `docs/ClientProtoGeneration.md` / `docs/Contracts.md` — record the pinned protoc version for descriptor generation.
|
||||
|
||||
**Verification.** Run `scripts/publish-client-proto-inputs.ps1 -Check` with the pinned protoc on Windows; the IPC-01 test on macOS.
|
||||
|
||||
---
|
||||
|
||||
## IPC-21 — `gateway.md` presents the unimplemented `Session` RPC inside the live service block `Low` · `P2`
|
||||
|
||||
**Finding.** Verified `gateway.md:328-345` sketches `rpc Session(stream ClientMessage) returns (stream ServerMessage)` inside the `service MxAccessGateway` block as "the best long-term shape," with a rollout plan whose step 3 is unimplemented; `mxaccess_gateway.proto:18-37` has no such RPC.
|
||||
|
||||
**Impact.** Intentional phasing, not a defect — but presenting it inside the service definition (alongside the five real RPCs) compounds IPC-06's staleness and reads as shipped surface. Low; roadmap item 15.
|
||||
|
||||
**Design.** Move the `Session` RPC out of the live service block into a clearly-labeled "Future work / not yet implemented" subsection, keeping the rollout plan. Do this in the same doc sweep as IPC-06/IPC-07. No proto or code change.
|
||||
|
||||
**Implementation.**
|
||||
- `gateway.md:328-345`: relabel; separate the bidi `Session` sketch from the shipped RPC list (which should also be reconciled to the actual seven RPCs — coordinate with IPC-07).
|
||||
|
||||
**Verification.** Doc-only.
|
||||
|
||||
---
|
||||
|
||||
## IPC-22 — Public error-detail model stops at status codes plus prose `Low (Info)` · `—`
|
||||
|
||||
**Finding.** `MxCommandReply` preserves MXAccess parity detail well (`hresult`, `statuses`, `diagnostic_message`, review cite `mxaccess_gateway.proto:518-529`), but transport-level failures surface only as gRPC status codes with message strings; no `google.rpc` error details are attached, and `AcknowledgeAlarmReply.status` is a permanently-unset placeholder documented as such (verified `mxaccess_gateway.proto:938-945`: `MxStatusProxy status = 5` with a comment that the by-name/by-GUID ack path produces only the int32 return code and the field is left UNSET; clients must read `hresult`/`protocol_status`).
|
||||
|
||||
**Impact.** Machine consumers must parse prose to distinguish sub-causes within a gRPC status code. Acceptable for the current in-repo client set. Info.
|
||||
|
||||
**Design.** No change now. If third-party clients appear, attach `google.rpc.ErrorInfo` (a stable `reason` enum + `metadata`) to transport failures so consumers branch on a machine-readable reason instead of message strings, and either populate or formally reserve `AcknowledgeAlarmReply.status`. The `status` placeholder is already honestly documented (parity-correct: the worker ack path genuinely produces only the int32), so leave it as-is unless the ack path grows a structured status. Recording the recommendation without acting keeps parity intact.
|
||||
|
||||
**Implementation.**
|
||||
- None now. Future: add `google.rpc.ErrorInfo` details in `MxAccessGatewayService` exception mapping; document reason codes in `docs/Grpc.md:217-241`.
|
||||
|
||||
**Verification.** N/A (no change).
|
||||
@@ -0,0 +1,537 @@
|
||||
# Security, Dashboard & Observability — Remediation Design & Implementation
|
||||
|
||||
Source review: [40-security-dashboard.md](../40-security-dashboard.md) · Generated: 2026-07-09
|
||||
|
||||
This document turns every finding in the Security/Dashboard/Observability review into a buildable remediation entry. The crypto and interceptor foundations are sound; the work here is policy tightening (loopback/`DisableLogin`/scope gaps), cross-platform path hygiene, hot-path auth cost, and documentation reconciliation. IDs are re-sequenced under the domain prefix `SEC-NN` (most-severe first); each entry names the original review tag (e.g. *review STA-1*) so evidence stays traceable. Roadmap tiers follow `00-overall.md`: the P1 "Security policy pass" (item 9) and "cross-platform paths" (item 10), and the P2 per-session-ACL (item 12) and documentation-drift sweep (item 15). Findings not itemized in that roadmap but belonging to the same security-hardening theme (§theme 6) are tagged P1 with a note; the remainder are `—`.
|
||||
|
||||
## Finding index
|
||||
|
||||
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||
|----|-----|-------|---------|--------|-----------|-------|
|
||||
| SEC-01 | Medium | Windows-absolute default paths become relative files off-Windows | P1 | M | — | Configuration/AuthenticationOptions.cs, Configuration/TlsOptions.cs, Configuration/GatewayOptionsValidator.cs, appsettings.json |
|
||||
| SEC-02 | Medium | `AllowAnonymousLocalhost` satisfies AdminOnly, not just Viewer | P1 | S | — | Dashboard/DashboardAuthorizationHandler.cs |
|
||||
| SEC-03 | Medium | Dashboard cookie lost its `__Host-` prefix; four docs still promise it | P2 | S | — | Dashboard/DashboardAuthenticationDefaults.cs, gateway.md, docs/GatewayDashboardDesign.md, CLAUDE.md |
|
||||
| SEC-04 | Medium | `DisableLogin` has no production guard | P1 | S | SEC-03 | Dashboard/DashboardServiceCollectionExtensions.cs, Configuration/GatewayOptionsValidator.cs |
|
||||
| SEC-05 | Medium | Hub bearer tokens irrevocable for 30 min, carried in query string | P1 | M | — | Dashboard/HubTokenService.cs, Dashboard/HubTokenAuthenticationHandler.cs, Dashboard/DashboardEndpointRouteBuilderExtensions.cs |
|
||||
| SEC-06 | Medium | LDAP plaintext-by-default with a committed service password | P1 | M | — | Configuration/LdapOptions.cs, appsettings.json, docs/GatewayConfiguration.md |
|
||||
| SEC-07 | Medium | `QueryActiveAlarmsRequest` missing from scope resolver; tests mislabeled | P1 | S | — | Security/Authorization/GatewayGrpcScopeResolver.cs, Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs, docs/Authorization.md |
|
||||
| SEC-08 | Medium | Per-RPC SQLite read + `last_used_utc` write; no verification cache | P1 | L | — | Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs, Security/Authentication/GatewayApiKeyIdentityMapper.cs |
|
||||
| SEC-09 | Medium | Dashboard design-doc `GroupToRole` sample now fails startup validation | P2 | S | — | docs/GatewayDashboardDesign.md |
|
||||
| SEC-10 | Medium | No API-key expiry | P1 | L | — | (shared `ZB.MOM.WW.Auth.ApiKeys`), Dashboard/DashboardApiKeyManagementService.cs |
|
||||
| SEC-11 | Medium | No rate limiting or lockout on either auth surface | P1 | M | SEC-08 | Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs, Dashboard/DashboardEndpointRouteBuilderExtensions.cs |
|
||||
| SEC-12 | Medium | Dashboard Close/Kill bypass the canonical audit store | P1 | M | — | Dashboard/DashboardSessionAdminService.cs |
|
||||
| SEC-13 | Low | Redactor's credential-command list omits secured-bulk variants | — | S | SEC-30 | Diagnostics/GatewayLogRedactor.cs |
|
||||
| SEC-14 | Low | `/metrics` + `/health` unauthenticated; a metric leaks session ids | — | S | SEC-02, SEC-20 | GatewayApplication.cs, Metrics/GatewayMetrics.cs |
|
||||
| SEC-15 | Low | GET `/logout` skips antiforgery | — | S | — | Dashboard/DashboardEndpointRouteBuilderExtensions.cs |
|
||||
| SEC-16 | Low | CLI accepts the pepper as a command-line argument | — | S | — | Program.cs |
|
||||
| SEC-17 | Low | Interceptor does not override client-streaming/duplex handlers | — | S | — | Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs |
|
||||
| SEC-18 | Low | Pepper-unavailable detection matches library message text | — | S | — | Dashboard/DashboardApiKeyManagementService.cs |
|
||||
| SEC-19 | Low | Canonical audit store re-issues `CREATE TABLE` per write/read | — | S | — | Security/Audit/SqliteCanonicalAuditStore.cs |
|
||||
| SEC-20 | Low | `session_id` metric tag is unbounded cardinality | P1 | S | — | Metrics/GatewayMetrics.cs, docs/Metrics.md |
|
||||
| SEC-21 | Low | Snapshot publisher works every second regardless of audience | — | M | — | Dashboard/DashboardSnapshotService.cs |
|
||||
| SEC-22 | Low | `docs/Authentication.md` documents types no longer in this repo | P2 | M | — | docs/Authentication.md |
|
||||
| SEC-23 | Low | Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` | — | S | SEC-03, SEC-04 | Configuration/GatewayOptionsValidator.cs |
|
||||
| SEC-24 | Low | Effective-config view omits the riskiest dashboard flags | — | S | SEC-04 | Configuration/GatewayConfigurationProvider.cs |
|
||||
| SEC-25 | Low | Per-session EventsHub ACL is an acknowledged TODO | P2 | M | SEC-02 | Dashboard/Hubs/EventsHub.cs |
|
||||
| SEC-26 | Low | Audit trail has no retention or pruning | — | M | — | Security/Audit/SqliteCanonicalAuditStore.cs |
|
||||
| SEC-27 | Low | Dashboard `GatewayStatus` is hardcoded Healthy | — | S | — | Dashboard/DashboardSnapshotService.cs |
|
||||
| SEC-28 | Info | Positive security observations (preserve under change) | — | S | — | (multiple) |
|
||||
| SEC-29 | Info | UI-stack rule verified compliant | — | S | — | wwwroot/lib/ |
|
||||
| SEC-30 | Info | Value-logging feature is unwired end-to-end | P2 | S | SEC-13 | Diagnostics/GatewayLogRedactor.cs, docs/Diagnostics.md |
|
||||
|
||||
---
|
||||
|
||||
## SEC-01 — Windows-absolute default paths become relative files off-Windows `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review SEC-1)* Three option defaults are Windows-absolute string literals: `AuthenticationOptions.SqlitePath` (`Configuration/AuthenticationOptions.cs:9`, mirrored `appsettings.json:17`), `TlsOptions.SelfSignedCertPath` (`Configuration/TlsOptions.cs:11-12`), and Galaxy `SnapshotCachePath` (`appsettings.json:80`). On Unix these contain no path separator, so SQLite/PFX writers treat the whole string `C:\ProgramData\MxGateway\gateway-auth.db` as a single filename relative to the content root — which is how the stray auth DB materialized under `src/ZB.MOM.WW.MxGateway.Server/`. The validator does not catch it: `AddIfInvalidPath` only requires `Path.GetFullPath` to *succeed* (`Configuration/GatewayOptionsValidator.cs:383-410`), and a bare filename is a valid relative path.
|
||||
|
||||
**Impact.** The auth DB (and, if an HTTPS endpoint is configured, a private-key PFX) silently lands in the working directory, its location varying by host OS and launch CWD. Keys created under one CWD are invisible under another. A materialized key DB in the tree is protected only by the generic `*.db` gitignore; a PFX would not be.
|
||||
|
||||
**Design.** Derive defaults from `Environment.GetFolderPath(SpecialFolder.CommonApplicationData)` + `Path.Combine`, which resolves to `C:\ProgramData` on Windows and `/usr/share` (or the container equivalent) elsewhere, and make the validator *reject* non-rooted paths for these keys so a relative override fails fast rather than writing into CWD. Rejected alternative: keeping the literals and only fixing the validator — that still leaves a broken default for any non-Windows run and diverges appsettings from code. Rooting enforcement is chosen over auto-rooting because the paths are security-sensitive (auth DB, private key) and silent relocation is worse than a boot error.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/AuthenticationOptions.cs:9`, `Configuration/TlsOptions.cs:11-12`: replace the literal initializers with a static helper, e.g. `Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData), "MxGateway", "gateway-auth.db")`. Keep `appsettings.json` values as the explicit Windows deployment path (they override the code default on the production hosts).
|
||||
- `Configuration/GatewayOptionsValidator.cs`: add an `AddIfNotRooted` check (`!Path.IsPathRooted(value)`) applied to `SqlitePath`, `SelfSignedCertPath`, and Galaxy `SnapshotCachePath`, alongside the existing `AddIfInvalidPath`.
|
||||
- Delete the stray `src/ZB.MOM.WW.MxGateway.Server/C:\ProgramData\MxGateway\gateway-auth.db` file and add a guard test that fails if any `*.db` appears under `src/` (a repo-hygiene test in `ZB.MOM.WW.MxGateway.Tests`).
|
||||
- Tests: extend `GatewayOptionsValidatorTests` with a non-rooted-path case per key; add the tree-hygiene test.
|
||||
- Docs: note the derived defaults in `docs/GatewayConfiguration.md` and `docs/Authentication.md`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-02 — `AllowAnonymousLocalhost` satisfies AdminOnly, not just Viewer `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review SEC-2)* `Dashboard/DashboardAuthorizationHandler.cs:32-37` calls `context.Succeed(requirement)` for any loopback request before the role loop, and the same handler serves both `AdminOnly` and `AnyDashboardRole`. Lines 25-30 do the same for `Authentication.Mode == Disabled` — for **remote** requests too. `AllowAnonymousLocalhost` defaults `true` (`Configuration/DashboardOptions.cs:9`).
|
||||
|
||||
**Impact.** Anything gated solely by `MxGateway.Dashboard.Admin` is authorized for an anonymous local process; `HubClientsPolicy` uses the same requirement, so a loopback process reaches every SignalR hub — the snapshot hub pushes the API-key inventory and effective config every second, and `EventsHub.SubscribeSession` joins any session's raw `MxEvent` feed. Destructive surfaces are saved today only by service-layer re-checks (`Dashboard/DashboardApiKeyAuthorization.cs`, `Dashboard/DashboardSessionAdminService.cs:33-39`), i.e. defense-in-depth, not the policy.
|
||||
|
||||
**Design.** Restrict the loopback bypass to the Viewer requirement: succeed only when `requirement.RequiredRoles` contains `DashboardRoles.Viewer` (the read-only requirement) and not when it is the Admin-only requirement. Decide deliberately whether anonymous loopback reaches hubs at all — recommended: keep hub read access for loopback (dashboard-on-the-box is the primary local-ops UX) but never Admin. This preserves the documented "anonymous localhost is read-only" contract at the policy layer instead of by accident. Note the loopback test trusts `Connection.RemoteIpAddress` (`:56-61`); if forwarded-headers middleware is ever added, this must be revisited — capture that as a comment.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/DashboardAuthorizationHandler.cs`: in the loopback branch, succeed only if `requirement.RequiredRoles.Contains(DashboardRoles.Viewer)` (the `AnyDashboardRole` requirement includes Viewer; `AdminOnly` does not). Leave the `Mode == Disabled` branch as-is but document that it is a global kill-switch, or gate it behind the same Viewer check for consistency (open question: whether `Disabled` should still grant admin — recommend no).
|
||||
- Tests: add cases to `DashboardAuthorizationHandlerTests` asserting loopback satisfies `AnyDashboardRole` but is denied `AdminOnly`.
|
||||
- Docs: reconcile `docs/GatewayDashboardDesign.md` and `CLAUDE.md`'s "anonymous localhost bypasses auth on loopback" wording to say "read-only".
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~DashboardAuthorizationHandler"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-03 — Dashboard cookie lost its `__Host-` prefix; four docs still promise it `Medium` · `P2`
|
||||
|
||||
**Finding.** *(review SEC-3)* `Dashboard/DashboardAuthenticationDefaults.cs:38` sets `CookieName = "MxGatewayDashboard"`, but `gateway.md:211`, `docs/GatewayDashboardDesign.md:425`, `docs/GatewayProcessDesign.md:686`, `docs/ImplementationPlanGateway.md:454`, and `CLAUDE.md` all still claim `__Host-MxGatewayDashboard`. Only `docs/GatewayConfiguration.md:170` is correct.
|
||||
|
||||
**Impact.** The `__Host-` browser guarantees (Secure required, no `Domain`, `Path=/`) are not in effect. The cookie is still HttpOnly/SameSite=Strict/SecurePolicy-controlled via `ZbCookieDefaults.Apply` (`Dashboard/DashboardServiceCollectionExtensions.cs:93-109`), and `RequireHttpsCookie=false` plus the configurable `CookieName` are legitimate reasons the prefix was dropped — so this is primarily doc drift, but security docs overstate protections.
|
||||
|
||||
**Design.** Prefer restoring the guarantee conditionally over a pure doc fix: when `RequireHttpsCookie` is true and no explicit `CookieName` override is set, default to `__Host-MxGatewayDashboard`; otherwise keep the plain name (the `__Host-` prefix is incompatible with `SameAsRequest`/HTTP). This is a small change in the PostConfigure at `DashboardServiceCollectionExtensions.cs:119-133` and recovers the documented protection for the default secure deployment. Either way, update the four stale docs in the *same* change (repo rule).
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/DashboardServiceCollectionExtensions.cs`: in the `CookieAuthenticationOptions` PostConfigure, when `RequireHttpsCookie` and `CookieName` is null/blank, set `cookieOptions.Cookie.Name = "__Host-MxGatewayDashboard"`. Guard: never apply `__Host-` when `SecurePolicy != Always` (would be silently dropped — this ties to SEC-23's consistency check).
|
||||
- `Dashboard/DashboardAuthenticationDefaults.cs`: keep `MxGatewayDashboard` as the non-secure fallback constant.
|
||||
- Docs: update `gateway.md`, `docs/GatewayDashboardDesign.md`, `docs/GatewayProcessDesign.md`, `docs/ImplementationPlanGateway.md`, `CLAUDE.md` to describe the actual conditional contract.
|
||||
- Tests: `DashboardServiceCollectionExtensionsTests` (or a cookie-options resolution test) asserting the name flips with `RequireHttpsCookie`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the dashboard-config test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-04 — `DisableLogin` has no production guard `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review SEC-4)* `Dashboard/DashboardServiceCollectionExtensions.cs:63-90` swaps in `DashboardAutoLoginAuthenticationHandler` under the cookie scheme for **all** clients when `MxGateway:Dashboard:DisableLogin` is true; the handler authenticates every request, remote included (`Dashboard/DashboardAutoLoginAuthenticationHandler.cs:56-62`). `GatewayOptionsValidator.ValidateDashboard` (`Configuration/GatewayOptionsValidator.cs:219-253`) never inspects `DisableLogin`, and the warning fires only on first `GatewayOptions` resolution (not guaranteed at process start).
|
||||
|
||||
**Impact.** One copied config flag turns the whole dashboard — API-key CRUD and worker Kill included — into an unauthenticated admin surface on a 0.0.0.0-bound port.
|
||||
|
||||
**Design.** Fail fast in production: in `ValidateDashboard`, when `IHostEnvironment.IsProduction()` and `DisableLogin` is true, emit a validator error so startup aborts like every other misconfiguration. Rejected alternative: silently forcing the flag off — hides operator intent and diverges config from behavior. The dev warning stays for non-production. Co-design with SEC-03 (cookie reconciliation) and SEC-23 (validator coverage) so the dashboard-security fixes land together.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/GatewayOptionsValidator.cs`: inject/accept the environment name (the validator already runs at startup; thread `IHostEnvironment` in, or pass an `isProduction` flag) and add the check in `ValidateDashboard`.
|
||||
- Tests: `GatewayOptionsValidatorTests` — `DisableLogin=true` + Production → invalid; + Development → valid.
|
||||
- Docs: `docs/GatewayConfiguration.md` DisableLogin row notes the production hard-stop.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayOptionsValidator"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-05 — Hub bearer tokens irrevocable for 30 min, carried in query string `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review SEC-5)* `Dashboard/HubTokenService.cs:29,44-52` mints a 30-minute data-protected token with no jti/revocation state and roles frozen at issue; `Dashboard/HubTokenAuthenticationHandler.cs:59-61` accepts `?access_token=` on the WebSocket upgrade. Logout (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:136-155`) clears the cookie but not outstanding tokens.
|
||||
|
||||
**Impact.** A token captured from an access log or proxy grants live snapshot/alarm/event access for up to 30 minutes after logout; role changes/revocations do not take effect until expiry.
|
||||
|
||||
**Design.** Two independent, low-cost mitigations: (1) shorten `TokenLifetime` to ~5 minutes — the factory refreshes per reconnect (`docs/GatewayDashboardDesign.md:497-499`), so the shorter window is transparent to clients and bounds exposure; (2) confirm no request-path logging captures query strings (Serilog request logging is not enabled — keep it off, or scrub `access_token` if it is ever added). Full server-side revocation (a jti denylist keyed on the data-protection payload) is the heavier option; recommend deferring it and relying on the short lifetime, since the tokens are already encrypted and single-purpose. State the open question: if per-session ACLs (SEC-25) land, tokens will need role/session binding and revocation becomes worthwhile.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/HubTokenService.cs:29`: `TokenLifetime = TimeSpan.FromMinutes(5)`.
|
||||
- Add an XML-doc note at `HubTokenAuthenticationHandler` that query-string carriage is the SignalR pattern and must not be request-logged.
|
||||
- Docs: `docs/GatewayDashboardDesign.md` token section — update the lifetime and the logout-vs-token caveat.
|
||||
- Tests: `HubTokenServiceTests` — token invalid after lifetime; issue/validate round-trip.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~HubTokenService"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-06 — LDAP plaintext-by-default with a committed service password `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review SEC-6)* `Configuration/LdapOptions.cs:49-61` defaults `Transport=None`, `AllowInsecure=true`, `ServiceAccountPassword="serviceaccount123"`; `appsettings.json:21-33` ships the same. `glauth.md:30,327` confirms dev LDAPS is disabled and binds send cleartext. The validator enforces the `Transport=None ⇒ AllowInsecure` consistency rule (`GatewayOptionsValidator.cs:82-85`) but nothing distinguishes dev from prod.
|
||||
|
||||
**Impact.** Every dashboard login sends the operator's password cleartext to `10.100.0.35:3893`, and a service-account credential is in source control.
|
||||
|
||||
**Design.** These are deliberate dev defaults (the shadow-options rationale at `LdapOptions.cs:20-28` is explicit that the shared library is secure-by-default). The fix is a production posture, not a default flip: (1) require `Transport=Ldaps`/`StartTls` + `AllowInsecure=false` in production deployment docs, and add an `IsProduction()` validator check (mirroring SEC-04) that rejects `Transport=None` in production; (2) move `ServiceAccountPassword` out of `appsettings.json` to env-var/secret configuration on the deployed hosts and rotate the committed value. LDAP-injection escaping is delegated to `ZB.MOM.WW.Auth.Ldap` (bind-then-search, `Dashboard/DashboardAuthenticator.cs:41-47`) and cannot be verified here — flag for review in the donor repo.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/GatewayOptionsValidator.cs`: in `ValidateLdap`, when Production and `Transport == None`, emit an error (co-locate with SEC-04's env plumbing).
|
||||
- Deployment: keep `serviceaccount123` only for local GLAuth dev; document env-var override (`MxGateway__Ldap__ServiceAccountPassword`) for the NSSM-wrapped hosts; rotate the dev credential's reuse.
|
||||
- Docs: `docs/GatewayConfiguration.md` Ldap section and a production hardening note referencing `glauth.md`.
|
||||
- Tests: `GatewayOptionsValidatorTests` — `Transport=None` + Production → invalid.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the validator test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-07 — `QueryActiveAlarmsRequest` missing from scope resolver; tests mislabeled `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review STA-1)* `Security/Authorization/GatewayGrpcScopeResolver.cs:15-29` has arms for `StreamAlarmsRequest` (→ `EventsRead`) and `AcknowledgeAlarmRequest` but **no** `QueryActiveAlarmsRequest` arm, so that RPC (`Grpc/MxAccessGatewayService.cs`, proto `mxaccess_gateway.proto`) falls to the `_ => GatewayScopes.Admin` default. The two `ServerStreamingServerHandler_QueryActiveAlarms…` tests construct `new StreamAlarmsRequest()` instead (`Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:322-359`), so they pass while testing the wrong message. `docs/Authorization.md`'s scope table omits the RPC.
|
||||
|
||||
**Impact.** Fail-closed (not a vulnerability) but a client holding `events:read` gets `PermissionDenied` demanding `admin` on `QueryActiveAlarms`, contradicting the stated design that alarm snapshot data shares the event surface.
|
||||
|
||||
**Design.** Add the missing arm `QueryActiveAlarmsRequest => GatewayScopes.EventsRead`, consistent with `StreamAlarmsRequest`. Fix both tests to use the real request type so they actually exercise the arm. Add the row to `docs/Authorization.md`. No parity or backward-compat concern — it only relaxes an over-strict requirement to the documented scope.
|
||||
|
||||
**Implementation.**
|
||||
- `Security/Authorization/GatewayGrpcScopeResolver.cs:22`: add `QueryActiveAlarmsRequest => GatewayScopes.EventsRead,` next to `StreamAlarmsRequest`.
|
||||
- `Tests/Security/Authorization/GatewayGrpcAuthorizationInterceptorTests.cs:322-359`: change `new StreamAlarmsRequest()` to `new QueryActiveAlarmsRequest()` in both `QueryActiveAlarms` tests; keep the `events:read`-passes / missing-scope-denies assertions.
|
||||
- Docs: `docs/Authorization.md` scope table — add `QueryActiveAlarms → events:read`; also `docs/Grpc.md` if it omits the RPC.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayGrpcAuthorizationInterceptor"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-08 — Per-RPC SQLite read + `last_used_utc` write; no verification cache `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review PERF-1)* The interceptor calls `IApiKeyVerifier.VerifyAsync` per call (`Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:69-71`); the verifier contract is find-by-id → hash → compare → `MarkKeyUsedAsync` per `docs/Authentication.md`, and the `last_used_utc` write "runs on every authenticated request". The interceptor also deserializes the constraints JSON per call (`Security/Authentication/GatewayApiKeyIdentityMapper.cs:41`).
|
||||
|
||||
**Impact.** WAL + busy-timeout keep it correct, but a per-call database **write** is the throughput ceiling on the hot path (bulk reads at high frequency are the primary workload), causing continuous WAL churn and making auth-store latency a per-RPC tail-latency contributor.
|
||||
|
||||
**Design.** Add a short-TTL (5–30 s) in-memory verification cache keyed by key id + presented-hash, invalidated on revoke/rotate — all mutations flow through in-process `ApiKeyAdminCommands`, so invalidation is a direct call, not a cross-process concern. Coalesce `last_used_utc` updates to at most one write per key per minute (track last-flushed-at in the cache entry). Cache the parsed constraints on the identity so the per-call JSON deserialize disappears. This is `L` effort and likely needs a small surface addition in `ZB.MOM.WW.Auth.ApiKeys` (a `MarkKeyUsed` batching hook or a verifier that returns the raw stored record so the gateway can cache it) — recommend coordinating with the donor library rather than shadowing its store. Open question: whether the cache lives gateway-side (wraps `IApiKeyVerifier`) or in the shared library; recommend gateway-side wrapper first (no library release dependency), keyed on a constant-time hash of the presented secret to avoid caching plaintext.
|
||||
|
||||
**Implementation.**
|
||||
- New gateway-side `CachingApiKeyVerifier : IApiKeyVerifier` decorator registered in `Security/Authentication/AuthStoreServiceCollectionExtensions.cs`; memory cache (`IMemoryCache`) with per-entry TTL; invalidation entry point called from `ApiKeyAdminCommands` on create/revoke/rotate.
|
||||
- Coalesce `MarkKeyUsedAsync`: only forward when the cached entry's last-marked timestamp is older than the coalesce window.
|
||||
- `GatewayApiKeyIdentityMapper.cs:41`: cache the deserialized constraints on the mapped identity.
|
||||
- Tests: `CachingApiKeyVerifierTests` — cache hit avoids store read; revoke invalidates; `last_used` coalesced to ≤1/min. Load-style assertion optional.
|
||||
- Docs: `docs/Authentication.md` and `docs/Metrics.md` — document the cache TTL/coalesce and any new config knob.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~CachingApiKeyVerifier"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-09 — Dashboard design-doc `GroupToRole` sample fails startup validation `Medium` · `P2`
|
||||
|
||||
**Finding.** *(review CON-1)* `docs/GatewayDashboardDesign.md:515-519` shows `"GroupToRole": { "GwAdmin": "Admin", … }`, but `GatewayOptionsValidator.cs:233-238` accepts only `DashboardRoles.Admin` = `"Administrator"` (`Dashboard/DashboardRoles.cs:14`) or `"Viewer"`. Live `appsettings.json:66-69` correctly uses `"Administrator"`. The doc also calls the role `Admin` in prose (lines 412-413, 434).
|
||||
|
||||
**Impact.** An operator copying the documented sample gets a fail-fast boot error.
|
||||
|
||||
**Design.** Doc-only fix: update the sample and prose to the canonical `Administrator` value (the rename is annotated in `glauth.md:79-83`). No code change — the validator is correct.
|
||||
|
||||
**Implementation.**
|
||||
- `docs/GatewayDashboardDesign.md`: change the `GroupToRole` sample values to `"Administrator"`/`"Viewer"` and align role prose.
|
||||
|
||||
**Verification.** Doc change; no build. Cross-check the sample against `Dashboard/DashboardRoles.cs`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-10 — No API-key expiry `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review UND-1)* The `api_keys` schema carries `created_utc`/`last_used_utc`/`revoked_utc` only (confirmed from the stray DB `sqlite_master` and `docs/Authentication.md`); no expiry field, no expiry check in the verification flow, no staleness surfacing beyond `LastUsedUtc`.
|
||||
|
||||
**Impact.** Keys live until an operator revokes; a leaked key is valid indefinitely.
|
||||
|
||||
**Design.** Add optional `expires_utc` to the shared `ZB.MOM.WW.Auth.ApiKeys` schema + verifier (an expired key verifies as failed, opaque to the client like every other auth failure). Surface age/staleness on the dashboard API Keys page. This is `L` because the store/verifier live in the donor package — coordinate a schema migration there; the gateway consumes it. Gateway-side work: expose an optional `--expires` on `apikey create`, display expiry/staleness in `DashboardApiKeyManagementService` summaries. Open question: default expiry (recommend none — opt-in — to preserve current behavior; backward-compatible).
|
||||
|
||||
**Implementation.**
|
||||
- Shared library: `expires_utc` column + migration + verifier check + `ApiKeyPepperUnavailableException`-style typing (see SEC-18).
|
||||
- Gateway: `Program.cs` apikey create option; `Dashboard/DashboardApiKeyManagementService.cs` summary projection; API Keys Razor page staleness badge.
|
||||
- Docs: `docs/Authentication.md` token lifecycle + `docs/Authorization.md` if scopes interplay.
|
||||
- Tests: verifier expiry test in the library; gateway summary test.
|
||||
|
||||
**Verification.** After the library bump: `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~DashboardApiKeyManagement"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-11 — No rate limiting or lockout on either auth surface `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review UND-2)* The gRPC interceptor verifies unconditionally per call (`GatewayGrpcAuthorizationInterceptor.cs:54-91`); `/auth/login` has no throttle (`Dashboard/DashboardEndpointRouteBuilderExtensions.cs:38-43,99-134`). The only brake is dev GLAuth's per-IP 3-fail lockout (`glauth.md:35`), absent in production AD and hazardous behind shared NAT.
|
||||
|
||||
**Impact.** Unbounded online guessing of API-key secrets (each guess costs a SQLite read — compounds with SEC-08) and unthrottled LDAP credential stuffing relayed to the directory.
|
||||
|
||||
**Design.** Add ASP.NET Core rate limiting on `/auth/login` (fixed-window per remote IP) and a cheap per-peer failure counter in front of `VerifyAsync` (sliding-window limiter keyed on peer + presented key id) that short-circuits with `ResourceExhausted`/`Unauthenticated` before the store read. Co-design with SEC-08 (both sit on the auth hot path; the cache and the limiter share the peer key). Recommend the built-in `Microsoft.AspNetCore.RateLimiting` middleware for the dashboard and an in-process counter (bounded LRU) for gRPC to avoid a dependency in the interceptor. Note the NAT caveat from `glauth.md:323-325`: prefer per-key-id over pure per-IP for the gRPC counter.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/DashboardEndpointRouteBuilderExtensions.cs`: apply a named rate-limiter policy to the login POST route.
|
||||
- `Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs` (or a decorator): per-peer failure counter checked before `VerifyAsync`; reset on success.
|
||||
- Config: new `MxGateway:Security:LoginRateLimit*` / `ApiKeyFailureLimit*` keys bound in `GatewayOptions`, validated.
|
||||
- Tests: interceptor test asserting Nth failed attempt short-circuits; dashboard integration test for 429 on burst.
|
||||
- Docs: `docs/GatewayConfiguration.md`, `docs/Authorization.md`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the interceptor + dashboard test filters.
|
||||
|
||||
---
|
||||
|
||||
## SEC-12 — Dashboard Close/Kill bypass the canonical audit store `Medium` · `P1`
|
||||
|
||||
**Finding.** *(review UND-3)* `Dashboard/DashboardSessionAdminService.cs:64-69,129-134` record close/kill via `ILogger` only, whereas API-key operations write `AuditEvent`s through `IAuditWriter` (`Dashboard/DashboardApiKeyManagementService.cs:242-264`).
|
||||
|
||||
**Impact.** Destructive operational actions (killing a worker mid-production) leave no durable, queryable audit row — invisible to the dashboard's recent-audit view and subject to log rotation.
|
||||
|
||||
**Design.** Emit `dashboard-close-session` / `dashboard-kill-worker` `AuditEvent`s through the existing `IAuditWriter`, mirroring the API-key pattern. Actor resolution helpers already exist in the service (`ResolveActor`, `ResolveRemoteAddress`). Keep the `ILogger` line too (operational tail). No new store — reuse `SqliteCanonicalAuditStore`.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/DashboardSessionAdminService.cs`: inject `IAuditWriter`; after a successful close/kill, write an `AuditEvent` with actor, session id, remote address, and `AlreadyClosed`. Failure paths write a denied/failed event as the API-key path does.
|
||||
- Tests: `DashboardSessionAdminServiceTests` — successful close/kill produces the expected `AuditEvent`; assert against a fake `IAuditWriter`.
|
||||
- Docs: `docs/Authorization.md` audit-event catalog — add the two action names.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~DashboardSessionAdmin"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-13 — Redactor's credential-command list omits secured-bulk variants `Low` · `—`
|
||||
|
||||
**Finding.** *(review SEC-7)* `Diagnostics/GatewayLogRedactor.cs:11-16` lists `AuthenticateUser`, `WriteSecured`, `WriteSecured2` but not `WriteSecuredBulk`/`WriteSecured2Bulk`, which exist as command kinds (`Security/Authorization/GatewayGrpcScopeResolver.cs:41-45`). Currently latent — `RedactCommandValue`/`IsCredentialBearingCommand` have no call sites (grep-verified), so no value logging occurs.
|
||||
|
||||
**Impact.** If value logging is ever wired (SEC-30), secured-bulk payloads (credential-bearing) would pass the unconditional-redaction check.
|
||||
|
||||
**Design.** Add the two bulk names now — trivial and forward-safe. Add a test asserting every `WriteSecured*` command kind is treated as credential-bearing, so the redactor cannot drift from the command catalog again.
|
||||
|
||||
**Implementation.**
|
||||
- `Diagnostics/GatewayLogRedactor.cs:13-15`: add `"WriteSecuredBulk"`, `"WriteSecured2Bulk"` to `SensitiveCommandMethods`.
|
||||
- Tests: `GatewayLogRedactorTests` — enumerate `MxCommandKind` names starting with `WriteSecured` and assert `IsCredentialBearingCommand` is true for each.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayLogRedactor"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-14 — `/metrics` + `/health` unauthenticated; a metric leaks session ids `Low` · `—`
|
||||
|
||||
**Finding.** *(review SEC-8)* `GatewayApplication.cs:196-197` maps `MapZbHealth()`/`MapZbMetrics()` (shared packages) with no `RequireAuthorization`; `Metrics/GatewayMetrics.cs:354` tags `mxgateway.heartbeats.failed` with `session_id`.
|
||||
|
||||
**Impact.** If the shared packages don't enforce auth internally (not verifiable here), an unauthenticated scraper on the gRPC/dashboard port can read telemetry including live session identifiers usable against the anonymous-localhost hub surface (SEC-02).
|
||||
|
||||
**Design.** Verify the shared packages' endpoint auth; if anonymous, either bind metrics to a loopback-only endpoint or gate with `RequireAuthorization`. Combine with SEC-20 (drop the `session_id` tag) so the leak closes even if the endpoint stays open. Depends-on SEC-02 (hub surface) and SEC-20 (tag).
|
||||
|
||||
**Implementation.**
|
||||
- Confirm `MapZbMetrics`/`MapZbHealth` auth behavior in the shared package; if open, add `.RequireAuthorization()` or bind to a separate loopback Kestrel endpoint.
|
||||
- Docs: `docs/Metrics.md`/`docs/Diagnostics.md` state the endpoint's auth posture.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; manual `curl` of `/metrics` on a running gateway to confirm the posture.
|
||||
|
||||
---
|
||||
|
||||
## SEC-15 — GET `/logout` skips antiforgery `Low` · `—`
|
||||
|
||||
**Finding.** *(review SEC-9)* `Dashboard/DashboardEndpointRouteBuilderExtensions.cs:53-58` intentionally skips antiforgery on GET `/logout` (comment acknowledges it). POST login/logout validate antiforgery (`:104,140`).
|
||||
|
||||
**Impact.** A third-party page can sign a dashboard operator out (nuisance CSRF; no state beyond the session).
|
||||
|
||||
**Design.** Acceptable as documented — logout is self-destructive with no side effects. No code change; if logout ever grows side effects, add a confirmation interstitial. Recorded so the tracking doc captures the accepted risk.
|
||||
|
||||
**Implementation.** None (documented acceptance). Optionally add a one-line note to `docs/GatewayDashboardDesign.md`.
|
||||
|
||||
**Verification.** N/A.
|
||||
|
||||
---
|
||||
|
||||
## SEC-16 — CLI accepts the pepper as a command-line argument `Low` · `—`
|
||||
|
||||
**Finding.** *(review SEC-10)* `Program.cs:37-40` maps `command.Pepper` → `MxGateway:ApiKeyPepper`.
|
||||
|
||||
**Impact.** The pepper lands in shell history and the process command line visible to other local users.
|
||||
|
||||
**Design.** Prefer an environment variable (`MxGateway__ApiKeyPepper`) or an interactive prompt over the CLI arg; keep the arg for scripted use but document the exposure in `--help`. Low-risk, small change.
|
||||
|
||||
**Implementation.**
|
||||
- `Program.cs`: fall back to the `MxGateway__ApiKeyPepper` env var when `--pepper` is absent; note the risk in the apikey subcommand help text.
|
||||
- Docs: CLAUDE.md apikey example already uses scopes; add a pepper-via-env note in `docs/Authentication.md`.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; run `apikey --help`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-17 — Interceptor does not override client-streaming/duplex handlers `Low` · `—`
|
||||
|
||||
**Finding.** *(review STA-2)* `Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs:21-47` overrides only `UnaryServerHandler` and `ServerStreamingServerHandler`. All current RPCs are unary or server-streaming (verified against both `.proto` files), so nothing bypasses auth today.
|
||||
|
||||
**Impact.** A future duplex/client-streaming RPC would run with no authentication and no scope check, silently.
|
||||
|
||||
**Design.** Override `ClientStreamingServerHandler` and `DuplexStreamingServerHandler` to run the same auth path (fail-closed to `admin` via the resolver default) — a loud, safe default so any future streaming RPC is authenticated by construction rather than accidentally open.
|
||||
|
||||
**Implementation.**
|
||||
- `Security/Authorization/GatewayGrpcAuthorizationInterceptor.cs`: add the two overrides calling `AuthenticateAndAuthorizeAsync` then the continuation, mirroring the existing two.
|
||||
- Tests: `GatewayGrpcAuthorizationInterceptorTests` — a client-streaming/duplex handler is denied without a key.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the interceptor test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-18 — Pepper-unavailable detection matches library message text `Low` · `—`
|
||||
|
||||
**Finding.** *(review STA-3)* `Dashboard/DashboardApiKeyManagementService.cs:21,281-282` catches `InvalidOperationException` whose `Message` contains `"pepper unavailable"`.
|
||||
|
||||
**Impact.** A wording change in `ZB.MOM.WW.Auth.ApiKeys` turns the friendly "pepper not configured" result into an unhandled exception on the Blazor circuit.
|
||||
|
||||
**Design.** Ask the library to expose a typed `ApiKeyPepperUnavailableException` (the pre-cutover code had one per `docs/Authentication.md:68`) and catch that instead of string-matching. Coordinate with SEC-10 (also a library change). Until the library ships it, keep the string match but add a test pinning the expected message so drift is caught in CI.
|
||||
|
||||
**Implementation.**
|
||||
- Shared library: add/restore the typed exception.
|
||||
- `Dashboard/DashboardApiKeyManagementService.cs`: catch the typed exception; drop the message match.
|
||||
- Tests: `DashboardApiKeyManagementServiceTests` — pepper-unavailable yields the friendly result, not an unhandled throw.
|
||||
|
||||
**Verification.** After library bump: `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the management-service test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-19 — Canonical audit store re-issues `CREATE TABLE` per write/read `Low` · `—`
|
||||
|
||||
**Finding.** *(review STA-4)* `Security/Audit/SqliteCanonicalAuditStore.cs:54,94,131-136` runs `CREATE TABLE IF NOT EXISTS audit_event` on every write and read.
|
||||
|
||||
**Impact.** An extra round-trip per audit op, and a schema defined outside the migrator — a future column change has no migration path (the `IF NOT EXISTS` silently keeps the old shape).
|
||||
|
||||
**Design.** Move `audit_event` creation into the startup migration path (alongside the api-key schema migrator) and drop the per-call ensure. Ties to UND-5/SEC-26 (retention) — do both in the audit-store pass.
|
||||
|
||||
**Implementation.**
|
||||
- `Security/Audit/SqliteCanonicalAuditStore.cs`: remove per-call `CREATE TABLE`; ensure the migration hosted service creates `audit_event`.
|
||||
- Tests: `SqliteCanonicalAuditStoreTests` — write/read succeed post-migration; no `CREATE TABLE` on the hot path (assert via a command-counting fake or integration check).
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~CanonicalAudit"`.
|
||||
|
||||
---
|
||||
|
||||
## SEC-20 — `session_id` metric tag is unbounded cardinality `Low` · `P1`
|
||||
|
||||
**Finding.** *(review PERF-2)* `Metrics/GatewayMetrics.cs:354` tags `mxgateway.heartbeats.failed` with `session_id`; `docs/Metrics.md:47` documents it. The in-memory `EventsBySession` map is correctly pruned on close (`:310-313`) — only the exported tag is the problem.
|
||||
|
||||
**Impact.** Every session ever opened mints a new exporter time series; long-running gateways with churn bloat Prometheus/OTLP storage.
|
||||
|
||||
**Design.** Drop the `session_id` tag from the exported counter (keep the aggregate). Per-session attribution stays available via the dashboard snapshot and log scope. Co-design with SEC-08 (both land in the observability pass) and closes the leak flagged in SEC-14.
|
||||
|
||||
**Implementation.**
|
||||
- `Metrics/GatewayMetrics.cs:354`: `_heartbeatFailuresCounter.Add(1)` without the tag.
|
||||
- Docs: `docs/Metrics.md:47` — remove the `session_id` dimension.
|
||||
- Tests: `GatewayMetricsTests` (if present) — counter increments without a `session_id` tag.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the metrics test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-21 — Snapshot publisher works every second regardless of audience `Low` · `—`
|
||||
|
||||
**Finding.** *(review PERF-3)* `Dashboard/DashboardSnapshotService.cs:134-166` runs the publisher loop; `:256-279` (`RefreshApiKeySummariesAsync`) calls `IApiKeyAdminStore.ListAsync` every tick (gated, 2 s-timeboxed); `:76-105` rebuilds session/worker/metric/fault/config projections each tick. The Galaxy breakdown is properly memoized by sequence (`:107-129`).
|
||||
|
||||
**Impact.** Constant background SQLite reads and allocation churn on an idle gateway; the full snapshot (API-key summaries + effective config) is serialized to all hub clients every second.
|
||||
|
||||
**Design.** Refresh the API-key summary on a slower cadence or on mutation (all mutations are in-process via `ApiKeyAdminCommands`, so an invalidation hook is exact — shares the mechanism with SEC-08's cache invalidation). Skip publication when the hub has no connections. Keep the 1 s cadence for live session/metric projections.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/DashboardSnapshotService.cs`: gate `RefreshApiKeySummariesAsync` behind a mutation flag (set by `ApiKeyAdminCommands`) or a longer interval; short-circuit the publish when `IHubContext` has zero clients.
|
||||
- Tests: `DashboardSnapshotServiceTests` — key summary refreshes on mutation, not every tick; no publish with zero connections.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the snapshot-service test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-22 — `docs/Authentication.md` documents types no longer in this repo `Low` · `P2`
|
||||
|
||||
**Finding.** *(review CON-2)* The doc presents `ApiKeyParser`, `ApiKeySecretGenerator`, `SqliteApiKeyStore`, and a parameterless `AddSqliteAuthStore()` (`docs/Authentication.md:9-28,36-48,253-272`), but the real registration is the package-delegating two-parameter method (`Security/Authentication/AuthStoreServiceCollectionExtensions.cs:41-105`) and the store/verifier live in `ZB.MOM.WW.Auth.ApiKeys`. The doc also says `api_key_audit` is written on every denial (`:122,131`), but the override redirects all writes to `audit_event`, leaving `api_key_audit` unused (`Security/Audit/CanonicalForwardingApiKeyAuditStore.cs:21-25`).
|
||||
|
||||
**Impact.** Violates the "no stale prose" rule; a maintainer auditing hashing/storage from the doc looks for code that isn't there.
|
||||
|
||||
**Design.** Rewrite `docs/Authentication.md` as a consumer-side doc: token format, options binding, the audit-store override (`api_key_audit` → `audit_event`), and a pointer to the donor library for internals. Fold in SEC-10/SEC-18 (typed exception, expiry) as they land.
|
||||
|
||||
**Implementation.**
|
||||
- `docs/Authentication.md`: rewrite per above; remove code excerpts of moved types.
|
||||
|
||||
**Verification.** Doc change; cross-check each referenced type against the current tree.
|
||||
|
||||
---
|
||||
|
||||
## SEC-23 — Validator ignores `DisableLogin`/`AutoLoginUser`/`RequireHttpsCookie`/`CookieName` `Low` · `—`
|
||||
|
||||
**Finding.** *(review CON-4)* `Configuration/GatewayOptionsValidator.cs:219-253` validates `GroupToRole`, snapshot interval, and the two limits, but ignores `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie`, and `CookieName` (all documented at `docs/GatewayConfiguration.md:169-177`). A `CookieName` beginning `__Host-` with `RequireHttpsCookie=false` yields a cookie browsers silently drop — no startup diagnosis.
|
||||
|
||||
**Impact.** Silent misconfiguration classes the validator exists to prevent.
|
||||
|
||||
**Design.** Add a `__Host-`/`RequireHttpsCookie` consistency check (reject `CookieName` starting `__Host-` unless `RequireHttpsCookie` is true) and the SEC-04 production guard for `DisableLogin`; validate `AutoLoginUser` semantics. Co-design with SEC-03 (which introduces the `__Host-` default) and SEC-04.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/GatewayOptionsValidator.cs` `ValidateDashboard`: add the checks.
|
||||
- Tests: `GatewayOptionsValidatorTests` — `__Host-` + `RequireHttpsCookie=false` → invalid.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the validator test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-24 — Effective-config view omits the riskiest dashboard flags `Low` · `—`
|
||||
|
||||
**Finding.** *(review CON-5)* `Configuration/GatewayConfigurationProvider.cs:57-64` projects `EffectiveDashboardConfiguration` without `DisableLogin`, `AutoLoginUser`, `RequireHttpsCookie`, or `CookieName`; TLS and Alarms sections are absent from `EffectiveGatewayConfiguration` entirely.
|
||||
|
||||
**Impact.** The Settings page cannot show an operator that login is disabled — the one flag they most need to see.
|
||||
|
||||
**Design.** Add the missing (non-secret) fields to the effective-config projection so the dashboard surfaces them. Depends-on SEC-04 (same flag). Optionally add TLS/Alarms sections; keep secrets redacted as the provider already does for the pepper/LDAP password.
|
||||
|
||||
**Implementation.**
|
||||
- `Configuration/GatewayConfigurationProvider.cs`: extend `EffectiveDashboardConfiguration` with the four flags; consider TLS/Alarms sections.
|
||||
- Tests: `GatewayConfigurationProviderTests` — effective config includes `DisableLogin` etc.; secrets still redacted.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the provider test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-25 — Per-session EventsHub ACL is an acknowledged TODO `Low` · `P2`
|
||||
|
||||
**Finding.** *(review UND-4)* `Dashboard/Hubs/EventsHub.cs:29-44` carries `TODO(per-session-acl)` — any Viewer (and anonymous localhost per SEC-02) can `SubscribeSession` to any session's raw event feed, bypassing the per-gRPC-subscriber filtering (`docs/GatewayDashboardDesign.md:170`).
|
||||
|
||||
**Impact.** Acceptable per the in-code v1 rationale, but it is the single seam where tag values reach the least-privileged principals; combined with `ShowTagValues=false` expectations it can surprise operators.
|
||||
|
||||
**Design.** Keep the TODO tied to the tracked per-session-ACL work item (P2 roadmap item 12). As a near-term hardening, redact event values in the dashboard mirror when `ShowTagValues` is false, so the seam cannot leak values regardless of the ACL. Depends-on SEC-02 (loopback reaching hubs). Full ACL: introduce a role/scope that scopes a Viewer to a session/tenant and enforce it at `SubscribeSession`.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/Hubs/EventsHub.cs` / `DashboardEventBroadcaster`: honor `ShowTagValues` by redacting values in the mirror now; add the session-access check when the scoping role lands.
|
||||
- Tests: broadcaster test asserting values redacted when `ShowTagValues=false`.
|
||||
- Docs: `docs/GatewayDashboardDesign.md` — clarify the current v1 posture.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the events-hub/broadcaster test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-26 — Audit trail has no retention or pruning `Low` · `—`
|
||||
|
||||
**Finding.** *(review UND-5)* `audit_event` is append-only with no cleanup path (`Security/Audit/SqliteCanonicalAuditStore.cs`); constraint denials append per denied bulk entry (`docs/Authorization.md:194-198`).
|
||||
|
||||
**Impact.** A misconfigured constrained client hammering denied reads grows the auth DB without bound.
|
||||
|
||||
**Design.** Add a retention sweep (age- or row-count-based, config-gated) run by a hosted service, or document the operational archive expectation. Recommend an age-based sweep with a generous default (e.g. 90 days) so the DB is self-limiting. Do this in the same audit-store pass as SEC-19.
|
||||
|
||||
**Implementation.**
|
||||
- `Security/Audit/`: add a retention sweep (hosted service or piggyback the migration service) with `MxGateway:Security:AuditRetentionDays` config.
|
||||
- Tests: sweep deletes rows older than the window; keeps recent.
|
||||
- Docs: `docs/Authorization.md`/`docs/Diagnostics.md` retention note; `docs/GatewayConfiguration.md` new key.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the audit test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-27 — Dashboard `GatewayStatus` is hardcoded Healthy `Low` · `—`
|
||||
|
||||
**Finding.** *(review UND-6)* `Dashboard/DashboardSnapshotService.cs:17,96` sets `GatewayStatus` to a `HealthyStatus` constant.
|
||||
|
||||
**Impact.** The home page headline status never reflects registered health checks (e.g. `AuthStoreHealthCheck` unhealthy while the banner says Healthy).
|
||||
|
||||
**Design.** Project `HealthCheckService` results into the snapshot so the banner reflects real health. Inject `HealthCheckService`, run/observe the aggregated status on the publish tick (cheap — health checks are cached), map to the dashboard status enum.
|
||||
|
||||
**Implementation.**
|
||||
- `Dashboard/DashboardSnapshotService.cs`: inject `HealthCheckService`; set `GatewayStatus` from the aggregated `HealthReport`.
|
||||
- Tests: `DashboardSnapshotServiceTests` — unhealthy check surfaces as non-Healthy status.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server` and the snapshot-service test filter.
|
||||
|
||||
---
|
||||
|
||||
## SEC-28 — Positive security observations (preserve under change) `Info` · `—`
|
||||
|
||||
**Finding.** *(review SEC-11)* Verified-good behaviors to preserve: fail-closed unrecognized-request → `admin` (`GatewayGrpcScopeResolver.cs:28`); opaque auth failures (`GatewayGrpcAuthorizationInterceptor.cs:66-78`); effective-config redacts pepper name + LDAP password (`GatewayConfigurationProvider.cs`); Galaxy connection string rebuilt field-by-field for display (`Dashboard/DashboardConnectionStringDisplay.cs`); global identity-property masking seam (`Diagnostics/GatewayLogRedactorSeam.cs`); self-signed PFX generation hardens permissions before writing and clears the buffer (`Security/Tls/SelfSignedCertificateProvider.cs:160-193`); scope strings validated against the canonical catalog on every creation path; dashboard key-id input constrained to a safe charset; single generic login-failure message; `SanitizeReturnUrl` blocks open redirects. No secret-logging call site found.
|
||||
|
||||
**Impact.** These are the load-bearing controls; regressions here are high-severity.
|
||||
|
||||
**Design.** No action. Guard against regression: the tests named in other entries (redactor, scope catalog, opaque-failure) already pin most of these — keep them.
|
||||
|
||||
**Implementation.** None. Reference in review sign-off.
|
||||
|
||||
**Verification.** N/A.
|
||||
|
||||
---
|
||||
|
||||
## SEC-29 — UI-stack rule verified compliant `Info` · `—`
|
||||
|
||||
**Finding.** *(review CON-3)* `wwwroot/lib/` contains only local Bootstrap CSS/JS; grep for MudBlazor/Radzen/Syncfusion/Telerik across `.csproj`/`.razor`/`.cs` returns nothing. The shared `ZB.MOM.WW.Theme` package supplies CSS only.
|
||||
|
||||
**Impact.** Confirms the CLAUDE.md "no Blazor UI component libraries" invariant holds.
|
||||
|
||||
**Design.** No action. Optionally add a build-time or test guard that fails if a banned package reference appears, to keep the invariant enforced rather than remembered.
|
||||
|
||||
**Implementation.** Optional repo-hygiene test asserting no MudBlazor/Radzen/FluentUI/Telerik/Syncfusion package references.
|
||||
|
||||
**Verification.** N/A (or run the hygiene test if added).
|
||||
|
||||
---
|
||||
|
||||
## SEC-30 — Value-logging feature is unwired end-to-end `Info` · `P2`
|
||||
|
||||
**Finding.** *(review UND-7)* `GatewayLogRedactor.RedactCommandValue`/`IsCredentialBearingCommand` have no call sites in the server (grep-verified); the opt-in value-logging flag described in `docs/Diagnostics.md:124-148` has no knob in `GatewayOptions`.
|
||||
|
||||
**Impact.** Currently the safest state (no values logged anywhere), but the doc implies a capability that doesn't exist end-to-end.
|
||||
|
||||
**Design.** Either wire the flag (add a `MxGateway:Diagnostics:LogCommandValues` option, thread it to the redactor call sites, default off) or trim the doc to match. Recommend trimming the doc for now and deferring wiring; if wiring proceeds, land SEC-13 (bulk-secured names) first so secured payloads are covered. Depends-on SEC-13.
|
||||
|
||||
**Implementation.**
|
||||
- If trimming: `docs/Diagnostics.md` — mark value logging as not-yet-implemented.
|
||||
- If wiring: add the option + validation, call `RedactCommandValue` at the command-log site with the flag, tests asserting credential-bearing commands stay redacted even with the flag on.
|
||||
|
||||
**Verification.** If wired: `dotnet build src/ZB.MOM.WW.MxGateway.Server` and `dotnet test ...MxGateway.Tests --filter "FullyQualifiedName~GatewayLogRedactor"`. Otherwise doc-only.
|
||||
@@ -0,0 +1,623 @@
|
||||
# Language Clients — Remediation Design & Implementation
|
||||
|
||||
Source review: [50-clients.md](../50-clients.md) · Generated: 2026-07-09
|
||||
|
||||
This document turns the Language Clients review into buildable remediation entries for all five clients (`clients/dotnet`, `clients/go`, `clients/java`, `clients/python`, `clients/rust`). Every finding carries the target language explicitly. The JDK 17 retarget is verified correct in build config (J1 in the source report is a no-defect verification and gets no remediation row); its only fallout is stale docs (CLI-12). Two operational constraints from prior work carry into these fixes: the Java client cannot build or test on the macOS tree (no local JRE — build/test on windev), and the Java Gradle build regenerates a tracked ~64k-line `MxaccessGateway.java` whose spurious protobuf-version churn must be reverted (`git checkout`) whenever no `.proto` changed. Python regen must pin `grpcio-tools` to the baseline (grpcio 1.80.0 / protobuf 6.31.1); the Rust `build.rs` proto path and `--no-verify` packaging are themselves a finding (CLI-02).
|
||||
|
||||
## Finding index
|
||||
|
||||
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||
|----|-----|-------|---------|--------|-----------|-------|
|
||||
| CLI-01 | High | Go `Session.Events()` silently closes stream on 16-slot overflow | P0 | M | — | clients/go/mxgateway/session.go |
|
||||
| CLI-02 | High | Rust crate unbuildable outside the repo (`build.rs` path + `--no-verify`) | P1 | M | — | clients/rust/build.rs, clients/rust/Cargo.toml, scripts/pack-clients.ps1 |
|
||||
| CLI-03 | High | Rust `invoke` never validates HRESULT / MXSTATUS_PROXY | P0 | M | — | clients/rust/src/error.rs, clients/rust/src/client.rs |
|
||||
| CLI-04 | High | Typed-command parity gap across all five clients (WriteSecured/AuthenticateUser/AdviseSupervisory/buffered) | P2 | L | CLI-15 | clients/{dotnet,go,java,python,rust} session surfaces |
|
||||
| CLI-05 | Medium | .NET session cannot be re-attached to an existing session id | — | S | — | clients/dotnet/.../MxGatewaySession.cs, MxGatewayClient.cs |
|
||||
| CLI-06 | Medium | .NET `DisposeAsync` blocks/throws on unreachable gateway | — | S | — | clients/dotnet/.../MxGatewaySession.cs |
|
||||
| CLI-07 | Medium | .NET retry budget self-defeats on `DeadlineExceeded` | — | S | — | clients/dotnet/.../MxGatewayClient.cs, MxGatewayClientRetryPolicy.cs |
|
||||
| CLI-08 | Medium | .NET/Go/Java treat any nonzero HRESULT as failure (should be `< 0`) | — | S | CLI-03 | clients/dotnet/.../MxCommandReplyExtensions.cs, clients/go/mxgateway/errors.go, clients/java/.../MxGatewayErrors.java |
|
||||
| CLI-09 | Medium | Go has no typed auth-error mapping (Unauthenticated vs PermissionDenied) | — | M | — | clients/go/mxgateway/errors.go, client.go |
|
||||
| CLI-10 | Medium | Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` | — | M | — | clients/go/mxgateway/client.go |
|
||||
| CLI-11 | Medium | Go CLI cannot opt into strict TLS validation | — | S | — | clients/go/cmd/mxgw-go/main.go |
|
||||
| CLI-12 | Medium | Java docs still say "Java 21" after the JDK 17 retarget | P2 | S | — | clients/java/README.md, clients/java/JavaClientDesign.md, docs/ClientPackaging.md |
|
||||
| CLI-13 | Medium | Java event-stream buffer hardcoded 16, cancel-on-overflow | — | M | — | clients/java/.../MxGatewayClient.java, MxEventStream.java |
|
||||
| CLI-14 | Medium | Python default TLS is blocking TOFU pin with silent `localhost` SNI | — | S | — | clients/python/.../options.py, clients/python/README |
|
||||
| CLI-15 | Medium | No client handles `ReplayGap` or offers a reconnect helper | P2 | M | — | clients/{dotnet,go,java,python,rust} + READMEs |
|
||||
| CLI-16 | Medium | `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` | P2 | S | CLI-12 | docs/ClientPackaging.md, docs/ClientLibrariesDesign.md |
|
||||
| CLI-17 | Medium | TLS default posture inconsistent across the five clients | — | M | CLI-14 | clients/{dotnet,go,java,python,rust} TLS paths |
|
||||
| CLI-18 | Low | .NET csproj records no `<Version>` | P2 | S | — | clients/dotnet/.../ZB.MOM.WW.MxGateway.Client.csproj |
|
||||
| CLI-19 | Low | .NET duplicate `InternalsVisibleTo` (AssemblyInfo + csproj) | — | S | — | clients/dotnet/.../Properties/AssemblyInfo.cs, csproj |
|
||||
| CLI-20 | Low | .NET `--tls` without CA installs accept-all callback | — | S | CLI-17 | clients/dotnet/.../MxGatewayClient.cs |
|
||||
| CLI-21 | Low | Go `ClientVersion = "0.1.0-dev"` stale vs tagged releases | P2 | S | — | clients/go/mxgateway/version.go, scripts/tag-go-module.ps1 |
|
||||
| CLI-22 | Low | Go `newCorrelationID` swallows `crypto/rand` error → empty id | — | S | — | clients/go/mxgateway/session.go |
|
||||
| CLI-23 | Low | Go nil-vs-empty bulk short-circuit asymmetry | — | S | — | clients/go/mxgateway/session.go |
|
||||
| CLI-24 | Low | Java `MxEventStream` single-consumer constraint undocumented | — | S | — | clients/java/.../MxEventStream.java |
|
||||
| CLI-25 | Low | Java `close()` does not await channel termination | — | S | — | clients/java/.../MxGatewayClient.java |
|
||||
| CLI-26 | Low | Python `version.py` (0.1.0) ≠ `pyproject.toml` (0.1.2) | P2 | S | — | clients/python/.../version.py, pyproject.toml |
|
||||
| CLI-27 | Low | Python `Session.close()` not concurrency-safe; synthesizes reply | — | S | — | clients/python/.../session.py |
|
||||
| CLI-28 | Low | Python circular-import workaround at end of `session.py` | — | S | — | clients/python/.../session.py |
|
||||
| CLI-29 | Low | Rust `CLIENT_VERSION` (0.1.0-dev) ≠ `Cargo.toml` (0.1.2) | P2 | S | — | clients/rust/src/version.rs |
|
||||
| CLI-30 | Low | Rust has no `unregister` typed helper | — | S | CLI-04 | clients/rust/src/session.rs |
|
||||
| CLI-31 | Low | Rust CLI is a single 2,699-line `main.rs` | — | M | — | clients/rust/crates/mxgw-cli/src/main.rs |
|
||||
| CLI-32 | Low | Client-side bulk caps differ (.NET/Java unbounded) | — | S | — | clients/dotnet, clients/java session surfaces |
|
||||
| CLI-33 | Low | Per-language event backpressure semantics undocumented | — | S | CLI-01, CLI-13 | docs/ClientBehaviorFixtures.md |
|
||||
| CLI-34 | Low | Python `build/`/`.pytest_cache/` present on disk (untracked) | — | S | — | clients/python/.gitignore |
|
||||
|
||||
---
|
||||
|
||||
## CLI-01 — Go `Session.Events()` silently closes stream on 16-slot overflow `High` · `P0`
|
||||
|
||||
**Finding.** In the Go client, `subscribeEventsAfter` (`clients/go/mxgateway/session.go:700-742`) spawns a goroutine that pushes each event through `sendEventResult` into a 16-slot channel. When the buffer is full and `cancelWhenBufferFull` is set (the `Events()`/`EventsAfter()` path), the `default:` arm cancels the stream and returns `false` with **no terminal error enqueued** (`session.go:751-768`); the goroutine then `close(results)` via `defer` (`:714`). The doc comment promises the stream runs "until … a terminal error is sent" (`:675-677`) but never mentions this drop.
|
||||
|
||||
**Impact.** A consumer that stalls for more than 16 queued events sees a closed channel indistinguishable from graceful server end. Events are lost with no signal — silent data loss, the single most serious runtime defect in the client tree. This is roadmap P0 item 6.
|
||||
|
||||
**Design.** Reserve one slot for a terminal `EventResult` so overflow is always observable. Add an exported sentinel `ErrSlowConsumer` and, on the overflow branch, cancel the stream and attempt a **non-blocking** send of `EventResult{Err: ...wrapping ErrSlowConsumer}`; the reserved capacity guarantees that final send succeeds even when the 16 data slots are full. Keep the blocking `SubscribeEvents` path (which rides gRPC flow control) unchanged. Rejected alternative: dropping the buffered-cancel variant entirely — it is public API and some callers depend on non-blocking semantics; a loud terminal error preserves the contract while removing the silent-loss failure mode.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/go/mxgateway/errors.go`: add `var ErrSlowConsumer = errors.New("mxgateway: event consumer fell behind; stream terminated")`.
|
||||
- `clients/go/mxgateway/session.go`: size the channel to `16+1` (or track a reserved terminal slot); in `sendEventResult`'s `default:` arm, call `cancel()` then a non-blocking `select { case results <- EventResult{Err: &GatewayError{Op: "stream events", Err: ErrSlowConsumer}}: default: }` before returning `false`. Update the `Events`/`EventsAfter` doc comments to state the slow-consumer termination contract.
|
||||
- Tests: extend the session event tests (`clients/go/mxgateway/*_test.go`, the fixture-driven event suite) with a slow-consumer case asserting the channel yields a final `EventResult` whose `errors.Is(res.Err, ErrSlowConsumer)` is true before close.
|
||||
- Docs: record the Go slow-consumer terminal-error behavior in `docs/ClientBehaviorFixtures.md` (see CLI-33).
|
||||
|
||||
**Verification.** From `clients/go`: `gofmt -l .`, `go build ./...`, `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-02 — Rust crate unbuildable outside the repository `High` · `P1`
|
||||
|
||||
**Finding.** The Rust `build.rs` resolves protos two directories above the crate — `repo_root.join("src/ZB.MOM.WW.MxGateway.Contracts/Protos")` with the hard assumption "clients/rust must live two levels below the repository root" (`clients/rust/build.rs:8-16`) — and `src/generated.rs` is only `tonic::include_proto!` of that build output. `Cargo.toml` does not vendor the `.proto` files. The packaging script hides the defect with `cargo package --no-verify` / `cargo publish --no-verify` (`scripts/pack-clients.ps1:190-211`). Any consumer of the published `zb-mom-ww-mxgateway-client 0.1.2` fails in `build.rs` on first `cargo build`.
|
||||
|
||||
**Impact.** The Gitea-published crate is dead on arrival for every external consumer — the most serious packaging finding. This is roadmap P1 item 11.
|
||||
|
||||
**Design.** Vendor the three protos into the crate and make `build.rs` prefer the in-repo source but fall back to the vendored copy, then drop `--no-verify` so `cargo package` proves standalone buildability. Copy `mxaccess_gateway.proto`, `mxaccess_worker.proto`, `galaxy_repository.proto` into `clients/rust/protos/` and list that dir in `Cargo.toml`'s `include`. In `build.rs`, probe the repo path first (keeps in-repo edits live) and fall back to `CARGO_MANIFEST_DIR/protos` when the repo path is absent. The vendored copies are build inputs, not the canonical source (which remains in Contracts) — add a note so they are refreshed on `.proto` change. Rejected alternative: publishing pre-generated `.rs` — tonic's generated code is tied to the tonic/prost version and would rot; regenerating from vendored protos at build time is the idiomatic Rust approach.
|
||||
|
||||
**Implementation.**
|
||||
- Add `clients/rust/protos/{mxaccess_gateway,mxaccess_worker,galaxy_repository}.proto` (copies).
|
||||
- `clients/rust/build.rs`: `let proto_root = if repo_proto_root.exists() { repo_proto_root } else { manifest_dir.join("protos") };` and compile from whichever exists; keep the descriptor-set output.
|
||||
- `clients/rust/Cargo.toml`: add `include = ["src/**/*.rs", "protos/*.proto", "build.rs", ...]` so the protos ship in the package.
|
||||
- `scripts/pack-clients.ps1:190-211`: remove `--no-verify` from the Rust `cargo package`/`cargo publish` invocations so verification runs.
|
||||
- Optional: a small `build.rs` staleness note or a repo script that copies Contracts protos → `clients/rust/protos/` to keep them in sync (tie into the codegen-freshness theme of report 30/60).
|
||||
- Docs: update the Rust section of `docs/ClientPackaging.md` and `clients/rust/README.md` to describe the vendored-proto layout.
|
||||
|
||||
**Verification.** From `clients/rust`: `cargo fmt`, `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`, and crucially `cargo package` (no `--no-verify`) to prove the crate builds from the packaged tarball.
|
||||
|
||||
---
|
||||
|
||||
## CLI-03 — Rust `invoke` never validates HRESULT / MXSTATUS_PROXY `High` · `P0`
|
||||
|
||||
**Finding.** In the Rust client, `ensure_command_success` checks only `protocol_status.code == Ok` (`clients/rust/src/error.rs:214-226`) and is the sole reply check on the `invoke` path (`clients/rust/src/client.rs:177-179`). It never inspects `hresult` or the `MXSTATUS_PROXY` array. Every other client performs a second MXAccess-level check (`MxCommandReplyExtensions.cs:27-41`, `errors.go:117-130`, `MxGatewayErrors.java`, `errors.py:122-148`).
|
||||
|
||||
**Impact.** A reply with an OK protocol envelope but failing per-item MXAccess statuses reads as success in Rust — a `session.write(...)` can report success while MXAccess rejected the write. There is also no distinct MXAccess error variant to catch. Violates MXAccess-parity-is-the-contract. Roadmap P0 item 6.
|
||||
|
||||
**Design.** Add an `ensure_mxaccess_success` pass mirroring the other clients and an `Error::MxAccess` variant carrying the reply, and call it after `ensure_command_success` on the typed command paths. Use the **correct COM semantics — `hresult < 0`** (see CLI-08), matching Python, rather than `!= 0`; then any per-item status where success is false raises. Keep the raw `invoke` escape hatch un-validated (callers opting out of typed helpers accept raw replies), but apply the check in the typed `session.write`/command helpers. This is co-designed with CLI-08 (HRESULT semantics) — Rust should land on `< 0` from the start rather than repeat the `!= 0` mistake.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/rust/src/error.rs`: add `Error::MxAccess(Box<MxAccessError>)` (boxed, consistent with the existing boxed `tonic::Status`/`CommandError` pattern) with a `thiserror` message summarizing hresult + status entries and credential-safe formatting; add `pub fn ensure_mxaccess_success(reply: MxCommandReply) -> Result<MxCommandReply, Error>` checking `hresult < 0` and each `MxStatusProxy.success`.
|
||||
- `clients/rust/src/client.rs` / `session.rs`: invoke `ensure_mxaccess_success` after `ensure_command_success` in the typed write/command wrappers.
|
||||
- Tests: add a Rust unit/fixture test asserting a reply with `protocol_status = Ok` but a failing status entry (and one with a negative hresult) yields `Error::MxAccess`; reuse the shared behavior fixtures if they carry such a case.
|
||||
- Docs: note the new `Error::MxAccess` variant in `clients/rust/README.md`.
|
||||
|
||||
**Verification.** From `clients/rust`: `cargo fmt`, `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-04 — Typed-command parity gap across all five clients `High` · `P2`
|
||||
|
||||
**Finding.** No client (`.NET`, `Go`, `Java`, `Python`, `Rust`) exposes typed session helpers for single-item `WriteSecured`/`WriteSecured2`, `AuthenticateUser`, `ArchestrAUserToId`, `AdviseSupervisory`, `AddBufferedItem`, `SetBufferedUpdateInterval`, `Suspend`, or `Activate`, though the wire contract defines all of them (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_gateway.proto:150-160`) and `gateway.md` documents `AdviseSupervisory` as a precondition for user-attributed plain writes. Only the .NET, Go, and Python CLIs offer `advise-supervisory` via hand-built raw commands (`clients/go/cmd/mxgw-go/main.go:364-391`, `clients/dotnet/.../MxGatewayClientCli.cs:456-470`, `clients/python/.../commands.py:287-291`); Java and Rust CLIs lack even that.
|
||||
|
||||
**Impact.** The secured-write parity path (AuthenticateUser → AdviseSupervisory → WriteSecured) and the buffered-event family (`OnBufferedDataChange`) are reachable only through raw `Invoke` — the single most important MXAccess parity surface is untyped in every language. Roadmap P2 item 13.
|
||||
|
||||
**Design.** Add the missing typed session helpers to all five clients, phased by parity value: (1) `adviseSupervisory`, single-item `writeSecured`/`writeSecured2`, `authenticateUser`, `archestrAUserToId`; (2) buffered family `addBufferedItem`/`setBufferedUpdateInterval` and `suspend`/`activate`. Each helper wraps the existing raw-`Invoke` machinery already used by the bulk variants, so no new wire surface is needed — the contract already carries the command kinds (`mxaccess_gateway.proto:150-160`). Follow each language's existing typed-helper idiom (e.g. `.NET` `InvokeCommandAsync` in `MxGatewaySession.cs`, Go `invokeCommand`, Rust `session.rs` wrappers, Java stub methods, Python async methods). Preserve MXAccess parity semantics exactly — do not "fix" `WriteSecured` requiring a prior authenticate/advise; surface the native failure. Credentials passed to `authenticateUser`/`writeSecured` must route through each client's existing secret-redaction seam (`MxGatewaySecrets.java`, Rust `error.rs` scrubbing, Go/Python redaction) so they never reach logs or error messages. Depends on CLI-15 (both touch the same session surfaces and READMEs — land as a coordinated session-surface pass).
|
||||
|
||||
**Implementation.**
|
||||
- `.NET`: add `WriteSecuredAsync`/`AuthenticateUserAsync`/`AdviseSupervisoryAsync`/etc. to `MxGatewaySession.cs`, mirroring the bulk-helper pattern; extend `MxGatewayClientCli.cs`.
|
||||
- `Go`: add methods to `session.go` alongside the bulk helpers; promote the CLI's hand-built `advise-supervisory` command into a typed method.
|
||||
- `Java`: add methods to `MxGatewaySession.java` and a CLI subcommand (`zb-mom-ww-mxgateway-cli`).
|
||||
- `Python`: add async methods to `session.py` and CLI commands in `commands.py`.
|
||||
- `Rust`: add methods to `session.rs` (see CLI-30 for `unregister` symmetry) and subcommands to `mxgw-cli`.
|
||||
- Tests: add per-client fixture/unit coverage for `writeSecured` (parity: fails without prior authenticate), `authenticateUser`, and `adviseSupervisory`; add these to the cross-language smoke matrix.
|
||||
- Docs: update each client README's capability list, the parity matrix in `docs/ClientLibrariesDesign.md`, and `docs/CrossLanguageSmokeMatrix.md`.
|
||||
|
||||
**Verification.** Per language, the CLAUDE.md per-area commands: `.NET` `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` + tests; `Go` `go build/test ./...`; `Rust` `cargo check/test/clippy`; `Python` `python -m pytest`; `Java` `gradle test` **on windev** (no local JRE on the Mac — revert the regenerated `MxaccessGateway.java` churn afterward, since no `.proto` changed).
|
||||
|
||||
---
|
||||
|
||||
## CLI-05 — .NET session cannot be re-attached to an existing session id `Medium` · `—`
|
||||
|
||||
**Finding.** In the .NET client, `MxGatewaySession`'s constructor is `internal` (`clients/dotnet/ZB.MOM.WW.MxGateway.Client/MxGatewaySession.cs:19`) and no public factory exists, unlike Go's `NewSessionForID` (`clients/go/mxgateway/session.go:70`), Java's `forSessionId`, Python's ctor, or Rust's `client.session()`.
|
||||
|
||||
**Impact.** After a client restart, the gateway's `DetachGraceSeconds`/replay features are usable from .NET only through the raw stub — all typed helpers are lost on reconnect. Parity gap; .NET is the only client that cannot re-attach.
|
||||
|
||||
**Design.** Add a public factory `MxGatewayClient.AttachSession(string sessionId)` returning an `MxGatewaySession` whose `OpenSessionReply` is synthesized to carry the given `SessionId` (no server round-trip; the session already exists). Keep the `internal` ctor for the open path. This aligns .NET with the other four clients and unblocks CLI-15's reconnect helper. Alternative rejected: making the ctor public — the factory name documents intent (re-attach vs open) and keeps `OpenSessionReply` construction internal.
|
||||
|
||||
**Implementation.**
|
||||
- `MxGatewayClient.cs`: add `public MxGatewaySession AttachSession(string sessionId)` building a minimal `OpenSessionReply { SessionId = sessionId }` and calling the internal ctor.
|
||||
- Tests: add a case in the .NET session test project asserting `AttachSession(id).SessionId == id` and that a subsequent `StreamEventsAsync(afterWorkerSequence)` targets that id.
|
||||
- Docs: note re-attach in `clients/dotnet/README.md` and flip the parity-matrix cell in `docs/ClientLibrariesDesign.md`.
|
||||
|
||||
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` and `dotnet test` the client test project.
|
||||
|
||||
---
|
||||
|
||||
## CLI-06 — .NET `DisposeAsync` blocks/throws on unreachable gateway `Medium` · `—`
|
||||
|
||||
**Finding.** In the .NET client, `MxGatewaySession.DisposeAsync` calls `CloseAsync()` with no cancellation token or timeout (`clients/dotnet/.../MxGatewaySession.cs:881-885`). When the gateway is unreachable, `await using` disposal blocks for the full retry-pipeline budget and then throws out of disposal, masking the original exception.
|
||||
|
||||
**Impact.** `await using` on a session whose gateway just died hangs and then surfaces a disposal exception instead of the caller's real error — a poor failure mode on the common teardown path.
|
||||
|
||||
**Design.** Time-bound and swallow close failures in the disposal path only. Wrap the `CloseAsync` call in a short linked-timeout CTS and catch (`OperationCanceledException`/`RpcException`/`MxGatewayException`), since throwing from `DisposeAsync` is an anti-pattern that masks in-flight exceptions. Explicit `CloseAsync()` remains fully surfacing for callers who want to observe close failures. Keep the `_closeLock.Dispose()` unconditional.
|
||||
|
||||
**Implementation.**
|
||||
- `MxGatewaySession.cs` `DisposeAsync`: `using var cts = new CancellationTokenSource(disposeCloseTimeout);` (a small fixed bound, e.g. 2 s, or `Options`-derived), `try { await CloseAsync(cts.Token); } catch (Exception ex) when (ex is OperationCanceledException or RpcException or MxGatewayException) { /* best-effort */ }`.
|
||||
- Tests: add a test using a transport that never responds, asserting `DisposeAsync` completes within the bound and does not throw.
|
||||
- Docs: note best-effort dispose semantics in `clients/dotnet/README.md`.
|
||||
|
||||
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` + client tests.
|
||||
|
||||
---
|
||||
|
||||
## CLI-07 — .NET retry budget self-defeats on `DeadlineExceeded` `Medium` · `—`
|
||||
|
||||
**Finding.** In the .NET client, `ExecuteSafeUnaryAsync` caps the whole retry pipeline with `CancelAfter(Options.DefaultCallTimeout)` (`clients/dotnet/.../MxGatewayClient.cs:305-316`) while each attempt's `CallOptions` also gets a `DefaultCallTimeout` deadline (`:290-303`). Because `DeadlineExceeded` is in the retryable set (`MxGatewayClientRetryPolicy.cs:62-67`), a first attempt that ends in `DeadlineExceeded` consumes the entire outer budget, so the configured retries never run.
|
||||
|
||||
**Impact.** Retries are silently inert for the exact transient class they were configured for; `Unavailable`/`ResourceExhausted` bursts shorter than one full timeout still retry, but any deadline-driven transient does not.
|
||||
|
||||
**Design.** Give the outer budget headroom so retries have room to execute. Recommended: size the outer CTS to cover the retry math — roughly `MaxAttempts × (DefaultCallTimeout + MaxDelay)` — rather than a single `DefaultCallTimeout`. Alternative (simpler, less faithful to intent): drop `DeadlineExceeded` from `IsTransientStatus`. Prefer the headroom fix because per-attempt deadlines remain meaningful and deadline transients still retry. Open question the review couldn't settle: whether an absolute wall-clock cap should exist at all — recommend keeping one but computed from the retry parameters.
|
||||
|
||||
**Implementation.**
|
||||
- `MxGatewayClient.cs` `ExecuteSafeUnaryAsync`: compute the outer `CancelAfter` from `Options.Retry` (attempts, delay, max-delay) plus per-attempt `DefaultCallTimeout`, instead of a bare `DefaultCallTimeout`.
|
||||
- Tests: add a retry test whose first attempt returns `DeadlineExceeded` and asserts a second attempt occurs (the existing retry-policy test project already exercises the pipeline).
|
||||
- Docs: clarify the timeout/retry interaction in `clients/dotnet/README.md`.
|
||||
|
||||
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` + retry test.
|
||||
|
||||
---
|
||||
|
||||
## CLI-08 — .NET/Go/Java treat any nonzero HRESULT as failure `Medium` · `—`
|
||||
|
||||
**Finding.** `EnsureMxAccessSuccess` in .NET tests `reply.HasHresult && reply.Hresult != 0` (`clients/dotnet/.../MxCommandReplyExtensions.cs:32`); Go tests `reply.GetHresult() != 0` (`clients/go/mxgateway/errors.go:121`); Java's `MxGatewayErrors` does the same. This misclassifies positive COM success codes (e.g. `S_FALSE = 1`). Python uses the correct `reply.hresult < 0` (`clients/python/.../errors.py:133`).
|
||||
|
||||
**Impact.** A parity-preserving gateway reply carrying a positive success HRESULT throws in three clients and passes in one — a cross-client inconsistency that can turn a legitimate MXAccess success into a raised error. Co-designed with CLI-03 (Rust must adopt `< 0` from the start).
|
||||
|
||||
**Design.** Align .NET, Go, and Java on the COM-correct `hresult < 0` check (negative = failure), matching Python and MXAccess semantics. Low-risk one-line change per client; keep the status-array check unchanged. Not individually enumerated in the roadmap, but shares the correctness theme of P0 item 6 (see CLI-03) — land alongside the Rust MXSTATUS work.
|
||||
|
||||
**Implementation.**
|
||||
- `.NET` `MxCommandReplyExtensions.cs:32`: `bool hResultFailure = reply.HasHresult && reply.Hresult < 0;`
|
||||
- `Go` `errors.go:121`: `if reply.Hresult != nil && reply.GetHresult() < 0 {`
|
||||
- `Java` `MxGatewayErrors.java` (~:50): change the `!= 0` comparison to `< 0`.
|
||||
- Tests: add a case in each client asserting a reply with `hresult = 1` (S_FALSE) passes and `hresult = -2147...` fails; Python's existing regression suite already covers `< 0`.
|
||||
- Docs: note the corrected semantics in each client README's error section.
|
||||
|
||||
**Verification.** `.NET` build+test; `Go` `go test ./...`; `Java` `gradle test` on windev (revert generated-file churn afterward).
|
||||
|
||||
---
|
||||
|
||||
## CLI-09 — Go has no typed auth-error mapping `Medium` · `—`
|
||||
|
||||
**Finding.** In the Go client, all RPC failures wrap into the generic `GatewayError` (`clients/go/mxgateway/errors.go:9-34`; call sites e.g. `client.go`), so distinguishing `Unauthenticated` from `PermissionDenied` requires `status.Code(errors.Unwrap(err))`. `docs/ClientLibrariesDesign.md:153` requires the two be treated distinctly, and the other four clients expose typed auth errors.
|
||||
|
||||
**Impact.** Go consumers cannot idiomatically branch on auth failure vs authorization failure — the documented contract is unmet only in Go.
|
||||
|
||||
**Design.** Add sentinel-backed typed errors `AuthenticationError` (gRPC `Unauthenticated`) and `AuthorizationError` (`PermissionDenied`) in `errors.go`, and map them at the transport boundary where `GatewayError` is currently constructed. Use `errors.Is`-friendly sentinels (`ErrUnauthenticated`, `ErrPermissionDenied`) plus wrapper structs preserving `Op` and the underlying `status.Status`. This mirrors the existing `MxAccessError`/`CommandError` layering already in the file.
|
||||
|
||||
**Implementation.**
|
||||
- `errors.go`: add the two error types + sentinels and a helper `classifyRPCError(op string, err error) error` that inspects `status.Code(err)`.
|
||||
- `client.go` (and other RPC wrappers): route transport errors through `classifyRPCError` instead of always constructing `GatewayError`.
|
||||
- Tests: add cases asserting `errors.Is(err, ErrUnauthenticated)` / `ErrPermissionDenied` for the corresponding gRPC codes.
|
||||
- Docs: update `clients/go/README.md` and flip the Go cell in the parity matrix (`docs/ClientLibrariesDesign.md`).
|
||||
|
||||
**Verification.** From `clients/go`: `gofmt -l .`, `go build ./...`, `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-10 — Go uses deprecated `grpc.DialContext` + `grpc.WithBlock()` `Medium` · `—`
|
||||
|
||||
**Finding.** In the Go client, `Dial` builds options including `grpc.WithBlock()` and calls `grpc.DialContext` (`clients/go/mxgateway/client.go:60-68`). grpc-go ≥1.63 deprecates both in favor of `grpc.NewClient` with lazy connection.
|
||||
|
||||
**Impact.** Future grpc-go upgrades and `go vet`/staticcheck flag the deprecations; blocking dial also hides the per-RPC connection-error semantics the ecosystem now expects.
|
||||
|
||||
**Design.** Migrate to `grpc.NewClient` (lazy, non-blocking) and drop `grpc.WithBlock()`. Because `NewClient` no longer surfaces connect failures at dial time, expose readiness via a first `Ping` in the connect helpers that currently rely on blocking dial (or document that the first RPC surfaces connection errors). Preserve the existing transport-credential and auth-interceptor wiring unchanged.
|
||||
|
||||
**Implementation.**
|
||||
- `client.go`: replace `grpc.DialContext(dialCtx, endpoint, dialOptions...)` with `grpc.NewClient(endpoint, dialOptions...)`, remove `grpc.WithBlock()`; if a caller-facing "connected" guarantee is desired, add an optional `Ping` in the `Dial`/connect wrapper.
|
||||
- Tests: existing dial/TLS tests should pass; add a case asserting a bad endpoint surfaces on first RPC rather than at construction.
|
||||
- Docs: note the lazy-connect behavior change in `clients/go/README.md`.
|
||||
|
||||
**Verification.** From `clients/go`: `go build ./...`, `go vet ./...`, `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-11 — Go CLI cannot opt into strict TLS validation `Medium` · `—`
|
||||
|
||||
**Finding.** In the Go client CLI, `dialForCommand` never sets `Options.RequireCertificateValidation` and exposes no flag for it (`clients/go/cmd/mxgw-go/main.go:1151-1158`), so every non-CA-pinned TLS run of `mxgw-go` is skip-verify even though the library supports strictness (`clients/go/mxgateway/client.go:231-241`).
|
||||
|
||||
**Impact.** `mxgw-go --tls` without a CA cannot authenticate the server at all, with no operator opt-in — a security-posture gap unique to the Go CLI. Ties into CLI-17 (cross-client TLS convergence).
|
||||
|
||||
**Design.** Add a `-require-certificate-validation` boolean flag that sets `Options.RequireCertificateValidation`, mirroring the library capability. Coordinate the default with CLI-17's convergence decision; at minimum emit a one-line stderr warning when TLS runs skip-verify (see CLI-17/CLI-20).
|
||||
|
||||
**Implementation.**
|
||||
- `clients/go/cmd/mxgw-go/main.go`: register the flag and thread it into `dialForCommand`'s `Options`.
|
||||
- Tests: extend the CLI/TLS tests to assert the flag sets the option.
|
||||
- Docs: document the flag in `clients/go/README.md` and the CLI usage.
|
||||
|
||||
**Verification.** From `clients/go`: `go build ./...`, `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-12 — Java docs still say "Java 21" after the JDK 17 retarget `Medium` · `P2`
|
||||
|
||||
**Finding.** After the verified-correct JDK 17 retarget (build.gradle `toolchain 17` + `options.release = 17`, `build.gradle:20-31`), three docs still claim Java 21: `clients/java/README.md:354`, `clients/java/JavaClientDesign.md:34-35`, and `docs/ClientPackaging.md:193`. This violates CLAUDE.md's docs-in-same-commit rule on the retarget's own branch.
|
||||
|
||||
**Impact.** Consumers targeting Ignition 8.3 (JDK 17) read contradictory prerequisites. Roadmap P2 item 15 (doc-drift sweep).
|
||||
|
||||
**Design.** Sweep all three references from "Java 21" to "Java 17" (noting a 17-targeted build still runs on 21+, as `build.gradle:20-21` already comments). Pure documentation change; no code. Land on the `feat/jdk17-client-retarget` branch before merge.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/java/README.md:354`, `clients/java/JavaClientDesign.md:34-35`, `docs/ClientPackaging.md:193`: change "Java 21" → "Java 17" (and correct the Gradle-toolchain wording). Note the `docs/ClientPackaging.md:193` line also carries the stale Java package/naming addressed in CLI-16 — fix together.
|
||||
- Docs only; no tests.
|
||||
|
||||
**Verification.** Grep confirms zero remaining "Java 21" under `clients/java` and `docs/`: `grep -rn "Java 21" clients/java docs/`. No build needed for the doc change.
|
||||
|
||||
---
|
||||
|
||||
## CLI-13 — Java event-stream buffer hardcoded 16, cancel-on-overflow `Medium` · `—`
|
||||
|
||||
**Finding.** In the Java client, `MxGatewayClient.streamEvents` constructs `new MxEventStream(16)` (`clients/java/.../MxGatewayClient.java:248`) with no configuration, and overflow cancels the RPC (`MxEventStream.java:124-132`). Unlike Go (CLI-01), the failure is at least surfaced as `MxGatewayException("…queue overflowed")`, but a consumer stalling for 16 events is disconnected even though gRPC has native flow control.
|
||||
|
||||
**Impact.** Slow Java consumers are force-disconnected at an arbitrary 16-event threshold; capacity is not tunable. Backpressure semantics diverge from the unbuffered .NET/Python/Rust clients (CLI-33).
|
||||
|
||||
**Design.** Make the buffer capacity an option on the stream-events call (default 16 for compatibility) and/or adopt manual flow control (`disableAutoRequestWithInitial` on the stub) so backpressure rides gRPC instead of cancel-on-overflow. Recommended minimal change: parameterize capacity via an overload / options object; the manual-flow-control migration is a larger follow-up. Keep the loud `MxGatewayException` on overflow (Java's surfaced-error behavior is already correct relative to Go).
|
||||
|
||||
**Implementation.**
|
||||
- `MxGatewayClient.java`: add an overload `streamEvents(StreamEventsRequest, int bufferCapacity)` (or an options field) feeding `new MxEventStream(capacity)`.
|
||||
- `MxEventStream.java`: no behavior change required for the capacity option; document the overflow contract (see CLI-24).
|
||||
- Tests: extend the Java event-stream tests with a custom-capacity case and an overflow case asserting the `MxGatewayException` message.
|
||||
- Docs: record Java's buffered/error-on-overflow behavior in `docs/ClientBehaviorFixtures.md` (CLI-33) and `clients/java/README.md`.
|
||||
|
||||
**Verification.** `gradle test` from `clients/java` **on windev** (no local JRE); revert the regenerated `MxaccessGateway.java` churn afterward since no `.proto` changed.
|
||||
|
||||
---
|
||||
|
||||
## CLI-14 — Python default TLS is blocking TOFU pin with silent `localhost` SNI `Medium` · `—`
|
||||
|
||||
**Finding.** In the Python client, `create_channel`'s default TLS path fetches the server certificate unverified and pins it trust-on-first-use (`clients/python/.../options.py:128-160`), and silently defaults the SNI/target-name override to `localhost` when none is supplied (`options.py:153-154`). It is documented and bounded (probe timeout; `asyncio.to_thread`), but it is the only client that opens a second out-of-band TCP+TLS connection per channel, and TOFU is exposed to first-contact interception.
|
||||
|
||||
**Impact.** A first-contact MITM can pin a rogue certificate; the silent `localhost` SNI can mask a hostname mismatch. Bounded and intentional for the internal-tool posture, but the threat window is undocumented in the client's own README. Ties into CLI-17.
|
||||
|
||||
**Design.** Keep the TOFU default (removing it would break the internal-tool ergonomics the other clients also target) but document the MITM window in the README threat model, prefer `ca_file` in examples, and make the `localhost` SNI default explicit in a log/warning rather than silent. No behavioral break; a warning + doc change. Convergence across languages is CLI-17.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/python/README` (or `docs/`): add a threat-model note describing the first-contact TOFU window and recommending `ca_file`; make `ca_file` the primary TLS example.
|
||||
- `options.py`: emit a one-line warning (via `logging`/`warnings`) when the TOFU path pins an unverified cert and when it auto-applies the `localhost` SNI override.
|
||||
- Tests: extend the Python TLS tests to assert the warning fires on the TOFU path.
|
||||
- Docs: cross-reference `docs/CrossLanguageSmokeMatrix.md`'s TLS divergence table.
|
||||
|
||||
**Verification.** `python -m pytest` from `clients/python`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-15 — No client handles `ReplayGap` or offers a reconnect helper `Medium` · `P2`
|
||||
|
||||
**Finding.** All five clients expose the `after_worker_sequence` resume cursor (`MxGatewaySession.cs:865-876`, `session.go:682-688`, `MxGatewaySession.java:718-722`, `session.py:571-582`, `session.rs:648`), but none handle the `ReplayGap` sentinel a resuming consumer must expect — `grep ReplayGap clients/` returns only build output (contracts XML, javadoc), no handwritten source. The gateway shipped `DetachGraceSeconds` + replay (on by default), so a resuming stream can legitimately emit a gap event that no client types or documents.
|
||||
|
||||
**Impact.** A client that reconnects with `after_worker_sequence` after the replay buffer overran receives a `ReplayGap` it does not recognize, silently mis-treating a lossy resume as continuous. Roadmap P2 item 12 (finish the session-resilience epic). Coordinates with CLI-04 (session-surface pass) and cross-domain with the gateway/testing reports.
|
||||
|
||||
**Design.** Two-stage: (1) document `ReplayGap` gap-detection per client README so consumers can branch on it via the raw event; (2) add a typed helper/observer that raises or signals gap detection during resume. Stage 1 is the P2-minimum. Recommended stage-2 shape: a per-client `on_replay_gap`/`ReplayGapError` hook surfaced from the event iterator when a `ReplayGap` event arrives, letting the consumer decide whether to re-snapshot. Do not synthesize or swallow the gap — forward the gateway's event faithfully (per the no-synthesized-events invariant); the client only makes it observable and typed.
|
||||
|
||||
**Implementation.**
|
||||
- Each client README: document that resuming with a stale `after_worker_sequence` can yield a `ReplayGap` event and what it means.
|
||||
- `.NET`/`Go`/`Java`/`Python`/`Rust` event surfaces: recognize the `ReplayGap` event kind and expose it as a distinct typed result/error alongside normal events.
|
||||
- Tests: add per-client fixture cases feeding a `ReplayGap` event and asserting the typed surface; add a resume-with-gap case to the cross-language smoke matrix.
|
||||
- Docs: `docs/ClientLibrariesDesign.md` (update the "no client-side reconnect" v1 non-goal, `:64-70`) and `docs/CrossLanguageSmokeMatrix.md`.
|
||||
|
||||
**Verification.** Per-language build/test as in CLI-04 (Java on windev, revert generated churn).
|
||||
|
||||
---
|
||||
|
||||
## CLI-16 — `docs/ClientPackaging.md` drifted from Python naming and .NET `.slnx` `Medium` · `P2`
|
||||
|
||||
**Finding.** `docs/ClientPackaging.md` names the Python package `mxaccess-gateway-client` and its generated dir `src/mxgateway/generated` (`:159-160`) vs the actual `zb-mom-ww-mxaccess-gateway-client` (`clients/python/pyproject.toml:8`) and `src/zb_mom_ww_mxgateway/generated`; the CLI module `python -m mxgateway_cli` (`:187`) vs actual `zb_mom_ww_mxgateway_cli`; the .NET solution `ZB.MOM.WW.MxGateway.Client.sln` (`:51-52`) vs actual `.slnx`; and Java 21 (`:193`, CLI-12). `docs/ClientLibrariesDesign.md:410` repeats the stale Python generated path.
|
||||
|
||||
**Impact.** Copy-paste build/run commands from the packaging doc fail — wrong package name, wrong module path, wrong solution extension. Roadmap P2 item 15.
|
||||
|
||||
**Design.** One documentation sweep aligning every reference to reality. Pure docs; no code. Land with CLI-12 (same file, `:193`).
|
||||
|
||||
**Implementation.**
|
||||
- `docs/ClientPackaging.md`: `:51-52` `.sln` → `.slnx`; `:159-160` package → `zb-mom-ww-mxaccess-gateway-client`, generated dir → `src/zb_mom_ww_mxgateway/generated`; `:187` CLI module → `python -m zb_mom_ww_mxgateway_cli`; `:193` Java 21 → 17 and package naming.
|
||||
- `docs/ClientLibrariesDesign.md:410`: generated path → `clients/python/src/zb_mom_ww_mxgateway/generated`.
|
||||
- Verify against `clients/python/pyproject.toml` and `clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx`.
|
||||
|
||||
**Verification.** Grep the corrected tokens back against the source: `grep -n "slnx\|zb_mom_ww_mxgateway" docs/ClientPackaging.md` and confirm no stale `mxgateway_cli`/`src/mxgateway/generated` remain.
|
||||
|
||||
---
|
||||
|
||||
## CLI-17 — TLS default posture inconsistent across the five clients `Medium` · `—`
|
||||
|
||||
**Finding.** The five clients diverge on default TLS: accept-any-cert in .NET (`MxGatewayClient.cs:361-364`), Go (`client.go:236-240`, `InsecureSkipVerify`), and Java (`InsecureTrustManagerFactory`); TOFU pinning in Python (`options.py:133-154`); strict pin-only in Rust. `docs/CrossLanguageSmokeMatrix.md:58-66` documents the divergence, but the same `--tls`-without-CA invocation authenticates the server in one language, half in another, and not at all in three.
|
||||
|
||||
**Impact.** Inconsistent, surprising security posture per language for identical invocations. Depends on CLI-14 (Python) and interacts with CLI-11/CLI-20.
|
||||
|
||||
**Design.** Converge, or at minimum make the weak default loud. Recommended target: the Python TOFU model as the shared default (pins on first contact, better than accept-all), and where a client keeps accept-all as an explicit opt-in, emit a one-line stderr/log warning whenever certificate verification is disabled. Full convergence is a larger cross-client effort; the review's minimum is the warning. Open question the review left: whether to make strict validation the default and require an explicit `--insecure`-style opt-out — recommended long-term but a breaking change for existing internal-tool callers, so stage it behind a major version.
|
||||
|
||||
**Implementation.**
|
||||
- `.NET` `MxGatewayClient.cs:361-364`, `Go` `client.go`, `Java` TLS path: emit a single warning when the accept-all callback / `InsecureSkipVerify` / `InsecureTrustManagerFactory` is installed (see CLI-20 for the .NET-specific note).
|
||||
- Optionally add TOFU pinning to .NET/Go/Java to match Python (larger; stage separately).
|
||||
- Tests: assert the warning fires when TLS runs without a CA and without strict validation.
|
||||
- Docs: reconcile `docs/CrossLanguageSmokeMatrix.md:58-66` with the chosen convergence and note the warning behavior in each README.
|
||||
|
||||
**Verification.** Per-language build/test (Java on windev).
|
||||
|
||||
---
|
||||
|
||||
## CLI-18 — .NET csproj records no `<Version>` `Low` · `P2`
|
||||
|
||||
**Finding.** `clients/dotnet/.../ZB.MOM.WW.MxGateway.Client.csproj` (packaging block lines 19-28) declares no `<Version>`; the published NuGet version (0.1.2) is supplied out-of-band at pack time, so the source tree does not record what ships.
|
||||
|
||||
**Impact.** The repo cannot be inspected to learn the shipped version; drift risk. Roadmap P2 item 16 (version alignment).
|
||||
|
||||
**Design.** Add `<Version>` to the csproj `PropertyGroup` so the source records the shipped version, keeping the pack-time override available for CI. Trivial.
|
||||
|
||||
**Implementation.**
|
||||
- `ZB.MOM.WW.MxGateway.Client.csproj`: add `<Version>0.1.2</Version>` (or the current release) to the packaging `PropertyGroup` (lines 19-28).
|
||||
- Docs: none beyond the version-alignment note in the packaging doc.
|
||||
|
||||
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-19 — .NET duplicate `InternalsVisibleTo` `Low` · `—`
|
||||
|
||||
**Finding.** `InternalsVisibleTo("ZB.MOM.WW.MxGateway.Client.Tests")` is declared both in `clients/dotnet/.../Properties/AssemblyInfo.cs:3` and as a csproj `AssemblyAttribute` (`ZB.MOM.WW.MxGateway.Client.csproj:35-39`). Harmless but redundant.
|
||||
|
||||
**Impact.** Cosmetic; two sources of truth for the same attribute.
|
||||
|
||||
**Design.** Keep one. Recommend the csproj `AssemblyAttribute` block (co-located with packaging config) and delete `Properties/AssemblyInfo.cs`, or vice versa.
|
||||
|
||||
**Implementation.**
|
||||
- Delete `clients/dotnet/.../Properties/AssemblyInfo.cs` (or remove the csproj block) — keep exactly one declaration.
|
||||
|
||||
**Verification.** `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` (confirms tests still see internals).
|
||||
|
||||
---
|
||||
|
||||
## CLI-20 — .NET `--tls` without CA installs accept-all callback `Low` · `—`
|
||||
|
||||
**Finding.** In the .NET client, with `UseTls` and no CA file, the default `RequireCertificateValidation = false` installs an accept-all certificate callback (`clients/dotnet/.../MxGatewayClient.cs:361-364`). Documented and intentional (`MxGatewayClientOptions.cs:31-36`), but `--tls` without a CA gives no server authentication.
|
||||
|
||||
**Impact.** Same class as CLI-17; called out separately for the .NET default. Low because documented/intentional.
|
||||
|
||||
**Design.** Emit a one-line warning when the accept-all callback is installed (co-designed with CLI-17). No behavior change to the default.
|
||||
|
||||
**Implementation.**
|
||||
- `MxGatewayClient.cs`: log a warning (via the client's `ILogger`) when the accept-all `RemoteCertificateValidationCallback` is installed (line 361-364 branch).
|
||||
- Docs: cross-reference CLI-17 in `clients/dotnet/README.md`.
|
||||
|
||||
**Verification.** `dotnet build` + client TLS-handler test asserting the warning.
|
||||
|
||||
---
|
||||
|
||||
## CLI-21 — Go `ClientVersion` stale vs tagged releases `Low` · `P2`
|
||||
|
||||
**Finding.** In the Go client, `ClientVersion = "0.1.0-dev"` (`clients/go/mxgateway/version.go:6`) is stale relative to the module releases published via `scripts/tag-go-module.ps1`.
|
||||
|
||||
**Impact.** `mxgw-go` and callers report a dev version regardless of the tagged release. Roadmap P2 item 16.
|
||||
|
||||
**Design.** Bump `ClientVersion` to the released value and update it as part of the tagging script so the two cannot drift.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/go/mxgateway/version.go:6`: set to the current release (e.g. `0.1.2`).
|
||||
- `scripts/tag-go-module.ps1`: update the constant (or fail the tag) when the version constant does not match the tag.
|
||||
- Docs: version-alignment note in the packaging doc.
|
||||
|
||||
**Verification.** From `clients/go`: `go build ./...`, `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-22 — Go `newCorrelationID` swallows `crypto/rand` error `Low` · `—`
|
||||
|
||||
**Finding.** In the Go client, `newCorrelationID` returns an empty string on `rand.Read` error (`clients/go/mxgateway/session.go:786-792`), silently dropping traceability.
|
||||
|
||||
**Impact.** A rare `crypto/rand` failure yields empty correlation ids, breaking request tracing with no signal.
|
||||
|
||||
**Design.** Fall back to a monotonic timestamp+counter id rather than empty, preserving uniqueness/traceability without introducing an error-return on a hot helper.
|
||||
|
||||
**Implementation.**
|
||||
- `session.go:786-792`: on `rand.Read` error, build a fallback id from `time.Now().UnixNano()` plus an atomic counter.
|
||||
- Tests: add a case (via an injectable rand source or by asserting non-empty output) that the id is never empty.
|
||||
|
||||
**Verification.** From `clients/go`: `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-23 — Go nil-vs-empty bulk short-circuit asymmetry `Low` · `—`
|
||||
|
||||
**Finding.** In the Go client, `WriteBulk`/`ReadBulk` short-circuit an empty (non-nil) slice locally (`session.go:399-407`, `:524-532`) while `AddItemBulk`/`SubscribeBulk` send an empty command to the wire (`session.go:255-274`). Harmless but inconsistent within one file.
|
||||
|
||||
**Impact.** Cosmetic; identical inputs produce a local no-op in some helpers and a wire round-trip in others.
|
||||
|
||||
**Design.** Pick one convention (recommend short-circuit empty locally, avoiding a pointless round-trip) and apply it uniformly across the bulk helpers.
|
||||
|
||||
**Implementation.**
|
||||
- `session.go`: make `AddItemBulk`/`SubscribeBulk` (and siblings) short-circuit empty input like `WriteBulk`/`ReadBulk`, or document the intentional difference.
|
||||
- Tests: assert empty-input behavior is consistent.
|
||||
|
||||
**Verification.** From `clients/go`: `go test ./...`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-24 — Java `MxEventStream` single-consumer constraint undocumented `Low` · `—`
|
||||
|
||||
**Finding.** In the Java client, `MxEventStream` is a single-consumer iterator with unsynchronized `next` state (`MxEventStream.java:31`, `:65-92`); the single-consumer constraint is not documented.
|
||||
|
||||
**Impact.** A caller iterating from two threads corrupts `next` state with no warning.
|
||||
|
||||
**Design.** Add a Javadoc note stating the iterator is single-consumer / not thread-safe. Documentation only (matching CLI-13's overflow doc).
|
||||
|
||||
**Implementation.**
|
||||
- `MxEventStream.java`: Javadoc on the class and `next` noting single-consumer usage.
|
||||
|
||||
**Verification.** `gradle test` on windev (revert generated churn); Javadoc-only change.
|
||||
|
||||
---
|
||||
|
||||
## CLI-25 — Java `close()` does not await channel termination `Low` · `—`
|
||||
|
||||
**Finding.** In the Java client, `close()` initiates `shutdown()` without awaiting termination (`MxGatewayClient.java:346-351`); `closeAndAwaitTermination()` exists (`:360-367`), but try-with-resources users can leak a channel briefly at JVM exit.
|
||||
|
||||
**Impact.** Minor resource leak window on `try (var client = ...)` teardown.
|
||||
|
||||
**Design.** Make `close()` await a short bounded termination (e.g. a few seconds) so try-with-resources users get clean teardown, keeping `closeAndAwaitTermination()` for explicit longer bounds.
|
||||
|
||||
**Implementation.**
|
||||
- `MxGatewayClient.java` `close()`: after `shutdown()`, `awaitTermination(shortBound, SECONDS)` (swallow/timeout gracefully).
|
||||
- Tests: assert `close()` completes and the channel is terminated.
|
||||
- Docs: note the bounded-await in `clients/java/README.md`.
|
||||
|
||||
**Verification.** `gradle test` on windev; revert generated churn.
|
||||
|
||||
---
|
||||
|
||||
## CLI-26 — Python `version.py` ≠ `pyproject.toml` `Low` · `P2`
|
||||
|
||||
**Finding.** In the Python client, `pyproject.toml:9` says `0.1.2` while `version.py:3` says `__version__ = "0.1.0"`, so `mxgw-py version` reports the wrong value.
|
||||
|
||||
**Impact.** CLI and callers report a stale version. Roadmap P2 item 16.
|
||||
|
||||
**Design.** Derive `__version__` from installed metadata so the two cannot drift: `importlib.metadata.version("zb-mom-ww-mxaccess-gateway-client")` with a fallback for editable/source runs. Aligns with the single-source-of-truth intent.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/python/src/zb_mom_ww_mxgateway/version.py`: replace the literal with `importlib.metadata.version(...)` guarded by `try/except PackageNotFoundError` falling back to a literal.
|
||||
- Tests: assert `version.__version__` matches `pyproject` (or the installed dist) in the packaging/regression suite.
|
||||
- Note: any Python regen for this work must pin `grpcio-tools` to the baseline (grpcio 1.80.0 / protobuf 6.31.1) — but this change touches no protos, so no regen is needed.
|
||||
|
||||
**Verification.** `python -m pytest` from `clients/python`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-27 — Python `Session.close()` not concurrency-safe `Low` · `—`
|
||||
|
||||
**Finding.** In the Python client, `Session.close()` has no lock around `_closed` (`session.py:38-55`) and repeated closes return a locally synthesized `CloseSessionReply` rather than the cached server reply — divergent from .NET/Go which cache the real reply.
|
||||
|
||||
**Impact.** Concurrent closes can double-invoke; repeated closes return a synthetic reply instead of the server's. Minor.
|
||||
|
||||
**Design.** Add an `asyncio.Lock` around the close transition and cache the real server `CloseSessionReply` to return on repeat, matching the .NET/Go cached-reply pattern.
|
||||
|
||||
**Implementation.**
|
||||
- `session.py`: guard `_closed` with an `asyncio.Lock`, store the first server reply, return it on subsequent calls.
|
||||
- Tests: assert concurrent `close()` calls invoke the server once and return the cached reply.
|
||||
|
||||
**Verification.** `python -m pytest` from `clients/python`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-28 — Python circular-import workaround at end of `session.py` `Low` · `—`
|
||||
|
||||
**Finding.** In the Python client, `from .client import GatewayClient` sits at the bottom of `session.py` with `# noqa: E402` (~`session.py:590`) to break a runtime import cycle.
|
||||
|
||||
**Impact.** Works, but the runtime cycle is fragile and lints noisily.
|
||||
|
||||
**Design.** Replace with a `TYPE_CHECKING`-guarded import plus a string annotation, removing the runtime cycle entirely.
|
||||
|
||||
**Implementation.**
|
||||
- `session.py`: `if TYPE_CHECKING: from .client import GatewayClient` at the top and annotate the reference as `"GatewayClient"`; delete the bottom-of-file import and `# noqa`.
|
||||
- Tests: existing import/session tests confirm no runtime cycle.
|
||||
|
||||
**Verification.** `python -m pytest` from `clients/python`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-29 — Rust `CLIENT_VERSION` ≠ `Cargo.toml` `Low` · `P2`
|
||||
|
||||
**Finding.** In the Rust client, `CLIENT_VERSION = "0.1.0-dev"` with a doc comment claiming it "Mirrors `Cargo.toml`" (`clients/rust/src/version.rs:6-7`) while `Cargo.toml:3` says `0.1.2`.
|
||||
|
||||
**Impact.** The advertised client version is wrong and contradicts its own doc. Roadmap P2 item 16.
|
||||
|
||||
**Design.** Source the constant from Cargo at compile time so it cannot drift: `env!("CARGO_PKG_VERSION")`.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/rust/src/version.rs:7`: `pub const CLIENT_VERSION: &str = env!("CARGO_PKG_VERSION");`.
|
||||
- Tests: add a trivial assertion that `CLIENT_VERSION` equals the expected release, or that it is non-empty and matches `Cargo.toml` via a build script check.
|
||||
|
||||
**Verification.** From `clients/rust`: `cargo check --workspace`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-30 — Rust has no `unregister` typed helper `Low` · `—`
|
||||
|
||||
**Finding.** In the Rust client, the session surface (`clients/rust/src/session.rs:119-231`) covers register/add/advise/remove but not `Unregister`; Go/Java/Python expose it (and .NET also lacks it per the parity matrix).
|
||||
|
||||
**Impact.** Rust (and .NET) users must drop to raw `Invoke` to unregister — a small parity gap.
|
||||
|
||||
**Design.** Add an `unregister` typed helper wrapping the existing raw-command machinery, for symmetry with the other clients. Fold into the CLI-04 session-surface pass.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/rust/src/session.rs`: add `pub async fn unregister(...)` mirroring `register`.
|
||||
- (Optionally add the .NET `UnregisterAsync` too, closing the matrix cell.)
|
||||
- Tests: fixture/unit coverage for `unregister`.
|
||||
- Docs: flip the parity-matrix cell in `docs/ClientLibrariesDesign.md`.
|
||||
|
||||
**Verification.** From `clients/rust`: `cargo check/test/clippy`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-31 — Rust CLI is a single 2,699-line `main.rs` `Low` · `—`
|
||||
|
||||
**Finding.** In the Rust client, the CLI is one 2,699-line `main.rs` (`clients/rust/crates/mxgw-cli/src/main.rs`), the largest single file in the client tree; the Windows stack-size workaround it forced (`clients/rust/.cargo/config.toml:1-19`) is evidence the command enum has outgrown one module.
|
||||
|
||||
**Impact.** Maintainability; large compile unit; the stack-size workaround is a symptom.
|
||||
|
||||
**Design.** Split subcommands into per-command modules (`mxgw-cli/src/commands/*.rs`) with `main.rs` reduced to arg parsing and dispatch. Mechanical refactor; behavior-preserving. Re-evaluate whether the `.cargo/config.toml` stack workaround can be relaxed after the split.
|
||||
|
||||
**Implementation.**
|
||||
- `clients/rust/crates/mxgw-cli/src/`: extract subcommand handlers into modules; keep the clap/enum wiring in `main.rs`.
|
||||
- Tests: existing CLI tests must pass unchanged.
|
||||
|
||||
**Verification.** From `clients/rust`: `cargo build`, `cargo test --workspace`, `cargo clippy --all-targets -- -D warnings`.
|
||||
|
||||
---
|
||||
|
||||
## CLI-32 — Client-side bulk caps differ `Low` · `—`
|
||||
|
||||
**Finding.** Go/Python/Rust enforce a 1,000-item client-side bulk cap (`session.go:19`, `session.py:11`, `session.rs:29`) while .NET and Java send unbounded lists and rely on the gateway. Same oversized call produces different error types per language.
|
||||
|
||||
**Impact.** Inconsistent error surface for oversized bulk calls across languages. Harmless functionally.
|
||||
|
||||
**Design.** Align: either all clients enforce the 1,000 cap locally (recommended — a fast, uniform client-side error) or none do. Add the cap to .NET and Java.
|
||||
|
||||
**Implementation.**
|
||||
- `.NET` (`MxGatewaySession.cs`) and `Java` (`MxGatewaySession.java`): add a client-side 1,000-item check on the bulk helpers, throwing the client's standard argument/validation error.
|
||||
- Tests: assert an oversized bulk call fails locally with the expected error type in .NET and Java.
|
||||
- Docs: note the uniform cap in the client READMEs.
|
||||
|
||||
**Verification.** `.NET` build+test; `Java` `gradle test` on windev (revert generated churn).
|
||||
|
||||
---
|
||||
|
||||
## CLI-33 — Per-language event backpressure semantics undocumented `Low` · `—`
|
||||
|
||||
**Finding.** Event-stream backpressure differs by language: Go buffered-16 / silent-cancel (CLI-01), Java buffered-16 / error-cancel (CLI-13), .NET/Python/Rust unbuffered (native gRPC flow control). This per-language slow-consumer behavior is a parity-relevant observable absent from `docs/ClientBehaviorFixtures.md`.
|
||||
|
||||
**Impact.** Consumers cannot predict slow-consumer behavior from the docs. Depends on CLI-01 and CLI-13 landing first (they change the Go/Java behavior being documented).
|
||||
|
||||
**Design.** After CLI-01 (Go now emits a terminal `ErrSlowConsumer`) and CLI-13 (Java capacity option), document the per-language slow-consumer contract in `docs/ClientBehaviorFixtures.md` as a parity fixture row. Documentation only.
|
||||
|
||||
**Implementation.**
|
||||
- `docs/ClientBehaviorFixtures.md`: add a backpressure/slow-consumer section stating each client's behavior (Go: terminal `ErrSlowConsumer`; Java: `MxGatewayException` on overflow, configurable capacity; .NET/Python/Rust: native gRPC flow control).
|
||||
- Optionally add a shared behavior-fixture case exercised by each client's fixture-driven tests.
|
||||
|
||||
**Verification.** Doc change; validated by the fixture-driven test suites already run per client.
|
||||
|
||||
---
|
||||
|
||||
## CLI-34 — Python `build/`/`.pytest_cache/` present on disk (untracked) `Low` · `—`
|
||||
|
||||
**Finding.** In the Python client, `build/` and `.pytest_cache/` exist on disk but are not git-tracked (report X7; generated-code hygiene is otherwise good across all clients — tracked generated dirs match the manifest, Rust uses `OUT_DIR` + `.gitkeep`, .NET references Contracts directly).
|
||||
|
||||
**Impact.** None functionally; risk of an accidental future commit of build artifacts.
|
||||
|
||||
**Design.** Confirm `clients/python/.gitignore` (or the repo root `.gitignore`) covers `build/` and `.pytest_cache/`; add entries if absent. No code change.
|
||||
|
||||
**Implementation.**
|
||||
- Verify/extend `.gitignore` to cover `clients/python/build/` and `clients/python/.pytest_cache/`.
|
||||
|
||||
**Verification.** `git status --ignored clients/python` shows the directories ignored, not untracked.
|
||||
@@ -0,0 +1,460 @@
|
||||
# Testing, Documentation & Underdeveloped Areas — Remediation Design & Implementation
|
||||
|
||||
Source review: [60-testing-docs-gaps.md](../60-testing-docs-gaps.md) · Generated: 2026-07-09
|
||||
|
||||
This domain has no source-code defects to fix — its findings are process, coverage, documentation-currency, and half-shipped-feature gaps. The remediation therefore centres on standing up CI, filling the highest-value test gaps, finishing (or explicitly re-scoping) the session-resilience epic, and a focused doc-drift + repo-root triage pass. Every citation below was re-verified against the working tree on 2026-07-09; corrections are noted inline.
|
||||
|
||||
## Finding index
|
||||
|
||||
| ID | Sev | Title | Roadmap | Effort | Depends on | Files |
|
||||
|----|-----|-------|---------|--------|-----------|-------|
|
||||
| TST-01 | High | Reconnect/replay has no e2e test and no client consumer | P2 | L | TST-04 | oldtasks.md, clients/*, EventStreamServiceTests.cs |
|
||||
| TST-02 | High | Reconnect owner re-validation not implemented | P0 | M | TST-04 | Sessions/SessionManager.cs, Grpc/EventStreamService.cs |
|
||||
| TST-03 | High | No CI exists | P1 | M | — | (new) .gitea/workflows/ or .github/workflows/ |
|
||||
| TST-04 | High | Session-resilience epic 16/28 tasks unfinished | P2 | L | — | oldtasks.md, docs/plans/2026-06-15-session-resilience.md.tasks.json |
|
||||
| TST-05 | Medium | Real-worker control/COM paths verified opt-in only | P1 | S | TST-03 | WorkerLiveMxAccessSmokeTests.cs |
|
||||
| TST-06 | Medium | Dashboard live-data path untested | — | M | — | Dashboard/DashboardLiveDataService.cs |
|
||||
| TST-07 | Medium | Real-clock sleeps with negative assertions are latent flakes | — | S | — | WorkerClientTests.cs:431, SessionManagerTests.cs:400 |
|
||||
| TST-08 | Medium | Full-suite orphaned testhost processes | P1 | M | — | Tests fixtures (pipe/hosted-service disposal) |
|
||||
| TST-09 | Medium | Health checks cover only the auth store | — | M | — | GatewayApplication.cs:71 |
|
||||
| TST-10 | Medium | Deployment/upgrade undocumented and hand-rolled | — | M | — | (new) docs/Deployment.md, scripts/ |
|
||||
| TST-11 | Medium | No version discipline on server; client versions drift | P2 | M | — | src/Directory.Build.props, clients/* |
|
||||
| TST-12 | Medium | CLAUDE.md misstates default retention behaviour | P0 | S | — | CLAUDE.md:79 |
|
||||
| TST-13 | Medium | gateway.md carries stale design-era sketches | P2 | S | — | gateway.md:291,738,330 |
|
||||
| TST-14 | Medium | Repo-root working artifacts need triage | P2 | S | — | *-docs-*.md, oldtasks.md, stillpending.md |
|
||||
| TST-15 | Medium | Dashboard EventsHub has no per-session ACL | P2 | M | TST-04 | Dashboard/Hubs/EventsHub.cs:39 |
|
||||
| TST-16 | Medium | `Dashboard:ShowTagValues` is a dead flag | — | S | — | Configuration/DashboardOptions.cs:62 |
|
||||
| TST-17 | Medium | Vendor-gated alarm parity residuals silently lossy | — | S | — | Worker/MxAccess/WnWrapAlarmConsumer.cs:261 |
|
||||
| TST-18 | Low | Hosted-service wrappers untested | — | S | — | SessionLeaseMonitorHostedService.cs, OrphanWorkerCleanupHostedService.cs |
|
||||
| TST-19 | Low | Keep FakeWorkerHarness canned replies in lockstep | — | S | — | Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs |
|
||||
| TST-20 | Low | E2E script papers over a real advise-without-consumer sharp edge | — | S | — | scripts/run-client-e2e-tests.ps1 |
|
||||
| TST-21 | Low | Log rotation configured but minimal | — | S | — | Server/appsettings.json:8 |
|
||||
| TST-22 | Low | Config-shape JSON block omits documented keys | — | S | — | docs/GatewayConfiguration.md:12 |
|
||||
| TST-23 | Low | Bidirectional `Session` RPC never built | P2 | S | — | gateway.md:330 |
|
||||
| TST-24 | Low | Client wire behaviour has no automated verification | — | M | TST-03 | clients/*, (new) in-process fake gateway |
|
||||
|
||||
---
|
||||
|
||||
## TST-01 — Reconnect/replay has no e2e test and no client consumer `High` · `P2`
|
||||
|
||||
**Finding.** The server emits the `ReplayGap` sentinel and replays the ring on reconnect (unit-covered in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Grpc/EventStreamServiceTests.cs` and `MxAccessGatewayServiceTests.cs`), but epic Task 15 (fake-worker reconnect integration test) and Task 14 (client `ReplayGap` handling in all five clients) remain pending — verified `oldtasks.md:41-44` (`⏳ Task 14 (#121)`, `⏳ Task 15 (#122)`). No client parses the `ReplayGap` message.
|
||||
|
||||
**Impact.** The default-on reconnect protocol (`DetachGraceSeconds = 30`, `ReplayBufferCapacity = 1024` — both verified in code, see TST-12) is unproven end-to-end and unusable by every official client: a client that reconnects silently drops the gap marker and cannot tell replayed events from live ones, or that a gap was truncated.
|
||||
|
||||
**Design.** Land Tasks 14 and 15 as the completion of Phase 3 (co-designed with TST-02, TST-04). Two parts:
|
||||
1. **Integration test (Task 15):** drive `FakeWorkerHarness` through advise → detach subscriber (keep session in grace) → emit events into the replay ring → reconnect a new `StreamEvents` with a `last_sequence` cursor → assert the exact replayed set plus a `ReplayGap` when the requested cursor predates the ring. This lives entirely in fakes; no live COM needed.
|
||||
2. **Client consumers (Task 14):** each client's event-stream reader must surface `ReplayGap` as a typed, observable signal (not a dropped frame). Carry the per-language `optional`/presence-check idiom note from `oldtasks.md:44` — proto3 `optional` message fields need a `HasField`/`Option`/`is not None` presence check per language.
|
||||
|
||||
Rejected alternative: advertise reconnect as shipped and document the gap. Rejected because the roadmap already lists it as shipped and CLAUDE.md documents it as a feature (TST-12); leaving clients unable to consume it is a latent correctness trap, not a documentation nuance.
|
||||
|
||||
**Implementation.**
|
||||
- Test: add `GatewayEndToEndReconnectReplayTests` under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/` reusing `FakeWorkerHarness` and `ManualTimeProvider` to control the detach-grace clock.
|
||||
- Clients: `clients/dotnet`, `clients/go`, `clients/rust`, `clients/python`, `clients/java` event readers — expose `ReplayGap` on the stream item type. Cross-reference the clients-domain report (`CLI`) for the shared stream-surface shape.
|
||||
- Docs: `docs/Sessions.md` "Reconnect and replay" — change "server-side" caveat to "supported end-to-end"; each client README documents the gap signal.
|
||||
|
||||
**Verification.** `dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~GatewayEndToEndReconnectReplayTests`; per-client build/test per CLAUDE.md's client table (`gradle test`, `cargo test --workspace`, `python -m pytest`, `go test ./...`, `dotnet build clients/dotnet/...`).
|
||||
|
||||
---
|
||||
|
||||
## TST-02 — Reconnect owner re-validation not implemented `High` · `P0`
|
||||
|
||||
**Finding.** Epic Task 13 ("Owner re-validation on reconnect") is pending — verified `oldtasks.md:40` (`⏳ Task 13 (#120): Owner re-validation on reconnect — blockedBy 12, 1`). Nothing ties a resuming `StreamEvents` call to the API key that opened the session beyond possessing the `event` scope and knowing the session id.
|
||||
|
||||
**Impact.** With fan-out (`AllowMultipleEventSubscribers`) or detach-grace enabled, any `event`-scoped key that learns a session id can attach to another key's session and receive its replayed and live data. This is a trust-boundary hole in a window that is **on by default** (`DetachGraceSeconds = 30`). It is a security control, not a resilience nicety.
|
||||
|
||||
**Design.** Persist the opening key's identity on the session and enforce it on every `StreamEvents` attach/reattach. Options:
|
||||
- **(recommended)** Store the authenticated key id (already available on the auth context from `GatewayGrpcAuthorizationInterceptor`) on the session record at `OpenSession`; in `EventStreamService`/`MxAccessGatewayService.StreamEvents`, reject an attach whose caller key id ≠ the session owner with `PermissionDenied`. Admin-scope override deferred to Phase 4 (TST-15).
|
||||
- Rejected: gate solely on the `event` scope (status quo) — that is exactly the hole.
|
||||
|
||||
This is a P0 item in the roadmap ("Close the reconnect owner re-validation gap or flip retention defaults off until it lands"). If Task 13 cannot land immediately, the safe interim is to flip `DetachGraceSeconds`/fan-out defaults off and correct CLAUDE.md (TST-12) to match — but the recommended path is to implement re-validation because the feature is already documented as shipped.
|
||||
|
||||
**Implementation.**
|
||||
- `src/ZB.MOM.WW.MxGateway.Server/Sessions/SessionManager.cs` (session record gains an owner key id) and `Sessions/GatewaySession.cs`.
|
||||
- `src/ZB.MOM.WW.MxGateway.Server/Gateway/Grpc/EventStreamService.cs` / `MxAccessGatewayService.cs` — owner check on attach.
|
||||
- Tests: `EventStreamServiceTests` (owner match / mismatch / admin override), plus an attach-by-foreign-key rejection case in the reconnect integration test (TST-01).
|
||||
- Docs: `docs/Sessions.md` and `docs/DesignDecisions.md` — document the owner-binding rule; CLAUDE.md Authentication section note that session attach is owner-scoped.
|
||||
|
||||
**Verification.** `dotnet test ... --filter FullyQualifiedName~EventStreamServiceTests` and the TST-01 integration filter; `dotnet build src/ZB.MOM.WW.MxGateway.Server`.
|
||||
|
||||
---
|
||||
|
||||
## TST-03 — No CI exists `High` · `P1`
|
||||
|
||||
**Finding.** No pipeline files anywhere — verified: `.github/` and `.gitea/` absent, no `*.yml`/`Jenkinsfile`/`azure-pipelines.yml` at repo root. Every documented verification step is manual.
|
||||
|
||||
**Impact.** The five-language, dual-runtime (net10 / net48-x86) matrix is precisely the shape that breaks silently. Repo history shows the failure class is routine: net48 CS0246 on unregenerated protos, Java tracked-generated-file churn, seven-week-stale client descriptors (cross-domain `IPC`/`CLI`).
|
||||
|
||||
**Design.** Stand up a minimal pipeline. Because the x86 Worker only builds on a Windows host with MXAccess installed, split into a portable job (any runner) and a Windows job (self-hosted runner on windev, `10.100.0.48`). Live-MXAccess runs stay opt-in and scheduled, never gating a push.
|
||||
|
||||
Minimal pipeline (Gitea Actions syntax; `.github/workflows/` is drop-in equivalent if the mirror is GitHub):
|
||||
|
||||
```yaml
|
||||
# .gitea/workflows/ci.yml
|
||||
name: ci
|
||||
on: [push, pull_request]
|
||||
jobs:
|
||||
portable: # any linux/mac runner — no MXAccess, no x86
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Build NonWindows solution
|
||||
run: dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx -c Release
|
||||
- name: Gateway fake-worker tests
|
||||
run: dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj -c Release
|
||||
- name: Descriptor / codegen freshness
|
||||
run: pwsh scripts/check-descriptors.ps1 # see below; fail if regen would change tracked output
|
||||
- name: Client builds/tests
|
||||
run: |
|
||||
dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx -c Release
|
||||
(cd clients/go && gofmt -l . && go build ./... && go test ./...)
|
||||
(cd clients/rust && cargo fmt --check && cargo test --workspace && cargo clippy --workspace --all-targets -- -D warnings)
|
||||
(cd clients/python && python -m pip install -e ".[dev]" && python -m pytest)
|
||||
- name: Java client
|
||||
run: (cd clients/java && gradle test) # needs a JDK 17 runner; Mac dev box has none
|
||||
windows: # self-hosted windev runner — x86 worker + net48 tests
|
||||
runs-on: [self-hosted, windows]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
|
||||
- run: dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86
|
||||
live-mxaccess: # scheduled only, never gates a push
|
||||
if: github.event_name == 'schedule'
|
||||
runs-on: [self-hosted, windows, mxaccess]
|
||||
env: { MXGATEWAY_RUN_LIVE_MXACCESS_TESTS: "1" }
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- run: dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj
|
||||
```
|
||||
|
||||
**Freshness check.** The `descriptors/codegen freshness` step is the highest-leverage single guard (it protects the Generated/-commit rule and the stale client descriptor set). Implement `scripts/check-descriptors.ps1` to: rebuild `ZB.MOM.WW.MxGateway.Contracts`, regenerate the published client descriptor set, then `git diff --exit-code` the tracked `Generated/` and `clients/proto/descriptors/` paths — non-empty diff fails the job. This addresses the `IPC` domain's stale-descriptor finding directly; coordinate the exact regen command with that report.
|
||||
|
||||
Note the two known-noisy interactions to encode as CI hygiene: net48 requires the regenerated `Generated/*.cs` to be **committed** (regen-then-commit), while the Java build **regenerates and dirties** `clients/java/src/main/generated/.../MxaccessGateway.java` with spurious version churn that must be reverted when no `.proto` changed. The Java job must `git checkout` that file after `gradle test` (or the freshness check must exclude it) so the build stays green.
|
||||
|
||||
**Implementation.** New `.gitea/workflows/ci.yml` (+ `.github/workflows/ci.yml` if mirrored); new `scripts/check-descriptors.ps1`. Register a self-hosted runner on windev with `windows`/`mxaccess` labels. Docs: add a "Continuous Integration" section to `docs/GatewayTesting.md` describing the jobs and the scheduled live cadence; note the Windows-runner requirement for the x86 worker.
|
||||
|
||||
**Verification.** Push a branch and confirm the `portable` and `windows` jobs go green; deliberately introduce a proto change without regen and confirm the freshness step fails.
|
||||
|
||||
---
|
||||
|
||||
## TST-04 — Session-resilience epic 16/28 tasks unfinished `High` · `P2`
|
||||
|
||||
**Finding.** Verified `oldtasks.md:39-63`: pending are Task 13 (owner re-validation, TST-02), 14–15 (client ReplayGap + reconnect test, TST-01), 16–19 (per-session dashboard ACL, TST-15), and 20–28 (orphan-worker reattach, incl. the `EnableOrphanReattach` flag at Task 26 that does not yet exist). The repo advertises reconnect and fan-out as shipped while their trust boundary and client support are absent.
|
||||
|
||||
**Impact.** The default-on retention window has no owner check (security, TST-02); no client can consume the protocol (TST-01); Phase 5 reattach would reverse a documented CLAUDE.md invariant ("Gateway restart does not reattach orphan workers"). The epic is in a state where "shipped" and "planned" are entangled in the same docs.
|
||||
|
||||
**Design.** Treat the epic as three decisions, not one:
|
||||
- **Phase 3 (Tasks 13–15): finish now, as one unit.** These make the already-shipped server behaviour real and safe — TST-01 + TST-02 are that work.
|
||||
- **Phase 4 (Tasks 16–19): scope explicitly.** Per-session dashboard ACL is TST-15; decide the Viewer default (admin-sees-all vs strict per-session, the open decision at `oldtasks.md:52`) before implementing.
|
||||
- **Phase 5 (Tasks 20–28): re-scope or defer explicitly.** Orphan-worker reattach reverses a CLAUDE.md invariant and adds a manifest store + phone-home protocol. Recommendation: mark Phase 5 as **deferred, not planned** unless a concrete requirement exists; do not leave it as implied backlog. The `EnableOrphanReattach` flag must not be referenced anywhere as if it exists until Task 26 lands.
|
||||
|
||||
This finding is the umbrella; TST-01/02/15 are its actionable slices. The remediation here is the governance action: update the epic's tracking source and the roadmap so "shipped" claims match reality.
|
||||
|
||||
**Implementation.**
|
||||
- Update `docs/plans/2026-06-15-session-resilience.md.tasks.json` and its mirror `oldtasks.md` to reflect the Phase-3-finish / Phase-4-scope / Phase-5-defer decision (retire the mirror once the epic resumes or is closed, per TST-14).
|
||||
- CLAUDE.md "Repository-Specific Conventions" — the reconnect/fan-out paragraph must state which parts are shipped-and-consumable vs server-only-pending-clients until TST-01 lands.
|
||||
|
||||
**Verification.** Documentation-only for the umbrella; the slices carry their own build/test verification (TST-01, TST-02, TST-15).
|
||||
|
||||
---
|
||||
|
||||
## TST-05 — Real-worker control/COM paths verified opt-in only `Medium` · `P1`
|
||||
|
||||
**Finding.** All eleven late-added command kinds are unit-tested against fakes and live-verified once on the dev rig (`stillpending.md` §1.1), but the default suite exercises `Ping`/`GetWorkerInfo`/`DrainEvents`/`ShutdownWorker` only through `FakeWorkerHarness.RespondToControlCommandAsync` (verify current line range in `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs`), which returns canned replies.
|
||||
|
||||
**Impact.** A worker-side regression in these paths is invisible until someone sets `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`.
|
||||
|
||||
**Design.** Co-designed with TST-03: schedule `WorkerLiveMxAccessSmokeTests` on the windev runner on a cadence (the `live-mxaccess` job above). No new test code needed beyond ensuring the existing live suite covers the eleven command kinds; if any are absent, add live cases. This converts "opt-in, run by memory" into "runs nightly, reports failures."
|
||||
|
||||
**Implementation.** Covered by the `live-mxaccess` scheduled job in TST-03. Audit `WorkerLiveMxAccessSmokeTests.cs` for coverage of each command kind; add missing `[LiveMxAccessFact]` cases. Docs: `docs/GatewayTesting.md` opt-in matrix notes the scheduled cadence.
|
||||
|
||||
**Verification.** On windev: `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1 dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/...`.
|
||||
|
||||
---
|
||||
|
||||
## TST-06 — Dashboard live-data path untested `Medium` · `—`
|
||||
|
||||
**Finding.** `Dashboard/DashboardLiveDataService.cs` (verified present; owns the shared lazily-opened gateway session backing `/browse` live values) has no test class; `EventsHub`/`AlarmsHubPublisher` hub methods are likewise untested.
|
||||
|
||||
**Impact.** The one dashboard component that holds a real worker session and can fault it has no regression net; a fault-recovery or double-open bug ships silently.
|
||||
|
||||
**Design.** Add a fake-worker-backed test for the three behaviours that matter: (1) session reuse across concurrent `/browse` requests (one shared session, not one per request), (2) fault recovery — after the backing session faults, the next request re-opens rather than throwing, (3) disposal releases the session. `FakeWorkerHarness` already models faults, so no live COM needed.
|
||||
|
||||
**Implementation.** New `DashboardLiveDataServiceTests` under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Dashboard/`. Optionally add hub-method tests for `EventsHub`/`AlarmsHubPublisher` group join/leave. Docs: none required (internal test coverage).
|
||||
|
||||
**Verification.** `dotnet test ... --filter FullyQualifiedName~DashboardLiveDataServiceTests`.
|
||||
|
||||
---
|
||||
|
||||
## TST-07 — Real-clock sleeps with negative assertions are latent flakes `Medium` · `—`
|
||||
|
||||
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs:431` sleeps `Task.Delay(150ms)` ("give the heartbeat monitor a few real check-intervals") then asserts `client.State == Ready` (state did *not* change); `SessionManagerTests.cs:400` races a 50 ms delayed state flip against a bounded wait. (Report cited :433 and :402; the sleep is at :431 and the race setup at :400 — minor drift.) 47 `Thread.Sleep`/`Task.Delay` occurrences across 22 test files, most benign.
|
||||
|
||||
**Impact.** Fixed-real-time negative assertions can pass spuriously (the awaited thing hadn't happened yet anyway) or fail under CI load — corrosive in an already-manual regime, and doubly so once TST-03 runs them on shared runners.
|
||||
|
||||
**Design.** Convert the negative-assertion sleeps to `ManualTimeProvider`-driven check-interval pumping (advance the fake clock across the monitor's interval deterministically) or poll-until-stable with a short quiescence window. The heartbeat monitor already accepts an injected clock in production, so `WorkerClientTests` can advance it instead of sleeping. Leave the benign cancellation-bounded polls and `ManualTimeProvider` clock uses alone.
|
||||
|
||||
**Implementation.** `WorkerClientTests.cs` (heartbeat within-ceiling case) and `SessionManagerTests.cs` (state-flip race) — drive via the injected `TimeProvider`. Docs: none.
|
||||
|
||||
**Verification.** `dotnet test ... --filter FullyQualifiedName~WorkerClientTests` and `~SessionManagerTests`; run each 10× to confirm no spurious pass/fail.
|
||||
|
||||
---
|
||||
|
||||
## TST-08 — Full-suite orphaned testhost processes `Medium` · `P1`
|
||||
|
||||
**Finding.** CLAUDE.md ("Source Update Workflow") documents that the full gateway suite "leaves orphaned testhost processes" and mandates filtered runs as a workaround. The workaround is procedural and is now baked into the repo's memory (`feedback_targeted_tests_only.md`).
|
||||
|
||||
**Impact.** Any future CI (TST-03) or an unaware contributor inherits zombie processes; the leak points at an undisposed `NamedPipeServerStream` or a hosted-service test fixture that never stops.
|
||||
|
||||
**Design.** Find and fix the leaking fixture rather than institutionalizing the workaround. Method: run the full suite locally, enumerate surviving `testhost`/pipe handles, bisect by test collection to the leaking fixture (prime suspects: fake-worker harness pipe servers, `SessionLeaseMonitorHostedService`/`OrphanWorkerCleanupHostedService` test hosts that start a timer but are never `StopAsync`/`Dispose`d). Ensure every `IAsyncDisposable`/`IDisposable` fixture is disposed and every started hosted service is stopped in teardown. This is prerequisite to a reliable CI full-suite run; keep the filtered-run guidance until the leak is closed.
|
||||
|
||||
**Implementation.** Likely `src/ZB.MOM.WW.MxGateway.Tests` fixtures under `Gateway/Workers/Fakes/` and any hosted-service test. Docs: once fixed, update CLAUDE.md "Source Update Workflow" to drop the "leaves orphaned testhost processes" rationale (keep filtered-run guidance for speed, not correctness).
|
||||
|
||||
**Verification.** Run the full `dotnet test src/ZB.MOM.WW.MxGateway.Tests/...` once and confirm no surviving `testhost` processes (`pgrep -f testhost` empty after exit). Note the pre-existing macOS `SessionWorkerClientFactoryFakeWorkerTests` timeout-message failures are environmental and out of scope (`project_macos_pipe_timeout_test_failures.md`).
|
||||
|
||||
---
|
||||
|
||||
## TST-09 — Health checks cover only the auth store `Medium` · `—`
|
||||
|
||||
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Server/GatewayApplication.cs:71-75` registers exactly one `AuthStoreHealthCheck` tagged `Ready`. No readiness check for Galaxy SQL reachability, LDAP bind, alarm-monitor session health, or worker-executable presence/launchability — each a documented startup dependency.
|
||||
|
||||
**Impact.** `/health` reports Ready while the gateway cannot open a session (missing/invalid worker exe) or browse Galaxy (SQL down). Deploys look healthy and fail on first real request.
|
||||
|
||||
**Design.** Add tagged health checks next to the existing one, cheapest-first:
|
||||
- **Worker-executable check** — reuse `WorkerExecutableValidator` (already unit-tested); pure filesystem/PE check, no side effects. Tag `Ready`.
|
||||
- **Galaxy SQL** and **LDAP bind** — network checks; cache the result (e.g. 30 s) so `/health` polling doesn't hammer SQL Server / GLAuth. Tag `Ready` (or a separate `Dependencies` tag if operators want liveness vs readiness split).
|
||||
- Alarm-monitor session health — expose the monitor's current state as a check if the alarm feature is enabled.
|
||||
|
||||
Gate the network checks behind config so a gateway with dashboard/alarms disabled doesn't fail readiness on an unconfigured dependency.
|
||||
|
||||
**Implementation.** `GatewayApplication.cs` health-check registration; new check classes under `src/ZB.MOM.WW.MxGateway.Server/` (e.g. `Health/WorkerExecutableHealthCheck.cs`, `Health/GalaxySqlHealthCheck.cs`, `Health/LdapHealthCheck.cs`). Tests: `...Tests/Gateway/Health/`. Docs: `docs/GatewayConfiguration.md` and `gateway.md` health section — list the checks and their tags; note the Wonder deployment exposes only `/health/live` (`project_wonder_deployment.md`) so keep a liveness-only endpoint working.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; unit tests for each check; manual `curl /health/ready` with a missing worker exe returns Unhealthy.
|
||||
|
||||
---
|
||||
|
||||
## TST-10 — Deployment/upgrade undocumented and hand-rolled `Medium` · `—`
|
||||
|
||||
**Finding.** Deployments are NSSM-wrapped services with configuration in NSSM environment variables (acknowledged in `docs/GatewayConfiguration.md` and the project-memory deploy notes). `scripts/` has packaging and e2e scripts but no deploy/upgrade script, and no runbook for the `MxAccessGw`→`OtOpcUa` service-dependency dance.
|
||||
|
||||
**Impact.** Deploys are tribal knowledge (captured only in the operator's private memory files, `project_deploy_mechanics.md`); a second operator cannot reproduce a deploy or upgrade from the repo.
|
||||
|
||||
**Design.** Commit `docs/Deployment.md` (runbook) plus a publish/deploy script that captures the NSSM env-var configuration as code. The runbook records: framework-dependent publish command, exe layout under `C:\publish\mxaccessgw\{Server,Worker}`, the ports (5120 gRPC / 5130 dashboard), the `Stop-Service -Force` + manual `OtOpcUa` restart sequence, and the Wonder-host HTTPS/self-signed specifics. The script parameterizes host + env-var set. This is documentation/scripting only — no source change, and it must not embed secrets (LDAP service password, API keys) per CLAUDE.md.
|
||||
|
||||
**Implementation.** New `docs/Deployment.md`; new `scripts/deploy-gateway.ps1` (idempotent: stop, copy, apply NSSM env vars from a config file, restart, restart dependents). Reference — do not copy secrets from — the operator memory notes. Docs cross-link from `gateway.md`.
|
||||
|
||||
**Verification.** Dry-run the script against windev (`10.100.0.48`); confirm the service restarts and `OtOpcUa` comes back. No unit tests (ops script).
|
||||
|
||||
---
|
||||
|
||||
## TST-11 — No version discipline on server; client versions drift `Medium` · `P2`
|
||||
|
||||
**Finding.** Verified: Server/Worker csproj carry no `Version` (assemblies stamp 1.0.0); Contracts is `0.1.2` (`ZB.MOM.WW.MxGateway.Contracts.csproj:10`); Python `0.1.2` (`pyproject.toml:9`), Rust `0.1.2` (`Cargo.toml:3`), Java `0.2.0` (`build.gradle:16`) — and the Java bump required a hand-maintained `CLIENT_VERSION` constant that drifted once (commit `b6a0d90`).
|
||||
|
||||
**Impact.** Support cannot correlate a deployed gateway or a client artifact to a commit; duplicated version constants have already drifted.
|
||||
|
||||
**Design.** Single-source versions:
|
||||
- **.NET side:** define `<Version>`/`<InformationalVersion>` in `src/Directory.Build.props` (which already exists and enforces build policy), stamping the git SHA into the informational version. This covers Server, Worker, Contracts uniformly.
|
||||
- **Clients:** generate the client version constant at build from the package manifest rather than hand-maintaining it (Java's `CLIENT_VERSION` is the proven failure). Where a build-time generation step is impractical, add the descriptor-freshness-style check (TST-03) that fails if the constant ≠ the manifest version.
|
||||
- Decide one version cadence: either lockstep all five clients + contracts, or document why Java leads (JDK-17 retarget shipped as 0.2.0). Recommendation: bring Python/Rust/dotnet/contracts to 0.2.0 to converge, then move in lockstep.
|
||||
|
||||
**Implementation.** `src/Directory.Build.props`; `clients/java/build.gradle` (generate constant from `version`); analogous generation/check for other clients. Docs: `docs/ClientPackaging.md` and each client README note the single-sourcing. Coordinate with the `CLI` domain report on client-version surface.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.slnx` and confirm assembly `InformationalVersion` carries the SHA (`dotnet --info` / reflection); per-client build confirms the constant matches the manifest.
|
||||
|
||||
---
|
||||
|
||||
## TST-12 — CLAUDE.md misstates default retention behaviour `Medium` · `P0`
|
||||
|
||||
**Finding.** Verified `CLAUDE.md:79` states "Default config preserves the original single-subscriber, no-retention behavior," but code defaults enable retention: `DetachGraceSeconds = 30` (`Configuration/SessionOptions.cs:46`), `ReplayBufferCapacity = 1024` (`Configuration/EventOptions.cs:21`), `ReplayRetentionSeconds = 300` (`EventOptions.cs:29`). Only `AllowMultipleEventSubscribers` defaults off.
|
||||
|
||||
**Impact.** An agent or operator following CLAUDE.md assumes detached sessions die immediately and nothing is buffered — the opposite of reality, and it obscures the TST-02 security window that is on by default.
|
||||
|
||||
**Design.** Straight documentation correction (the roadmap lists this as P0 alongside owner re-validation). Replace the sentence with an accurate one.
|
||||
|
||||
**Implementation.** `CLAUDE.md:79` — change:
|
||||
|
||||
> Default config preserves the original single-subscriber, no-retention behavior.
|
||||
|
||||
to:
|
||||
|
||||
> Default config is single-subscriber (`AllowMultipleEventSubscribers` off), but detach-grace and replay retention are **on** by default (`DetachGraceSeconds=30`, `ReplayBufferCapacity=1024`, `ReplayRetentionSeconds=300`): a detached session is retained for 30 s and recent events are buffered for reconnect.
|
||||
|
||||
If TST-02's interim mitigation (flip retention off) is chosen instead of implementing owner re-validation, this sentence must instead document the flipped defaults — keep the two changes in the same commit. Docs-change-in-same-commit rule applies.
|
||||
|
||||
**Verification.** Read-back; no build. Cross-check the reworded claim against `SessionOptions.cs`/`EventOptions.cs` defaults.
|
||||
|
||||
---
|
||||
|
||||
## TST-13 — gateway.md carries stale design-era sketches `Medium` · `P2`
|
||||
|
||||
**Finding.** Verified in `gateway.md`:
|
||||
- The `WorkerEnvelope` snippet (~`:291-309`) shows `uint64 correlation_id = 4` and body tags `worker_hello=10 … command=20 … fault=26` with **no** `WorkerShutdownAck`. The actual proto uses `string correlation_id = 4` and `gateway_hello = 10 … worker_command = 13 … worker_shutdown_ack = 17 … worker_fault = 20` (`src/ZB.MOM.WW.MxGateway.Contracts/Protos/mxaccess_worker.proto:24,27,30,34,37`).
|
||||
- `gateway.md:738` still states the gateway "allow[s] one active client event subscriber per session, reject[s] a second subscriber" as unconditional policy — contradicted by config-gated fan-out later in the same file and by `docs/Sessions.md`.
|
||||
- `gateway.md:330` includes `rpc Session(stream ClientMessage) returns (stream ServerMessage)` which was never built; the real service is unary + server-streaming only (`mxaccess_gateway.proto:18-22`, no bidi RPC).
|
||||
|
||||
**Impact.** gateway.md is the mandated pre-change reading; internally inconsistent load-bearing sections cost every reader a reconciliation pass and can mislead a change.
|
||||
|
||||
**Design.** Don't re-inline the wire types (they drift). Replace the `WorkerEnvelope` sketch with a one-line pointer to `mxaccess_worker.proto` as the source of truth. Mark the single-subscriber paragraph as the *default* mode with a pointer to the fan-out section and `docs/Sessions.md`. Label the `Session` bidi RPC explicitly as unbuilt/deferred future work (see TST-23).
|
||||
|
||||
**Implementation.** `gateway.md` — three edits at the cited locations. Docs-only; no source change. This is part of the roadmap's P2 documentation-drift sweep and should land with the wider sweep (cookie name, GroupToRole sample, Java 17, Grpc.md RPC table) tracked in the `SEC`/`IPC`/`CLI` domains — cross-reference so the sweep is one commit where practical.
|
||||
|
||||
**Verification.** Read-back against the three proto/code citations above.
|
||||
|
||||
---
|
||||
|
||||
## TST-14 — Repo-root working artifacts need triage `Medium` · `P2`
|
||||
|
||||
**Finding.** Verified: `MxAccessGateway-docs-{issues,fixed,final}.md` and `MxGatewayClient-docs-{issues,fixed}.md` are **untracked** (`git status` shows them untracked; `*-docs-{issues,fixed,final}.md` are gitignored at `.gitignore:152-154`), largest 586 KB / ~15 k lines. `stillpending.md` (tracked) is a 2026-06-15 audit snapshot serving as the de-facto backlog; `oldtasks.md` (tracked) is a human-readable mirror of the epic tasks.json; `REVIEW-PROCESS.md` + `code-reviews/` is a completed review system; `A2-galaxyrepository-adoption-handoff.md` is a completed migration handoff referenced from CLAUDE.md.
|
||||
|
||||
**Impact.** The dead untracked `*-docs-*.md` files (15 k+ lines) trip greps and agents. The tracked snapshots blur "backlog" vs "historical record."
|
||||
|
||||
**Design.** Triage by evidence, not blanket deletion:
|
||||
- **Delete** the five untracked `*-docs-*.md` working files — dead local outputs of a finished docs-review pass. They are gitignored, so deletion is local only.
|
||||
- **`stillpending.md`:** either promote to a maintained register (drop the "Generated: … Commit:" snapshot framing) or migrate its open items (§1.3, §1.4, §3.x, epic remainder) to the Gitea tracker `docs/ImplementationPlanIndex.md` describes. Recommendation: migrate, since a tracker already exists in the plan.
|
||||
- **`oldtasks.md`:** keep only until the epic resumes/closes (TST-04), then delete — it is a mirror of the tasks.json.
|
||||
- **`REVIEW-PROCESS.md` + `code-reviews/`:** keep (living, self-consistent, zero open findings).
|
||||
- **`A2-galaxyrepository-adoption-handoff.md`:** archive under `docs/plans/` once its cross-repo follow-ups land; update the CLAUDE.md reference if moved.
|
||||
|
||||
**Implementation.** `rm` the untracked files; move/rewrite `stillpending.md`; update CLAUDE.md if `A2-…` moves. No source change.
|
||||
|
||||
**Verification.** `git status` clean of the `*-docs-*.md` files; `grep -rn` for those filenames returns nothing.
|
||||
|
||||
---
|
||||
|
||||
## TST-15 — Dashboard EventsHub has no per-session ACL `Medium` · `P2`
|
||||
|
||||
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/EventsHub.cs:39` carries `TODO(per-session-acl)` — the only production-source TODO in the repo. Any authenticated dashboard Viewer may join any `session:{id}` group (group name built at `EventsHub.cs:21`) and observe that session's event stream.
|
||||
|
||||
**Impact.** Acceptable for a single-tenant dashboard; wrong the moment `GroupToRole` admits low-trust viewers. It is the dashboard-side twin of the gRPC owner-revalidation gap (TST-02).
|
||||
|
||||
**Design.** Covered by epic Phase 4 (Tasks 16–19, TST-04). Introduce a role/scope that scopes a hub connection to permitted sessions, plus a hub-token session tag (Task 18). The open design decision (`oldtasks.md:52`) — Viewer default of admin-sees-all vs strict per-session — must be settled first. Recommendation: admin-sees-all, Viewer strictly scoped to sessions they own/are granted, matching TST-02's gRPC owner binding for consistency. Until Phase 4 lands, keep the TODO (it correctly documents the accepted single-tenant assumption); do not silently remove it.
|
||||
|
||||
**Implementation.** `Dashboard/Hubs/EventsHub.cs` (ACL check on group join), hub-token minting to carry the session tag, `Configuration/DashboardOptions.cs` for any group-to-tag config (Task 17). Tests: `...Tests/Gateway/Dashboard/` hub ACL cases incl. live-LDAP users (Task 19). Docs: `docs/Sessions.md`/`gateway.md` dashboard section document the ACL model; CLAUDE.md dashboard-auth paragraph.
|
||||
|
||||
**Verification.** `dotnet test ... --filter FullyQualifiedName~EventsHub`; `dotnet build src/ZB.MOM.WW.MxGateway.Server`.
|
||||
|
||||
---
|
||||
|
||||
## TST-16 — `Dashboard:ShowTagValues` is a dead flag `Medium` · `—`
|
||||
|
||||
**Finding.** Verified bound at `Configuration/DashboardOptions.cs:62`, projected into effective config (`EffectiveDashboardConfiguration.cs:9`, `GatewayConfigurationProvider.cs:63`), and displayed read-only on the settings page (`Dashboard/Components/Pages/SettingsPage.razor:65`) — but it gates no value-display behaviour anywhere. `docs/GatewayConfiguration.md:174` calls it "Reserved". (Note `docs/GatewayDashboardDesign.md:305` claims it "continues to govern" value display — that prose is aspirational/stale.)
|
||||
|
||||
**Impact.** An operator toggling a flag that renders on the settings page reasonably expects an effect; there is none.
|
||||
|
||||
**Design.** Decide implement-or-remove. Recommendation: **implement** — the flag has an obvious intended meaning (gate live tag-value display in `/browse`, consistent with the redaction philosophy in CLAUDE.md "Don't log secrets or full tag values by default"). Wire `ShowTagValues` into `DashboardLiveDataService`/the browse value-rendering path so `false` suppresses live values. If the team prefers not to implement, remove it from the options, the projector, the settings page, and downgrade the docs row — a settings-page-visible no-op is the worse outcome. Open question: default value (currently `false`) and whether it interacts with the per-session ACL (TST-15).
|
||||
|
||||
**Implementation.** `Dashboard/DashboardLiveDataService.cs` + the browse value component to honour the flag (implement path); or remove from `DashboardOptions.cs:62`, `EffectiveDashboardConfiguration.cs`, `GatewayConfigurationProvider.cs`, `SettingsPage.razor` (remove path). Tests: extend the TST-06 `DashboardLiveDataServiceTests` for the flag. Docs: `docs/GatewayConfiguration.md:174` and `docs/GatewayDashboardDesign.md:305` — replace "Reserved"/stale prose with the real behaviour.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Server`; test toggling the flag suppresses/shows values; docs match.
|
||||
|
||||
---
|
||||
|
||||
## TST-17 — Vendor-gated alarm parity residuals silently lossy `Medium` · `—`
|
||||
|
||||
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/WnWrapAlarmConsumer.cs:261-278`: the 8-arg `AlarmAckByName` `domain`/`full_name` fields are accepted on the wire then wrapped into the 6-arg call because the 8-arg form returns −55 on this AVEVA build (comments at `:119`, `:265-270`). `AlarmAckByGUID` is `E_NOTIMPL`. The `provider_switches` metric reason tagging has never been exercised live (`stillpending.md` §1.3–1.4, §3.4–3.5).
|
||||
|
||||
**Impact.** Contract fields that silently do nothing, invisible to callers — a parity gap dressed as a working feature.
|
||||
|
||||
**Design.** This is a genuine vendor limitation, not a bug to "fix" — per CLAUDE.md, MXAccess behaviour is the baseline and must not be papered over. The remediation is **observability, not behaviour change**: surface the drop so callers know. Add a diagnostic string to the ack reply (e.g. reply `message`/diagnostic field) noting that operator `domain`/`full_name` were not applied on this build (8-arg returns −55), and that `AlarmAckByGUID` is `E_NOTIMPL`. Do not synthesize success or invent a result — just report the degrade. Keep the TODO/comment referencing the AVEVA stub so a future runtime that implements it can drop the workaround.
|
||||
|
||||
**Implementation.** `Worker/MxAccess/WnWrapAlarmConsumer.cs` — populate a diagnostic in the ack reply when the 8-arg path is downgraded. Confirm the gateway forwards that diagnostic to the client reply (`Gateway/Grpc/...`). Tests: worker ack-consumer test asserting the diagnostic is present on the downgraded path (`src/ZB.MOM.WW.MxGateway.Worker.Tests`). Docs: `gateway.md`/`docs/` alarm section and the relevant client READMEs note the vendor-gated residual. Cross-reference the `WRK` domain report for the alarm-consumer specifics.
|
||||
|
||||
**Verification.** `dotnet build src/ZB.MOM.WW.MxGateway.Worker -p:Platform=x86`; `dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/... -p:Platform=x86` (Windows).
|
||||
|
||||
---
|
||||
|
||||
## TST-18 — Hosted-service wrappers untested `Low` · `—`
|
||||
|
||||
**Finding.** `SessionLeaseMonitorHostedService` and `OrphanWorkerCleanupHostedService` delegate to well-tested cores (sweep logic covered via `SessionManagerTests.CloseExpiredLeasesAsync` cases), but their timer/startup wiring has no direct tests.
|
||||
|
||||
**Impact.** Low — failure mode (a timer that never fires) is obvious at startup and in integration.
|
||||
|
||||
**Design.** Add a thin `StartAsync` → tick-once (via injected `TimeProvider`) → assert the core method was invoked → `StopAsync` disposes cleanly test for each. This also directly exercises the disposal path relevant to TST-08 (verify the hosted service stops its timer on `StopAsync`), so it doubles as a leak-regression guard.
|
||||
|
||||
**Implementation.** New small test classes under `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Sessions/` and `.../Workers/`. Docs: none.
|
||||
|
||||
**Verification.** `dotnet test ... --filter FullyQualifiedName~HostedService`.
|
||||
|
||||
---
|
||||
|
||||
## TST-19 — Keep FakeWorkerHarness canned replies in lockstep `Low` · `—`
|
||||
|
||||
**Finding.** `FakeWorkerHarness` shares `WorkerFrameReader`/`WorkerFrameWriter`/`WorkerEnvelope` with production and scripts malformed/oversized frames, heartbeats, faults, shutdown acks, and control-command reply shapes. What it cannot represent (STA pumping, COM HRESULT semantics, process exit codes, event timing under load) is honestly delegated to the live suite. History (`stillpending.md` §1.1) shows the canned replies drifted once: green fakes masked an unimplemented real worker for months.
|
||||
|
||||
**Impact.** Low today; the risk is future drift where fake replies stop matching `WorkerPipeSession`, hiding a real-worker regression.
|
||||
|
||||
**Design.** No code fix — this is a process guard. The durable protection is TST-05's scheduled live suite (it catches fake/real drift automatically) plus a code-review checklist note. Add a comment/README note in the fakes directory pointing at `WorkerPipeSession` as the reply-shape source of truth, so any control-command change updates both.
|
||||
|
||||
**Implementation.** `src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/Fakes/` — add a short README or header comment. Rely on TST-05 (scheduled live) for the real guard. Docs: `docs/GatewayTesting.md` already states the fidelity charter; add the "keep in lockstep, live suite catches drift" note.
|
||||
|
||||
**Verification.** Covered by TST-05's scheduled live run; no standalone test.
|
||||
|
||||
---
|
||||
|
||||
## TST-20 — E2E script papers over a real advise-without-consumer sharp edge `Low` · `—`
|
||||
|
||||
**Finding.** `scripts/run-client-e2e-tests.ps1` (~1,715 lines) interleaves `StreamEvents` drains every ~15 advised tags because advising without a consumer overflows the worker event channel and faults the session under `FailFast` (`docs/GatewayTesting.md` "Client E2E Scripts" phase 3).
|
||||
|
||||
**Impact.** Low as a test detail, but it is real product feedback: any client doing bulk-advise-then-stream will fault sessions.
|
||||
|
||||
**Design.** Treat as product feedback, not a script fix. The clean solution is a subscribe-time channel policy (buffer/backpressure the event channel until a subscriber attaches, or a bounded pre-subscription buffer) — but that is a cross-domain backpressure decision owned by the `IPC`/`GWC`/`WRK` reports' "coordinated size/backpressure pass" (roadmap P1 #8). This finding's action is to **record** the sharp edge as an input to that pass and, until then, document the advise-then-stream ordering requirement in the client READMEs so real clients don't fault sessions.
|
||||
|
||||
**Implementation.** No script change. Docs: client READMEs and `gateway.md` event section note "attach `StreamEvents` before bulk advise, or drain periodically." Cross-reference the backpressure-pass finding in the other domains.
|
||||
|
||||
**Verification.** Docs read-back; the behavioural fix is verified under the coordinated backpressure pass, not here.
|
||||
|
||||
---
|
||||
|
||||
## TST-21 — Log rotation configured but minimal `Low` · `—`
|
||||
|
||||
**Finding.** Verified `src/ZB.MOM.WW.MxGateway.Server/appsettings.json:8-11` rolls the Serilog file sink daily (`logs/mxgateway-.log`, relative to the service working directory) with no `retainedFileCountLimit` or size cap (Serilog defaults: 31 files, unbounded size per day).
|
||||
|
||||
**Impact.** Acceptable, but a high-rate event day can produce an unbounded single file, and a relative path lands logs in the service CWD.
|
||||
|
||||
**Design.** Set `fileSizeLimitBytes` + `rollOnFileSizeLimit: true` and an absolute log path for the deployed service. Keep it in the deployment config (NSSM env-var overlay / deploy script from TST-10) rather than only in the committed `appsettings.json`, since the absolute path is host-specific.
|
||||
|
||||
**Implementation.** `src/ZB.MOM.WW.MxGateway.Server/appsettings.json` Serilog file sink (add size cap + retained-count) and the TST-10 deploy config for the absolute path. Docs: `docs/GatewayConfiguration.md` logging notes; `docs/Deployment.md`.
|
||||
|
||||
**Verification.** `dotnet run --project src/ZB.MOM.WW.MxGateway.Server/...` and confirm rollover on size; no unit test.
|
||||
|
||||
---
|
||||
|
||||
## TST-22 — Config-shape JSON block omits documented keys `Low` · `—`
|
||||
|
||||
**Finding.** Verified the Configuration Shape JSON block (`docs/GatewayConfiguration.md:12-83`) omits `DisableLogin`/`AutoLoginUser`/`CookieName`, the `Tls` block, and the `Alarms:Fallback` block — all of which the tables below it document. (The rest of `docs/GatewayConfiguration.md` is current: table defaults match `Configuration/*.cs`.)
|
||||
|
||||
**Impact.** Low — the tables are authoritative and correct; the shape block is a convenience that is now incomplete, so a copy-paste starting config misses those keys.
|
||||
|
||||
**Design.** Add the missing keys to the JSON block so it round-trips with the tables. Pure documentation.
|
||||
|
||||
**Implementation.** `docs/GatewayConfiguration.md:12-83` — add `Dashboard:DisableLogin`, `Dashboard:AutoLoginUser`, `Dashboard:CookieName`, the `Tls` section, and `Alarms:Fallback`. Verify against `Configuration/DashboardOptions.cs`, `TlsOptions.cs`, and the alarms fallback options.
|
||||
|
||||
**Verification.** Read-back against the cited options classes.
|
||||
|
||||
---
|
||||
|
||||
## TST-23 — Bidirectional `Session` RPC never built `Low` · `P2`
|
||||
|
||||
**Finding.** Verified `gateway.md:330` presents `rpc Session(stream ClientMessage) returns (stream ServerMessage)` as the "best long-term shape"; the shipped service stopped at unary + server-streaming (`mxaccess_gateway.proto:18-22`, no bidi RPC).
|
||||
|
||||
**Impact.** None functional — purely a docs-currency/roadmap-clarity issue (the twin of TST-13).
|
||||
|
||||
**Design.** Fold into the TST-13 gateway.md edit: label the `Session` RPC explicitly as unbuilt future work (or remove it if it is no longer the intended direction). Decide whether bidi `Session` is still on the roadmap; if not, drop the "best long-term shape" framing so it doesn't read as an imminent deliverable.
|
||||
|
||||
**Implementation.** `gateway.md` around `:328-345` — one clarifying edit, made with TST-13. Docs-only.
|
||||
|
||||
**Verification.** Read-back; consistent with `mxaccess_gateway.proto`.
|
||||
|
||||
---
|
||||
|
||||
## TST-24 — Client wire behaviour has no automated verification `Low` · `—`
|
||||
|
||||
**Finding.** All five clients have unit tests (13/8/3/13/7 files for dotnet/go/rust/python/java) but no in-process or containerized gateway integration tests; the only cross-language verification is the operator-run `scripts/run-client-e2e-tests.ps1`. `CrossLanguageSmokeMatrixTests` checks shapes only.
|
||||
|
||||
**Impact.** Low-to-moderate: a gateway contract change can pass every default suite and break all five clients (partly mitigated by shared-proto codegen). The Java CLI already proves the cheap pattern — `InProcessGatewayHarness` (`stillpending.md` §8).
|
||||
|
||||
**Design.** Replicate the in-process fake-gateway pattern per client so each client's happy path (open → invoke → stream) runs against a lightweight fake gateway in the client's own test suite, wired into CI (TST-03). Java has the template; port the idea to the other four. This is the cheapest way to catch contract breaks without a live rig. Full parity is out of scope — target the core session/invoke/stream shapes and the `ReplayGap` handling added in TST-01.
|
||||
|
||||
**Implementation.** Per-client test harness (e.g. an in-process gRPC server serving canned replies) under each `clients/<lang>/` test tree; add to the CI client jobs. Coordinate the fake-gateway contract with the `CLI` domain report. Docs: each client README notes the harness.
|
||||
|
||||
**Verification.** Per-client test command (CLAUDE.md client table); the harness tests run inside CI (TST-03).
|
||||
Reference in New Issue
Block a user