# MxAccessGateway — Remediation Tracking (2026-07-12 review)

Master progress tracker for the 2026-07-12 follow-up architecture review. Generated 2026-07-13.

Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation designs are linked below and hold the full **Finding / Impact / Design / Implementation / Verification** for every entry here. The first-cycle tracker ([`../../remediation/00-tracking.md`](../../remediation/00-tracking.md)) remains the record for the original 153 findings; this document tracks only the 47 new IDs (GWC-24+, WRK-21+, IPC-23+, SEC-31+, CLI-35+, TST-25+) plus the old-tracker actions listed at the end.

## How to use this document

- Each finding has a stable ID that never changes. Cite it in commits, branches, and PRs (e.g. `fix(GWC-25): empty-ring ReplayGap sentinel`).
- The **Status** column is the single source of truth for progress. Update it in the same change that lands the fix.
- Do the work in **roadmap-tier order** (P0 → P1 → P2), respecting the `Dep` column and the cross-cutting clusters below — several fixes are one change set across two domains and must land together.
- When a fix lands: flip Status to `Done` and, per the repo rule, update the affected docs in the same commit.

**Status legend:** `Not started` · `In progress` · `In review` · `Done` · `Won't fix` (record why in the domain doc) · `N/A` (informational / decision recorded, no action).

## Severity roll-up

| Domain | Doc | High | Medium | Low | Info | Total |
|---|---|:-:|:-:|:-:|:-:|:-:|
| Gateway core | [10-gateway-core.md](10-gateway-core.md) | — | 2 | 4 | 1 | 7 |
| Worker | [20-worker.md](20-worker.md) | — | 1 | 7 | — | 8 |
| Contracts & IPC | [30-contracts-ipc.md](30-contracts-ipc.md) | — | 3 | 5 | 2 | 10 |
| Security & dashboard | [40-security-dashboard.md](40-security-dashboard.md) | — | 1 | 4 | 1 | 6 |
| Clients | [50-clients.md](50-clients.md) | — | 5 | 6 | — | 11 |
| Testing, docs & gaps | [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | 1 | 2 | 3 | — | 6 |
| **Total** | | **1** | **14** | **29** | **4** | **48** |

## Roadmap-tier roll-up

| Tier | Meaning | Count | Findings |
|---|---|:-:|---|
| **P0** | Correctness & safety — all small-to-medium | 10 | GWC-25, WRK-21, IPC-23, IPC-24, IPC-25, IPC-30, SEC-31, SEC-32, CLI-35, CLI-36 |
| **P1** | Process & hardening | 12 | GWC-24, WRK-26, SEC-33, SEC-36, CLI-37, CLI-38, CLI-39, CLI-42, CLI-45, TST-25, TST-26, TST-27 |
| **P2** | Completeness & polish | 10 | GWC-26, GWC-27, GWC-28, WRK-25, IPC-26, IPC-27, SEC-34, TST-28, TST-29, TST-30 |
| **—** | Not individually in the roadmap (Lows/Infos) | 16 | GWC-29/30, WRK-22/23/24/27/28, IPC-28/29/31/32, SEC-35, CLI-40/41/43/44 |

### P0 — do first (10)

Sequenced by cluster; a cluster is one change set.

| ID | Sev | Eff | Dep | Status | Title |
|---|---|:-:|---|---|---|
| GWC-25 | Medium | S | CLI-35/36 (coord) | Done | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
| CLI-35 | Medium | S | GWC-25 (coord) | Done | Python CLI `stream-events` crashes on a ReplayGap |
| CLI-36 | Medium | S | GWC-25 (coord) | Done | Go CLI `stream-events` silently destroys the ReplayGap signal |
| WRK-21 | Medium | M | owns IPC-23 fix; WRK-28 same batch | Done | DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events |
| IPC-23 | Medium | S | WRK-21 | Done | DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave |
| IPC-30 | Low | M | WRK-21 (same batch) | Done | Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity) |
| SEC-31 | Medium | M | — | Done | Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS) |
| SEC-32 | Low | S | SEC-31 | Done | Failure-limiter LRU flushable by junk-token spray; token prefix never validated |
| IPC-24 | Medium | S | — | Done | CI's unconditional Java churn-revert masks real drift |
| IPC-25 | Medium | M | — | Done | Regenerate stale Go/Python worker bindings + add binding-freshness guard (Check 4) to check-codegen.ps1 |

## Finding registers by domain

Full design + implementation for each row lives in the linked domain doc under its ID heading.

### Gateway core — [10-gateway-core.md](10-gateway-core.md)

| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| GWC-24 | Medium | P1 | M | GWC-21 (coord, old tracker) | Done | Unbounded event staging channel: sustained slow drain grows memory silently and invisibly |
| GWC-25 | Medium | P0 | S | CLI-35/36 (coord) | Done | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0` |
| GWC-26 | Low | P2 | M | GWC-27 | Done | Alarm monitor attaches its subscriber after SubscribeAlarms; window transitions bypass the feed |
| GWC-27 | Low | P2 | S | GWC-26 | Done | `AttachInternalEventSubscriber` bypasses the readiness gate; premature attach poisons the distributor |
| GWC-28 | Low | P2 | S | GWC-10 (coord, old tracker) | Done | Gateway→worker envelope `sequence` stamped at creation, not at write |
| GWC-29 | Low | — | S | — | Done | `Invoke` deep-clones the entire request only to discard the cloned command |
| GWC-30 | Info | — | S | — | Done | Frame reader allocates a fresh 4-byte length-prefix array per frame |

### Worker — [20-worker.md](20-worker.md)

| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| WRK-21 | Medium | P0 | M | IPC-23 (fix owned here); WRK-28 | Done | DrainEvents bound count-based only; oversized reply kills session and loses drained events |
| WRK-22 | Low | — | S | IPC-26 (fix owned here) | Done | Cancelled `WriteAsync` leaves its frame queued; it is still written later |
| WRK-23 | Low | — | S | WRK-21 | Done | Rejected frames consume sequence numbers, producing wire gaps |
| WRK-24 | Low | — | S | — | Done | `AdoptNegotiatedMaxMessageBytes` has no lower-bound sanity check |
| WRK-25 | Low | P2 | S | WRK-22 (shared seam) | Done | WRK-12 flush coalescing never engages on the event hot path |
| WRK-26 | Low | P1 | S | WRK-23 (soft); discharges IPC-29 | Done | Write-priority and overflow doc drift from the WRK-07 change |
| WRK-27 | Low | — | S | — | Done | Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s) |
| WRK-28 | Low | — | S | WRK-21 (same batch) | Done | 10,000 drain cap is a duplicated magic constant |

### Contracts & IPC — [30-contracts-ipc.md](30-contracts-ipc.md)

| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| IPC-23 | Medium | P0 | S | WRK-21 (mechanics) | Done | DrainEvents byte-blindness — contract requirements + proto-comment/doc wave (mechanics in WRK-21; proto-comment/regen wave landed 2026-08-07 with the codegen cluster; the byte-cap doc rule lives in `docs/WorkerFrameProtocol.md` — `docs/Grpc.md` intentionally carries no DrainEvents row since it scopes to the seven public gRPC RPCs and DrainEvents is a worker IPC diagnostic) |
| IPC-24 | Medium | P0 | S | — | Done | CI's unconditional Java churn-revert masks real drift |
| IPC-25 | Medium | P0 | M | — | Done | Stale Go/Python worker bindings: regenerate (pinned toolchains) + check-codegen Check 4 |
| IPC-26 | Low | P2 | S | WRK-22 (mechanics) | Done (mechanics in WRK-22) | Cancelled write leaves ghost frame — cancelled means never written |
| IPC-27 | Low | P2 | S | — | Done | Descriptor-freshness test blind to enums/services/galaxy descriptor |
| IPC-28 | Low | — | S | — | Done | docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping |
| IPC-29 | Low | — | S | WRK-26 (discharged by) | Done (discharged by WRK-26) | WorkerFrameProtocol.md missing write-scheduling/sequencing section |
| IPC-30 | Low | P0 | M | WRK-21 (same batch) | Done | Oversized event frame: keep session-fatal, make the death structured |
| IPC-31 | Info | — | — | — | N/A | Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded |
| IPC-32 | Info | — | S | IPC-25 (folded in) | Done | check-codegen banner relabel 1/4…4/4 |

### Security & dashboard — [40-security-dashboard.md](40-security-dashboard.md)

| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| SEC-31 | Medium | P0 | M | — | Done | Failure limiter: composite (peer, key-id) partitions + cross-peer aggregate with probe admission |
| SEC-32 | Low | P0 | S | SEC-31 | Done | Limiter LRU flushable by junk-token spray; validate token shape, cap per-peer partitions |
| SEC-33 | Low | P1 | M | old SEC-23 (co-locate) | Done | Host-meaningful path rooting; drop Windows literals from appsettings; validate Galaxy `SnapshotCachePath` |
| SEC-34 | Low | P2 | S | — | Done | Verification cache: expiry outlives TTL; `Invalidate` races in-flight repopulation |
| SEC-35 | Info | — | S | — | N/A (doc note discharged 2026-08-07) | Production hard-stops key on exact `Production` environment name (doc-only) |
| SEC-36 | Low | P1 | M | cross-repo `scadaproj/infra/glauth` | Done | Committed dev LDAP service-account password: rotate, remove, move dev channel to user-secrets |

### Clients — [50-clients.md](50-clients.md)

| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| CLI-35 | Medium | P0 | S | GWC-25 (coord) | Done | Python CLI `stream-events` crashes on a ReplayGap |
| CLI-36 | Medium | P0 | S | GWC-25 (coord) | Done | Go CLI `stream-events` silently destroys the ReplayGap signal |
| CLI-37 | Medium | P1 | M | CLI-38 (co-land) | Done | Status-array validation must branch on `category` per the proto contract (4-vs-1 divergence) |
| CLI-38 | Medium | P1 | S | — | Done | Align .NET/Go/Java on `hresult < 0` — lands old CLI-08, cures design-doc drift |
| CLI-39 | Medium | P1 | S | CLI-35..38, CLI-45 (land last) | Done | Bump client versions off published 0.1.2 (converge on 0.2.0); registry-collision guard in pack-clients.ps1 |
| CLI-40 | Low | — | M | — | Done | Port the exact-secret credential scrub to Rust/Java/.NET |
| CLI-41 | Low | — | M | — | Done | Uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/AddBufferedItem |
| CLI-42 | Low | P1 | S | — | Done | Document the vendored Rust proto layout (CLI-02's missing doc half) |
| CLI-43 | Low | — | S | — | Done | Java style guide still prescribes "Java 21 preferred" |
| CLI-44 | Low | — | S | — | Done | Go event goroutine can mislabel a genuine terminal error as `ErrSlowConsumer` |
| CLI-45 | Low | P1 | M | — | Done | Standardize CLI credential env-var names; fail fast on missing/empty passwords |

### Testing, docs & gaps — [60-testing-docs-gaps.md](60-testing-docs-gaps.md)

| ID | Sev | Tier | Eff | Dep | Status | Title |
|---|---|:-:|:-:|---|---|---|
| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | Done | Windows/x86 test tier has zero automation — SSH-driven windev CI job |
| TST-26 | Medium | P1 | S | TST-25 (same commit) | Done | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen |
| TST-27 | Medium | P1 | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
| TST-28 | Low | P2 | S | relates IPC-02 (old) | Done | Gateway-side `max_frame_bytes` handshake untested in the CI-run suite |
| TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` (fold Phase-5 governance into DesignDecisions.md); delete root artifacts |
| TST-30 | Low | P2 | M | — | Done | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete API) |

## Cross-cutting clusters

Sequence these together rather than piecemeal — several are one change set spanning two domains:

- **Drain / oversized-frame cluster (P0):** WRK-21 + IPC-23 + IPC-30, with WRK-28 and WRK-23 in the same batch. One worker change set in `WorkerPipeSession.cs`/`WorkerFrameWriter.cs`; WRK-21 owns the byte-budgeted drain and must satisfy IPC-23's requirements R1–R3; IPC-30's structured-fault seam lands in the same drain loop. IPC-23's proto-comment edits trigger the full regen fan-out (Generated/, Rust vendored, Go/Java, descriptor set) — land with WRK-21 so the wave happens once. One windev x86 verification run for the cluster.
- **ReplayGap end-to-end (P0):** GWC-25 (server sentinel arithmetic) + CLI-35 (Python CLI) + CLI-36 (Go CLI). Independently landable; the e2e resume walk validates only when all three are in.
- **Auth limiter (P0):** SEC-31 + SEC-32 — same component, one change set, one test suite.
- **Codegen freshness (P0):** IPC-24 + IPC-25 (+ IPC-32 folded in). Both edit `.gitea/workflows/ci.yml` — coordinate the branch with TST-25, which touches the same file.
- **Windows-tier automation (P1):** TST-25 + TST-26 (same commit). Unlocks old TST-05/TST-24 and provides CI evidence for every windev-verified cluster above; until it lands, record windev runs in this tracker's change log.
- **Client conformance + release train (P1):** CLI-37 + CLI-38 co-land (one conformance commit; closes old CLI-08), then CLI-45, with CLI-40/41 fixtures as follow-ups; **CLI-39 lands last** so published 0.2.0 carries the conformant behavior — `Done` 2026-08-07 (Rust/Python/Go/.NET → 0.2.0, Java → 0.2.1; see CLI-39 change-log entry). Shared fixtures under `clients/proto/fixtures/behavior/`; update CrossLanguageSmokeMatrix.md/ClientLibrariesDesign.md same-commit. IPC-25's regenerated Go/Python bindings are still not republished — CLI-39 bumped the version constants but did not run an actual publish (explicitly out of scope; see the change-log entry).
- **Doc-drift batch (P1):** TST-27 + WRK-26 (discharges IPC-29) + CLI-42 + IPC-28 + SEC-35's doc note — one sweep commit is fine.
- **Backpressure follow-on (P1):** GWC-24 — `Done` 2026-08-07. Landed without a new config key (the staging bound is derived as `2 × EventChannelCapacity`), so still-open old GWC-21 (`EventChannelFullModeTimeout` configurability) remains the natural companion knob but was not a blocker.

## Old-tracker actions ([`../../remediation/00-tracking.md`](../../remediation/00-tracking.md))

- Close **CLI-24** and **CLI-34** as `Done` (incidentally fixed; evidence in [../50-clients.md](../50-clients.md)).
- ~~When CLI-38 lands, close old **CLI-08** with a pointer here.~~ Done 2026-08-07: CLI-38 landed and old CLI-08 is now `Done` in the first-cycle tracker, pointing at [CLI-38](50-clients.md#cli-38--align-netgojava-on-hresult--0-lands-prior-cli-08-cures-the-doc-drift---medium--p1).
- ~~When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap~~ Done 2026-08-07: WRK-26 landed and **IPC-29** is discharged, pointing at [WRK-26](20-worker.md#wrk-26--write-priority-and-overflow-doc-drift-from-the-wrk-07-change---low--p1). ~~When TST-25 lands, revisit old **TST-05** (scheduled live smoke) and **TST-24** (client wire tests), which it unlocks.~~ TST-05 revisited 2026-08-10: `Partially done` in the first-cycle tracker — the `nightly-windev` job closes the scheduled-cadence half, but the finding's coverage-audit half stays open (the live suite reaches all six late-added COM commands and none of the five control commands). TST-24 revisited in the same change and closed `Done`: Go/Rust/Java already had real-server wire tests, and the two genuine gaps (.NET, Python) now have them plus a CI step.

## Change log

| Date | Change |
|---|---|
| 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). |
| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. |
| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). **Superseded 2026-08-10 — Check 6 is now Done; see the 2026-08-10 change-log entry in `archreview/remediation/00-tracking.md`.** |
| 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
| 2026-08-07 | **TST-29 → `Done`:** migrated the Phase-5 (orphan-worker reattach) deferred-not-planned governance record and the settled Phase-4 Viewer-default decision from `oldtasks.md` into a new "Session-Resilience Epic Scope" entry in `docs/DesignDecisions.md`; repointed CLAUDE.md and `stillpending.md:7,165` from `oldtasks.md` to `docs/DesignDecisions.md` / `docs/plans/2026-06-15-session-resilience.md.tasks.json`; `git rm oldtasks.md`. The five untracked root docs-review artifacts (`MxAccessGateway-docs-{issues,fixed,final}.md`, `MxGatewayClient-docs-{issues,fixed}.md`) were absent from this worktree — delete from the main working tree separately. |
| 2026-08-07 | **GWC-24 → `Done`** (branch `fix/gwc-24-staging-bound`). `WorkerClient._eventStaging` is now `Channel.CreateBounded` at `2 × EventChannelCapacity` (`Wait`, single reader/writer, no sync continuations); a rejected staging `TryWrite` faults the client `ProtocolViolation` with `QueueOverflow("worker-event-staging")` unless `IsTerminalState()` (shutdown stays a silent drop), so a consumer draining slower than its worker produces dies at a fixed ceiling instead of growing gateway memory. Queue-depth accounting moved from `EnqueueWorkerEventAsync` to `StageWorkerEvent`, so the single gauge reports staged + queued; the timed-write fault (`EventChannelFullModeTimeout` / `QueueOverflow("worker-events")`) is unchanged and still catches the full-stall case first. No new config key — total gateway-side buffering is `3 × MxGateway:Events:QueueCapacity`, derived; coordination with still-open old **GWC-21** (`EventChannelFullModeTimeout` configurability) remains open and was not blocked on. Docs same commit: `GatewayProcessDesign.md` (two overflow faults), `MxAccessWorkerInstanceDesign.md`, `GatewayConfiguration.md`, `Metrics.md`. Tests: `WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout` and `.WorkerEventQueueDepthGaugeCountsStagedEvents`; `WorkerClientTests` 22/22 green, `NonWindows.slnx` builds with 0 warnings. |
| 2026-08-07 | **ReplayGap end-to-end cluster (GWC-25 + CLI-35 + CLI-36) → `Done`** on `fix/gwc-25-replaygap-trio`. GWC-25: `SessionEventDistributor.RegisterWithReplay`'s empty-ring branch now reports `oldestAvailableSequence = _highestSequenceSeen + 1` when `gap == true` (still `0` when no gap), so the universal `oldest - 1` resume formula no longer wraps to `ulong.MaxValue` and dead-stream the subscriber; `docs/Sessions.md` documents the empty-ring value. CLI-35: the Python CLI renders a `ReplayGap` as a `{"replayGap": {...}}` row via a new `_event_row` helper instead of crashing in `MessageToDict`. CLI-36: the Go CLI branches on `result.IsReplayGap()` and prints the typed `REPLAY_GAP requested_after=<n> oldest_available=<n>` line / `replayGap` JSON row instead of formatting the library's cleared `Event`. `docs/CrossLanguageSmokeMatrix.md` gained a per-CLI gap-rendering table (one edit covering both client findings). Four new tests as designed (3 × `SessionEventDistributorTests`, `GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWithSentinelFormula`) plus `test_stream_events_renders_replay_gap` (Python) and `TestRunStreamEventsPrintsReplayGap` (Go); all written red first and each reproducing its defect verbatim. **Deferred:** GWC-25's `ReplayGap.oldest_available_sequence` proto-comment amendment is **not** in this change — it is comment-only but triggers the full five-client regen fan-out, so it lands with the later codegen wave (alongside IPC-23's proto-comment edits) rather than forcing a regen for one sentence. Note for that wave: the fake-worker gateway e2e suite cannot run on the macOS worktree without `TMPDIR` shortened (macOS caps the Unix-domain-socket path backing .NET named pipes at 104 chars; `TMPDIR=/tmp dotnet test …` works and was used here). |
| 2026-08-07 | **GWC-27 → `Done`, GWC-26 → `Done`** (branch `fix/gwc-26-27-alarm-attach`). GWC-27: `GatewaySession.AttachInternalEventSubscriber` now mirrors `AttachEventSubscriber`'s readiness gate under `_syncRoot`, before `EnsureDistributorCreated`, so a premature attach can no longer latch a poisoned distributor. GWC-26: the alarm monitor takes its internal lease directly from the session **before** `SubscribeAlarms` and drains it after the first reconcile; `ISessionManager.ReadAlarmEventsAsync` removed (zero remaining callers); `ApplyReconcile` now broadcasts an `Acknowledge` feed transition for a both-present alarm whose state advanced to `ActiveAcked` (feed-level repair on `AlarmFeedMessage`, not `MxEvent` synthesis). New tests `GatewaySessionTests.AttachInternalEventSubscriberBeforeReadyThrowsAndDoesNotPoisonDistributor` and `GatewayAlarmMonitorAttachOrderTests` (`TransitionsDuringSubscribeWindow_StillReachTheAlarmFeed`, `ApplyReconcileBroadcastsAcknowledgeDelta`); the alarm-monitor fakes now hand the monitor a real Ready `GatewaySession` with a dashboard mirror so the window is actually reproducible. Verification: NonWindows build 0 warnings/0 errors; `GatewayAlarmMonitor` 16 passed, `SessionManagerTests` 38 passed, `GatewaySessionTests` 19 passed, `AlarmFailoverEndToEndTests` 2 passed. |
| 2026-08-07 | Code review of `fix/gwc-26-27-alarm-attach` surfaced a **known pre-existing characteristic, now documented**: the alarm monitor's reconcile-derived feed repairs are **at-least-once, not exactly-once**. A reconcile reads the worker's current state while the matching live transition may still be buffered in the monitor's internal lease, so both broadcast and the duplicates are indistinguishable on the alarm feed (`StreamAlarms` + dashboard alarm hub). This pre-dates GWC-26 — the Raise/Clear presence repair has always had it, since nothing serializes a reconcile pass against the in-flight live stream — so closing it (reconcile/live serialization or transition-timestamp dedup) was ruled out of scope for a P2 fix. Documented instead in `GatewayAlarmMonitor.ApplyReconcile`, `gateway.md`, and `docs/Sessions.md`, with the consumer-side contract stated explicitly (apply transitions idempotently — "set this alarm to this state", never increment/toggle). **Candidate finding for the next review cycle.** |
| 2026-08-07 | **CLI-37 + CLI-38 -> `Done`** (branch `fix/cli-37-38-conformance`), one cross-client conformance commit; **closes old-tracker CLI-08**. Canonical rules landed everywhere: an `MxStatusProxy` entry fails iff `category != MX_STATUS_CATEGORY_OK` (`success` is the raw COM member, diagnostics only; absent entry = success, present entry with `UNSPECIFIED` = failure), and a reply fails on HRESULT iff `hresult` is present and `< 0` (so `S_FALSE = 1` passes). Edits: .NET `MxStatusProxyExtensions.IsSuccess` (drop the `Success != 0` conjunct) + `MxCommandReplyExtensions` (`!= 0` -> `< 0`); Go `StatusSucceeded` (category) + `errors.go` (`< 0`); Java `MxStatuses.succeeded` (category, Javadoc corrected) + `MxGatewayErrors` (`< 0`); Python `errors.py` (category); Rust `ensure_mxaccess_success` (category, doc comment corrected). Four shared fixtures added under `clients/proto/fixtures/behavior/command-replies/` (`write.status-category-{error-success-set,ok-success-zero}.reply.json`, `write.hresult-{s-false,e-fail}.reply.json`) + manifest + `docs/ClientBehaviorFixtures.md`; each of the five suites now runs all four fixture-driven, plus a per-language table test for the two edges fixtures cannot express (nil/null entry, `UNSPECIFIED` category). Docs same commit: `ClientLibrariesDesign.md` per-item rule sentence (its existing HRESULT `< 0` claim is now true), .NET/Go/Java README error sections. Also fixed a Java test fake that built a status with a bare `setSuccess(1)` and no category. Verification: dotnet build 0 warnings + 110 passed/1 skipped; `gofmt -l` clean, `go build ./...`, `go test ./...` all ok; `gradle test` BUILD SUCCESSFUL with **no** generated-file churn to revert this time (no `.proto` changed and `generateProto` stayed up to date); `python -m pytest` 155 passed/1 skipped; `cargo fmt` (no unrelated reformat), `cargo check`, `cargo test --workspace` 100 passed, `cargo clippy --all-targets -- -D warnings` clean. Gateway-side `ClientBehaviorFixtureTests` 8/8 re-run because the new fixtures are validated there. |
| 2026-08-07 | **CLI-45 → `Done`** on `fix/cli-45-credential-envvar`. All five CLIs now share one credential contract for `authenticate-user`: flags `--password` / `--password-env` (Go: `-password` / `-password-env`) defaulting to env `MXGATEWAY_VERIFY_PASSWORD`, resolution flag-then-env, and a resolved credential that is missing **or empty** is a usage error naming the flag and the variable — never the value, never sent to the wire. Go and Java previously authenticated with an empty password: Go now returns the guard error before dialing, Java throws a picocli `ParameterException` instead of falling back to `""`. Python's `--password-env` gained the canonical default (its `UsageError` was already conformant) and its message now names the resolved variable. Rust treats an empty `--password` or empty env value as missing (resolution extracted into a testable `resolve_verify_user_password`). .NET adopted the canonical flags and keeps its pre-existing names as **deprecated aliases for one release** — order: `--password`, `--verify-user-password`, the variable named by `--password-env` (or the deprecated `--verify-user-password-env`; default `MXGATEWAY_VERIFY_PASSWORD`), then `MXGATEWAY_VERIFY_USER_PASSWORD`. Tests: `TestRunAuthenticateUser{RejectsEmptyPassword,ReadsPasswordFromCanonicalEnv}` (Go), 3 picocli cases (Java), 3 click cases (Python), 2 clap/resolver cases (Rust), 4 xUnit cases covering the canonical flag, both env-name paths, the deprecated flag+env aliases, and the missing/empty failure (.NET). Docs same commit: `docs/CrossLanguageSmokeMatrix.md` gained a "Credential contract for `authenticate-user`" section **and** the per-CLI subcommand-coverage table — the half of this finding that is documented rather than fixed (.NET exposes all nine single-item session commands; Rust `unregister` + the credential pair; Go/Python/Java the credential pair only; verified against each dispatch table, and every gap is CLI surface only since all five *libraries* implement all nine helpers). All five client READMEs name the canonical variable and the fail-fast rule; the .NET README gained an `authenticate-user` credentials section carrying the deprecation note. **Deviation:** Java keeps `isBlank()` (per this design's "null or blank" wording for Java) where the other four test emptiness, so a whitespace-only credential is additionally rejected there. Verification (all five, on macOS): Go `gofmt -l .` clean, `go build ./...` clean, `go test ./...` ok; Java `JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` BUILD SUCCESSFUL, CLI suite 51 tests / 0 failures — **no generated-tree churn appeared this run**, `git status` for `clients/java/**/generated` clean with no revert needed (no `.proto` changed); Python `python -m pytest` 148 passed / 1 skipped (TLS opt-in); .NET `dotnet build …Client.slnx` 0 warnings / 0 errors and client tests 108 passed / 1 skipped (live-gateway opt-in); Rust `cargo fmt` (diff confined to the new code), `cargo check --workspace`, `cargo test --workspace` 100 tests across 6 targets all green, `cargo clippy --all-targets -- -D warnings` clean. |
| 2026-08-07 | **SEC-33 + SEC-34 → `Done`, SEC-35 discharged (doc-only)** (branch `fix/sec-33-34`). SEC-33: `IsRootedForAnyPlatform` deleted and rooting made host-meaningful (`Path.IsPathRooted`); `AddIfNotRooted`/`AddIfInvalidPath` promoted to a shared `GatewayConfigPathRules` helper; both `C:\ProgramData\...` literals (`Authentication:SqlitePath`, `Galaxy:SnapshotCachePath`) removed from `appsettings.json` so the `CommonApplicationData`-derived code defaults take over; new `GalaxyRepositoryOptionsValidator` (`ValidateOnStart`) enforces a valid, host-rooted `SnapshotCachePath` when `PersistSnapshot`; the Galaxy default is seeded as a configuration value before `AddZbGalaxyRepository` (deviation: the package's `SnapshotCachePath` is init-only, so a `PostConfigure` mutation would not compile — same effect). Stray-file root cause: host start eagerly builds `AuthSqliteConnectionFactory`, which under the Windows literal materialized a junk-named relative auth DB under the test `bin/` on macOS; the three real-host-start tests now pin `SqlitePath` to a temp path (`find src -name 'C:*'` empty). SEC-34: window-3 `Invalidate` race fixed with a per-key generation counter (bump-before-evict, snapshot-then-recheck); window-2 expiry cap took the **documented fallback** because the library verification identity carries no `ExpiresUtc` (donor-library ask recorded) — so only `Invalidate_DuringInFlightVerification_DiscardsStaleRepopulation` was added, not the two expiry-cap tests. GWC-24 rider: `MxGateway:Events:QueueCapacity` gained an `int.MaxValue/2` upper bound so `checked(2 * EventChannelCapacity)` in `WorkerClient` cannot overflow at session creation (+ two validator tests). SEC-35: doc note added to `docs/GatewayConfiguration.md` (`IsProduction()` env-name semantics). Docs same commit: `GatewayConfiguration.md`, `Authentication.md`. Evidence (macOS): `dotnet build …Server` 0 warnings/0 errors; `--filter ~GatewayOptionsValidator` 69/69, `~GalaxyRepositoryOptionsValidator` 5/5, `~CachingApiKeyVerifier` 10/10, `~GatewayTreeHygiene` 1/1. **Code-review follow-ups (same branch):** (a) two missed same-commit docs updated — `docs/GalaxyRepository.md` (SnapshotCachePath per-OS default + rooting validator) and `A2-galaxyrepository-adoption-handoff.md` (the "NSSM `SnapshotCachePath` required or persistence silently no-ops" caveat corrected — the override is now optional, blank → rooted host default; the option-validation item repointed to the new validator). (b) SEC-34 `TryParseKeyId` guard confirmed: the `_` split cannot truncate a key id because both — and the only — gateway key-creation paths (`ApiKeyAdminCommandLineParser.IsValidKeyId`, `DashboardApiKeyManagementService.ValidateKeyId`) restrict key ids to `IsAsciiLetterOrDigit || '.' || '-'` and key ids are never library-generated; a citing comment was added. (c) the three host-start `SqlitePath` overrides were consolidated into `TestHostEnvironmentInitializer` (per-process temp store, mirroring `Secrets__SqlitePath`) so future host-start tests auto-cover. Re-verified: 69/5/10 filters green, 17 host-start/hygiene tests green, `find src -name 'C:*'` empty. |
| 2026-08-07 | **SEC-31 + SEC-32 → `Done`** (branch `fix/sec-31-32-limiter`, one change set as planned). `ApiKeyFailureLimiter` reworked from `IsBlocked/RecordFailure/Reset(string peer)` to a partition-pair API (`Check/RecordFailure/Reset(ApiKeyThrottlePartition)` returning `ApiKeyThrottleDecision`): layer 1 is the composite `(transport peer, key id)` partition, layer 2 a per-key-id aggregate across peers (`ApiKeyFailureAggregateLimit`, default 30), and an over-limit state now admits one probe per `ApiKeyFailureProbeIntervalSeconds` (default 5) instead of blocking absolutely — so a success can reset the state while throttled, killing the 10-packets-per-minute lockout. SEC-32 rides along: the interceptor validates token shape (`mxgw` prefix, ≥3 non-empty `_` segments, key id ≤ 64 chars) before minting a key-id partition, each peer may mint at most 32 of them (overflow collapses to its fallback partition), and eviction prefers expired windows, never dropping an over-limit partition below a 2× transient overshoot ceiling. New counter `mxgateway.auth.throttled` tagged `stage=peer\|aggregate` only (no key material — `/metrics` is still unauthenticated per open SEC-14). Docs updated in the same commit (`docs/GatewayConfiguration.md` limiter rows + two new keys, `docs/Authentication.md` hot-path paragraph, `docs/Authorization.md` SEC-11 section, limiter/`SecurityOptions` XML remarks). Evidence: `dotnet build …Server` clean; `--filter ~ApiKeyFailureLimiter` 11/11 passed (new `ApiKeyFailureLimiterTests`), `--filter ~GatewayGrpcAuthorizationInterceptor` 20/20 passed (incl. the four SEC-31 contract tests and `NonMxgwToken_FallsBackToTransportPeerPartition`), `--filter ~GatewayOptionsValidator` 66/66 passed. Full suite on macOS: 804 passed / 44 failed — all 44 are the pre-existing named-pipe fake-worker classes (`WorkerClientTests`, `FakeWorkerHarnessTests`, `SessionWorkerClientFactoryFakeWorkerTests`, `GatewayEndToEnd*`), verified identical (44) on the unmodified tree. Follow-up unchanged: the new `MxGateway:Security` keys belong in old **SEC-24**'s effective-config projection when that is picked up. **Code review of the branch found two defects in the first pass, both fixed before merge:** (1) probe admission was check-then-act across two lock scopes, so a burst arriving at an interval boundary could all observe "due" and all be admitted — the claim is now a single critical section (`TryConsumeProbe`), and because the two layers are claimed one at a time, a slot claimed on the partition is compensated (`ReleaseProbe`) when the aggregate then refuses; (2) `Reset` on a success whose key id had been collapsed into the address's shared fallback partition removed that shared partition, letting one authentication wipe an in-progress spray from the same address — it is now left to decay by window expiry, while the key's aggregate is still cleared. Tests added: `ProbeAdmission_UnderConcurrentArrivals_GrantsExactlyOneSlot`, `ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot`, `Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition`. **A second review pass found a residual defect in that compensation path:** the release identified its own reservation by comparing `NextProbeAtTicks` to `now + interval`, the identical expression a failure re-arm writes — so a concurrent `RecordFailure` on the same state sharing a clock tick (routine at ~1 ms resolution) was mistaken for the caller's own claim and stomped back to the stale, already-due value, prematurely reopening the probe slot. Replaced with a monotonic per-state `ProbeVersion` bumped by every writer of `NextProbeAtTicks` (claim and re-arm alike); the release restores only when the version still matches the one its claim stamped, and bumps it again on restore so no other stale release can match. Covered by `ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick`, made deterministic by a new `internal ProbeReleaseInterleaveHook` test seam (null in production, one null check on the refused path) because the claim-to-release window is nanoseconds wide and racing threads cannot hit it reliably — verified as a genuine red against the timestamp guard (`Expected: ThrottledByPeer / Actual: ProbeAdmitted`). Limiter suite 11 → 15. |
| 2026-08-07 | **CLI-40 + CLI-41 + CLI-44 → `Done`** (branch `fix/cli-40-41-44`), one change set; two fast-follow riders from the CLI-37/38 review landed alongside. **CLI-40** (exact-secret credential scrub, ported to Rust/Java/.NET; Go/Python already did it): every credential-bearing helper (`AuthenticateUser` password, `WriteSecured`/`WriteSecured2` string payloads) now scrubs the **exact** caller-supplied secret from any surfaced error text, on top of the by-construction guarantee — Rust `MxAccessError` gained a `secrets: Vec<String>` field whose `Display` scrubs exact-then-pattern (and a **hand-written redacting `Debug`**, since the derived `Debug` would have leaked the reply verbatim — caught by the existing Debug regression test); Java added `MxGatewaySecrets.redactExact` + a private `invokeCommandRedacted(command, secrets…)` that rebuilds the same exception type with the redacted message and **does not chain the secret-bearing original as cause**; .NET added an internal `MxGatewaySecretRedaction` (rebuilds the same concrete `MxGateway*Exception` type via a type switch) wired into the three credential helpers — and it carries the original's **inner** cause forward rather than the secret-bearing original, so `ToString()` (what loggers emit) is scrubbed too, not just `Message` (locked by a `ToString()` assertion). **CLI-41** (uniform malformed-reply contract for `AuthenticateUser`/`ArchestrAUserToId`/`AddBufferedItem` across all five): typed payload → present `return_value` with the int32 variant → else a typed malformed-reply error (`MalformedReplyError` Go/Python, `MxGatewayMalformedReplyException` Java/.NET, existing `Error::MalformedReply` Rust) — never a proto3 default `0`, never an NRE (fixes the Go/Java silent-`0`, .NET NRE, and Rust's own internal inconsistency by giving `authenticate_user_id`/`archestra_user_id` the same `return_value` fallback `add_buffered_item_handle` already had). **CLI-44** (Go): the event goroutine's Recv-error path now uses a new non-blocking `sendTerminalEventResult` on the reserved slot instead of `sendEventResult`, so a genuine terminal gRPC error is reported as itself even when the 16 data slots are full, rather than being mislabeled `ErrSlowConsumer`; test `TestEventsFullBufferTerminalErrorKeepsRootCause` was confirmed red-first (a 250 ms settle after `streamDone` is required to make the buffer genuinely full at error time). Three shared fixtures added under `clients/proto/fixtures/behavior/command-replies/` (`authenticate-user.{echoed-credential,missing-payload,return-value-only}.reply.json`; the echoed-credential reply uses an OK envelope + negative HRESULT + the credential in `protocolStatus.message` / `statuses[0].diagnosticText` / `diagnosticMessage` so all five clients route it to their MXAccess error uniformly) + manifest + `docs/ClientBehaviorFixtures.md` + `docs/ClientLibrariesDesign.md`. **Rider (a):** .NET `ToDiagnosticSummary` and Python `_mxaccess_message` now surface the raw `success` member (Rust already did), for diagnostics-only parity. **Rider (b):** the status-conversion fixture gained an independent `wantSuccess` boolean per case; the Go `TestStatusConversionFixtures` and .NET `FixtureStatuses_ProjectSuccessAndPreserveRawFields` now assert against it instead of recomputing `category == OK` (the formula under test). **Deviation:** the `<redacted>` marker is not universal — Go/Rust/Java use `<redacted>`, Python and the .NET CLI use `[redacted]`; each suite asserts its own client's marker plus the exact-secret absence (marker unification was out of scope). Verification (all five, on macOS): Go `gofmt -l` clean + `go build ./...` + `go test ./...` ok; Python `python -m pytest` 162 passed / 1 skipped; .NET `dotnet build …Client.slnx` 0 warnings + client tests 120 passed / 1 skipped; Rust `cargo fmt` + `cargo check --workspace` + `cargo test --workspace` (all targets pass) + `cargo clippy --all-targets -- -D warnings` clean; Java `JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` BUILD SUCCESSFUL with **no** generated-tree churn to revert (no `.proto` changed). **Code review of the branch found three defects, all fixed before merge:** (1, all five, critical) the message-only scrub left the server-echoed credential exposed on the redacted error's **structured reply accessor** (`.NET MxAccessException.Reply`/`Statuses`, Java `reply()`/`protocolStatus()`, Go `MxAccessError.Reply` via `errors.As`, Rust `reply()`/`into_reply()`, Python `raw_reply`) — the redacted error now carries a **scrubbed clone** of the reply (`protocol_status.message`, `diagnostic_message`, `statuses[].diagnostic_text` all redacted), with per-language tests asserting the reply accessor is clean; docs/ClientLibrariesDesign.md updated to make the "never reaches exception text" claim true for structured fields too. (2, Rust, critical) `ensure_command_success` routed `PROTOCOL_STATUS_CODE_MXACCESS_FAILURE` to `Error::Command` (unlike the other four clients), where `attach_secrets` did not patch it and its derived `Debug`/`Display` leaked the secret — now `MxaccessFailure` routes to `Error::MxAccess` (fixing a real cross-client inconsistency; an existing test flipped from `Error::Command` to `Error::MxAccess`). (3, Go, important) the CLI-44 `sendTerminalEventResult` was unconditionally non-blocking, so on the never-drop `SubscribeEvents`/`SubscribeEventsAfter` path (`cancelWhenResultBufferFull=false`) a genuine terminal error under a full buffer hit the `default:` and was silently dropped — now the terminal send is reserved-slot-non-blocking only for the cancel-on-overflow path and **blocking** for the never-drop path. A new shared fixture `authenticate-user.echoed-credential-mxaccess-failure.reply.json` (the echo under `MXACCESS_FAILURE`) is wired into all five suites. Minors also landed: whitespace-only-secret guard on the .NET/Java redact helpers; Java `invokeCommandRedacted` now preserves the exception subtype (mirroring .NET's type switch) instead of collapsing to the base type; dedicated redaction-helper unit tests (multiple occurrences, substring-overlap, empty/blank secrets) in Go/Java/.NET. Re-verified all five green (Go `go test ./...` ok + gofmt clean; Python 163 passed/1 skipped; .NET build 0 warnings + 128 passed/1 skipped; Rust fmt/check/`test --workspace`/clippy `-D warnings` all clean; Java gradle BUILD SUCCESSFUL, no generated churn). |
| 2026-08-07 | **GWC-28, GWC-29, GWC-30, TST-28 → `Done`** (branch `fix/gwc-28-29-30-polish`). GWC-28: `WorkerClient.WriteLoopAsync` now stamps `envelope.Sequence = unchecked(++_nextSequence)` immediately before `_writer.WriteAsync`, and `CreateEnvelope` leaves it unset; `_nextSequence` dropped from `long` + `Interlocked` to a plain `ulong` touched only by the write loop (the channel's single consumer, `SingleReader = true`), so wire order and sequence order are the same thing by construction. Mirrors the worker's WRK-04 stamping, which the gateway half had never received; `gateway.md`'s envelope-sequence rule now states that both sides stamp at write inside their single write path and that inbound enforcement (still open, old **GWC-10**) would rely on it. New `WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequencesOnTheWire` (32 parallel invokes, sequences asserted strictly increasing in wire order) failed 3/3 pre-fix. GWC-29: added `MxAccessGrpcMapper.MapCommand(MxCommand)`; `Invoke` no longer deep-clones the whole `MxCommandRequest` just to overwrite and discard its command. The one clone inside `MapCommand` stays and is documented as required — `commandToInvoke` may be the gRPC-owned `request.Command` and is read again after dispatch by `TrackCommandReply`, so it is what keeps `CreateCommandEnvelope`'s no-aliasing invariant true. New `MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload` (isolation + both overloads equal under a `FakeTimeProvider`). GWC-30: `WorkerFrameReader` reuses a per-instance `_lengthPrefix` scratch buffer instead of allocating 4 bytes per frame, with a class remark that `ReadAsync` is not reentrant (single read loop per `WorkerClient`; handshake reads complete before the loop starts); guarded by new `WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_ParsesEveryFrame` (5 frames, varying payload lengths, one reader). TST-28: new `[Theory] WorkerClientTests.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes` over the default and a 2 MiB override via `FakeWorkerHarness.CreateConnectedPairAsync(maxMessageBytes:)` — test-only, and the mutation check (hard-code `MaxFrameBytes = 0`) failed both cases before being reverted. Verification: `NonWindows.slnx` 0 warnings/0 errors; `WorkerClientTests` 25 passed, `WorkerFrameProtocolTests` 11 passed, `MxAccessGrpcMapperTests` 6 passed, `MxAccessGatewayService*` 29 passed, full gateway suite 844 passed / 0 failed (`TMPDIR=/tmp` on macOS). |
| 2026-08-07 | **WRK-21 + WRK-28 + WRK-23 + IPC-30 → `Done`** (branch `fix/wrk-21-drain-cluster`, commits `33ba612` + test-fixture follow-ups `7c2eaf0`/`a256560`). WRK-21: `MxAccessEventQueue` gains a byte-budgeted `Drain(maxEvents, maxTotalBytes)` returning the new `WorkerEventDrainResult`, sizing inside the queue lock so an event that will not fit is never dequeued; `CreateDrainEventsReply` budgets against the negotiated frame max less a 64 KiB wrapper reserve and reports truncation through the existing `DiagnosticMessage` (no proto change), satisfying IPC-23 R1–R3; both reply-write seams (`HandleControlCommandAsync`, `ProcessCommandAsync`) now catch `MessageTooLarge` and answer the correlation with an `InvalidRequest` reply instead of unwinding/faulting the session. WRK-28: the 10,000 ceiling moved to `GatewayContractInfo.MaxDrainEventsPerCommand`, referenced by the gateway validator and the worker clamp (C# const, no `.proto` change). WRK-23: `WorkerFrameWriter` peek-stamps then commits `Sequence` only immediately before the stream write, so rejections leave no wire gap. IPC-30: an oversized event frame stays session-fatal but writes a `PROTOCOL_VIOLATION` `WorkerFault` with `command_method = EventDrain` naming family/handles/sequence/sizes (never the value) before exiting. Docs same commit: `MxAccessWorkerInstanceDesign.md`, `WorkerFrameProtocol.md`, `gateway.md`. **IPC-23 → `In progress`** — mechanics landed here; the proto-comment/doc wave (and its regen fan-out) is still pending and must not be folded into this branch. **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings/0 errors, `dotnet test …MxGateway.Tests --filter FullyQualifiedName~MxAccessGrpcRequestValidator` 4/4 passed. windev (`scripts/ci/windev-worker-ci.ps1 -Sha a2565604 -Mode test`, 2026-08-07 06:47): x86 Worker build 0 warnings/0 errors, `Worker.Tests` **367 passed / 0 failed / 11 skipped** (skips are the live-MXAccess/dev-rig opt-ins), script exit 0. **Harness note:** `PipePair` runs both pipe ends in one process with blocking `FlushFileBuffers` per frame, so it wedges on multi-MB frames or after ~85 large round trips; the pipe tests therefore negotiate a 128 KiB frame maximum and walk 1,000 events to empty, while the full 10,000-event drain-to-empty no-loss proof runs at the queue layer (`MxAccessEventQueueTests`). |
| 2026-08-07 | Code-review follow-ups on the same branch (commit `6bc3f9b`). (1) **Important** — `ResolveDrainReplyByteBudget` was a step, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (exactly 1024 at the validator floor `MaxMessageBytes = 1024 + 64 KiB`), so a byte-heavy `DrainEvents` truncated on every call and the drain-until-empty loop never terminated. Now `Math.Max(frameMax - reserve, frameMax / 2)` — monotonic, never below half the frame max. New test `WorkerPipeSessionTests.DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates` drives a byte-heavy queue at the exact validator floor and asserts drain-to-empty with no head reported oversized. (2) **Hardening** — the reply-too-large fallback write is now itself size-guarded (`WriteReplyTooLargeFallbackAsync`, shared by the control and STA reply seams) so a pathologically tiny negotiated max below the gateway floor (the WRK-24 gap) cannot make even the backstop session-fatal; log-and-swallow, comment points at WRK-24. (3) **Comment** — corrected the `RepeatedFieldOverheadBytes` docs: `WorkerEvent.CalculateSize()` already includes the event's tag+length, so the 8 bytes is pure slack, not wrapper compensation. **Evidence** — macOS build 0/0, validator filter 4/4. windev (`windev-worker-ci.ps1 -Sha 6bc3f9b -Mode test`, 07:07): x86 Worker build 0/0, `Worker.Tests` **368 passed / 0 failed / 11 skipped**, script exit 0. (An earlier run of the same SHA flaked on the pre-existing `RunAsync_WhenStaActivityIsStale_WritesWatchdogFault` — a 5 s CTS timeout under first-run load, untouched by this change; it passed on the clean re-run and in both prior full runs.) |
| 2026-08-07 | **SEC-36 → `Done` (repo-side); live rotation operator-pending** (branch `fix/sec-36-ldap-secret`). **State reconciliation:** the plaintext-in-`appsettings.json` half was already discharged before this branch — HEAD ships `"ServiceAccountPassword": "${secret:ldap/mxgateway/bind}"` (a fail-closed encrypted-store reference, documented at `GatewayConfiguration.md:252`, tested by `PreHostSecretExpansionTests`), not the literal the spec's stale baseline assumed. **Deviation from spec step 2:** the `${secret:}` reference was **kept, not deleted** — deleting it would regress the shipped/documented/tested secret-store channel and the finding (committed plaintext) is already resolved for `appsettings.json`. The live residual leak was the literal value in `glauth.md`, `docs/GatewayTesting.md`, and the historical `archreview/*` SEC-06 evidence — all scrubbed to `<service-account-password>` placeholders with a source-of-truth pointer (`scadaproj/infra/glauth/`). Added `<UserSecretsId>mxaccessgw-server</UserSecretsId>` to the server csproj (dev channel) and extended the `ValidateLdap` blank-password message to name both channels (dev user-secrets, deployed `MxGateway__Ldap__ServiceAccountPassword`) — asserted by the updated `Validate_Fails_WhenLdapEnabledAndServiceAccountPasswordBlank`. Docs same commit: `GatewayConfiguration.md` (three-channel + rotation note), `glauth.md` (placeholders + rotation-required note + runbook pointer), `docs/GatewayTesting.md` (live-LDAP env-var example de-secreted). New operator runbook `docs/runbooks/SEC-36-ldap-credential-rotation.md` carries the cutover order (generate in glauth → pre-stage NEW env var on `10.100.0.48` and on `wonder-app-vd03` only if `Ldap.Enabled` → rotate GLAuth on `10.100.0.35` → verify dashboard login → devs set user-secrets). **The live GLAuth rotation + NSSM staging remain the operator's to execute** and are tracked in that runbook. Verification (macOS): `dotnet build …Server` 0 warnings/0 errors; `--filter ~GatewayOptionsValidator` green; `git grep -i` for the old value returns nothing across tracked files. |
| 2026-08-07 | **Worker-seam batch → `Done`: WRK-22 (mechanics for IPC-26), WRK-24, WRK-25, WRK-27** (branch `fix/wrk-22-25-seam`). **WRK-22/IPC-26**: `WorkerFrameWriter.PendingFrame` gained a `Claimed` field; a `WriteAsync`/`WriteBatchAsync` cancelled while waiting for the write lock tombstones its still-unclaimed frame (`TrySetCanceled` under `_gate`) and `DequeueNext` skips cancelled frames and marks the one it returns `Claimed`, so a cancelled write never reaches the wire — except the documented, by-design residual where a lock-holder claimed the frame first (mid-write, cannot be recalled; caller still observes cancellation). **WRK-25**: new `WriteBatchAsync(IReadOnlyList<WorkerEnvelope>, priority, ct)` enqueues a whole batch under one `_gate` acquisition, takes the lock once, drains, then observes every completion (surfacing the first per-frame rejection); `RunEventDrainLoopAsync` now submits the drained event batch through it, so a burst of N events costs one flush not N — the WRK-12 coalescing now engages on the event hot path. IPC-30's oversized-event structured fault is preserved (`FindOversizedEvent` maps the batch rejection back to the offending event). **WRK-24**: `WorkerFrameProtocolOptions.MinNegotiableFrameBytes = 1024` (matches `GatewayOptionsValidator.MinimumMaxMessageBytes`); `AdoptNegotiatedMaxMessageBytes` now faults a below-floor negotiated value at the handshake, closing the [1024, 256 MiB] accepted range. **WRK-27**: alarm poll runs outside the dispatcher, so `MxAccessStaSession` sets a `volatile staAlarmPollInProgress` around the `PollOnce` COM call and surfaces it on the new `WorkerRuntimeHeartbeatSnapshot.StaCallInProgress`; `ReportWatchdogFaultIfNeededAsync` honors it alongside `CurrentCommandCorrelationId`, so a healthy-but-slow poll gets grace-to-ceiling suppression (not the 15 s grace) but still faults past the 75 s ceiling. Docs same commit: `docs/WorkerFrameProtocol.md` (accepted-range paragraph, flush-coalescing sentence flipped to coalesced-on-drain, cancellation-tombstone contract replacing the WRK-26 "pending" placeholder), `docs/MxAccessWorkerInstanceDesign.md` (watchdog alarm-poll paragraph). New tests: `WorkerFrameProtocolTests.{WriteAsync_CancelledWhileWaitingForLock_FrameIsNeverWritten, WriteAsync_CancelledEventFrame_DoesNotTrailShutdownAck, WriteBatchAsync_FlushesOnceAndPreservesOrder, WriteBatchAsync_ControlFrameQueuedDuringBatch_JumpsRemainingEvents, AdoptNegotiatedMaxMessageBytes_BelowFloor_ThrowsInvalidConfiguration}`, `WorkerPipeSessionTests.{Watchdog_StaCallInProgress_SuppressedUntilCeiling, EventBurst_DrainLoopCoalescesFlushes, Handshake_GatewayHelloWithTinyMaxFrameBytes_FaultsAtHandshake}`, `MxAccessStaSessionTests.CaptureHeartbeat_DuringAlarmPoll_ReportsStaCallInProgress` (+ `FakeRuntimeSession.EnqueueEvents` bulk helper, `staCallInProgress` snapshot ctor param). **IPC-26 → `Done`** (mechanics owned here). **Evidence** — macOS: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings / 0 errors (worker excluded there; proves the shared/server side is intact). windev (`scripts/ci/windev-worker-ci.ps1 -Sha 8df35cd -Mode test`, 2026-08-07 07:45): x86 Worker build 0 warnings / 0 errors, `Worker.Tests` **377 passed / 0 failed / 11 skipped** (skips are the live-MXAccess/dev-rig opt-ins; +9 over the prior 368 = the nine new tests, all green), script exit 0. |
| 2026-08-07 | **P1 doc-drift batch → `Done`: TST-27, WRK-26 (discharges IPC-29), CLI-42, CLI-43, IPC-28** (branch `fix/doc-drift-batch`). Doc-only; no source, proto, or test changes — cross-checked against HEAD in this worktree. **TST-27**: `docs/GatewayConfiguration.md`'s `ShowTagValues` row no longer says "Reserved" — it now states what `false` (default) does (`DashboardEventBroadcaster` blanks tag values from a deep-cloned `MxEvent` before the SignalR events-hub mirror, metadata still renders), the security relevance (the per-session hub ACL, SEC-25 roadmap item 12, still does not exist, so this redaction is the only thing between a low-trust Viewer and other sessions' tag values), and the honest scope limit (the flag does **not** cover `/browse`). **WRK-26** (discharges **IPC-29**): `docs/MxAccessWorkerInstanceDesign.md`'s "Outbound Queues" section rewritten from the stale five-level priority list to the two-class `Control`/`Event` scheduler actually shipped (`WorkerFrameWriter`/`WorkerFrameWritePriority.cs`), with the collapsed-decision rationale recorded, and the overflow paragraph rewritten to the implemented fail-fast (`WorkerFault` category `QueueOverflow` → fault frame written → `RunAsync` unwinds → generic `WorkerExitCode.UnexpectedFailure`, dedicated code still open). `docs/WorkerFrameProtocol.md` gained a new "Write Scheduling And Sequencing" section: the two priority classes, enqueue-then-contend/single-lock-holder-drains-all, write-time peek-stamp-commit sequencing, per-frame-rejection vs. stream-failure semantics, and flush coalescing — stated truthfully as landed (WRK-23's peek-stamp-commit is live at HEAD) or not (the drain loop still awaits each event `WriteAsync` individually, so WRK-25's N-events-one-flush batching has **not** landed and the section says so explicitly). Cancellation is deliberately **not** documented as a firm contract — a one-paragraph placeholder notes it is pending WRK-22, which has not landed (confirmed by reading `WorkerFrameWriter.cs`: no `Claimed`/tombstone machinery exists yet). `gateway.md:328-330` was cross-checked and left unchanged — its sequence prose (both sides stamp at write, per GWC-28) already reads true. **CLI-42**: `clients/rust/README.md` and `docs/ClientPackaging.md`'s Rust section now document the vendored proto layout matching `clients/rust/build.rs` exactly — repo-path-first resolution (`../../src/ZB.MOM.WW.MxGateway.Contracts/Protos`) falling back to `clients/rust/protos/` when the canonical path is absent (published-tarball case), the same-commit refresh rule enforced by `scripts/check-codegen.ps1` Check 3, and why `cargo package`/`cargo publish` run without `--no-verify` (matches `scripts/pack-clients.ps1:190-192`). **CLI-43**: `docs/style-guides/JavaStyleGuide.md` line 8 now says "Target Java 17 (the Ignition 8.3 baseline...)" mirroring the CLI-12 wording, matching the shipped `clients/java/build.gradle` toolchain-17 build. **IPC-28**: `docs/Grpc.md`'s exception-mapping prose gained `CommandTooLarge` → `ResourceExhausted` (verified against the live `switch` in `Grpc/MxAccessGatewayService.cs:950-960`), and the `Invoke` section gained one sentence on the oversized-payload path (`WorkerClient.InvokeAsync` rejects at the enqueue boundary per-correlation, session not faulted — verified against `WorkerClient.cs:220-234`), cross-referencing the headroom rule already documented in `docs/GatewayConfiguration.md:120-129`. Did not touch the DrainEvents-truncation row or the proto/`Generated/` trees — those belong to a parallel codegen task per the handoff note. **Source files cross-read for accuracy** (no edits): `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs`, `.../WorkerFrameWritePriority.cs`, `.../WorkerPipeSession.cs` (confirmed two-class scheduler, WRK-21/23/28/30 landed, WRK-25/WRK-22 not landed), `src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs` (exit-code mapping), `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs` (overflow fault path), `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardEventBroadcaster.cs` + `Configuration/DashboardOptions.cs` + `docs/GatewayDashboardDesign.md:170` (ShowTagValues), `src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGatewayService.cs:940-963` + `Workers/WorkerClient.cs:205-244` + `Workers/WorkerClientErrorCode.cs` (CommandTooLarge mapping), `clients/rust/build.rs`, `clients/rust/Cargo.toml`, `scripts/check-codegen.ps1`, `scripts/pack-clients.ps1` (Rust vendoring), `gateway.md:326-360` (sequence-prose cross-check). Verification (greps, doc-only — no build required): `grep -n 'Reserved' docs/GatewayConfiguration.md` no longer matches the `ShowTagValues` row; `grep -n 'faults' docs/MxAccessWorkerInstanceDesign.md` shows no remaining five-level list; `grep -n 'scheduling' docs/WorkerFrameProtocol.md` finds the new section; `grep -rn 'Java 21' docs/style-guides/` empty; `grep -i vendored docs/ClientPackaging.md clients/rust/README.md` non-empty in both; `grep -n 'CommandTooLarge' docs/Grpc.md` shows the mapping. |
| 2026-08-07 | **IPC-23 + IPC-24 + IPC-25 + IPC-32 → `Done`; GWC-25 deferred proto-comment resolved** (branch `fix/ipc-24-25-codegen`). The proto-comment/doc + regen wave for the P0 codegen-freshness cluster. **Proto comments (comment-only):** `mxaccess_worker.proto` `GatewayHello.max_frame_bytes` gained the every-frame-must-fit / reply-builders-truncate sentence (IPC-23); `mxaccess_gateway.proto` `DrainEventsReply` gained the count-cap+byte-cap / drain-until-empty comment (IPC-23); `mxaccess_gateway.proto` `ReplayGap.oldest_available_sequence` gained the empty-ring value definition (`highest observed + 1`, `oldest − 1` formula stays valid) resolving GWC-25's deferred amendment. **Regen wave:** `Contracts/Generated/{MxaccessGateway,MxaccessWorker}.cs` (XML-doc only, no descriptor delta), `clients/rust/protos/{mxaccess_gateway,mxaccess_worker}.proto` byte-copied, Go `internal/generated/{mxaccess_gateway,mxaccess_worker}.pb.go` (worker binding was genuinely stale — it lacked the whole `MaxFrameBytes` field/accessor/rawDesc, now refreshed; gateway.pb.go got the new doc comments), Python `generated/mxaccess_worker_pb2.py` (real descriptor delta — `max_frame_bytes` field + cascaded offsets), Java aggregates `MxaccessGateway.java`/`MxaccessWorker.java` (javadoc from the new proto comments — **zero** protobuf-version churn under the pinned toolchain), and the client descriptor set `mxaccessgw-client-v1.protoset`. **IPC-24:** regenerating Java on the pinned toolchain (grpc 1.76.0 / protobuf 4.33.1, Homebrew JDK 17) produced only the legitimate javadoc delta with no `validateProtobufGencodeVersion`/`major=/minor=/patch=` churn — so the preferred path was taken: the unconditional churn-revert step and its comment were **deleted** from `.gitea/workflows/ci.yml` (the `git diff --exit-code` gate is now a true message-level drift gate for the single-file Java aggregates). **IPC-25:** pinned the Go generators in `clients/go/generate-proto.ps1` (`protoc-gen-go` v1.36.11 assert, `protoc-gen-go-grpc` 1.6.2 assert, protoc warn-only; also fixed a latent `(if …)`→`$(if …)` parse bug that broke the script under pwsh 7); added **Check 4** to `scripts/check-codegen.ps1` (regenerate Go+Python bindings, fail on any diff, tool-missing FAILS not skips) and relabeled the banners `1/4`…`4/4` + header comment (**IPC-32** folded in); added the pinned-generator installs to the `portable` CI job before the codegen step. **Docs same commit:** `ClientProtoGeneration.md` (pinned-versions table + Go/Python Check-4 guard + Java-revert-deleted note), `Contracts.md` (four-check enumeration), `GatewayTesting.md` (java job + four-check prose), `clients/java/zb-mom-ww-mxgateway-client/build.gradle` checkGeneratedClean caveat. **IPC-23 DrainEvents-truncation doc deviation:** `docs/Grpc.md` has no DrainEvents row (it documents only the seven public RPCs; DrainEvents is a worker diagnostic), and `docs/WorkerFrameProtocol.md` (owned by the parallel doc batch) already carries the byte-cap/drain-until-empty prose via WRK-21 — so no Grpc.md edit was made. **Evidence (macOS):** `pwsh scripts/check-codegen.ps1` all four checks green (banners `1/4`…`4/4`); `dotnet build …NonWindows.slnx` 0 warnings/0 errors; `dotnet test …Tests --filter ~ClientProtoInputTests` green; `grep -c max_frame_bytes` → Go worker 2, Python worker 1 (both non-zero, were 0); `clients/go` gofmt clean + `go build` + `go test` ok; `clients/python` `pytest` 163 passed/1 skipped; Java `gradle generateProto` clean under JDK 17. **Negative-path proof:** added a scratch field to `mxaccess_worker.proto`, regenerated only `Contracts/Generated/`, reran `check-codegen.ps1` and confirmed **Check 4 fails naming both `clients/go/internal/generated` and `clients/python/…/generated`**, then reverted. **Not published** (CLI-39 gates Go/Python republish). |
| 2026-08-07 | **IPC-27 → `Done`** (branch `fix/ipc-27-descriptor-test`). `ClientProtoInputTests.Descriptor_ContainsEveryContractMessageAndField` renamed to `Descriptor_ContainsEveryContractSymbol` and extended: the reflection walk now also collects enums (top-level `file.EnumTypes` + nested `message.EnumTypes`, as `{enumFullName}` and `{enumFullName}/{valueName}`) and services/methods (`file.Services`, as `{serviceFullName}` and `{serviceFullName}/{methodName}`), and `GalaxyRepositoryReflection.Descriptor` was added to the enumerated contract files alongside the existing gateway/worker descriptors. The published-side `CollectPublishedSymbols` grew matching walks over `FileDescriptorProto.EnumType`/`Service` and `DescriptorProto.EnumType`; the comparison stays a flat, order-insensitive string-set diff with no protoc dependency. Docs: `docs/ClientProtoGeneration.md` and `docs/Contracts.md` updated from "message or field" to full symbol coverage (message/field/enum/enum value/service/method), both now naming `galaxy_repository.proto` explicitly. **Red-path proof performed:** pointed the test at the pre-IPC-01 stale protoset (`git cat-file -p 0f88a95:clients/proto/descriptors/mxaccessgw-client-v1.protoset`), confirmed it failed naming — among the missing symbols — `mxaccess_worker.v1.GatewayHello/max_frame_bytes`, enum values (`MxCommandKind/MX_COMMAND_KIND_SUBSCRIBE_ALARMS`, `AlarmProviderMode/*`, `MxEventFamily/MX_EVENT_FAMILY_ON_ALARM_PROVIDER_MODE_CHANGED`), services/methods (`MxAccessGateway/StreamAlarms`, `GalaxyRepository/BrowseChildren`), and the full `galaxy_repository.v1.*` surface; restored the real descriptor path afterward and re-ran green. Verification: `dotnet build src/ZB.MOM.WW.MxGateway.NonWindows.slnx` 0 warnings/0 errors; `dotnet test …Tests --filter FullyQualifiedName~ClientProtoInputTests` 5/5 passed. No proto, config, or CI change, matching the design's stated scope. |
| 2026-08-07 | **TST-30 → `Done` (doc half); runner registration operator-pending** (branch `fix/tst-30-runner-docs`). Doc + operator-runbook task only — no source/tests, per the finding's P2 scope. `docs/GatewayTesting.md`'s Continuous Integration section gained a "Runner capacity is shared and finite" subsection: the co-located `gitea-runner` on `10.100.0.35` runs `maxParallel=1` and is registered at the Gitea **instance** level (`GET /repos/dohertj2/mxaccessgw/actions/runners` → `total_count: 0`), so it interleaves with `dohertj2/lmxopcua` and every job in a run serializes — ~20–30 minute queue depth was observed under cross-repo contention during TST-25 acceptance. Documented that Gitea 1.26 exposes no run cancel/delete (`POST .../cancel` → 404, `DELETE` → 400), so a superseded or hung run holds its slot to completion. The existing "windev tier down" degraded-mode paragraph now also names "runner contended" as a reason to use the bypass — `CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` or the manual windev worktree flow — generalizing it per the finding's design note (was tier-down-only). New operator runbook `docs/runbooks/TST-30-second-ci-runner.md`: implementation step 1 (register a second `act_runner` on `10.100.0.35`), preference-ordered per the design — (a) second instance with the same `container.network: traefik` config, **recommended**; (b) dedicated labelled runner + `ci.yml` `runs-on` gate, escalation only; (c) runner on windev, **rejected** (loses `gitea:3000` resolution TST-03 depends on) — plus the verification checklist (concurrent back-to-back pushes, `GET .../actions/runners` shows ≥2) and a reminder the no-cancel reality persists regardless of runner count. The optional workflow-level `concurrency` group (design step 3) is documented in the runbook as unverified, framed explicitly as "verify this Gitea deployment honors it before relying on it," and left **unimplemented** in `.gitea/workflows/ci.yml` — no `runs-on` or workflow edits were made, consistent with the task scoping the actual runner registration as operator/infra work outside this repo's tree. Tracking: both registers' TST-30 rows and this row. Verification (doc-only, no build required): `grep -n 'maxParallel\|shared\|cancel' docs/GatewayTesting.md` shows the new prose (the "Runner capacity is shared and finite" subsection, the no-cancel API detail, and the degraded-mode "runner contended" generalization — plus pre-existing unrelated `shared` hits in the LDAP section); `docs/runbooks/TST-30-second-ci-runner.md` exists. |
| 2026-08-07 | **CLI-39 → `Done`** (branch `fix/cli-39-version-train`). The last client task in the P1 release train — lands after CLI-35/36/37/38/40/41/44/45 so the next publish carries the conformant behavior, not the pre-remediation API. Bumped all five client version constants off the already-published 0.1.2: Rust `Cargo.toml` `[package]` + `[workspace.package]` → 0.2.0 (`crates/mxgw-cli` inherits via `version.workspace = true`; `CLIENT_VERSION` already derived via `env!("CARGO_PKG_VERSION")` from CLI-29, no separate edit needed); Python `pyproject.toml` + `version.py` → 0.2.0, plus a new `test_version_matches_pyproject_toml` (`clients/python/tests/test_cli.py`) that parses `pyproject.toml` with `tomllib` and asserts `__version__` matches it — closes the CLI-26 residual (the old test only checked self-consistency against `__version__`, so the two literals could still drift from each other undetected); Go `mxgateway/version.go` `ClientVersion` → 0.2.0; .NET `ZB.MOM.WW.MxGateway.Client.csproj` `<Version>` → 0.2.0. **Java is the one exception, landed at 0.2.1, not 0.2.0**: queried the live Gitea package API (`GET /api/v1/packages/dohertj2/maven/com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client/0.2.0` — reachable and credentialed in this environment) and confirmed Java 0.2.0 was already published on 2026-06-26, before the CLI-37/38/40/41 conformance fixes changed the client's observable behavior; reusing 0.2.0 for the now-conformant build would have labeled two different APIs identically, so `build.gradle`'s `subprojects { version = '0.2.1' }`, the hand-maintained `MxGatewayClientVersion.CLIENT_VERSION` constant, and the two `MxGatewayCliTests` version assertions all moved to 0.2.1 together (`clients/java/README.md`'s Maven dependency example updated to match — it had drifted to a stale `0.1.2` example). Confirmed via the same live API that Python (`pypi`) and Rust (`cargo`) had only ever reached 0.1.1 in the registry (0.1.2 was set in source but never actually published), while `nuget` (`ZB.MOM.WW.MxGateway.Client`/`.Contracts`) reached 0.1.2 — recorded as historical context in the new "Versioning" section of `docs/ClientPackaging.md` rather than left implicit. **Publish-pipeline guards:** `scripts/tag-go-module.ps1` gained the CLI-21 guard that was designed but never implemented — after semver validation it now reads `clients/go/mxgateway/version.go`'s `ClientVersion` and throws if it does not equal the requested tag version (stripped of `v`), before touching git. `scripts/pack-clients.ps1` gained a Gitea package-registry collision guard (`Test-GiteaPackageExists`/`Assert-GiteaPackageNotPublished`, hitting the documented `GET /api/v1/packages/{owner}/{type}/{name}/{version}` package API) wired into all four artifact-producing `-Publish` blocks (.NET nupkg identity parsed from the packed filename so both `Client` and `Contracts` are covered; Python from `pyproject.toml`; Rust from `Cargo.toml`; Java from `build.gradle`'s `group`+`version`, **required for a real check** since Gitea's Maven package identity is `groupId:artifactId`, not the bare artifact id — an early version of this guard used the bare id and would have silently never matched anything, caught by testing it live against the known-published 0.2.0 before landing); Go is unaffected (no artifact producer in this script; the version-match guard above is its collision protection). Every non-404 registry response (including a plain 401 from an unauthenticated call) is treated as "cannot verify" and aborts — the guard fails closed, never proceeds on ambiguity. **No `.proto` changes; no publish performed.** Docs same commit: `docs/ClientPackaging.md` gained a "Versioning" section (single-source-of-truth list per client, the bump-before-publish rule, the Java exception with evidence); `clients/dotnet/README.md`, `clients/rust/README.md`, `clients/go/README.md` (install example + `tag-go-module.ps1` example + guard behavior sentence), `clients/java/README.md` had their stale `0.1.1`/`0.1.2` example version strings updated to the new released numbers. **Verification** (macOS, all four non-Java languages against the live tree): `cargo check --workspace` clean (`zb-mom-ww-mxgateway-client v0.2.0`, `mxgw-cli v0.2.0`), `cargo test --workspace` version tests pass (`version_json_output_has_protocol_versions`); `python -m pytest` 164 passed / 1 skipped (was 163; the one new test); Go `gofmt -l .` clean, `go build ./...` clean, `go test ./...` ok; `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` 0 warnings/0 errors, client tests 128 passed / 1 skipped (unchanged). `JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test` BUILD SUCCESSFUL, CLI suite 51/51 passed including both version-string assertions now at `0.2.1`, **no generated-tree churn** to revert (no `.proto` changed). Script guards: `pwsh -NoProfile` dry-run of `tag-go-module.ps1` — mismatched version (`v1.2.3`) failed with the new guard message before touching git; matched version (`v0.2.0`) passed the guard and reached the pre-existing dirty-working-tree check (expected, uncommitted changes present) — proves the new check runs first and does not block a correct version. `pack-clients.ps1`'s guard functions were exercised directly (dot-sourced in isolation, no pack/publish side effects) against the **live** Gitea registry with real credentials already present in this environment (`GITEA_USERNAME`/`GITEA_TOKEN` from `~/.zshenv`): correctly refused on every known-published artifact (nuget Client 0.1.2, maven 0.2.0, pypi 0.1.1, cargo 0.1.1) and correctly passed on every unpublished target version (0.2.0/0.2.1 across all four). **No actual publish was attempted or performed** — the task explicitly forbids it, and the sandbox additionally refuses any Bash command that inlines `$GITEA_TOKEN`/`$GITEA_USERNAME` into the command text (a credential-use guard, not a git-safety one), so the live checks above ran only by having the script's own functions read the already-exported env vars internally rather than the invoking command line referencing them. |
| 2026-08-07 | Code-review follow-up on the same branch (`fix/cli-39-version-train`). **Important gap fixed:** `src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj` was left at `<Version>0.1.2</Version>` — the original CLI-39 pass bumped the .NET Client but missed that `Invoke-PackDotnet` in `scripts/pack-clients.ps1` packs and publishes **both** `ZB.MOM.WW.MxGateway.Contracts` and `.Client` through the same `-Publish` loop, and the new collision guard runs every `ZB.MOM.WW.MxGateway.*.nupkg` it finds through `Assert-GiteaPackageNotPublished`. Left as-is, the next real `.NET` publish would have packed Contracts at the already-published 0.1.2, the guard would have correctly refused it, and the bare `ForEach-Object` would have aborted mid-loop — with `Client` (0.2.0, alphabetically first) potentially already pushed and `Contracts` stuck at 0.1.2, the two published nupkgs out of lockstep. Fixed: bumped `Contracts.csproj` to `0.2.0` alongside the Client (they have always released together). Also updated the now-stale comment in `src/Directory.Build.props` (it previously said the repo-wide `0.1.2` default was "kept ... to match the Contracts package"; that's no longer true now that Contracts overrides it — the comment now explains the override and that Server/Worker/Tests staying at `0.1.2` is a separate, not-yet-made decision; the `Directory.Build.props` `<Version>` value itself was **not** changed, since Server/Worker are not published packages and were out of scope for CLI-39). `docs/ClientPackaging.md`'s "Versioning" section gained `Contracts.csproj` as a fifth manifest (explaining why it must move with the .NET Client) and a note on the near-miss. **Cheap hardening also applied:** the Python (`pyproject.toml`) and Rust (`Cargo.toml`) version-extraction regexes in `pack-clients.ps1` now scope to the `[project]`/`[package]` section header respectively (matching the first `version = "..."` line anywhere in the file was luck of ordering — Cargo.toml has an identical second one under `[workspace.package]`), plus a one-line comment on the nuget filename-parse regex explaining the lazy-id/anchored-version split. **Verification:** `dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csproj` 0 warnings/0 errors; `dotnet pack` of that project confirmed the artifact is literally named `ZB.MOM.WW.MxGateway.Contracts.0.2.0.nupkg`; `dotnet build clients/dotnet/ZB.MOM.WW.MxGateway.Client.slnx` still 0 warnings/0 errors. Re-ran the pack-clients.ps1 guard function-harness live against the real Gitea registry: `ZB.MOM.WW.MxGateway.Contracts`/`.Client` **0.1.2 both still refused**, **0.2.0 both now pass** — then went one step further and packed both projects for real (`dotnet pack`, no `-Publish`) and ran the exact `Invoke-PackDotnet` filename-parse-then-guard logic against the two real `.nupkg` files, confirming both parse to `id`/`version` correctly and both clear the live collision check. The scoped Python/Rust regexes were also exercised live against the real `pyproject.toml`/`Cargo.toml` and still resolve to `0.2.0`. No publish performed. |
| 2026-08-07 | **TST-30 → fully `Done`** — the operator half (register a second CI runner) executed live on `10.100.0.35` per `docs/runbooks/TST-30-second-ci-runner.md` option (a). A second **instance-level** `act_runner` container, `gitea-runner-2` (runner id 5, capacity 2, labels `ubuntu-latest`/`ubuntu-22.04`), was added to the `/opt/gitea` compose stack with the same `container.network: traefik` setting as the original, its registration token mounted from a `0600` file rather than inlined in compose. The existing `gitea-runner` (id 1, capacity 4) was **not** touched — total capacity went 4 → 6 purely by addition, so the change reverts by removing one container. **Concurrency verified live** (the finding's acceptance test): HEAD `a346d51` was pushed to two scratch branches, `scratch/tst30-a` (run 661) and `scratch/tst30-b` (run 662), while an unrelated run 660 was already in flight; at 13:07:53Z jobs from **three** runs were `in_progress` at once — run 660 `portable` plus run 662 `portable`/`java` on runner 1, and run 661 `portable`/`java` on `gitea-runner-2` — a state the pre-change single-runner topology could not produce. Runner attribution came from `GET /repos/dohertj2/mxaccessgw/actions/runs/{id}/jobs`, which exposes `runner_id`/`runner_name` per job, and was cross-checked against `docker logs gitea-runner-2` (`task 1145 …`, `task 1146 …` pickups at 13:05:22–23Z). **`gitea:3000` resolution holds on the new instance** — the property option (c) was rejected for losing: run 661's `portable` job (task 1145, scheduled on `gitea-runner-2`) logged `git remote add origin http://gitea:3000/dohertj2/mxaccessgw` then a successful `fetch … From http://gitea:3000/dohertj2/mxaccessgw`, and its job container's `/workspace/dohertj2/mxaccessgw` was confirmed checked out at `a346d514dd24e775640e5667aa7cd8e561fec68a`. **Runbook fact corrected:** its checklist expected `GET /repos/dohertj2/mxaccessgw/actions/runners` to show ≥2, but that endpoint still returns `total_count: 0` — correctly, since both runners are instance-scoped, which is exactly what this finding documented; the working check is `GET /api/v1/admin/actions/runners` (lists ids 1, 4 = an unrelated local macOS runner, and 5). A dated "Executed 2026-08-07" note recording this was added to the top of the runbook rather than rewriting its history. No source, workflow, or `docs/GatewayTesting.md` change was needed — the no-cancel reality and the `run-windev-ci.sh` bypass are unaffected by runner count, as that prose already stated. Scratch branches deleted after verification. Tracking: both registers' TST-30 rows, the pending-operator-actions list in `90-candidate-findings-next-cycle.md`, and this row. |
| 2026-08-07 | **CLI-39 publish executed** — the release the version train was cut for is now live on `gitea.dohertylan.com`. Published from HEAD `a346d51` via `scripts/pack-clients.ps1`: `nuget` `ZB.MOM.WW.MxGateway.Client` **0.2.0** + `ZB.MOM.WW.MxGateway.Contracts` **0.2.0**, `pypi` `zb-mom-ww-mxaccess-gateway-client` **0.2.0**, `cargo` `zb-mom-ww-mxgateway-client` **0.2.0**, `maven` `com.zb.mom.ww.mxgateway:zb-mom-ww-mxgateway-client` **0.2.1** (the Java exception recorded in the CLI-39 row above — 0.2.0 was already taken by the pre-conformance build). All five coordinates re-queried through `GET /api/v1/packages/dohertj2/{type}/{name}/{version}` after the push and confirmed `200`. Go ships no artifact: module tag `clients/go/v0.2.0` was created and pushed at `a346d51` by `scripts/tag-go-module.ps1` (its CLI-21 version-match guard passed against `ClientVersion = "0.2.0"`), and `go list -m …@v0.2.0` resolves. Full suites ran pre-push with no failures: 128 .NET, 164 Python, 107 Rust, 79 Java. **Deviation found and fixed:** the first `cargo publish` returned `401` because `CARGO_REGISTRIES_DOHERTJ2_GITEA_TOKEN` held the raw token — cargo forwards that value as the `Authorization` header verbatim and Gitea's cargo registry requires `Bearer <token>` there (confirmed both ways against `/api/v1/user`: bare → 401, `Bearer`-prefixed → 200), unlike the other feeds' username/token basic-auth pairs. The publish succeeded on retry with the prefix exported shell-locally; the permanent fix was then applied to the operator's `~/.zshenv` (outside this repo; prior file backed up `0600` as `~/.zshenv.bak-cli39`) and re-verified via a `Bearer`-authed `GET /api/v1/user` → `200`. The requirement is now documented in `docs/ClientPackaging.md`'s Rust section, and that doc's "Versioning" section carries the published-coordinates record so the next bump starts from the real registry state rather than a re-check. No source, proto, script, or test change — publish execution plus docs. Tracking: `docs/ClientPackaging.md` and this row (CLI-39's register rows in both this file and `50-clients.md` were already plain `Done` with no publish-pending annotation to clear). |
| 2026-08-07 | **SEC-36 → fully `Done`** — the operator half (rotate the dev LDAP service-account credential) executed live per `docs/runbooks/SEC-36-ldap-credential-rotation.md`. A new password was generated, the `serviceaccount` `passsha256` in `scadaproj/infra/glauth/config.toml` replaced, and the shared GLAuth recreated on `10.100.0.35`, so the value this repo's git history permanently discloses (live in the directory since 2026-06-04; not reproduced here) **no longer binds** `dc=zb,dc=local` — the load-bearing half of the finding. The new value lives only in the three channels the design named: the GLAuth `passsha256` (committed in `scadaproj`, commit `aada53b`), the NSSM environment on `10.100.0.48`, and each dev box's user-secrets (set on this Mac); it is in no file of this repo. The retired plaintext was also scrubbed from the `scadaproj` glauth comments (`config.toml`, `docker-compose.yml`, `README.md`) and from the docker host's live `docker-compose.yml`; the host's `*.bak-sec36` rollback copies deliberately keep it. **Three runbook facts corrected** in a dated block at its top: (1) step 3's `cd ~/Desktop/scadaproj/infra/glauth` on the docker host is stale — the stack runs from `/home/dohertj2/zb-glauth` (container `zb-shared-glauth`), fed by the `scp` deploy in `scadaproj/infra/glauth/README.md`; (2) `wonder-app-vd03` is out of scope **on documentary evidence**, not merely unchecked — its gateway binds the ScadaBridge/ScadaLink local GLAuth under `dc=scadalink`/`dc=scadabridge`, a different directory that never held this credential (and the host is unreachable from the dev network), so no env var was staged there; (3) the "3-fail / 10-minute per-IP lockout" caution is inert for this instance — `config.toml:14` sets `LimitFailedBinds = false`. **One Done criterion is met with a caveat:** the new value is staged on windev (10th `AppEnvironmentExtra` entry on the `MxAccessGw` NSSM service) but the runbook's primary dashboard `/login` check **could not run** — windev's gateway is crash-looping on an unrelated pre-existing fault (deployed Server binary of 2026-06-25 predates the 2026-07-15 auth-DB migration, so it opens a schema-3 DB it supports only at schema 2 and aborts; ~10k Hosting-failed events/day since at least 08-06). Verified instead by the equivalent primitive: a direct `ldapsearch` bind as `cn=serviceaccount,dc=zb,dc=local` with the new value against `10.100.0.35:3893` succeeded and returned the `multi-role` entry — the same search bind the dashboard performs. The deferred check is recorded in the runbook with its completion path. **Two new next-cycle candidate findings** filed in `90-candidate-findings-next-cycle.md`: NEXT-06 (`DashboardLdapLiveTests` fixture drift — its only positive test binds `admin`/`admin123` while the directory's `admin` password is the standard dev one, and fixture user `readonly` does not exist in the GLAuth config, so the suite carries **no positive-proof coverage** of the service-account bind and could not have substituted for the dashboard check) and NEXT-07 (the windev stale deployment / crash loop above). Tracking: both registers' SEC-36 rows, the pending-operator-actions list in `90-candidate-findings-next-cycle.md`, and this row. |
| 2026-08-07 | **NEXT-07 resolved and SEC-36's deferred dashboard check discharged** — the two loose ends the SEC-36 row above left open. windev (`10.100.0.48`) was repaired by a fresh portable framework-dependent publish of `origin/main` (`a346d51`) built in a clean clone at `C:\build\mxgw-redeploy`, deployed to `C:\publish\mxaccessgw\Server-20260807` with the `MxAccessGw` NSSM service repointed; the service holds a stable PID with `5120`/`5130` listening, a worker spawned, the Galaxy snapshot restored (129 objects / 56,731 attributes) and a clean event log. The predicted root cause is confirmed: the deployed 2026-06-25 build carried `ZB.MOM.WW.Auth.ApiKeys` 0.1.2.0 (auth-DB schema 2) against a database stamped at schema 3 on 2026-07-15 by an ephemeral run of newer code — schema 3 is the **current** shared-lib version (`SqliteAuthSchema.CurrentVersion=3` in Auth 0.1.5), so redeploying forward was the fix and the DB was left untouched; `gateway-auth.db.bak-next07` (+`-wal`/`-shm`) and the previous `Server` publish directory stand as rollback. Two side effects: the old deploy's `appsettings.json` carried the LDAP bind password in **plaintext on disk**, while the new one keeps the repo's `${secret:ldap/mxgateway/bind}` token with the NSSM environment supplying the value; and the redeploy tripped the SEC-06 `Ldap:Transport=None` production hard-stop (`GatewayOptionsValidator.cs:178`), resolved by relabelling `DOTNET_ENVIRONMENT` to `Staging` (that entry only — the other nine preserved byte-identical), which matches what windev actually is: a host running `Dashboard:DisableLogin=true` against a GLAuth with no TLS. SEC-06 is untouched for real production hosts, and the posture gap that relabelling defers is filed as NEXT-08. With windev live, **SEC-36's deferred check ran for real**: a foreground run of the new build with `Dashboard:DisableLogin=false` as a process-env-only override served `GET /login` (200 + antiforgery token), accepted `POST /auth/login` as `multi-role`/`password` (302 to `/` + `MxGatewayDashboard` cookie), rendered the admin nav on the authenticated `GET /`, and redirected the anonymous control to `/login?ReturnUrl=%2F` — proving the rotated credential through the real `DashboardAuthenticator` search bind on the deployed host, not just by `ldapsearch`. Also executed host-side: runner-1 on `10.100.0.35` moved to the same `0600` token-file mount as runner-2 (identity preserved), leaving the shared instance registration token — proven still live — as a **pending operator UI reset**. Three new candidate findings filed: NEXT-08 (GLAuth has no TLS, so SEC-06 makes `docs/GatewayConfiguration.md`'s "deployed hosts must set `Ldaps` or `StartTls`" unsatisfiable), NEXT-09 (`src/Directory.Build.props:29` stamps git's error text into `InformationalVersion` on Windows), NEXT-10 (`glauth.md`'s pre-provisioned-user table contradicts the directory and the rest of its own file — the root cause of NEXT-06, deliberately deferred because reconciling it sweeps the OPC-UA group taxonomy). Tracking: `90-candidate-findings-next-cycle.md` (NEXT-07 struck, NEXT-08/09/10 added, operator-action list), `docs/runbooks/SEC-36-ldap-credential-rotation.md` Correction 3, `docs/runbooks/TST-30-second-ci-runner.md`, `docs/GatewayConfiguration.md` SEC-35 section. |
