Files
mxaccessgw/archreview/2026-07-12/remediation/00-tracking.md
T
Joseph Doherty 0d874f91ee fix(CLI-40): scrub the credential from the redacted error's structured reply, route MXACCESS_FAILURE to MxAccess (Rust), fix Go Subscribe terminal-error drop
Code-review follow-up on the CLI-40/41/44 branch.

ISSUE 1 (all five, critical): the message-only scrub still leaked the
server-echoed credential through the redacted error's structured reply accessor
(.NET Reply/Statuses, Java reply()/protocolStatus(), Go MxAccessError.Reply via
errors.As, Rust reply()/into_reply(), Python raw_reply). The redacted error now
carries a scrubbed clone of the reply (protocol_status.message,
diagnostic_message, statuses[].diagnostic_text), with per-language tests asserting
the reply accessor no longer contains the credential.

ISSUE 2 (Rust, critical): ensure_command_success routed MXACCESS_FAILURE to
Error::Command (unlike the other four clients), bypassing attach_secrets and
leaking via derived Debug/Display. MXACCESS_FAILURE now routes to Error::MxAccess,
fixing the cross-client inconsistency.

ISSUE 3 (Go, important): the CLI-44 terminal send was unconditionally
non-blocking, dropping a genuine terminal error under a full buffer on the
never-drop SubscribeEvents path. It is now reserved-slot-non-blocking only for the
cancel-on-overflow path and blocking for the never-drop path.

New shared fixture authenticate-user.echoed-credential-mxaccess-failure.reply.json
wired into all five suites. Minors: whitespace-secret guard on .NET/Java redact
helpers; Java preserves exception subtype on redaction; redaction-helper unit
tests (Go/Java/.NET). Docs (ClientBehaviorFixtures.md, ClientLibrariesDesign.md)
updated to make the structured-field claim true.
2026-08-07 07:04:56 -04:00

42 KiB
Raw Blame History

MxAccessGateway — Remediation Tracking (2026-07-12 review)

Master progress tracker for the 2026-07-12 follow-up architecture review. Generated 2026-07-13.

Source review: ../00-overall.md · Per-domain remediation designs are linked below and hold the full Finding / Impact / Design / Implementation / Verification for every entry here. The first-cycle tracker (../../remediation/00-tracking.md) remains the record for the original 153 findings; this document tracks only the 47 new IDs (GWC-24+, WRK-21+, IPC-23+, SEC-31+, CLI-35+, TST-25+) plus the old-tracker actions listed at the end.

How to use this document

  • Each finding has a stable ID that never changes. Cite it in commits, branches, and PRs (e.g. fix(GWC-25): empty-ring ReplayGap sentinel).
  • The Status column is the single source of truth for progress. Update it in the same change that lands the fix.
  • Do the work in roadmap-tier order (P0 → P1 → P2), respecting the Dep column and the cross-cutting clusters below — several fixes are one change set across two domains and must land together.
  • When a fix lands: flip Status to Done and, per the repo rule, update the affected docs in the same commit.

Status legend: Not started · In progress · In review · Done · Won't fix (record why in the domain doc) · N/A (informational / decision recorded, no action).

Severity roll-up

Domain Doc High Medium Low Info Total
Gateway core 10-gateway-core.md 2 4 1 7
Worker 20-worker.md 1 7 8
Contracts & IPC 30-contracts-ipc.md 3 5 2 10
Security & dashboard 40-security-dashboard.md 1 4 1 6
Clients 50-clients.md 5 6 11
Testing, docs & gaps 60-testing-docs-gaps.md 1 2 3 6
Total 1 14 29 4 48

Roadmap-tier roll-up

Tier Meaning Count Findings
P0 Correctness & safety — all small-to-medium 10 GWC-25, WRK-21, IPC-23, IPC-24, IPC-25, IPC-30, SEC-31, SEC-32, CLI-35, CLI-36
P1 Process & hardening 12 GWC-24, WRK-26, SEC-33, SEC-36, CLI-37, CLI-38, CLI-39, CLI-42, CLI-45, TST-25, TST-26, TST-27
P2 Completeness & polish 10 GWC-26, GWC-27, GWC-28, WRK-25, IPC-26, IPC-27, SEC-34, TST-28, TST-29, TST-30
Not individually in the roadmap (Lows/Infos) 16 GWC-29/30, WRK-22/23/24/27/28, IPC-28/29/31/32, SEC-35, CLI-40/41/43/44

P0 — do first (10)

Sequenced by cluster; a cluster is one change set.

ID Sev Eff Dep Status Title
GWC-25 Medium S CLI-35/36 (coord) Done Empty-ring ReplayGap sentinel carries oldest_available_sequence = 0, dead-streaming a compliant client
CLI-35 Medium S GWC-25 (coord) Done Python CLI stream-events crashes on a ReplayGap
CLI-36 Medium S GWC-25 (coord) Done Go CLI stream-events silently destroys the ReplayGap signal
WRK-21 Medium M owns IPC-23 fix; WRK-28 same batch Not started DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events
IPC-23 Medium S WRK-21 Not started DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave
IPC-30 Low M WRK-21 (same batch) Not started Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity)
SEC-31 Medium M Done Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS)
SEC-32 Low S SEC-31 Done Failure-limiter LRU flushable by junk-token spray; token prefix never validated
IPC-24 Medium S Not started CI's unconditional Java churn-revert masks real drift
IPC-25 Medium M Not started Regenerate stale Go/Python worker bindings + add binding-freshness guard (Check 4) to check-codegen.ps1

Finding registers by domain

Full design + implementation for each row lives in the linked domain doc under its ID heading.

Gateway core — 10-gateway-core.md

ID Sev Tier Eff Dep Status Title
GWC-24 Medium P1 M GWC-21 (coord, old tracker) Done Unbounded event staging channel: sustained slow drain grows memory silently and invisibly
GWC-25 Medium P0 S CLI-35/36 (coord) Done Empty-ring ReplayGap sentinel carries oldest_available_sequence = 0
GWC-26 Low P2 M GWC-27 Done Alarm monitor attaches its subscriber after SubscribeAlarms; window transitions bypass the feed
GWC-27 Low P2 S GWC-26 Done AttachInternalEventSubscriber bypasses the readiness gate; premature attach poisons the distributor
GWC-28 Low P2 S GWC-10 (coord, old tracker) Done Gateway→worker envelope sequence stamped at creation, not at write
GWC-29 Low S Done Invoke deep-clones the entire request only to discard the cloned command
GWC-30 Info S Done Frame reader allocates a fresh 4-byte length-prefix array per frame

Worker — 20-worker.md

ID Sev Tier Eff Dep Status Title
WRK-21 Medium P0 M IPC-23 (fix owned here); WRK-28 Not started DrainEvents bound count-based only; oversized reply kills session and loses drained events
WRK-22 Low S IPC-26 (fix owned here) Not started Cancelled WriteAsync leaves its frame queued; it is still written later
WRK-23 Low S WRK-21 Not started Rejected frames consume sequence numbers, producing wire gaps
WRK-24 Low S Not started AdoptNegotiatedMaxMessageBytes has no lower-bound sanity check
WRK-25 Low P2 S WRK-22 (shared seam) Not started WRK-12 flush coalescing never engages on the event hot path
WRK-26 Low P1 S WRK-23 (soft); discharges IPC-29 Not started Write-priority and overflow doc drift from the WRK-07 change
WRK-27 Low S Not started Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s)
WRK-28 Low S WRK-21 (same batch) Not started 10,000 drain cap is a duplicated magic constant

Contracts & IPC — 30-contracts-ipc.md

ID Sev Tier Eff Dep Status Title
IPC-23 Medium P0 S WRK-21 (mechanics) Not started DrainEvents byte-blindness — contract requirements + proto-comment/doc wave
IPC-24 Medium P0 S Not started CI's unconditional Java churn-revert masks real drift
IPC-25 Medium P0 M Not started Stale Go/Python worker bindings: regenerate (pinned toolchains) + check-codegen Check 4
IPC-26 Low P2 S WRK-22 (mechanics) Not started Cancelled write leaves ghost frame — cancelled means never written
IPC-27 Low P2 S Not started Descriptor-freshness test blind to enums/services/galaxy descriptor
IPC-28 Low S Not started docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping
IPC-29 Low S WRK-26 (discharged by) Not started WorkerFrameProtocol.md missing write-scheduling/sequencing section
IPC-30 Low P0 M WRK-21 (same batch) Not started Oversized event frame: keep session-fatal, make the death structured
IPC-31 Info N/A Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded
IPC-32 Info S IPC-25 (folded in) Not started check-codegen banner relabel 1/4…4/4

Security & dashboard — 40-security-dashboard.md

ID Sev Tier Eff Dep Status Title
SEC-31 Medium P0 M Done Failure limiter: composite (peer, key-id) partitions + cross-peer aggregate with probe admission
SEC-32 Low P0 S SEC-31 Done Limiter LRU flushable by junk-token spray; validate token shape, cap per-peer partitions
SEC-33 Low P1 M old SEC-23 (co-locate) Not started Host-meaningful path rooting; drop Windows literals from appsettings; validate Galaxy SnapshotCachePath
SEC-34 Low P2 S Not started Verification cache: expiry outlives TTL; Invalidate races in-flight repopulation
SEC-35 Info S N/A Production hard-stops key on exact Production environment name (doc-only)
SEC-36 Low P1 M cross-repo scadaproj/infra/glauth Not started Committed dev LDAP service-account password: rotate, remove, move dev channel to user-secrets

Clients — 50-clients.md

ID Sev Tier Eff Dep Status Title
CLI-35 Medium P0 S GWC-25 (coord) Done Python CLI stream-events crashes on a ReplayGap
CLI-36 Medium P0 S GWC-25 (coord) Done Go CLI stream-events silently destroys the ReplayGap signal
CLI-37 Medium P1 M CLI-38 (co-land) Done Status-array validation must branch on category per the proto contract (4-vs-1 divergence)
CLI-38 Medium P1 S Done Align .NET/Go/Java on hresult < 0 — lands old CLI-08, cures design-doc drift
CLI-39 Medium P1 S CLI-35..38, CLI-45 (land last) Not started Bump client versions off published 0.1.2 (converge on 0.2.0); registry-collision guard in pack-clients.ps1
CLI-40 Low M Done Port the exact-secret credential scrub to Rust/Java/.NET
CLI-41 Low M Done Uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/AddBufferedItem
CLI-42 Low P1 S Not started Document the vendored Rust proto layout (CLI-02's missing doc half)
CLI-43 Low S Not started Java style guide still prescribes "Java 21 preferred"
CLI-44 Low S Done Go event goroutine can mislabel a genuine terminal error as ErrSlowConsumer
CLI-45 Low P1 M Done Standardize CLI credential env-var names; fail fast on missing/empty passwords

Testing, docs & gaps — 60-testing-docs-gaps.md

ID Sev Tier Eff Dep Status Title
TST-25 High P1 M unlocks old TST-05, TST-24 Done Windows/x86 test tier has zero automation — SSH-driven windev CI job
TST-26 Medium P1 S TST-25 (same commit) Done Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen
TST-27 Medium P1 S Not started ShowTagValues config row still says "Reserved" after SEC-25 made the flag live
TST-28 Low P2 S relates IPC-02 (old) Done Gateway-side max_frame_bytes handshake untested in the CI-run suite
TST-29 Low P2 S Done Retire oldtasks.md (fold Phase-5 governance into DesignDecisions.md); delete root artifacts
TST-30 Low P2 M Not started Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete API)

Cross-cutting clusters

Sequence these together rather than piecemeal — several are one change set spanning two domains:

  • Drain / oversized-frame cluster (P0): WRK-21 + IPC-23 + IPC-30, with WRK-28 and WRK-23 in the same batch. One worker change set in WorkerPipeSession.cs/WorkerFrameWriter.cs; WRK-21 owns the byte-budgeted drain and must satisfy IPC-23's requirements R1R3; IPC-30's structured-fault seam lands in the same drain loop. IPC-23's proto-comment edits trigger the full regen fan-out (Generated/, Rust vendored, Go/Java, descriptor set) — land with WRK-21 so the wave happens once. One windev x86 verification run for the cluster.
  • ReplayGap end-to-end (P0): GWC-25 (server sentinel arithmetic) + CLI-35 (Python CLI) + CLI-36 (Go CLI). Independently landable; the e2e resume walk validates only when all three are in.
  • Auth limiter (P0): SEC-31 + SEC-32 — same component, one change set, one test suite.
  • Codegen freshness (P0): IPC-24 + IPC-25 (+ IPC-32 folded in). Both edit .gitea/workflows/ci.yml — coordinate the branch with TST-25, which touches the same file.
  • Windows-tier automation (P1): TST-25 + TST-26 (same commit). Unlocks old TST-05/TST-24 and provides CI evidence for every windev-verified cluster above; until it lands, record windev runs in this tracker's change log.
  • Client conformance + release train (P1): CLI-37 + CLI-38 co-land (one conformance commit; closes old CLI-08), then CLI-45, with CLI-40/41 fixtures as follow-ups; CLI-39 lands last so published 0.2.0 carries the conformant behavior. Shared fixtures under clients/proto/fixtures/behavior/; update CrossLanguageSmokeMatrix.md/ClientLibrariesDesign.md same-commit. Do not republish regenerated bindings (IPC-25) before CLI-39 resolves.
  • Doc-drift batch (P1): TST-27 + WRK-26 (discharges IPC-29) + CLI-42 + IPC-28 + SEC-35's doc note — one sweep commit is fine.
  • Backpressure follow-on (P1): GWC-24 — Done 2026-08-07. Landed without a new config key (the staging bound is derived as 2 × EventChannelCapacity), so still-open old GWC-21 (EventChannelFullModeTimeout configurability) remains the natural companion knob but was not a blocker.

Old-tracker actions (../../remediation/00-tracking.md)

  • Close CLI-24 and CLI-34 as Done (incidentally fixed; evidence in ../50-clients.md).
  • When CLI-38 lands, close old CLI-08 with a pointer here. Done 2026-08-07: CLI-38 landed and old CLI-08 is now Done in the first-cycle tracker, pointing at CLI-38.
  • When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap; when TST-25 lands, revisit old TST-05 (scheduled live smoke) and TST-24 (client wire tests), which it unlocks.

Change log

Date Change
2026-07-13 Initial tracking doc generated from the six domain remediation designs. All 47 findings Not started (IPC-31, SEC-35 N/A).
2026-07-13 TST-25/TST-26 → In progress (branch fix/tst-25-windev-ci). Added scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}, windows-x86 (per-push) + nightly-windev (scheduled) jobs in ci.yml, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: build→0, bogus-SHA→nonzero (lock released), test→356 passed/0 failed in ~50s (per-push stays test, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed.
2026-07-13 Operator bring-up complete: dedicated CI ed25519 key installed in windev administrators_authorized_keys (authorized into dohertj2, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets WINDEV_SSH_KEY/WINDEV_SSH_KNOWN_HOSTS + variable WINDEV_SSH_USER=dohertj2 stored, runner→10.100.0.48:22 egress verified on the traefik net, issue-write confirmed. TST-25/TST-26 → Done: credentialed windows-x86 ran GREEN on d769244 (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in C:\build\mxaccessgw-ci under lock, ran the x86 Worker build + Worker.Tests, exit 0; nightly-windev correctly skipped on the push event. Branch merged to main. Follow-ups (old tracker): revisit TST-05 (scheduled live smoke — now covered by nightly-windev) and TST-24 (client wire tests) which this unlocks.
2026-07-13 Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught two real CI defects, both fixed on fix/tst-25-ci-key-log-leak: (1) CI SSH key leaked in cleartext in the windows-x86 step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key base64-encoded so the masker redacts it to *** (confirmed on run #38), taught run-windev-ci.sh to decode, dropped the redundant public known-hosts secret from the job env; (2) bootstrap lock racerun-windev-ci.sh's pre-hand-off git fetch/checkout ran outside the worktree lock, so concurrent runs collided on .git/index.lock; the bootstrap now holds the lock (ps1 re-uses it via MXGW_CI_LOCK_HELD), retest confirmed clean serialization. Also deflaked SessionManagerTests fail-fast timing assertions (absolute <100ms wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target df7e20d verified GREEN via the local windev path (Worker build + 356 tests); merged to main 19cbf7b. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual if: failure() gating is standard Actions).
2026-07-13 New finding TST-30 (Low/P2) added — surfaced during TST-25 acceptance: CI runs on a single shared gitea-runner (maxParallel=1, co-located 10.100.0.35) interleaved with dohertj2/lmxopcua, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the run-windev-ci.sh bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10).
2026-08-07 TST-29 → Done: migrated the Phase-5 (orphan-worker reattach) deferred-not-planned governance record and the settled Phase-4 Viewer-default decision from oldtasks.md into a new "Session-Resilience Epic Scope" entry in docs/DesignDecisions.md; repointed CLAUDE.md and stillpending.md:7,165 from oldtasks.md to docs/DesignDecisions.md / docs/plans/2026-06-15-session-resilience.md.tasks.json; git rm oldtasks.md. The five untracked root docs-review artifacts (MxAccessGateway-docs-{issues,fixed,final}.md, MxGatewayClient-docs-{issues,fixed}.md) were absent from this worktree — delete from the main working tree separately.
2026-08-07 GWC-24 → Done (branch fix/gwc-24-staging-bound). WorkerClient._eventStaging is now Channel.CreateBounded at 2 × EventChannelCapacity (Wait, single reader/writer, no sync continuations); a rejected staging TryWrite faults the client ProtocolViolation with QueueOverflow("worker-event-staging") unless IsTerminalState() (shutdown stays a silent drop), so a consumer draining slower than its worker produces dies at a fixed ceiling instead of growing gateway memory. Queue-depth accounting moved from EnqueueWorkerEventAsync to StageWorkerEvent, so the single gauge reports staged + queued; the timed-write fault (EventChannelFullModeTimeout / QueueOverflow("worker-events")) is unchanged and still catches the full-stall case first. No new config key — total gateway-side buffering is 3 × MxGateway:Events:QueueCapacity, derived; coordination with still-open old GWC-21 (EventChannelFullModeTimeout configurability) remains open and was not blocked on. Docs same commit: GatewayProcessDesign.md (two overflow faults), MxAccessWorkerInstanceDesign.md, GatewayConfiguration.md, Metrics.md. Tests: WorkerClientTests.StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout and .WorkerEventQueueDepthGaugeCountsStagedEvents; WorkerClientTests 22/22 green, NonWindows.slnx builds with 0 warnings.
2026-08-07 ReplayGap end-to-end cluster (GWC-25 + CLI-35 + CLI-36) → Done on fix/gwc-25-replaygap-trio. GWC-25: SessionEventDistributor.RegisterWithReplay's empty-ring branch now reports oldestAvailableSequence = _highestSequenceSeen + 1 when gap == true (still 0 when no gap), so the universal oldest - 1 resume formula no longer wraps to ulong.MaxValue and dead-stream the subscriber; docs/Sessions.md documents the empty-ring value. CLI-35: the Python CLI renders a ReplayGap as a {"replayGap": {...}} row via a new _event_row helper instead of crashing in MessageToDict. CLI-36: the Go CLI branches on result.IsReplayGap() and prints the typed REPLAY_GAP requested_after=<n> oldest_available=<n> line / replayGap JSON row instead of formatting the library's cleared Event. docs/CrossLanguageSmokeMatrix.md gained a per-CLI gap-rendering table (one edit covering both client findings). Four new tests as designed (3 × SessionEventDistributorTests, GatewayEndToEndReconnectReplayTests.ReconnectAfterFullAgeEvictionResumesWithSentinelFormula) plus test_stream_events_renders_replay_gap (Python) and TestRunStreamEventsPrintsReplayGap (Go); all written red first and each reproducing its defect verbatim. Deferred: GWC-25's ReplayGap.oldest_available_sequence proto-comment amendment is not in this change — it is comment-only but triggers the full five-client regen fan-out, so it lands with the later codegen wave (alongside IPC-23's proto-comment edits) rather than forcing a regen for one sentence. Note for that wave: the fake-worker gateway e2e suite cannot run on the macOS worktree without TMPDIR shortened (macOS caps the Unix-domain-socket path backing .NET named pipes at 104 chars; TMPDIR=/tmp dotnet test … works and was used here).
2026-08-07 GWC-27 → Done, GWC-26 → Done (branch fix/gwc-26-27-alarm-attach). GWC-27: GatewaySession.AttachInternalEventSubscriber now mirrors AttachEventSubscriber's readiness gate under _syncRoot, before EnsureDistributorCreated, so a premature attach can no longer latch a poisoned distributor. GWC-26: the alarm monitor takes its internal lease directly from the session before SubscribeAlarms and drains it after the first reconcile; ISessionManager.ReadAlarmEventsAsync removed (zero remaining callers); ApplyReconcile now broadcasts an Acknowledge feed transition for a both-present alarm whose state advanced to ActiveAcked (feed-level repair on AlarmFeedMessage, not MxEvent synthesis). New tests GatewaySessionTests.AttachInternalEventSubscriberBeforeReadyThrowsAndDoesNotPoisonDistributor and GatewayAlarmMonitorAttachOrderTests (TransitionsDuringSubscribeWindow_StillReachTheAlarmFeed, ApplyReconcileBroadcastsAcknowledgeDelta); the alarm-monitor fakes now hand the monitor a real Ready GatewaySession with a dashboard mirror so the window is actually reproducible. Verification: NonWindows build 0 warnings/0 errors; GatewayAlarmMonitor 16 passed, SessionManagerTests 38 passed, GatewaySessionTests 19 passed, AlarmFailoverEndToEndTests 2 passed.
2026-08-07 Code review of fix/gwc-26-27-alarm-attach surfaced a known pre-existing characteristic, now documented: the alarm monitor's reconcile-derived feed repairs are at-least-once, not exactly-once. A reconcile reads the worker's current state while the matching live transition may still be buffered in the monitor's internal lease, so both broadcast and the duplicates are indistinguishable on the alarm feed (StreamAlarms + dashboard alarm hub). This pre-dates GWC-26 — the Raise/Clear presence repair has always had it, since nothing serializes a reconcile pass against the in-flight live stream — so closing it (reconcile/live serialization or transition-timestamp dedup) was ruled out of scope for a P2 fix. Documented instead in GatewayAlarmMonitor.ApplyReconcile, gateway.md, and docs/Sessions.md, with the consumer-side contract stated explicitly (apply transitions idempotently — "set this alarm to this state", never increment/toggle). Candidate finding for the next review cycle.
2026-08-07 CLI-37 + CLI-38 -> Done (branch fix/cli-37-38-conformance), one cross-client conformance commit; closes old-tracker CLI-08. Canonical rules landed everywhere: an MxStatusProxy entry fails iff category != MX_STATUS_CATEGORY_OK (success is the raw COM member, diagnostics only; absent entry = success, present entry with UNSPECIFIED = failure), and a reply fails on HRESULT iff hresult is present and < 0 (so S_FALSE = 1 passes). Edits: .NET MxStatusProxyExtensions.IsSuccess (drop the Success != 0 conjunct) + MxCommandReplyExtensions (!= 0 -> < 0); Go StatusSucceeded (category) + errors.go (< 0); Java MxStatuses.succeeded (category, Javadoc corrected) + MxGatewayErrors (< 0); Python errors.py (category); Rust ensure_mxaccess_success (category, doc comment corrected). Four shared fixtures added under clients/proto/fixtures/behavior/command-replies/ (write.status-category-{error-success-set,ok-success-zero}.reply.json, write.hresult-{s-false,e-fail}.reply.json) + manifest + docs/ClientBehaviorFixtures.md; each of the five suites now runs all four fixture-driven, plus a per-language table test for the two edges fixtures cannot express (nil/null entry, UNSPECIFIED category). Docs same commit: ClientLibrariesDesign.md per-item rule sentence (its existing HRESULT < 0 claim is now true), .NET/Go/Java README error sections. Also fixed a Java test fake that built a status with a bare setSuccess(1) and no category. Verification: dotnet build 0 warnings + 110 passed/1 skipped; gofmt -l clean, go build ./..., go test ./... all ok; gradle test BUILD SUCCESSFUL with no generated-file churn to revert this time (no .proto changed and generateProto stayed up to date); python -m pytest 155 passed/1 skipped; cargo fmt (no unrelated reformat), cargo check, cargo test --workspace 100 passed, cargo clippy --all-targets -- -D warnings clean. Gateway-side ClientBehaviorFixtureTests 8/8 re-run because the new fixtures are validated there.
2026-08-07 CLI-45 → Done on fix/cli-45-credential-envvar. All five CLIs now share one credential contract for authenticate-user: flags --password / --password-env (Go: -password / -password-env) defaulting to env MXGATEWAY_VERIFY_PASSWORD, resolution flag-then-env, and a resolved credential that is missing or empty is a usage error naming the flag and the variable — never the value, never sent to the wire. Go and Java previously authenticated with an empty password: Go now returns the guard error before dialing, Java throws a picocli ParameterException instead of falling back to "". Python's --password-env gained the canonical default (its UsageError was already conformant) and its message now names the resolved variable. Rust treats an empty --password or empty env value as missing (resolution extracted into a testable resolve_verify_user_password). .NET adopted the canonical flags and keeps its pre-existing names as deprecated aliases for one release — order: --password, --verify-user-password, the variable named by --password-env (or the deprecated --verify-user-password-env; default MXGATEWAY_VERIFY_PASSWORD), then MXGATEWAY_VERIFY_USER_PASSWORD. Tests: TestRunAuthenticateUser{RejectsEmptyPassword,ReadsPasswordFromCanonicalEnv} (Go), 3 picocli cases (Java), 3 click cases (Python), 2 clap/resolver cases (Rust), 4 xUnit cases covering the canonical flag, both env-name paths, the deprecated flag+env aliases, and the missing/empty failure (.NET). Docs same commit: docs/CrossLanguageSmokeMatrix.md gained a "Credential contract for authenticate-user" section and the per-CLI subcommand-coverage table — the half of this finding that is documented rather than fixed (.NET exposes all nine single-item session commands; Rust unregister + the credential pair; Go/Python/Java the credential pair only; verified against each dispatch table, and every gap is CLI surface only since all five libraries implement all nine helpers). All five client READMEs name the canonical variable and the fail-fast rule; the .NET README gained an authenticate-user credentials section carrying the deprecation note. Deviation: Java keeps isBlank() (per this design's "null or blank" wording for Java) where the other four test emptiness, so a whitespace-only credential is additionally rejected there. Verification (all five, on macOS): Go gofmt -l . clean, go build ./... clean, go test ./... ok; Java JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test BUILD SUCCESSFUL, CLI suite 51 tests / 0 failures — no generated-tree churn appeared this run, git status for clients/java/**/generated clean with no revert needed (no .proto changed); Python python -m pytest 148 passed / 1 skipped (TLS opt-in); .NET dotnet build …Client.slnx 0 warnings / 0 errors and client tests 108 passed / 1 skipped (live-gateway opt-in); Rust cargo fmt (diff confined to the new code), cargo check --workspace, cargo test --workspace 100 tests across 6 targets all green, cargo clippy --all-targets -- -D warnings clean.
2026-08-07 SEC-31 + SEC-32 → Done (branch fix/sec-31-32-limiter, one change set as planned). ApiKeyFailureLimiter reworked from IsBlocked/RecordFailure/Reset(string peer) to a partition-pair API (Check/RecordFailure/Reset(ApiKeyThrottlePartition) returning ApiKeyThrottleDecision): layer 1 is the composite (transport peer, key id) partition, layer 2 a per-key-id aggregate across peers (ApiKeyFailureAggregateLimit, default 30), and an over-limit state now admits one probe per ApiKeyFailureProbeIntervalSeconds (default 5) instead of blocking absolutely — so a success can reset the state while throttled, killing the 10-packets-per-minute lockout. SEC-32 rides along: the interceptor validates token shape (mxgw prefix, ≥3 non-empty _ segments, key id ≤ 64 chars) before minting a key-id partition, each peer may mint at most 32 of them (overflow collapses to its fallback partition), and eviction prefers expired windows, never dropping an over-limit partition below a 2× transient overshoot ceiling. New counter mxgateway.auth.throttled tagged stage=peer|aggregate only (no key material — /metrics is still unauthenticated per open SEC-14). Docs updated in the same commit (docs/GatewayConfiguration.md limiter rows + two new keys, docs/Authentication.md hot-path paragraph, docs/Authorization.md SEC-11 section, limiter/SecurityOptions XML remarks). Evidence: dotnet build …Server clean; --filter ~ApiKeyFailureLimiter 11/11 passed (new ApiKeyFailureLimiterTests), --filter ~GatewayGrpcAuthorizationInterceptor 20/20 passed (incl. the four SEC-31 contract tests and NonMxgwToken_FallsBackToTransportPeerPartition), --filter ~GatewayOptionsValidator 66/66 passed. Full suite on macOS: 804 passed / 44 failed — all 44 are the pre-existing named-pipe fake-worker classes (WorkerClientTests, FakeWorkerHarnessTests, SessionWorkerClientFactoryFakeWorkerTests, GatewayEndToEnd*), verified identical (44) on the unmodified tree. Follow-up unchanged: the new MxGateway:Security keys belong in old SEC-24's effective-config projection when that is picked up. Code review of the branch found two defects in the first pass, both fixed before merge: (1) probe admission was check-then-act across two lock scopes, so a burst arriving at an interval boundary could all observe "due" and all be admitted — the claim is now a single critical section (TryConsumeProbe), and because the two layers are claimed one at a time, a slot claimed on the partition is compensated (ReleaseProbe) when the aggregate then refuses; (2) Reset on a success whose key id had been collapsed into the address's shared fallback partition removed that shared partition, letting one authentication wipe an in-progress spray from the same address — it is now left to decay by window expiry, while the key's aggregate is still cleared. Tests added: ProbeAdmission_UnderConcurrentArrivals_GrantsExactlyOneSlot, ProbeAdmission_WhenAggregateRefuses_ReturnsTheClaimedPeerSlot, Reset_WithOverCapKeyId_DoesNotClearSharedFallbackPartition. A second review pass found a residual defect in that compensation path: the release identified its own reservation by comparing NextProbeAtTicks to now + interval, the identical expression a failure re-arm writes — so a concurrent RecordFailure on the same state sharing a clock tick (routine at ~1 ms resolution) was mistaken for the caller's own claim and stomped back to the stale, already-due value, prematurely reopening the probe slot. Replaced with a monotonic per-state ProbeVersion bumped by every writer of NextProbeAtTicks (claim and re-arm alike); the release restores only when the version still matches the one its claim stamped, and bumps it again on restore so no other stale release can match. Covered by ProbeSlotRestore_DoesNotStompConcurrentRearmAtSameTick, made deterministic by a new internal ProbeReleaseInterleaveHook test seam (null in production, one null check on the refused path) because the claim-to-release window is nanoseconds wide and racing threads cannot hit it reliably — verified as a genuine red against the timestamp guard (Expected: ThrottledByPeer / Actual: ProbeAdmitted). Limiter suite 11 → 15.
2026-08-07 CLI-40 + CLI-41 + CLI-44 → Done (branch fix/cli-40-41-44), one change set; two fast-follow riders from the CLI-37/38 review landed alongside. CLI-40 (exact-secret credential scrub, ported to Rust/Java/.NET; Go/Python already did it): every credential-bearing helper (AuthenticateUser password, WriteSecured/WriteSecured2 string payloads) now scrubs the exact caller-supplied secret from any surfaced error text, on top of the by-construction guarantee — Rust MxAccessError gained a secrets: Vec<String> field whose Display scrubs exact-then-pattern (and a hand-written redacting Debug, since the derived Debug would have leaked the reply verbatim — caught by the existing Debug regression test); Java added MxGatewaySecrets.redactExact + a private invokeCommandRedacted(command, secrets…) that rebuilds the same exception type with the redacted message and does not chain the secret-bearing original as cause; .NET added an internal MxGatewaySecretRedaction (rebuilds the same concrete MxGateway*Exception type via a type switch) wired into the three credential helpers — and it carries the original's inner cause forward rather than the secret-bearing original, so ToString() (what loggers emit) is scrubbed too, not just Message (locked by a ToString() assertion). CLI-41 (uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/AddBufferedItem across all five): typed payload → present return_value with the int32 variant → else a typed malformed-reply error (MalformedReplyError Go/Python, MxGatewayMalformedReplyException Java/.NET, existing Error::MalformedReply Rust) — never a proto3 default 0, never an NRE (fixes the Go/Java silent-0, .NET NRE, and Rust's own internal inconsistency by giving authenticate_user_id/archestra_user_id the same return_value fallback add_buffered_item_handle already had). CLI-44 (Go): the event goroutine's Recv-error path now uses a new non-blocking sendTerminalEventResult on the reserved slot instead of sendEventResult, so a genuine terminal gRPC error is reported as itself even when the 16 data slots are full, rather than being mislabeled ErrSlowConsumer; test TestEventsFullBufferTerminalErrorKeepsRootCause was confirmed red-first (a 250 ms settle after streamDone is required to make the buffer genuinely full at error time). Three shared fixtures added under clients/proto/fixtures/behavior/command-replies/ (authenticate-user.{echoed-credential,missing-payload,return-value-only}.reply.json; the echoed-credential reply uses an OK envelope + negative HRESULT + the credential in protocolStatus.message / statuses[0].diagnosticText / diagnosticMessage so all five clients route it to their MXAccess error uniformly) + manifest + docs/ClientBehaviorFixtures.md + docs/ClientLibrariesDesign.md. Rider (a): .NET ToDiagnosticSummary and Python _mxaccess_message now surface the raw success member (Rust already did), for diagnostics-only parity. Rider (b): the status-conversion fixture gained an independent wantSuccess boolean per case; the Go TestStatusConversionFixtures and .NET FixtureStatuses_ProjectSuccessAndPreserveRawFields now assert against it instead of recomputing category == OK (the formula under test). Deviation: the <redacted> marker is not universal — Go/Rust/Java use <redacted>, Python and the .NET CLI use [redacted]; each suite asserts its own client's marker plus the exact-secret absence (marker unification was out of scope). Verification (all five, on macOS): Go gofmt -l clean + go build ./... + go test ./... ok; Python python -m pytest 162 passed / 1 skipped; .NET dotnet build …Client.slnx 0 warnings + client tests 120 passed / 1 skipped; Rust cargo fmt + cargo check --workspace + cargo test --workspace (all targets pass) + cargo clippy --all-targets -- -D warnings clean; Java JAVA_HOME=/opt/homebrew/opt/openjdk@17 gradle test BUILD SUCCESSFUL with no generated-tree churn to revert (no .proto changed). Code review of the branch found three defects, all fixed before merge: (1, all five, critical) the message-only scrub left the server-echoed credential exposed on the redacted error's structured reply accessor (.NET MxAccessException.Reply/Statuses, Java reply()/protocolStatus(), Go MxAccessError.Reply via errors.As, Rust reply()/into_reply(), Python raw_reply) — the redacted error now carries a scrubbed clone of the reply (protocol_status.message, diagnostic_message, statuses[].diagnostic_text all redacted), with per-language tests asserting the reply accessor is clean; docs/ClientLibrariesDesign.md updated to make the "never reaches exception text" claim true for structured fields too. (2, Rust, critical) ensure_command_success routed PROTOCOL_STATUS_CODE_MXACCESS_FAILURE to Error::Command (unlike the other four clients), where attach_secrets did not patch it and its derived Debug/Display leaked the secret — now MxaccessFailure routes to Error::MxAccess (fixing a real cross-client inconsistency; an existing test flipped from Error::Command to Error::MxAccess). (3, Go, important) the CLI-44 sendTerminalEventResult was unconditionally non-blocking, so on the never-drop SubscribeEvents/SubscribeEventsAfter path (cancelWhenResultBufferFull=false) a genuine terminal error under a full buffer hit the default: and was silently dropped — now the terminal send is reserved-slot-non-blocking only for the cancel-on-overflow path and blocking for the never-drop path. A new shared fixture authenticate-user.echoed-credential-mxaccess-failure.reply.json (the echo under MXACCESS_FAILURE) is wired into all five suites. Minors also landed: whitespace-only-secret guard on the .NET/Java redact helpers; Java invokeCommandRedacted now preserves the exception subtype (mirroring .NET's type switch) instead of collapsing to the base type; dedicated redaction-helper unit tests (multiple occurrences, substring-overlap, empty/blank secrets) in Go/Java/.NET. Re-verified all five green (Go go test ./... ok + gofmt clean; Python 163 passed/1 skipped; .NET build 0 warnings + 128 passed/1 skipped; Rust fmt/check/test --workspace/clippy -D warnings all clean; Java gradle BUILD SUCCESSFUL, no generated churn).
2026-08-07 GWC-28, GWC-29, GWC-30, TST-28 → Done (branch fix/gwc-28-29-30-polish). GWC-28: WorkerClient.WriteLoopAsync now stamps envelope.Sequence = unchecked(++_nextSequence) immediately before _writer.WriteAsync, and CreateEnvelope leaves it unset; _nextSequence dropped from long + Interlocked to a plain ulong touched only by the write loop (the channel's single consumer, SingleReader = true), so wire order and sequence order are the same thing by construction. Mirrors the worker's WRK-04 stamping, which the gateway half had never received; gateway.md's envelope-sequence rule now states that both sides stamp at write inside their single write path and that inbound enforcement (still open, old GWC-10) would rely on it. New WorkerClientTests.ConcurrentInvokesEmitStrictlyIncreasingSequencesOnTheWire (32 parallel invokes, sequences asserted strictly increasing in wire order) failed 3/3 pre-fix. GWC-29: added MxAccessGrpcMapper.MapCommand(MxCommand); Invoke no longer deep-clones the whole MxCommandRequest just to overwrite and discard its command. The one clone inside MapCommand stays and is documented as required — commandToInvoke may be the gRPC-owned request.Command and is read again after dispatch by TrackCommandReply, so it is what keeps CreateCommandEnvelope's no-aliasing invariant true. New MxAccessGrpcMapperTests.MapCommandFromCommandClonesPayload (isolation + both overloads equal under a FakeTimeProvider). GWC-30: WorkerFrameReader reuses a per-instance _lengthPrefix scratch buffer instead of allocating 4 bytes per frame, with a class remark that ReadAsync is not reentrant (single read loop per WorkerClient; handshake reads complete before the loop starts); guarded by new WorkerFrameProtocolTests.ReadAsync_WithMultipleFramesOnOneReader_ParsesEveryFrame (5 frames, varying payload lengths, one reader). TST-28: new [Theory] WorkerClientTests.StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes over the default and a 2 MiB override via FakeWorkerHarness.CreateConnectedPairAsync(maxMessageBytes:) — test-only, and the mutation check (hard-code MaxFrameBytes = 0) failed both cases before being reverted. Verification: NonWindows.slnx 0 warnings/0 errors; WorkerClientTests 25 passed, WorkerFrameProtocolTests 11 passed, MxAccessGrpcMapperTests 6 passed, MxAccessGatewayService* 29 passed, full gateway suite 844 passed / 0 failed (TMPDIR=/tmp on macOS).