mxaccessgw/archreview/2026-07-12/remediation/00-tracking.md

MxAccessGateway — Remediation Tracking (2026-07-12 review)

Master progress tracker for the 2026-07-12 follow-up architecture review. Generated 2026-07-13.

Source review: ../00-overall.md · Per-domain remediation designs are linked below and hold the full Finding / Impact / Design / Implementation / Verification for every entry here. The first-cycle tracker (../../remediation/00-tracking.md) remains the record for the original 153 findings; this document tracks only the 47 new IDs (GWC-24+, WRK-21+, IPC-23+, SEC-31+, CLI-35+, TST-25+) plus the old-tracker actions listed at the end.

How to use this document

• Each finding has a stable ID that never changes. Cite it in commits, branches, and PRs (e.g. fix(GWC-25): empty-ring ReplayGap sentinel).
• The Status column is the single source of truth for progress. Update it in the same change that lands the fix.
• Do the work in roadmap-tier order (P0 → P1 → P2), respecting the Dep column and the cross-cutting clusters below — several fixes are one change set across two domains and must land together.
• When a fix lands: flip Status to Done and, per the repo rule, update the affected docs in the same commit.

Status legend: Not started · In progress · In review · Done · Won't fix (record why in the domain doc) · N/A (informational / decision recorded, no action).

Severity roll-up

Domain	Doc	High	Medium	Low	Info	Total
Gateway core	10-gateway-core.md	—	2	4	1	7
Worker	20-worker.md	—	1	7	—	8
Contracts & IPC	30-contracts-ipc.md	—	3	5	2	10
Security & dashboard	40-security-dashboard.md	—	1	4	1	6
Clients	50-clients.md	—	5	6	—	11
Testing, docs & gaps	60-testing-docs-gaps.md	1	2	3	—	6
Total		1	14	29	4	48

Roadmap-tier roll-up

Tier	Meaning	Count	Findings
P0	Correctness & safety — all small-to-medium	10	GWC-25, WRK-21, IPC-23, IPC-24, IPC-25, IPC-30, SEC-31, SEC-32, CLI-35, CLI-36
P1	Process & hardening	12	GWC-24, WRK-26, SEC-33, SEC-36, CLI-37, CLI-38, CLI-39, CLI-42, CLI-45, TST-25, TST-26, TST-27
P2	Completeness & polish	10	GWC-26, GWC-27, GWC-28, WRK-25, IPC-26, IPC-27, SEC-34, TST-28, TST-29, TST-30
—	Not individually in the roadmap (Lows/Infos)	16	GWC-29/30, WRK-22/23/24/27/28, IPC-28/29/31/32, SEC-35, CLI-40/41/43/44

P0 — do first (10)

Sequenced by cluster; a cluster is one change set.

ID	Sev	Eff	Dep	Status	Title
GWC-25	Medium	S	CLI-35/36 (coord)	Not started	Empty-ring ReplayGap sentinel carries oldest_available_sequence = 0, dead-streaming a compliant client
CLI-35	Medium	S	GWC-25 (coord)	Not started	Python CLI stream-events crashes on a ReplayGap
CLI-36	Medium	S	GWC-25 (coord)	Not started	Go CLI stream-events silently destroys the ReplayGap signal
WRK-21	Medium	M	owns IPC-23 fix; WRK-28 same batch	Done	DrainEvents bound is count-based only; oversized reply kills the session and loses the drained events
IPC-23	Medium	S	WRK-21	In progress — mechanics landed with WRK-21; proto-comment/doc wave pending	DrainEvents contract requirements (reply fits negotiated max, no event loss, drain-until-empty) + proto-comment/doc wave
IPC-30	Low	M	WRK-21 (same batch)	Done	Oversized event frame stays session-fatal by design, but the death becomes structured (fault frame + logged identity)
SEC-31	Medium	M	—	Not started	Failure limiter partitions on attacker-controlled key id and blocks before verification (lockout DoS)
SEC-32	Low	S	SEC-31	Not started	Failure-limiter LRU flushable by junk-token spray; token prefix never validated
IPC-24	Medium	S	—	Not started	CI's unconditional Java churn-revert masks real drift
IPC-25	Medium	M	—	Not started	Regenerate stale Go/Python worker bindings + add binding-freshness guard (Check 4) to check-codegen.ps1

Finding registers by domain

Full design + implementation for each row lives in the linked domain doc under its ID heading.

Gateway core — 10-gateway-core.md

ID	Sev	Tier	Eff	Dep	Status	Title
GWC-24	Medium	P1	M	GWC-21 (coord, old tracker)	Not started	Unbounded event staging channel: sustained slow drain grows memory silently and invisibly
GWC-25	Medium	P0	S	CLI-35/36 (coord)	Not started	Empty-ring ReplayGap sentinel carries oldest_available_sequence = 0
GWC-26	Low	P2	M	GWC-27	Not started	Alarm monitor attaches its subscriber after SubscribeAlarms; window transitions bypass the feed
GWC-27	Low	P2	S	GWC-26	Not started	AttachInternalEventSubscriber bypasses the readiness gate; premature attach poisons the distributor
GWC-28	Low	P2	S	GWC-10 (coord, old tracker)	Not started	Gateway→worker envelope sequence stamped at creation, not at write
GWC-29	Low	—	S	—	Not started	Invoke deep-clones the entire request only to discard the cloned command
GWC-30	Info	—	S	—	Not started	Frame reader allocates a fresh 4-byte length-prefix array per frame

Worker — 20-worker.md

ID	Sev	Tier	Eff	Dep	Status	Title
WRK-21	Medium	P0	M	IPC-23 (fix owned here); WRK-28	Done	DrainEvents bound count-based only; oversized reply kills session and loses drained events
WRK-22	Low	—	S	IPC-26 (fix owned here)	Not started	Cancelled WriteAsync leaves its frame queued; it is still written later
WRK-23	Low	—	S	WRK-21	Done	Rejected frames consume sequence numbers, producing wire gaps
WRK-24	Low	—	S	—	Not started	AdoptNegotiatedMaxMessageBytes has no lower-bound sanity check
WRK-25	Low	P2	S	WRK-22 (shared seam)	Not started	WRK-12 flush coalescing never engages on the event hot path
WRK-26	Low	P1	S	WRK-23 (soft); discharges IPC-29	Not started	Write-priority and overflow doc drift from the WRK-07 change
WRK-27	Low	—	S	—	Not started	Alarm poll bypasses the watchdog's in-flight suppression (15 s vs 75 s)
WRK-28	Low	—	S	WRK-21 (same batch)	Done	10,000 drain cap is a duplicated magic constant

Contracts & IPC — 30-contracts-ipc.md

ID	Sev	Tier	Eff	Dep	Status	Title
IPC-23	Medium	P0	S	WRK-21 (mechanics)	In progress — mechanics landed with WRK-21; proto-comment/doc wave pending	DrainEvents byte-blindness — contract requirements + proto-comment/doc wave
IPC-24	Medium	P0	S	—	Not started	CI's unconditional Java churn-revert masks real drift
IPC-25	Medium	P0	M	—	Not started	Stale Go/Python worker bindings: regenerate (pinned toolchains) + check-codegen Check 4
IPC-26	Low	P2	S	WRK-22 (mechanics)	Not started	Cancelled write leaves ghost frame — cancelled means never written
IPC-27	Low	P2	S	—	Not started	Descriptor-freshness test blind to enums/services/galaxy descriptor
IPC-28	Low	—	S	—	Not started	docs/Grpc.md missing CommandTooLarge → ResourceExhausted mapping
IPC-29	Low	—	S	WRK-26 (discharged by)	Not started	WorkerFrameProtocol.md missing write-scheduling/sequencing section
IPC-30	Low	P0	M	WRK-21 (same batch)	Done	Oversized event frame: keep session-fatal, make the death structured
IPC-31	Info	—	—	—	N/A	Gateway creation-time sequence stamping accepted; diagnostic-only, decision recorded
IPC-32	Info	—	S	IPC-25 (folded in)	Not started	check-codegen banner relabel 1/4…4/4

Security & dashboard — 40-security-dashboard.md

ID	Sev	Tier	Eff	Dep	Status	Title
SEC-31	Medium	P0	M	—	Not started	Failure limiter: composite (peer, key-id) partitions + cross-peer aggregate with probe admission
SEC-32	Low	P0	S	SEC-31	Not started	Limiter LRU flushable by junk-token spray; validate token shape, cap per-peer partitions
SEC-33	Low	P1	M	old SEC-23 (co-locate)	Not started	Host-meaningful path rooting; drop Windows literals from appsettings; validate Galaxy SnapshotCachePath
SEC-34	Low	P2	S	—	Not started	Verification cache: expiry outlives TTL; Invalidate races in-flight repopulation
SEC-35	Info	—	S	—	N/A	Production hard-stops key on exact Production environment name (doc-only)
SEC-36	Low	P1	M	cross-repo scadaproj/infra/glauth	Not started	Committed dev LDAP service-account password: rotate, remove, move dev channel to user-secrets

Clients — 50-clients.md

ID	Sev	Tier	Eff	Dep	Status	Title
CLI-35	Medium	P0	S	GWC-25 (coord)	Not started	Python CLI stream-events crashes on a ReplayGap
CLI-36	Medium	P0	S	GWC-25 (coord)	Not started	Go CLI stream-events silently destroys the ReplayGap signal
CLI-37	Medium	P1	M	CLI-38 (co-land)	Not started	Status-array validation must branch on category per the proto contract (4-vs-1 divergence)
CLI-38	Medium	P1	S	—	Not started	Align .NET/Go/Java on hresult < 0 — lands old CLI-08, cures design-doc drift
CLI-39	Medium	P1	S	CLI-35..38, CLI-45 (land last)	Not started	Bump client versions off published 0.1.2 (converge on 0.2.0); registry-collision guard in pack-clients.ps1
CLI-40	Low	—	M	—	Not started	Port the exact-secret credential scrub to Rust/Java/.NET
CLI-41	Low	—	M	—	Not started	Uniform malformed-reply contract for AuthenticateUser/ArchestrAUserToId/AddBufferedItem
CLI-42	Low	P1	S	—	Not started	Document the vendored Rust proto layout (CLI-02's missing doc half)
CLI-43	Low	—	S	—	Not started	Java style guide still prescribes "Java 21 preferred"
CLI-44	Low	—	S	—	Not started	Go event goroutine can mislabel a genuine terminal error as ErrSlowConsumer
CLI-45	Low	P1	M	—	Not started	Standardize CLI credential env-var names; fail fast on missing/empty passwords

Testing, docs & gaps — 60-testing-docs-gaps.md

ID	Sev	Tier	Eff	Dep	Status	Title
TST-25	High	P1	M	unlocks old TST-05, TST-24	Done	Windows/x86 test tier has zero automation — SSH-driven windev CI job
TST-26	Medium	P1	S	TST-25 (same commit)	Done	Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen
TST-27	Medium	P1	S	—	Not started	ShowTagValues config row still says "Reserved" after SEC-25 made the flag live
TST-28	Low	P2	S	relates IPC-02 (old)	Not started	Gateway-side max_frame_bytes handshake untested in the CI-run suite
TST-29	Low	P2	S	—	Not started	Retire oldtasks.md (fold Phase-5 governance into DesignDecisions.md); delete root artifacts
TST-30	Low	P2	M	—	Not started	Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete API)

Cross-cutting clusters

Sequence these together rather than piecemeal — several are one change set spanning two domains:

• Drain / oversized-frame cluster (P0): WRK-21 + IPC-23 + IPC-30, with WRK-28 and WRK-23 in the same batch. One worker change set in WorkerPipeSession.cs/WorkerFrameWriter.cs; WRK-21 owns the byte-budgeted drain and must satisfy IPC-23's requirements R1–R3; IPC-30's structured-fault seam lands in the same drain loop. IPC-23's proto-comment edits trigger the full regen fan-out (Generated/, Rust vendored, Go/Java, descriptor set) — land with WRK-21 so the wave happens once. One windev x86 verification run for the cluster.
• ReplayGap end-to-end (P0): GWC-25 (server sentinel arithmetic) + CLI-35 (Python CLI) + CLI-36 (Go CLI). Independently landable; the e2e resume walk validates only when all three are in.
• Auth limiter (P0): SEC-31 + SEC-32 — same component, one change set, one test suite.
• Codegen freshness (P0): IPC-24 + IPC-25 (+ IPC-32 folded in). Both edit .gitea/workflows/ci.yml — coordinate the branch with TST-25, which touches the same file.
• Windows-tier automation (P1): TST-25 + TST-26 (same commit). Unlocks old TST-05/TST-24 and provides CI evidence for every windev-verified cluster above; until it lands, record windev runs in this tracker's change log.
• Client conformance + release train (P1): CLI-37 + CLI-38 co-land (one conformance commit; closes old CLI-08), then CLI-45, with CLI-40/41 fixtures as follow-ups; CLI-39 lands last so published 0.2.0 carries the conformant behavior. Shared fixtures under clients/proto/fixtures/behavior/; update CrossLanguageSmokeMatrix.md/ClientLibrariesDesign.md same-commit. Do not republish regenerated bindings (IPC-25) before CLI-39 resolves.
• Doc-drift batch (P1): TST-27 + WRK-26 (discharges IPC-29) + CLI-42 + IPC-28 + SEC-35's doc note — one sweep commit is fine.
• Backpressure follow-on (P1): GWC-24, coordinating with still-open old GWC-21 (EventChannelFullModeTimeout configurability).

Old-tracker actions (../../remediation/00-tracking.md)

• Close CLI-24 and CLI-34 as Done (incidentally fixed; evidence in ../50-clients.md).
• When CLI-38 lands, close old CLI-08 with a pointer here.
• When WRK-26 lands, its doc section also discharges the WorkerFrameProtocol gap; when TST-25 lands, revisit old TST-05 (scheduled live smoke) and TST-24 (client wire tests), which it unlocks.

Change log

Date	Change
2026-07-13	Initial tracking doc generated from the six domain remediation designs. All 47 findings Not started (IPC-31, SEC-35 N/A).
2026-07-13	TST-25/TST-26 → In progress (branch fix/tst-25-windev-ci). Added scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}, windows-x86 (per-push) + nightly-windev (scheduled) jobs in ci.yml, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: build→0, bogus-SHA→nonzero (lock released), test→356 passed/0 failed in ~50s (per-push stays test, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed.
2026-07-13	Operator bring-up complete: dedicated CI ed25519 key installed in windev administrators_authorized_keys (authorized into dohertj2, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets WINDEV_SSH_KEY/WINDEV_SSH_KNOWN_HOSTS + variable WINDEV_SSH_USER=dohertj2 stored, runner→10.100.0.48:22 egress verified on the traefik net, issue-write confirmed. TST-25/TST-26 → Done: credentialed windows-x86 ran GREEN on d769244 (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in C:\build\mxaccessgw-ci under lock, ran the x86 Worker build + Worker.Tests, exit 0; nightly-windev correctly skipped on the push event. Branch merged to main. Follow-ups (old tracker): revisit TST-05 (scheduled live smoke — now covered by nightly-windev) and TST-24 (client wire tests) which this unlocks.
2026-07-13	Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught two real CI defects, both fixed on fix/tst-25-ci-key-log-leak: (1) CI SSH key leaked in cleartext in the windows-x86 step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key base64-encoded so the masker redacts it to *** (confirmed on run #38), taught run-windev-ci.sh to decode, dropped the redundant public known-hosts secret from the job env; (2) bootstrap lock race — run-windev-ci.sh's pre-hand-off git fetch/checkout ran outside the worktree lock, so concurrent runs collided on .git/index.lock; the bootstrap now holds the lock (ps1 re-uses it via MXGW_CI_LOCK_HELD), retest confirmed clean serialization. Also deflaked SessionManagerTests fail-fast timing assertions (absolute <100ms wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target df7e20d verified GREEN via the local windev path (Worker build + 356 tests); merged to main 19cbf7b. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual if: failure() gating is standard Actions).
2026-07-13	New finding TST-30 (Low/P2) added — surfaced during TST-25 acceptance: CI runs on a single shared gitea-runner (maxParallel=1, co-located 10.100.0.35) interleaved with dohertj2/lmxopcua, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the run-windev-ci.sh bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10).