Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):
- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
classes migrating the one per-process secrets store. Tests now build the gateway through
TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.
Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- The metrics scheme skips its dashboard-cookie fallback while
Dashboard:DisableLogin is on, so the dev auto-login handler cannot
satisfy RequireAuthentication=true.
- Invoke logs client_correlation_id <-> worker envelope id once at Debug.
- MetricsEndpointAuthenticationTests drives the real pipeline: anonymous
loopback 401 (no /login redirect), Viewer cookie 200, admin key 200,
non-admin key 403, DisableLogin 401, /health/active 200.
- Docs: DisableLogin behaviour; the /metrics limiter partition is per IP
vs gRPC per connection, so only the per-key aggregate is shared.