Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6): 7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and burst-coalescing failures (#126/#127/#139) last failed before50322bac/7da52b65(Aug 2026). - 2026-10-15..20 was the NU1903 package-audit restore failure, fixed infb20997. Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Windows/x86 CI tier (TST-25)
The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows
act_runner is broken (host-mode writes each step's script to a path it can't resolve; a
runs-on gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to
windev (10.100.0.48) and runs the Worker build/tests there, propagating the remote exit
code back so a regression turns the job red.
Pieces
| File | Runs on | Role |
|---|---|---|
run-windev-ci.sh <build|test|live> |
Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. |
windev-worker-ci.ps1 -Sha -Mode |
windev | Locks the CI clone, checks out the SHA, and runs the requested cumulative Windows tiers. |
windev.known_hosts |
committed | Pinned windev host keys so StrictHostKeyChecking=yes has no TOFU prompt. Host keys are public; committing is intentional. |
Modes are cumulative: build = x86 Worker build; test = build + Worker.Tests;
live = test + full-slnx build + the net10 gateway suite + live-MXAccess smoke
(MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1). The live order is x86 Worker build,
Worker.Tests, full solution build, gateway tests, then live smoke. One host lock covers
checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive
(FileShare.None) handle on C:\build\mxaccessgw-ci.ci-lock held by the SSH bootstrap; the OS
releases it if the run is killed, so it never goes stale. (It replaced a New-Item -ItemType Directory lock that two runs starting in the same second could both take — mxaccessgw#150.)
Runs of SHAs older than that change still use the old directory lock and do not exclude new-style
runs; that window closes as branches rebase. each native command is checked
immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.
CI jobs (.gitea/workflows/ci.yml): windows-x86 runs test per push/PR (not on the schedule,
where it would duplicate the nightly's first two tiers and start a second windev session in the
same second); nightly-windev runs live on the 0 6 * * * schedule and opens a Gitea issue on
failure.
One-time bring-up (operator — required before this lands on main)
Until these are done, windows-x86 is red on every push (no key → SSH fails). Do them
first, then merge.
- CI clone —
C:\build\mxaccessgw-ci, a fresh clone of the Gitea origin (done during TST-25 verification; recreate withgit clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ciif missing). Never the dirty Desktop checkout. - Dedicated CI key — generate a NEW ed25519 keypair for CI (do not reuse the operator's
personal windev key):
ssh-keygen -t ed25519 -f ci_windev -N ''. Appendci_windev.pubto the windev CI account'sauthorized_keys(optionally with a forcedcommand=limiting it to this bootstrap). - Gitea repo secret
WINDEV_SSH_KEY— store the private key base64-encoded on a single line:base64 -w0 < ci_windev(macOS:base64 < ci_windev | tr -d '\n'). This is required, not cosmetic — Gitea's secret masker is line-oriented, so a raw multiline PEM is not redacted in the stepenv:echo and leaks in cleartext; the single-line base64 masks to***.run-windev-ci.shauto-decodes it. Host keys are public and come from the committedwindev.known_hostspin, so noWINDEV_SSH_KNOWN_HOSTSsecret is wired intoci.yml.run-windev-ci.shconnects as userciby default; set theWINDEV_SSH_USERvariable (not a secret — it is public) to the actual account, and ensure that account exists on windev. - Network precheck — confirm the Gitea runner container (on the
traefikdocker network, host10.100.0.35) can reach10.100.0.48:22: a one-offnc -z -w5 10.100.0.48 22step. The network was created for gitea name resolution, not LAN egress, so verify. - Actions token — the
nightly-windevon-failure step creates a Gitea issue with the built-in token; confirm Actions tokens have issue-write on this repo.
Verifying it end-to-end (design's acceptance checks)
- Push a branch →
portable,java,windows-x86all green; windev's clone sits at the pushed SHA. - Deliberate red: add a failing
[Fact]toWorker.Tests, push, confirmwindows-x86goes red; revert. (Proves exit-code propagation — the whole point.) - Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
- Nightly: trigger the schedule path, confirm
liveruns and a forced failure opens an issue. Done 2026-08-10 (Check 6). Verified two ways: (a) production — every red nightly since 2026-07-17 has auto-filed an issue (#126–#139) authored by thegitea-actionsbot, e.g. run 672 → issue #139, with the built-in token masked to***in the job log; (b) a forced-failure probe on the throwaway branchtest/tst25-check6-nightly-issue(run 677 → issue #140, since closed and the branch deleted), which reproduced the job shape withexit 1in place of the live step and confirmed theif: failure()step fires, the token carries issue-write, and the payload is well-formed. The probe also caught the one defect:${{ github.server_url }}is the runner-internalhttp://gitea:3000, so the run link in the issue body was unreachable from a browser — the body now uses thePUBLIC_SERVER_URLjob env instead (the API call still targetsgithub.server_url, which is what the job container can resolve). - Confirm no key material appears in job logs.
Degraded mode
If the tier is down for infra reasons, fall back to the manual windev worktree procedure
(isolated C:\build worktree, dotnet build/test -p:Platform=x86 by hand) per merge for
worker-touching changes until windows-x86 is green again. See docs/GatewayTesting.md.