ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6): 7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and burst-coalescing failures (#126/#127/#139) last failed before50322bac/7da52b65(Aug 2026). - 2026-10-15..20 was the NU1903 package-audit restore failure, fixed infb20997. Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
+21
-1
@@ -639,7 +639,8 @@ runtime because the x86 Worker cannot build on Linux:
|
||||
client (IPC-24 deleted the old unconditional churn-revert step, which masked exactly that
|
||||
class). The dev Mac has a Homebrew JDK 17, so `gradle generateProto` can be run there to refresh
|
||||
the Java aggregates when a `.proto` changes.
|
||||
- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and
|
||||
- **`windows-x86`** (Linux runner, per push/PR; skipped on the schedule, where
|
||||
`nightly-windev` runs the same tiers) — builds the **x86 / net48 Worker and
|
||||
Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a
|
||||
Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where
|
||||
`scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci`
|
||||
@@ -658,6 +659,25 @@ runtime because the x86 Worker cannot build on Linux:
|
||||
Gated `if: github.event_name == 'schedule'`, so it never gates a push. On failure it opens a
|
||||
Gitea issue via the Actions token, since nobody watches the Actions page.
|
||||
|
||||
The windev host lock is an exclusive file handle on `C:\build\mxaccessgw-ci.ci-lock`, held by
|
||||
the SSH bootstrap for the whole run. It replaced a `New-Item -ItemType Directory` lock that two
|
||||
runs starting in the same second could both take (PowerShell checks for the directory, then
|
||||
creates it), which is how the 2026-09-22 nightly collided with the scheduled `windows-x86` run on
|
||||
`.git/index.lock` (mxaccessgw#150). The OS drops the handle when its holder dies, so a killed run
|
||||
leaves no stale lock, and a holder may delete a leftover `index.lock` because no other CI git can
|
||||
be running. Each run also writes a `host (start)` line — and a `host (failure)` line when a tier
|
||||
fails — to `C:\build\mxaccessgw-ci\ci.log` with free commit, physical memory, disk and the
|
||||
number of `testhost` processes, so a host-resource failure can be matched to the host's state.
|
||||
|
||||
When a nightly goes red, sort the cause before touching code: a test that fails with an
|
||||
assertion or test exception is a test or product fault; `OutOfMemoryException` from
|
||||
MSBuild/csc/protoc, `csc.exe exited with code -1073741502` (0xC0000142, a process that could not
|
||||
initialize), a remote exit code with no `ci.log` lines at all, or an SSH `kex_exchange_identification`
|
||||
reset is the host, as is `No route to host`. The 2026-07-24 to 2026-08-03 red streak was the
|
||||
second kind: windev's System log carries a `Resource-Exhaustion-Detector` event 2004 (low virtual
|
||||
memory) every few minutes from 2026-07-23 until the box hard-reset on 2026-08-03, with no single
|
||||
process holding most of the commit.
|
||||
|
||||
### Runner capacity is shared and finite
|
||||
|
||||
CI runs on two co-located runner containers on docker host `10.100.0.35` — `gitea-runner`
|
||||
|
||||
Reference in New Issue
Block a user