ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m34s
ci / java (push) Successful in 2m37s
ci / portable (push) Failing after 5m0s

Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):

- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
  New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
  succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
  2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
  second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
  Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
  handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
  harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
  removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
  Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
  Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
  until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
  csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
  window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
  memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
  in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
  classes migrating the one per-process secrets store. Tests now build the gateway through
  TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
  the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
  7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
  burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.

Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:35:54 -05:00
parent 09b3ecddf2
commit de86f8b80e
18 changed files with 234 additions and 65 deletions
+29 -11
View File
@@ -84,19 +84,33 @@ echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_S
#
# The bootstrap fetch/checkout mutates the shared clone's git index, so it MUST hold the
# worktree lock — otherwise two concurrent runs collide on .git/index.lock before either
# reaches windev-worker-ci.ps1's lock (the exact race the lock exists to prevent). We take the
# same mkdir lock here (atomic; retry to a 20-min deadline) and export MXGW_CI_LOCK_HELD so the
# handed-off script re-uses this lock instead of re-acquiring/releasing it. A standalone/manual
# run of windev-worker-ci.ps1 (degraded mode) has no such env and self-locks as before.
# reaches windev-worker-ci.ps1 (the exact race the lock exists to prevent). We export
# MXGW_CI_LOCK_HELD so the handed-off script re-uses this lock instead of re-acquiring it; a
# standalone/manual run of windev-worker-ci.ps1 (degraded mode) has no such env and takes the
# same lock itself.
#
# The lock is an exclusive (FileShare.None) handle on '<clone>.ci-lock', held by this
# bootstrap process for the whole run (the handle is not inherited by the child powershell).
# It replaced a `New-Item -ItemType Directory` lock that was NOT mutually exclusive:
# PowerShell's directory provider checks for existence and then calls CreateDirectory (which
# succeeds on an existing directory), so two bootstraps starting together both "acquired" it.
# A scheduled run starts windows-x86 and nightly-windev in the same second, and on 2026-09-22
# (run 827) both got in, ran git concurrently and the nightly died on
# ".git/index.lock: File exists" (mxaccessgw#148/#150). A file handle is also released by the
# OS if the holder is killed (ssh drop, host reboot), so a dead run can never leave a stale
# lock behind; the old directory had to be removed by hand.
#
# Holding the lock also proves no other CI git command can be running in the clone, so a
# leftover .git/index.lock can only be debris from a killed git and is removed.
read -r -d '' BOOTSTRAP <<PS || true
\$ErrorActionPreference = 'Continue'
\$lockDir = '$REMOTE_CLONE.lock'
\$lockFile = '$REMOTE_CLONE.ci-lock'
\$deadline = (Get-Date).AddMinutes(20)
\$haveLock = \$false
while (-not \$haveLock) {
try { New-Item -Path \$lockDir -ItemType Directory -ErrorAction Stop | Out-Null; \$haveLock = \$true }
\$lock = \$null
while (\$null -eq \$lock) {
try { \$lock = [System.IO.File]::Open(\$lockFile, 'OpenOrCreate', 'ReadWrite', 'None') }
catch {
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockDir"; exit 75 }
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockFile : \$(\$_.Exception.Message)"; exit 75 }
Start-Sleep -Seconds 10
}
}
@@ -104,16 +118,20 @@ while (-not \$haveLock) {
\$rc = 1
try {
Set-Location '$REMOTE_CLONE'
if (Test-Path '.git\\index.lock') {
Write-Warning 'removing stale .git\\index.lock left by a killed git (the CI lock is held, so no CI git is running)'
Remove-Item -Force '.git\\index.lock'
}
git fetch --prune origin
git checkout --force --detach $CI_SHA
if (\$LASTEXITCODE -ne 0) { Write-Error "bootstrap checkout failed"; \$rc = \$LASTEXITCODE }
if (\$LASTEXITCODE -ne 0) { \$rc = \$LASTEXITCODE; Write-Error "bootstrap checkout failed (exit \$rc)" }
else {
powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
\$rc = \$LASTEXITCODE
}
}
finally {
Remove-Item -Path \$lockDir -Recurse -Force -ErrorAction SilentlyContinue
\$lock.Dispose()
}
exit \$rc
PS