ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6): 7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and burst-coalescing failures (#126/#127/#139) last failed before50322bac/7da52b65(Aug 2026). - 2026-10-15..20 was the NU1903 package-audit restore failure, fixed infb20997. Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
+10
-3
@@ -18,11 +18,18 @@ Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests
|
||||
`live` = test + full-`slnx` build + the net10 gateway suite + live-MXAccess smoke
|
||||
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The `live` order is x86 Worker build,
|
||||
`Worker.Tests`, full solution build, gateway tests, then live smoke. One host lock covers
|
||||
checkout and every tier, so concurrent runs cannot overlap; each native command is checked
|
||||
checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive
|
||||
(`FileShare.None`) handle on `C:\build\mxaccessgw-ci.ci-lock` held by the SSH bootstrap; the OS
|
||||
releases it if the run is killed, so it never goes stale. (It replaced a `New-Item -ItemType
|
||||
Directory` lock that two runs starting in the same second could both take — mxaccessgw#150.)
|
||||
Runs of SHAs older than that change still use the old directory lock and do not exclude new-style
|
||||
runs; that window closes as branches rebase. each native command is checked
|
||||
immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.
|
||||
|
||||
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev`
|
||||
runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure.
|
||||
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR (not on the schedule,
|
||||
where it would duplicate the nightly's first two tiers and start a second windev session in the
|
||||
same second); `nightly-windev` runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on
|
||||
failure.
|
||||
|
||||
## One-time bring-up (operator — required before this lands on `main`)
|
||||
|
||||
|
||||
+29
-11
@@ -84,19 +84,33 @@ echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_S
|
||||
#
|
||||
# The bootstrap fetch/checkout mutates the shared clone's git index, so it MUST hold the
|
||||
# worktree lock — otherwise two concurrent runs collide on .git/index.lock before either
|
||||
# reaches windev-worker-ci.ps1's lock (the exact race the lock exists to prevent). We take the
|
||||
# same mkdir lock here (atomic; retry to a 20-min deadline) and export MXGW_CI_LOCK_HELD so the
|
||||
# handed-off script re-uses this lock instead of re-acquiring/releasing it. A standalone/manual
|
||||
# run of windev-worker-ci.ps1 (degraded mode) has no such env and self-locks as before.
|
||||
# reaches windev-worker-ci.ps1 (the exact race the lock exists to prevent). We export
|
||||
# MXGW_CI_LOCK_HELD so the handed-off script re-uses this lock instead of re-acquiring it; a
|
||||
# standalone/manual run of windev-worker-ci.ps1 (degraded mode) has no such env and takes the
|
||||
# same lock itself.
|
||||
#
|
||||
# The lock is an exclusive (FileShare.None) handle on '<clone>.ci-lock', held by this
|
||||
# bootstrap process for the whole run (the handle is not inherited by the child powershell).
|
||||
# It replaced a `New-Item -ItemType Directory` lock that was NOT mutually exclusive:
|
||||
# PowerShell's directory provider checks for existence and then calls CreateDirectory (which
|
||||
# succeeds on an existing directory), so two bootstraps starting together both "acquired" it.
|
||||
# A scheduled run starts windows-x86 and nightly-windev in the same second, and on 2026-09-22
|
||||
# (run 827) both got in, ran git concurrently and the nightly died on
|
||||
# ".git/index.lock: File exists" (mxaccessgw#148/#150). A file handle is also released by the
|
||||
# OS if the holder is killed (ssh drop, host reboot), so a dead run can never leave a stale
|
||||
# lock behind; the old directory had to be removed by hand.
|
||||
#
|
||||
# Holding the lock also proves no other CI git command can be running in the clone, so a
|
||||
# leftover .git/index.lock can only be debris from a killed git and is removed.
|
||||
read -r -d '' BOOTSTRAP <<PS || true
|
||||
\$ErrorActionPreference = 'Continue'
|
||||
\$lockDir = '$REMOTE_CLONE.lock'
|
||||
\$lockFile = '$REMOTE_CLONE.ci-lock'
|
||||
\$deadline = (Get-Date).AddMinutes(20)
|
||||
\$haveLock = \$false
|
||||
while (-not \$haveLock) {
|
||||
try { New-Item -Path \$lockDir -ItemType Directory -ErrorAction Stop | Out-Null; \$haveLock = \$true }
|
||||
\$lock = \$null
|
||||
while (\$null -eq \$lock) {
|
||||
try { \$lock = [System.IO.File]::Open(\$lockFile, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch {
|
||||
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockDir"; exit 75 }
|
||||
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockFile : \$(\$_.Exception.Message)"; exit 75 }
|
||||
Start-Sleep -Seconds 10
|
||||
}
|
||||
}
|
||||
@@ -104,16 +118,20 @@ while (-not \$haveLock) {
|
||||
\$rc = 1
|
||||
try {
|
||||
Set-Location '$REMOTE_CLONE'
|
||||
if (Test-Path '.git\\index.lock') {
|
||||
Write-Warning 'removing stale .git\\index.lock left by a killed git (the CI lock is held, so no CI git is running)'
|
||||
Remove-Item -Force '.git\\index.lock'
|
||||
}
|
||||
git fetch --prune origin
|
||||
git checkout --force --detach $CI_SHA
|
||||
if (\$LASTEXITCODE -ne 0) { Write-Error "bootstrap checkout failed"; \$rc = \$LASTEXITCODE }
|
||||
if (\$LASTEXITCODE -ne 0) { \$rc = \$LASTEXITCODE; Write-Error "bootstrap checkout failed (exit \$rc)" }
|
||||
else {
|
||||
powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
|
||||
\$rc = \$LASTEXITCODE
|
||||
}
|
||||
}
|
||||
finally {
|
||||
Remove-Item -Path \$lockDir -Recurse -Force -ErrorAction SilentlyContinue
|
||||
\$lock.Dispose()
|
||||
}
|
||||
exit \$rc
|
||||
PS
|
||||
|
||||
@@ -44,7 +44,7 @@ param(
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$lockDir = "$RepoRoot.lock"
|
||||
$lockFile = "$RepoRoot.ci-lock"
|
||||
$logFile = Join-Path $RepoRoot 'ci.log'
|
||||
$lockTimeout = [TimeSpan]::FromMinutes(20)
|
||||
|
||||
@@ -67,37 +67,70 @@ function Invoke-Native {
|
||||
}
|
||||
}
|
||||
|
||||
# --- acquire the worktree lock (mkdir is atomic; retry until timeout) ---
|
||||
# Logs the host's memory/commit/disk headroom and any leftover test processes, so a
|
||||
# host-resource failure (OOM in csc/MSBuild/protoc, 0xC0000142 process-start failures, a
|
||||
# PowerShell that dies before logging) can be correlated with the host's state at the time
|
||||
# rather than reconstructed afterwards from the event log (mxaccessgw#150). Never fatal.
|
||||
function Write-HostResources {
|
||||
param([string]$When)
|
||||
try {
|
||||
$os = Get-CimInstance -ClassName Win32_OperatingSystem
|
||||
$drive = Get-PSDrive -Name ($RepoRoot.Substring(0, 1)) -ErrorAction Stop
|
||||
$testhosts = @(Get-Process -Name 'testhost*' -ErrorAction SilentlyContinue).Count
|
||||
Write-Log ('host ({0}): commit free {1:N1} of {2:N1} GB, physical free {3:N1} of {4:N1} GB, {5}: free {6:N1} GB, testhost processes {7}, up since {8:yyyy-MM-dd HH:mm}' -f
|
||||
$When,
|
||||
($os.FreeVirtualMemory / 1MB), ($os.TotalVirtualMemorySize / 1MB),
|
||||
($os.FreePhysicalMemory / 1MB), ($os.TotalVisibleMemorySize / 1MB),
|
||||
$drive.Name, ($drive.Free / 1GB), $testhosts, $os.LastBootUpTime)
|
||||
}
|
||||
catch {
|
||||
Write-Log "host ($When): resource probe failed: $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
|
||||
# --- acquire the worktree lock ---
|
||||
# The lock is an exclusive (FileShare.None) handle on "$RepoRoot.ci-lock". It replaced a
|
||||
# `New-Item -ItemType Directory` lock that was not mutually exclusive (PowerShell checks for
|
||||
# the directory and then calls CreateDirectory, which succeeds on an existing one), so two
|
||||
# runs starting together both got in and collided on .git/index.lock (mxaccessgw#148/#150).
|
||||
# The OS releases the handle if the holder is killed, so a dead run cannot leave it stale.
|
||||
# run-windev-ci.sh's bootstrap already holds this lock around its pre-hand-off fetch/checkout
|
||||
# (MXGW_CI_LOCK_HELD=1) so that stage can't race a concurrent run; re-use it rather than
|
||||
# re-acquire (and leave releasing to the bootstrap). A standalone/manual run (degraded mode)
|
||||
# has no such env and self-locks here as before.
|
||||
# (MXGW_CI_LOCK_HELD=1); re-use it rather than re-acquire (and leave releasing to the
|
||||
# bootstrap). A standalone/manual run (degraded mode) has no such env and self-locks here.
|
||||
$deadline = (Get-Date).Add($lockTimeout)
|
||||
$haveLock = $false
|
||||
$lock = $null
|
||||
if ($env:MXGW_CI_LOCK_HELD -eq '1') {
|
||||
Write-Log "Worktree lock already held by bootstrap (mode=$Mode sha=$Sha)"
|
||||
}
|
||||
else {
|
||||
Write-Log "Acquiring worktree lock $lockDir (mode=$Mode sha=$Sha)"
|
||||
while (-not $haveLock) {
|
||||
Write-Log "Acquiring worktree lock $lockFile (mode=$Mode sha=$Sha)"
|
||||
while ($null -eq $lock) {
|
||||
try {
|
||||
New-Item -Path $lockDir -ItemType Directory -ErrorAction Stop | Out-Null
|
||||
$haveLock = $true
|
||||
$lock = [System.IO.File]::Open($lockFile, 'OpenOrCreate', 'ReadWrite', 'None')
|
||||
}
|
||||
catch {
|
||||
if ((Get-Date) -ge $deadline) {
|
||||
throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockDir. A stuck run may have left it; if no CI job is active, remove it by hand."
|
||||
throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockFile ($($_.Exception.Message)). Another CI run is holding it."
|
||||
}
|
||||
Start-Sleep -Seconds 10
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Write-HostResources -When 'start'
|
||||
|
||||
try {
|
||||
Push-Location $RepoRoot
|
||||
try {
|
||||
# Re-fetch + re-checkout under the lock so a concurrent run's bootstrap checkout
|
||||
# (done before we held the lock) cannot leave us on the wrong tree.
|
||||
# Holding the lock proves no other CI git command is running in this clone, so a
|
||||
# leftover index.lock can only be debris from a killed git; it would fail the checkout.
|
||||
if (Test-Path '.git\index.lock') {
|
||||
Write-Log 'Removing stale .git\index.lock left by a killed git'
|
||||
Remove-Item -Force '.git\index.lock'
|
||||
}
|
||||
|
||||
# Re-fetch + re-checkout under the lock so the tree is the requested SHA even when the
|
||||
# bootstrap's checkout was skipped (standalone run) or left residue.
|
||||
Invoke-Native 'git fetch origin' { git fetch --prune origin }
|
||||
Invoke-Native "git checkout --force $Sha" { git checkout --force --detach $Sha }
|
||||
# Drop any stray tracked/untracked residue from a prior run so the build is clean.
|
||||
@@ -151,11 +184,12 @@ try {
|
||||
}
|
||||
catch {
|
||||
Write-Log "FAILURE mode=$Mode sha=$Sha : $($_.Exception.Message)"
|
||||
Write-HostResources -When 'failure'
|
||||
throw
|
||||
}
|
||||
finally {
|
||||
if ($haveLock) {
|
||||
Remove-Item -Path $lockDir -Recurse -Force -ErrorAction SilentlyContinue
|
||||
Write-Log "Released worktree lock $lockDir"
|
||||
if ($null -ne $lock) {
|
||||
$lock.Dispose()
|
||||
Write-Log "Released worktree lock $lockFile"
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user