de86f8b80e
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6): 7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and burst-coalescing failures (#126/#127/#139) last failed before50322bac/7da52b65(Aug 2026). - 2026-10-15..20 was the NU1903 package-audit restore failure, fixed infb20997. Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
150 lines
6.7 KiB
Bash
Executable File
150 lines
6.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
#
|
|
# TST-25: Linux side of the Windows/x86 Worker CI tier.
|
|
#
|
|
# Runs in the `windows-x86` / nightly Gitea jobs on a Linux runner (which always schedules,
|
|
# unlike a broken Windows act_runner). SSHes to windev (10.100.0.48), where it fetches +
|
|
# checks out the SHA under test in the isolated CI clone C:\build\mxaccessgw-ci and runs
|
|
# scripts/ci/windev-worker-ci.ps1 there. The remote exit code propagates through
|
|
# ssh -> this script -> the CI step, so a Worker regression turns the job red.
|
|
#
|
|
# Usage: run-windev-ci.sh <build|test|live>
|
|
#
|
|
# Environment:
|
|
# CI_SHA commit to test (default: `git rev-parse HEAD` in this checkout)
|
|
# WINDEV_SSH_KEY private key (Gitea secret). Store it BASE64-ENCODED (one line):
|
|
# Gitea's secret masker is line-oriented, so a raw multiline PEM in
|
|
# the step `env:` echo is NOT redacted and leaks in cleartext, whereas
|
|
# the single-line base64 is masked to *** (TST-25 acceptance: no key
|
|
# material in logs). A raw PEM is still accepted for local hand-testing
|
|
# (auto-detected). If empty, falls back to the runner's default identity.
|
|
# WINDEV_SSH_KNOWN_HOSTS pinned host keys (Gitea secret). If empty, uses the committed
|
|
# scripts/ci/windev.known_hosts.
|
|
# WINDEV_SSH_USER ssh user on windev (default: ci)
|
|
# WINDEV_SSH_HOST windev host/IP (default: 10.100.0.48)
|
|
#
|
|
# The private key is never echoed (written 0600 to a temp file, removed on exit). git on
|
|
# windev writes progress to stderr, so the remote bootstrap does NOT use -ErrorActionPreference
|
|
# Stop around git — it checks $LASTEXITCODE instead (known windev gotcha).
|
|
|
|
set -euo pipefail
|
|
|
|
MODE="${1:-}"
|
|
case "$MODE" in
|
|
build|test|live) ;;
|
|
*) echo "usage: $0 <build|test|live>" >&2; exit 2 ;;
|
|
esac
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
CI_SHA="${CI_SHA:-$(git -C "$SCRIPT_DIR" rev-parse HEAD)}"
|
|
WINDEV_SSH_USER="${WINDEV_SSH_USER:-ci}"
|
|
WINDEV_SSH_HOST="${WINDEV_SSH_HOST:-10.100.0.48}"
|
|
REMOTE_CLONE='C:\build\mxaccessgw-ci'
|
|
REMOTE_SCRIPT='scripts\ci\windev-worker-ci.ps1'
|
|
|
|
# Validate the SHA shape before it reaches a remote shell command.
|
|
if ! printf '%s' "$CI_SHA" | grep -Eq '^[0-9a-fA-F]{7,40}$'; then
|
|
echo "run-windev-ci: refusing to run — CI_SHA '$CI_SHA' is not a hex commit id" >&2
|
|
exit 2
|
|
fi
|
|
|
|
WORK="$(mktemp -d)"
|
|
cleanup() { rm -rf "$WORK"; }
|
|
trap cleanup EXIT
|
|
|
|
SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=15)
|
|
|
|
# known_hosts: secret if provided, else the committed pins.
|
|
KNOWN_HOSTS="$WORK/known_hosts"
|
|
if [ -n "${WINDEV_SSH_KNOWN_HOSTS:-}" ]; then
|
|
printf '%s\n' "$WINDEV_SSH_KNOWN_HOSTS" > "$KNOWN_HOSTS"
|
|
else
|
|
cp "$SCRIPT_DIR/windev.known_hosts" "$KNOWN_HOSTS"
|
|
fi
|
|
SSH_OPTS+=(-o "UserKnownHostsFile=$KNOWN_HOSTS")
|
|
|
|
# Private key: secret if provided, else fall back to the default identity/agent.
|
|
# CI stores it base64-encoded (single line) so Gitea's line-oriented secret masker redacts it
|
|
# in the step env echo — a raw multiline PEM leaks in cleartext. Decode when it is valid base64
|
|
# wrapping a PEM; otherwise treat the value as a raw PEM (local hand-testing convenience).
|
|
if [ -n "${WINDEV_SSH_KEY:-}" ]; then
|
|
KEY="$WORK/id_ci"
|
|
if printf '%s' "$WINDEV_SSH_KEY" | base64 -d 2>/dev/null | grep -q 'PRIVATE KEY'; then
|
|
( umask 077; printf '%s' "$WINDEV_SSH_KEY" | base64 -d > "$KEY" )
|
|
else
|
|
( umask 077; printf '%s\n' "$WINDEV_SSH_KEY" > "$KEY" )
|
|
fi
|
|
SSH_OPTS+=(-o IdentitiesOnly=yes -i "$KEY")
|
|
fi
|
|
|
|
echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}"
|
|
|
|
# Remote PowerShell bootstrap: fetch + checkout enough to load the versioned CI script, then
|
|
# hand off to it. Not Stop around git (stderr progress).
|
|
#
|
|
# The bootstrap fetch/checkout mutates the shared clone's git index, so it MUST hold the
|
|
# worktree lock — otherwise two concurrent runs collide on .git/index.lock before either
|
|
# reaches windev-worker-ci.ps1 (the exact race the lock exists to prevent). We export
|
|
# MXGW_CI_LOCK_HELD so the handed-off script re-uses this lock instead of re-acquiring it; a
|
|
# standalone/manual run of windev-worker-ci.ps1 (degraded mode) has no such env and takes the
|
|
# same lock itself.
|
|
#
|
|
# The lock is an exclusive (FileShare.None) handle on '<clone>.ci-lock', held by this
|
|
# bootstrap process for the whole run (the handle is not inherited by the child powershell).
|
|
# It replaced a `New-Item -ItemType Directory` lock that was NOT mutually exclusive:
|
|
# PowerShell's directory provider checks for existence and then calls CreateDirectory (which
|
|
# succeeds on an existing directory), so two bootstraps starting together both "acquired" it.
|
|
# A scheduled run starts windows-x86 and nightly-windev in the same second, and on 2026-09-22
|
|
# (run 827) both got in, ran git concurrently and the nightly died on
|
|
# ".git/index.lock: File exists" (mxaccessgw#148/#150). A file handle is also released by the
|
|
# OS if the holder is killed (ssh drop, host reboot), so a dead run can never leave a stale
|
|
# lock behind; the old directory had to be removed by hand.
|
|
#
|
|
# Holding the lock also proves no other CI git command can be running in the clone, so a
|
|
# leftover .git/index.lock can only be debris from a killed git and is removed.
|
|
read -r -d '' BOOTSTRAP <<PS || true
|
|
\$ErrorActionPreference = 'Continue'
|
|
\$lockFile = '$REMOTE_CLONE.ci-lock'
|
|
\$deadline = (Get-Date).AddMinutes(20)
|
|
\$lock = \$null
|
|
while (\$null -eq \$lock) {
|
|
try { \$lock = [System.IO.File]::Open(\$lockFile, 'OpenOrCreate', 'ReadWrite', 'None') }
|
|
catch {
|
|
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockFile : \$(\$_.Exception.Message)"; exit 75 }
|
|
Start-Sleep -Seconds 10
|
|
}
|
|
}
|
|
\$env:MXGW_CI_LOCK_HELD = '1'
|
|
\$rc = 1
|
|
try {
|
|
Set-Location '$REMOTE_CLONE'
|
|
if (Test-Path '.git\\index.lock') {
|
|
Write-Warning 'removing stale .git\\index.lock left by a killed git (the CI lock is held, so no CI git is running)'
|
|
Remove-Item -Force '.git\\index.lock'
|
|
}
|
|
git fetch --prune origin
|
|
git checkout --force --detach $CI_SHA
|
|
if (\$LASTEXITCODE -ne 0) { \$rc = \$LASTEXITCODE; Write-Error "bootstrap checkout failed (exit \$rc)" }
|
|
else {
|
|
powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
|
|
\$rc = \$LASTEXITCODE
|
|
}
|
|
}
|
|
finally {
|
|
\$lock.Dispose()
|
|
}
|
|
exit \$rc
|
|
PS
|
|
|
|
# PowerShell -EncodedCommand wants UTF-16LE base64 on one line (avoids all ssh/quoting hazards).
|
|
ENCODED="$(printf '%s' "$BOOTSTRAP" | iconv -t UTF-16LE | base64 | tr -d '\n')"
|
|
|
|
set +e
|
|
ssh "${SSH_OPTS[@]}" "${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}" \
|
|
"powershell -NoProfile -NonInteractive -EncodedCommand $ENCODED"
|
|
RC=$?
|
|
set -e
|
|
|
|
echo "run-windev-ci: remote exit code $RC"
|
|
exit $RC
|