Files
mxaccessgw/scripts/ci/run-windev-ci.sh
T
Joseph Doherty 5c8075996f
ci / portable (push) Successful in 6m58s
ci / java (push) Successful in 1m52s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m8s
fix(TST-25): serialize the CI bootstrap fetch/checkout under the worktree lock
Acceptance-check finding (concurrency): the worktree lock in windev-worker-ci.ps1
guarded the build stage, but run-windev-ci.sh's bootstrap did git fetch + git
checkout on the shared C:\build\mxaccessgw-ci clone BEFORE that lock was taken.
Two concurrent runs therefore collided on .git/index.lock at the bootstrap stage
(the second failed 'Another git process seems to be running', exit 1) instead of
the second waiting — defeating the lock's purpose for manual/degraded-mode overlap
or a future second runner. (The single Gitea runner serializes jobs, so CI pushes
never actually overlapped; this is defense-in-depth being restored.)

Fix: the bootstrap now acquires the same mkdir worktree lock around its fetch/
checkout and exports MXGW_CI_LOCK_HELD; windev-worker-ci.ps1 re-uses that lock
(skips re-acquire/release) when the env is set, and still self-locks for a
standalone/manual run. Now the second concurrent run waits at the bootstrap.
2026-07-13 10:50:22 -04:00

132 lines
5.6 KiB
Bash
Executable File

#!/usr/bin/env bash
#
# TST-25: Linux side of the Windows/x86 Worker CI tier.
#
# Runs in the `windows-x86` / nightly Gitea jobs on a Linux runner (which always schedules,
# unlike a broken Windows act_runner). SSHes to windev (10.100.0.48), where it fetches +
# checks out the SHA under test in the isolated CI clone C:\build\mxaccessgw-ci and runs
# scripts/ci/windev-worker-ci.ps1 there. The remote exit code propagates through
# ssh -> this script -> the CI step, so a Worker regression turns the job red.
#
# Usage: run-windev-ci.sh <build|test|live>
#
# Environment:
# CI_SHA commit to test (default: `git rev-parse HEAD` in this checkout)
# WINDEV_SSH_KEY private key (Gitea secret). Store it BASE64-ENCODED (one line):
# Gitea's secret masker is line-oriented, so a raw multiline PEM in
# the step `env:` echo is NOT redacted and leaks in cleartext, whereas
# the single-line base64 is masked to *** (TST-25 acceptance: no key
# material in logs). A raw PEM is still accepted for local hand-testing
# (auto-detected). If empty, falls back to the runner's default identity.
# WINDEV_SSH_KNOWN_HOSTS pinned host keys (Gitea secret). If empty, uses the committed
# scripts/ci/windev.known_hosts.
# WINDEV_SSH_USER ssh user on windev (default: ci)
# WINDEV_SSH_HOST windev host/IP (default: 10.100.0.48)
#
# The private key is never echoed (written 0600 to a temp file, removed on exit). git on
# windev writes progress to stderr, so the remote bootstrap does NOT use -ErrorActionPreference
# Stop around git — it checks $LASTEXITCODE instead (known windev gotcha).
set -euo pipefail
MODE="${1:-}"
case "$MODE" in
build|test|live) ;;
*) echo "usage: $0 <build|test|live>" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CI_SHA="${CI_SHA:-$(git -C "$SCRIPT_DIR" rev-parse HEAD)}"
WINDEV_SSH_USER="${WINDEV_SSH_USER:-ci}"
WINDEV_SSH_HOST="${WINDEV_SSH_HOST:-10.100.0.48}"
REMOTE_CLONE='C:\build\mxaccessgw-ci'
REMOTE_SCRIPT='scripts\ci\windev-worker-ci.ps1'
# Validate the SHA shape before it reaches a remote shell command.
if ! printf '%s' "$CI_SHA" | grep -Eq '^[0-9a-fA-F]{7,40}$'; then
echo "run-windev-ci: refusing to run — CI_SHA '$CI_SHA' is not a hex commit id" >&2
exit 2
fi
WORK="$(mktemp -d)"
cleanup() { rm -rf "$WORK"; }
trap cleanup EXIT
SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=15)
# known_hosts: secret if provided, else the committed pins.
KNOWN_HOSTS="$WORK/known_hosts"
if [ -n "${WINDEV_SSH_KNOWN_HOSTS:-}" ]; then
printf '%s\n' "$WINDEV_SSH_KNOWN_HOSTS" > "$KNOWN_HOSTS"
else
cp "$SCRIPT_DIR/windev.known_hosts" "$KNOWN_HOSTS"
fi
SSH_OPTS+=(-o "UserKnownHostsFile=$KNOWN_HOSTS")
# Private key: secret if provided, else fall back to the default identity/agent.
# CI stores it base64-encoded (single line) so Gitea's line-oriented secret masker redacts it
# in the step env echo — a raw multiline PEM leaks in cleartext. Decode when it is valid base64
# wrapping a PEM; otherwise treat the value as a raw PEM (local hand-testing convenience).
if [ -n "${WINDEV_SSH_KEY:-}" ]; then
KEY="$WORK/id_ci"
if printf '%s' "$WINDEV_SSH_KEY" | base64 -d 2>/dev/null | grep -q 'PRIVATE KEY'; then
( umask 077; printf '%s' "$WINDEV_SSH_KEY" | base64 -d > "$KEY" )
else
( umask 077; printf '%s\n' "$WINDEV_SSH_KEY" > "$KEY" )
fi
SSH_OPTS+=(-o IdentitiesOnly=yes -i "$KEY")
fi
echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}"
# Remote PowerShell bootstrap: fetch + checkout enough to load the versioned CI script, then
# hand off to it. Not Stop around git (stderr progress).
#
# The bootstrap fetch/checkout mutates the shared clone's git index, so it MUST hold the
# worktree lock — otherwise two concurrent runs collide on .git/index.lock before either
# reaches windev-worker-ci.ps1's lock (the exact race the lock exists to prevent). We take the
# same mkdir lock here (atomic; retry to a 20-min deadline) and export MXGW_CI_LOCK_HELD so the
# handed-off script re-uses this lock instead of re-acquiring/releasing it. A standalone/manual
# run of windev-worker-ci.ps1 (degraded mode) has no such env and self-locks as before.
read -r -d '' BOOTSTRAP <<PS || true
\$ErrorActionPreference = 'Continue'
\$lockDir = '$REMOTE_CLONE.lock'
\$deadline = (Get-Date).AddMinutes(20)
\$haveLock = \$false
while (-not \$haveLock) {
try { New-Item -Path \$lockDir -ItemType Directory -ErrorAction Stop | Out-Null; \$haveLock = \$true }
catch {
if ((Get-Date) -ge \$deadline) { Write-Error "timed out waiting for worktree lock \$lockDir"; exit 75 }
Start-Sleep -Seconds 10
}
}
\$env:MXGW_CI_LOCK_HELD = '1'
\$rc = 1
try {
Set-Location '$REMOTE_CLONE'
git fetch --prune origin
git checkout --force --detach $CI_SHA
if (\$LASTEXITCODE -ne 0) { Write-Error "bootstrap checkout failed"; \$rc = \$LASTEXITCODE }
else {
powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
\$rc = \$LASTEXITCODE
}
}
finally {
Remove-Item -Path \$lockDir -Recurse -Force -ErrorAction SilentlyContinue
}
exit \$rc
PS
# PowerShell -EncodedCommand wants UTF-16LE base64 on one line (avoids all ssh/quoting hazards).
ENCODED="$(printf '%s' "$BOOTSTRAP" | iconv -t UTF-16LE | base64 | tr -d '\n')"
set +e
ssh "${SSH_OPTS[@]}" "${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}" \
"powershell -NoProfile -NonInteractive -EncodedCommand $ENCODED"
RC=$?
set -e
echo "run-windev-ci: remote exit code $RC"
exit $RC