dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 3 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(TST-25,TST-26): restore Windows/x86 CI tier via SSH-driven windev job
ci / java (push) Successful in 2m2s
Details
ci / windows-x86 (push) Successful in 1m10s
Details
ci / nightly-windev (push) Has been skipped
Details
ci / portable (push) Successful in 8m52s
Details

Browse Source 
The `windows`/`live-mxaccess` CI jobs were removed in abb0930 because Gitea
act_runner host-mode on Windows is broken and a runs-on gate with no runner
wedges the queue. This left the entire x86/net48 Worker + Worker.Tests tier
(and the live-MXAccess smoke) unguarded by CI.

Restore it without a native Windows runner: Linux jobs on ubuntu-latest (which
always schedule) SSH to windev (10.100.0.48), fetch+checkout the SHA under test
in an isolated clone C:\build\mxaccessgw-ci, run the x86 build/tests there, and
propagate the remote exit code back — so a Worker regression turns the job red
and an unreachable host fails loud (never stuck).

- scripts/ci/run-windev-ci.sh: Linux driver (key/known-hosts, UTF-16LE base64
  EncodedCommand bootstrap, ssh, exit-code passthrough).
- scripts/ci/windev-worker-ci.ps1: windev stage — worktree lock, re-fetch/
  checkout under lock, build|test|live modes; PS 5.1-safe, checks $LASTEXITCODE.
- scripts/ci/windev.known_hosts: pinned host keys for StrictHostKeyChecking.
- scripts/ci/README.md: operator bring-up + acceptance checklist.
- ci.yml: windows-x86 (per-push `test`) + nightly-windev (scheduled `live` +
  on-failure Gitea issue); drop the removal note; fix header/cron comments.

TST-26 (same commit, by rule): correct docs/scripts that still describe the
removed jobs — GatewayTesting.md, Contracts.md, check-codegen.ps1 — and
reattribute the Generated/-commit guard (primary = check-codegen diff in the
portable job; secondary = windows-x86 net48 compile).

Mechanism hand-verified on windev: build->0, bogus-SHA->nonzero (lock released),
test->356 passed/0 failed in ~50s, ssh exit-code propagation confirmed. Merge
requires operator bring-up first (dedicated ci@ key + Gitea secrets) or the
per-push job is red every push — see scripts/ci/README.md.
This commit is contained in:
Joseph Doherty
2026-07-13 10:02:26 -04:00
parent 1a5a12a416
commit d769244ac0
10 changed files with 405 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/check-codegen.ps1
+2 -1
View File
@@ -14,7 +14,8 @@
# protos. A drift means a .proto was edited without refreshing the vendored copies, which would
# publish a stale wire contract to crate consumers while the in-repo build stays correct.
#
# The x86 Worker + Worker.Tests are Windows-only and are guarded by the Windows CI job, not here.
# The x86 Worker + Worker.Tests are Windows-only and are guarded by the SSH-driven `windows-x86`
# CI job (see docs/GatewayTesting.md, Continuous Integration), not here.
[CmdletBinding()]
param()
scripts/ci/README.md
+58
View File
@@ -0,0 +1,58 @@
# Windows/x86 CI tier (TST-25)
The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows
`act_runner` is broken (host-mode writes each step's script to a path it can't resolve; a
`runs-on` gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to
windev (`10.100.0.48`) and runs the Worker build/tests there, propagating the remote exit
code back so a regression turns the job red.
## Pieces
| File | Runs on | Role |
|---|---|---|
| `run-windev-ci.sh <build\|test\|live>` | Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. |
| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, runs the x86 build / `Worker.Tests` / live smoke. |
| `windev.known_hosts` | committed | Pinned windev host keys so `StrictHostKeyChecking=yes` has no TOFU prompt. Host keys are public; committing is intentional. |
Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests`;
`live` = test + full-`slnx` build + live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`).
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev`
runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure.
## One-time bring-up (operator — required before this lands on `main`)
Until these are done, `windows-x86` is **red on every push** (no key → SSH fails). Do them
first, then merge.
1. **CI clone** — `C:\build\mxaccessgw-ci`, a fresh clone of the Gitea origin (done during
TST-25 verification; recreate with `git clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ci` if missing). Never the dirty Desktop checkout.
2. **Dedicated CI key** — generate a NEW ed25519 keypair for CI (do not reuse the operator's
personal windev key): `ssh-keygen -t ed25519 -f ci_windev -N ''`. Append `ci_windev.pub`
to the windev CI account's `authorized_keys` (optionally with a forced `command=` limiting
it to this bootstrap).
3. **Gitea repo secrets** — store the private key as `WINDEV_SSH_KEY` and, optionally, the
pinned host keys as `WINDEV_SSH_KNOWN_HOSTS` (the committed `windev.known_hosts` is the
fallback). `run-windev-ci.sh` connects as user `ci` by default (`WINDEV_SSH_USER` to
override); ensure that account exists on windev, or set `WINDEV_SSH_USER` to the intended one.
4. **Network precheck** — confirm the Gitea runner container (on the `traefik` docker network,
host `10.100.0.35`) can reach `10.100.0.48:22`: a one-off `nc -z -w5 10.100.0.48 22` step.
The network was created for gitea name resolution, not LAN egress, so verify.
5. **Actions token** — the `nightly-windev` on-failure step creates a Gitea issue with the
built-in token; confirm Actions tokens have issue-write on this repo.
## Verifying it end-to-end (design's acceptance checks)
- Push a branch → `portable`, `java`, `windows-x86` all green; windev's clone sits at the pushed SHA.
- Deliberate red: add a failing `[Fact]` to `Worker.Tests`, push, confirm `windows-x86` goes
red; revert. (Proves exit-code propagation — the whole point.)
- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
- Nightly: trigger the schedule path, confirm `live` runs and a forced failure opens an issue.
- Confirm no key material appears in job logs.
## Degraded mode
If the tier is down for infra reasons, fall back to the manual windev worktree procedure
(isolated `C:\build` worktree, `dotnet build`/`test` `-p:Platform=x86` by hand) per merge for
worker-touching changes until `windows-x86` is green again. See `docs/GatewayTesting.md`.
scripts/ci/run-windev-ci.sh
Executable
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
#
# TST-25: Linux side of the Windows/x86 Worker CI tier.
#
# Runs in the `windows-x86` / nightly Gitea jobs on a Linux runner (which always schedules,
# unlike a broken Windows act_runner). SSHes to windev (10.100.0.48), where it fetches +
# checks out the SHA under test in the isolated CI clone C:\build\mxaccessgw-ci and runs
# scripts/ci/windev-worker-ci.ps1 there. The remote exit code propagates through
# ssh -> this script -> the CI step, so a Worker regression turns the job red.
#
# Usage: run-windev-ci.sh <build|test|live>
#
# Environment:
# CI_SHA commit to test (default: `git rev-parse HEAD` in this checkout)
# WINDEV_SSH_KEY private key PEM (Gitea secret). If empty, falls back to the
# runner's default ssh identity/agent (used for local hand-testing).
# WINDEV_SSH_KNOWN_HOSTS pinned host keys (Gitea secret). If empty, uses the committed
# scripts/ci/windev.known_hosts.
# WINDEV_SSH_USER ssh user on windev (default: ci)
# WINDEV_SSH_HOST windev host/IP (default: 10.100.0.48)
#
# The private key is never echoed (written 0600 to a temp file, removed on exit). git on
# windev writes progress to stderr, so the remote bootstrap does NOT use -ErrorActionPreference
# Stop around git — it checks $LASTEXITCODE instead (known windev gotcha).
set -euo pipefail
MODE="${1:-}"
case "$MODE" in
build|test|live) ;;
*) echo "usage: $0 <build|test|live>" >&2; exit 2 ;;
esac
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CI_SHA="${CI_SHA:-$(git -C "$SCRIPT_DIR" rev-parse HEAD)}"
WINDEV_SSH_USER="${WINDEV_SSH_USER:-ci}"
WINDEV_SSH_HOST="${WINDEV_SSH_HOST:-10.100.0.48}"
REMOTE_CLONE='C:\build\mxaccessgw-ci'
REMOTE_SCRIPT='scripts\ci\windev-worker-ci.ps1'
# Validate the SHA shape before it reaches a remote shell command.
if ! printf '%s' "$CI_SHA" | grep -Eq '^[0-9a-fA-F]{7,40}$'; then
echo "run-windev-ci: refusing to run — CI_SHA '$CI_SHA' is not a hex commit id" >&2
exit 2
fi
WORK="$(mktemp -d)"
cleanup() { rm -rf "$WORK"; }
trap cleanup EXIT
SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=15)
# known_hosts: secret if provided, else the committed pins.
KNOWN_HOSTS="$WORK/known_hosts"
if [ -n "${WINDEV_SSH_KNOWN_HOSTS:-}" ]; then
printf '%s\n' "$WINDEV_SSH_KNOWN_HOSTS" > "$KNOWN_HOSTS"
else
cp "$SCRIPT_DIR/windev.known_hosts" "$KNOWN_HOSTS"
fi
SSH_OPTS+=(-o "UserKnownHostsFile=$KNOWN_HOSTS")
# Private key: secret if provided, else fall back to the default identity/agent.
if [ -n "${WINDEV_SSH_KEY:-}" ]; then
KEY="$WORK/id_ci"
( umask 077; printf '%s\n' "$WINDEV_SSH_KEY" > "$KEY" )
SSH_OPTS+=(-o IdentitiesOnly=yes -i "$KEY")
fi
echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}"
# Remote PowerShell bootstrap: fetch + checkout enough to load the versioned CI script, then
# hand off to it. Not Stop around git (stderr progress). windev-worker-ci.ps1 re-fetches and
# re-checks-out under a lock, so this checkout only needs to load the right script version.
read -r -d '' BOOTSTRAP <<PS || true
\$ErrorActionPreference = 'Continue'
Set-Location '$REMOTE_CLONE'
git fetch --prune origin
git checkout --force --detach $CI_SHA
if (\$LASTEXITCODE -ne 0) { Write-Error "bootstrap checkout failed"; exit \$LASTEXITCODE }
powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
exit \$LASTEXITCODE
PS
# PowerShell -EncodedCommand wants UTF-16LE base64 on one line (avoids all ssh/quoting hazards).
ENCODED="$(printf '%s' "$BOOTSTRAP" | iconv -t UTF-16LE | base64 | tr -d '\n')"
set +e
ssh "${SSH_OPTS[@]}" "${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}" \
"powershell -NoProfile -NonInteractive -EncodedCommand $ENCODED"
RC=$?
set -e
echo "run-windev-ci: remote exit code $RC"
exit $RC
scripts/ci/windev-worker-ci.ps1
+147
View File
@@ -0,0 +1,147 @@
<#
.SYNOPSIS
Windows/x86 Worker CI stage — runs ON windev (10.100.0.48), invoked over SSH by
scripts/ci/run-windev-ci.sh at the exact SHA under test. Restores the x86/net48
Worker test tier that the removed `windows` CI job used to cover (TST-25).
.DESCRIPTION
The Linux `windows-x86` / nightly CI jobs cannot build the x86/net48 Worker, so they
ssh here and run this script inside the isolated CI clone C:\build\mxaccessgw-ci.
The script is deliberately self-contained and versioned with the code under test: the
SSH bootstrap only fetches + checks out the SHA (enough to load *this* file), then this
script re-fetches and re-checks-out $Sha INSIDE the worktree lock so the checkout+build
are fully serialized against a concurrent run and can never stomp each other's tree.
Modes (cumulative):
build x86 Worker build only (net48/CS0246/x86 compile guard — always fast).
test build + Worker.Tests (x86). The per-push default.
live test + the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full
slnx build (surfaces Windows-only analyzer diagnostics, e.g. xUnit1030). Nightly.
Runs under Windows PowerShell 5.1 (`powershell.exe`), so it avoids PS7-only syntax and
checks $LASTEXITCODE after every native command rather than relying on stderr-as-error
(git writes progress to stderr — the known windev gotcha).
Exits nonzero on the first failure so the SSH exit code propagates to the CI job.
.PARAMETER Sha
The commit SHA under test. HEAD is force-checked-out to it and verified.
.PARAMETER Mode
build | test | live.
.PARAMETER RepoRoot
The CI clone root. Defaults to C:\build\mxaccessgw-ci.
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)][string]$Sha,
[Parameter(Mandatory = $true)][ValidateSet('build', 'test', 'live')][string]$Mode,
[string]$RepoRoot = 'C:\build\mxaccessgw-ci'
)
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$lockDir = "$RepoRoot.lock"
$logFile = Join-Path $RepoRoot 'ci.log'
$lockTimeout = [TimeSpan]::FromMinutes(20)
function Write-Log {
param([string]$Message)
$line = ('[{0}] {1}' -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Message)
Write-Host $line
# Best-effort file log for the "is the tier down?" diagnosis in docs/GatewayTesting.md.
try { Add-Content -Path $logFile -Value $line -ErrorAction SilentlyContinue } catch { }
}
# Run a native command and throw on nonzero exit. Native stderr (git progress) does NOT
# set $LASTEXITCODE, so it is not treated as failure — only a real nonzero exit is.
function Invoke-Native {
param([Parameter(Mandatory = $true)][string]$What, [Parameter(Mandatory = $true)][scriptblock]$Command)
Write-Log "RUN: $What"
& $Command
if ($LASTEXITCODE -ne 0) {
throw "$What failed with exit code $LASTEXITCODE"
}
}
# --- acquire the worktree lock (mkdir is atomic; retry until timeout) ---
$deadline = (Get-Date).Add($lockTimeout)
$haveLock = $false
Write-Log "Acquiring worktree lock $lockDir (mode=$Mode sha=$Sha)"
while (-not $haveLock) {
try {
New-Item -Path $lockDir -ItemType Directory -ErrorAction Stop | Out-Null
$haveLock = $true
}
catch {
if ((Get-Date) -ge $deadline) {
throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockDir. A stuck run may have left it; if no CI job is active, remove it by hand."
}
Start-Sleep -Seconds 10
}
}
try {
Push-Location $RepoRoot
try {
# Re-fetch + re-checkout under the lock so a concurrent run's bootstrap checkout
# (done before we held the lock) cannot leave us on the wrong tree.
Invoke-Native 'git fetch origin' { git fetch --prune origin }
Invoke-Native "git checkout --force $Sha" { git checkout --force --detach $Sha }
# Drop any stray tracked/untracked residue from a prior run so the build is clean.
Invoke-Native 'git reset --hard' { git reset --hard $Sha }
Invoke-Native 'git clean -fdx (build outputs)' { git clean -fdx -e ci.log }
$head = (& git rev-parse HEAD).Trim()
if ($head -ne $Sha) {
throw "Worktree HEAD $head does not match requested SHA $Sha after checkout."
}
Write-Log "Worktree at $head"
$workerProj = 'src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj'
$workerTests = 'src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj'
$integrationTests = 'src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj'
# build: always. The x86/net48 compile is the floor guard (catches CS0246 on
# unregenerated Generated/, x86-only breaks) even when the test step is demoted.
Invoke-Native 'x86 Worker build' { dotnet build $workerProj -c Release -p:Platform=x86 }
if ($Mode -eq 'test' -or $Mode -eq 'live') {
Invoke-Native 'x86 Worker.Tests' { dotnet test $workerTests -c Release -p:Platform=x86 }
}
if ($Mode -eq 'live') {
# Full-solution build surfaces Windows-only analyzer diagnostics (xUnit1030 etc.)
# that never fire on the macOS/Linux NonWindows build.
Invoke-Native 'full slnx build' { dotnet build src/ZB.MOM.WW.MxGateway.slnx -c Release }
$env:MXGATEWAY_RUN_LIVE_MXACCESS_TESTS = '1'
try {
Invoke-Native 'live-MXAccess smoke' {
dotnet test $integrationTests -c Release --filter 'FullyQualifiedName~WorkerLiveMxAccessSmokeTests'
}
}
finally {
Remove-Item Env:\MXGATEWAY_RUN_LIVE_MXACCESS_TESTS -ErrorAction SilentlyContinue
}
}
Write-Log "SUCCESS mode=$Mode sha=$Sha"
}
finally {
Pop-Location
}
}
catch {
Write-Log "FAILURE mode=$Mode sha=$Sha : $($_.Exception.Message)"
throw
}
finally {
if ($haveLock) {
Remove-Item -Path $lockDir -Recurse -Force -ErrorAction SilentlyContinue
Write-Log "Released worktree lock $lockDir"
}
}
scripts/ci/windev.known_hosts
+12
View File
@@ -0,0 +1,12 @@
# Pinned SSH host keys for the windev CI target (10.100.0.48), used by the
# `windows-x86` / nightly jobs so `StrictHostKeyChecking=yes` can verify the host
# without an interactive TOFU prompt. Host keys are public data — committing them is
# intentional (greppable, versioned) per TST-25's design.
#
# Regenerate if windev's host keys rotate:
# ssh-keyscan -t ed25519,rsa,ecdsa 10.100.0.48 > scripts/ci/windev.known_hosts
# (then re-add this header). run-windev-ci.sh prefers the WINDEV_SSH_KNOWN_HOSTS
# secret when set and falls back to this file.
10.100.0.48 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHRqxpRq/G0BR5vAomHMNc/H7eyTkdqliHNrFlMQ9HCb
10.100.0.48 ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIvTHAgf7Q+rK2dCyjSUgkYgos4j2eqyJN/AAfs2TXGJgrpp9AfOMnkbPtmMX3CvsrPoZxzBttIYcBYbf9KXb98JAu6fWqeglI/CeXH4oHw6oqMRtM0K2hl2gm3DicKKCQ==
10.100.0.48 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxKOhjdVawf3ziO9MB4VfLA1zDwnNK028siIzjTHZec4uJumInvSFD5S3tkJvSKGfCzXHtLu9m0zHmvJ4gIvLx4gt60TPABPC/QrdRG8mP4uOlPCciukWNIBhMWFU6oxHpZlQgrjQ3a+6WZxIjZBtaZQt3S5f9zEvOu6NUrLmP9cmrrXtAByMYyjISQf2BswxZ5ugeDHn0rVo3p3qYZ3M8lbXV+CRTwKj0I+VVceffJrYwLT0LyWxi0tDxmHxK1PgkhcIjLDWKhk4UxOA6gdO6MjuHvXZv5+UXJzjgGxom+xqxGhAge0F66p5IEJ/xrJMWoPBtFQ1by7tDZAsRhcbQJVeCbBjEDv7ShCbyh2Gq+PfJhWVVaYnMKcqBWkhPrZH4BE9yhAtSlV8QZsZ1iYUYKNAt9Kcg3ugRYx414KUf2fvUWSuoaRkJnBqQq1Mi0yK1JtTZOoYvN4XEYIr56hXQG1RfA8xWqYzjmbHYTgO8zv6tJJTeXLMqubmDdLP9H+U=