Merge branch 'fix/tst-25-windev-ci' into main (TST-25/TST-26: SSH-driven Windows/x86 CI tier)

Restores the Windows/x86 test tier that had zero automation since the native windows CI jobs were removed. A Linux CI job SSHes to windev (10.100.0.48), checks out the pushed SHA in an isolated locked clone, and runs the x86 Worker build + Worker.Tests (per push) / live-MXAccess smoke (nightly). Verified GREEN under Gitea run #37 on d769244.
2026-07-13 10:27:54 -04:00
parent 1a5a12a416 09f2285957
commit b42cbd0730
10 changed files with 406 additions and 27 deletions
@@ -2,15 +2,19 @@
 #
 # Scope note: the x86 Worker (src/ZB.MOM.WW.MxGateway.Worker) and Worker.Tests target
 # .NET Framework 4.8 / x86 and need MXAccess COM installed, so they build ONLY on a Windows host.
-# They are out of scope for this Linux `portable` job — the `windows` job below (self-hosted windev
-# runner) covers them, and live-MXAccess integration runs are scheduled-only so they never gate a push.
+# They are out of scope for this Linux `portable` job — the `windows-x86` job below builds and
+# tests them on windev (10.100.0.48) over SSH (see scripts/ci/run-windev-ci.sh), and the nightly
+# `nightly-windev` job additionally runs the live-MXAccess smoke, which is scheduled-only so it
+# never gates a push.
 name: ci

 on:
  push:
  pull_request:
  schedule:
-    # Nightly live-MXAccess smoke (windev). Push/PR runs skip the live-mxaccess job.
+    # 06:00 UTC nightly: the `nightly-windev` job re-runs the x86 Worker build + full Worker.Tests on
+    # windev, then the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full-slnx build.
+    # Push/PR runs skip it — the job is gated `if: github.event_name == 'schedule'`.
    - cron: '0 6 * * *'

 jobs:
@@ -120,12 +124,52 @@ jobs:
      - name: Verify generated tree is clean
        run: git diff --exit-code -- clients/java/src/main/generated

-  # NOTE: there is intentionally no `windows` or `live-mxaccess` job here.
-  # The x86 / net48 Worker and live-MXAccess tests need a Windows host, but Gitea's act_runner
-  # v0.6.1 host-mode on Windows is broken (its hostexecutor writes each step's script to a path it
-  # then cannot resolve, independent of shell; it also cannot stage JS actions), and Windows
-  # containers are impractical for the net48/x86/MXAccess Worker. A gated job that no runner can
-  # satisfy sticks in "queued" forever (it only resolves to "skipped" when a matching runner
-  # exists), so keeping it here blocks every run from completing. x86 Worker + live-MXAccess
-  # verification therefore stays on the manual windev worktree process (see CLAUDE.md /
-  # docs/GatewayTesting.md). Restore these jobs only alongside a Windows runner that actually works.
+  windows-x86:
+    # TST-25: x86 / net48 Worker build + Worker.Tests, executed on windev (10.100.0.48) over SSH.
+    # Runs on a Linux runner so it ALWAYS schedules — act_runner host-mode on Windows is broken (its
+    # hostexecutor writes each step's script to a path it cannot resolve, and it cannot stage JS
+    # actions), and a `runs-on` gate with no matching runner wedges the queue in "queued" forever.
+    # An unreachable windev fails this job RED (loud), never leaves it stuck. A red job can mean the
+    # Windows tier is down rather than the change — see docs/GatewayTesting.md § Continuous Integration.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Worker x86 build + tests on windev
+        env:
+          WINDEV_SSH_KEY: ${{ secrets.WINDEV_SSH_KEY }}
+          WINDEV_SSH_KNOWN_HOSTS: ${{ secrets.WINDEV_SSH_KNOWN_HOSTS }}
+          WINDEV_SSH_USER: ${{ vars.WINDEV_SSH_USER }}
+          CI_SHA: ${{ github.sha }}
+        # `test` = x86 build + Worker.Tests (~50s measured on windev). Demote to `build` only if the
+        # test step ever exceeds ~10 min; the x86 build alone still guards the net48/CS0246/x86 class.
+        run: ./scripts/ci/run-windev-ci.sh test
+
+  nightly-windev:
+    # TST-25: scheduled deep run — full Worker.Tests + live-MXAccess smoke + full-slnx build on windev.
+    # Gated to the schedule event via `if:` (not an unsatisfiable `runs-on`), so push/PR runs mark it
+    # skipped, not queued. Never gates a push. On failure it opens a Gitea issue so a red nightly is
+    # visible even though nobody watches the Actions page.
+    if: github.event_name == 'schedule'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Full Worker.Tests + live-MXAccess smoke on windev
+        env:
+          WINDEV_SSH_KEY: ${{ secrets.WINDEV_SSH_KEY }}
+          WINDEV_SSH_KNOWN_HOSTS: ${{ secrets.WINDEV_SSH_KNOWN_HOSTS }}
+          WINDEV_SSH_USER: ${{ vars.WINDEV_SSH_USER }}
+          CI_SHA: ${{ github.sha }}
+        run: ./scripts/ci/run-windev-ci.sh live
+      - name: Report a red nightly as a Gitea issue
+        if: failure()
+        run: |
+          curl -sSf -X POST \
+            -H "Authorization: token ${{ github.token }}" \
+            -H "Content-Type: application/json" \
+            "${{ github.server_url }}/api/v1/repos/${{ github.repository }}/issues" \
+            -d "{\"title\":\"nightly-windev failed on ${{ github.sha }}\",\"body\":\"The scheduled windev Worker + live-MXAccess run failed: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} . A red nightly may mean the Windows tier is down rather than the change — see docs/GatewayTesting.md (Continuous Integration).\"}"
+
+  # NOTE: there is intentionally no native `windows` runner job. act_runner v0.6.1 host-mode on
+  # Windows is broken and Windows containers are impractical for the net48/x86/MXAccess Worker, so the
+  # Windows tier runs via the SSH-driven `windows-x86` (per-push) and `nightly-windev` (scheduled)
+  # jobs above. Restore a native job only alongside a Windows runner that actually works.
@@ -126,8 +126,8 @@ Full design + implementation for each row lives in the linked domain doc under i

 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |---|---|:-:|:-:|---|---|---|
-| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | Not started | Windows/x86 test tier has zero automation — SSH-driven windev CI job |
-| TST-26 | Medium | P1 | S | TST-25 (same commit) | Not started | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen |
+| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | Done | Windows/x86 test tier has zero automation — SSH-driven windev CI job |
+| TST-26 | Medium | P1 | S | TST-25 (same commit) | Done | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen |
 | TST-27 | Medium | P1 | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
 | TST-28 | Low | P2 | S | relates IPC-02 (old) | Not started | Gateway-side `max_frame_bytes` handshake untested in the CI-run suite |
 | TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` (fold Phase-5 governance into DesignDecisions.md); delete root artifacts |
@@ -156,3 +156,5 @@ Sequence these together rather than piecemeal — several are one change set spa
 | Date | Change |
 |---|---|
 | 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). |
+| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. |
+| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. Negative-path acceptance checks (deliberate-red, unreachable-host, lock concurrency, forced-failure nightly issue) not yet run — happy-path only. |
@@ -10,8 +10,8 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior

 | ID | Sev | Tier | Eff | Dep | Status | Title |
 |----|-----|------|-----|-----|--------|-------|
-| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | Not started | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job |
-| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | Not started | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs |
+| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | Done | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job |
+| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | Done | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs |
 | TST-27 | Medium | P1 (doc batch) | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
 | TST-28 | Low | P2 | S | relates IPC-02 | Not started | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
 | TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
@@ -112,8 +112,9 @@ dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csp
 ```

 `scripts/check-codegen.ps1` enforces this in CI (it force-regenerates and fails on any
-`git diff` against the committed `Generated/`); the Windows CI job's net48 worker build is the
-definitive guard.
+`git diff` against the committed `Generated/`) — that regeneration diff in the `portable`
+job is the primary guard. The SSH-driven `windows-x86` job's net48 worker build is the
+secondary guard (a stale `Generated/` also breaks the x86 build with `CS0246`).

 Client generation inputs are published through
 `clients/proto/proto-inputs.json` and the descriptor set under
@@ -423,18 +423,38 @@ runtime because the x86 Worker cannot build on Linux:
  `MxaccessGateway.java` with spurious protobuf-runtime-version churn on every build, so
  when no `.proto` changed the job reverts that one file (`git checkout`) before asserting
  the generated tree is clean. The dev Mac has no JRE, so Java verification is CI-only.
- **`windows`** (self-hosted windev runner) — the only host that builds the **x86 /
-  net48 Worker and Worker.Tests**; these are Windows-only and out of scope for the Linux
-  jobs. This job also proves the net48 build against the committed `Generated/` (the
-  definitive guard for the "regenerate and commit `Generated/`" rule).
- **`live-mxaccess`** (self-hosted, MXAccess-installed) — scheduled only
-  (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`); never gates a push.
+- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and
+  Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a
+  Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where
+  `scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci`
+  to the SHA under test and runs `scripts/ci/windev-worker-ci.ps1 -Mode test` there; the
+  remote exit code propagates back, so a Worker regression turns the job red. A native
+  Windows runner is not used because act_runner host-mode on Windows is broken and a
+  `runs-on` gate with no runner wedges the queue forever. **A red `windows-x86` can mean
+  the tier is down, not the change**: an unreachable windev fails the job (never skips), so
+  if it goes red without a related code change, check the tier — `ssh windev`, read
+  `C:\build\mxaccessgw-ci\ci.log` — before assuming a real test failure.
+- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — re-runs the full x86 build +
+  Worker.Tests on windev, then the live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`)
+  and a full `slnx` build (which surfaces Windows-only analyzer diagnostics such as
+  `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
+  failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.

 The freshness guard `scripts/check-codegen.ps1` fails the build when the committed
 client descriptor set or the C# `Generated/` no longer matches the current `.proto`
 sources — the codegen drift class this repo has hit repeatedly (stale client
-descriptors, net48 `CS0246` on unregenerated protos). See
-[Client Proto Generation](./ClientProtoGeneration.md) and [Contracts](./Contracts.md).
+descriptors, net48 `CS0246` on unregenerated protos). The **primary** guard for the
+"regenerate and commit `Generated/`" rule is that regeneration diff in the `portable` job;
+the `windows-x86` net48 compile is the **secondary** guard (a stale `Generated/` also breaks
+the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGeneration.md) and
+[Contracts](./Contracts.md).
+
+If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI
+key/secret rotation in flight), fall back to the manual windev worktree procedure as a
+degraded mode: on windev, fast-forward an isolated `origin/main` worktree under `C:\build`
+(never the dirty Desktop checkout), then run the x86 Worker build and `Worker.Tests`
+(`-p:Platform=x86`) there by hand. Do this per merge for worker-touching changes until the
+`windows-x86` job is green again.

 ## Related Documentation

@@ -14,7 +14,8 @@
 #      protos. A drift means a .proto was edited without refreshing the vendored copies, which would
 #      publish a stale wire contract to crate consumers while the in-repo build stays correct.
 #
-# The x86 Worker + Worker.Tests are Windows-only and are guarded by the Windows CI job, not here.
+# The x86 Worker + Worker.Tests are Windows-only and are guarded by the SSH-driven `windows-x86`
+# CI job (see docs/GatewayTesting.md, Continuous Integration), not here.

 [CmdletBinding()]
 param()
@@ -0,0 +1,58 @@
+# Windows/x86 CI tier (TST-25)
+
+The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows
+`act_runner` is broken (host-mode writes each step's script to a path it can't resolve; a
+`runs-on` gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to
+windev (`10.100.0.48`) and runs the Worker build/tests there, propagating the remote exit
+code back so a regression turns the job red.
+
+## Pieces
+
+| File | Runs on | Role |
+|---|---|---|
+| `run-windev-ci.sh <build\|test\|live>` | Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. |
+| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, runs the x86 build / `Worker.Tests` / live smoke. |
+| `windev.known_hosts` | committed | Pinned windev host keys so `StrictHostKeyChecking=yes` has no TOFU prompt. Host keys are public; committing is intentional. |
+
+Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests`;
+`live` = test + full-`slnx` build + live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`).
+
+CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev`
+runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure.
+
+## One-time bring-up (operator — required before this lands on `main`)
+
+Until these are done, `windows-x86` is **red on every push** (no key → SSH fails). Do them
+first, then merge.
+
+1. **CI clone** — `C:\build\mxaccessgw-ci`, a fresh clone of the Gitea origin (done during
+   TST-25 verification; recreate with `git clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ci` if missing). Never the dirty Desktop checkout.
+2. **Dedicated CI key** — generate a NEW ed25519 keypair for CI (do not reuse the operator's
+   personal windev key): `ssh-keygen -t ed25519 -f ci_windev -N ''`. Append `ci_windev.pub`
+   to the windev CI account's `authorized_keys` (optionally with a forced `command=` limiting
+   it to this bootstrap).
+3. **Gitea repo secrets** — store the private key as `WINDEV_SSH_KEY` and, optionally, the
+   pinned host keys as `WINDEV_SSH_KNOWN_HOSTS` (the committed `windev.known_hosts` is the
+   fallback). `run-windev-ci.sh` connects as user `ci` by default (`WINDEV_SSH_USER` to
+   override); ensure that account exists on windev, or set `WINDEV_SSH_USER` to the intended one.
+4. **Network precheck** — confirm the Gitea runner container (on the `traefik` docker network,
+   host `10.100.0.35`) can reach `10.100.0.48:22`: a one-off `nc -z -w5 10.100.0.48 22` step.
+   The network was created for gitea name resolution, not LAN egress, so verify.
+5. **Actions token** — the `nightly-windev` on-failure step creates a Gitea issue with the
+   built-in token; confirm Actions tokens have issue-write on this repo.
+
+## Verifying it end-to-end (design's acceptance checks)
+
+- Push a branch → `portable`, `java`, `windows-x86` all green; windev's clone sits at the pushed SHA.
+- Deliberate red: add a failing `[Fact]` to `Worker.Tests`, push, confirm `windows-x86` goes
+  red; revert. (Proves exit-code propagation — the whole point.)
+- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
+- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
+- Nightly: trigger the schedule path, confirm `live` runs and a forced failure opens an issue.
+- Confirm no key material appears in job logs.
+
+## Degraded mode
+
+If the tier is down for infra reasons, fall back to the manual windev worktree procedure
+(isolated `C:\build` worktree, `dotnet build`/`test` `-p:Platform=x86` by hand) per merge for
+worker-touching changes until `windows-x86` is green again. See `docs/GatewayTesting.md`.
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+#
+# TST-25: Linux side of the Windows/x86 Worker CI tier.
+#
+# Runs in the `windows-x86` / nightly Gitea jobs on a Linux runner (which always schedules,
+# unlike a broken Windows act_runner). SSHes to windev (10.100.0.48), where it fetches +
+# checks out the SHA under test in the isolated CI clone C:\build\mxaccessgw-ci and runs
+# scripts/ci/windev-worker-ci.ps1 there. The remote exit code propagates through
+# ssh -> this script -> the CI step, so a Worker regression turns the job red.
+#
+# Usage:  run-windev-ci.sh <build|test|live>
+#
+# Environment:
+#   CI_SHA                  commit to test (default: `git rev-parse HEAD` in this checkout)
+#   WINDEV_SSH_KEY          private key PEM (Gitea secret). If empty, falls back to the
+#                           runner's default ssh identity/agent (used for local hand-testing).
+#   WINDEV_SSH_KNOWN_HOSTS  pinned host keys (Gitea secret). If empty, uses the committed
+#                           scripts/ci/windev.known_hosts.
+#   WINDEV_SSH_USER         ssh user on windev (default: ci)
+#   WINDEV_SSH_HOST         windev host/IP (default: 10.100.0.48)
+#
+# The private key is never echoed (written 0600 to a temp file, removed on exit). git on
+# windev writes progress to stderr, so the remote bootstrap does NOT use -ErrorActionPreference
+# Stop around git — it checks $LASTEXITCODE instead (known windev gotcha).
+
+set -euo pipefail
+
+MODE="${1:-}"
+case "$MODE" in
+  build|test|live) ;;
+  *) echo "usage: $0 <build|test|live>" >&2; exit 2 ;;
+esac
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CI_SHA="${CI_SHA:-$(git -C "$SCRIPT_DIR" rev-parse HEAD)}"
+WINDEV_SSH_USER="${WINDEV_SSH_USER:-ci}"
+WINDEV_SSH_HOST="${WINDEV_SSH_HOST:-10.100.0.48}"
+REMOTE_CLONE='C:\build\mxaccessgw-ci'
+REMOTE_SCRIPT='scripts\ci\windev-worker-ci.ps1'
+
+# Validate the SHA shape before it reaches a remote shell command.
+if ! printf '%s' "$CI_SHA" | grep -Eq '^[0-9a-fA-F]{7,40}$'; then
+  echo "run-windev-ci: refusing to run — CI_SHA '$CI_SHA' is not a hex commit id" >&2
+  exit 2
+fi
+
+WORK="$(mktemp -d)"
+cleanup() { rm -rf "$WORK"; }
+trap cleanup EXIT
+
+SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=15)
+
+# known_hosts: secret if provided, else the committed pins.
+KNOWN_HOSTS="$WORK/known_hosts"
+if [ -n "${WINDEV_SSH_KNOWN_HOSTS:-}" ]; then
+  printf '%s\n' "$WINDEV_SSH_KNOWN_HOSTS" > "$KNOWN_HOSTS"
+else
+  cp "$SCRIPT_DIR/windev.known_hosts" "$KNOWN_HOSTS"
+fi
+SSH_OPTS+=(-o "UserKnownHostsFile=$KNOWN_HOSTS")
+
+# Private key: secret if provided, else fall back to the default identity/agent.
+if [ -n "${WINDEV_SSH_KEY:-}" ]; then
+  KEY="$WORK/id_ci"
+  ( umask 077; printf '%s\n' "$WINDEV_SSH_KEY" > "$KEY" )
+  SSH_OPTS+=(-o IdentitiesOnly=yes -i "$KEY")
+fi
+
+echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}"
+
+# Remote PowerShell bootstrap: fetch + checkout enough to load the versioned CI script, then
+# hand off to it. Not Stop around git (stderr progress). windev-worker-ci.ps1 re-fetches and
+# re-checks-out under a lock, so this checkout only needs to load the right script version.
+read -r -d '' BOOTSTRAP <<PS || true
+\$ErrorActionPreference = 'Continue'
+Set-Location '$REMOTE_CLONE'
+git fetch --prune origin
+git checkout --force --detach $CI_SHA
+if (\$LASTEXITCODE -ne 0) { Write-Error "bootstrap checkout failed"; exit \$LASTEXITCODE }
+powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -File '$REMOTE_SCRIPT' -Sha '$CI_SHA' -Mode '$MODE'
+exit \$LASTEXITCODE
+PS
+
+# PowerShell -EncodedCommand wants UTF-16LE base64 on one line (avoids all ssh/quoting hazards).
+ENCODED="$(printf '%s' "$BOOTSTRAP" | iconv -t UTF-16LE | base64 | tr -d '\n')"
+
+set +e
+ssh "${SSH_OPTS[@]}" "${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}" \
+  "powershell -NoProfile -NonInteractive -EncodedCommand $ENCODED"
+RC=$?
+set -e
+
+echo "run-windev-ci: remote exit code $RC"
+exit $RC
@@ -0,0 +1,147 @@
+<#
+.SYNOPSIS
+    Windows/x86 Worker CI stage — runs ON windev (10.100.0.48), invoked over SSH by
+    scripts/ci/run-windev-ci.sh at the exact SHA under test. Restores the x86/net48
+    Worker test tier that the removed `windows` CI job used to cover (TST-25).
+
+.DESCRIPTION
+    The Linux `windows-x86` / nightly CI jobs cannot build the x86/net48 Worker, so they
+    ssh here and run this script inside the isolated CI clone C:\build\mxaccessgw-ci.
+
+    The script is deliberately self-contained and versioned with the code under test: the
+    SSH bootstrap only fetches + checks out the SHA (enough to load *this* file), then this
+    script re-fetches and re-checks-out $Sha INSIDE the worktree lock so the checkout+build
+    are fully serialized against a concurrent run and can never stomp each other's tree.
+
+    Modes (cumulative):
+      build  x86 Worker build only (net48/CS0246/x86 compile guard — always fast).
+      test   build + Worker.Tests (x86). The per-push default.
+      live   test + the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full
+             slnx build (surfaces Windows-only analyzer diagnostics, e.g. xUnit1030). Nightly.
+
+    Runs under Windows PowerShell 5.1 (`powershell.exe`), so it avoids PS7-only syntax and
+    checks $LASTEXITCODE after every native command rather than relying on stderr-as-error
+    (git writes progress to stderr — the known windev gotcha).
+
+    Exits nonzero on the first failure so the SSH exit code propagates to the CI job.
+
+.PARAMETER Sha
+    The commit SHA under test. HEAD is force-checked-out to it and verified.
+
+.PARAMETER Mode
+    build | test | live.
+
+.PARAMETER RepoRoot
+    The CI clone root. Defaults to C:\build\mxaccessgw-ci.
+#>
+[CmdletBinding()]
+param(
+    [Parameter(Mandatory = $true)][string]$Sha,
+    [Parameter(Mandatory = $true)][ValidateSet('build', 'test', 'live')][string]$Mode,
+    [string]$RepoRoot = 'C:\build\mxaccessgw-ci'
+)
+
+Set-StrictMode -Version Latest
+$ErrorActionPreference = 'Stop'
+
+$lockDir = "$RepoRoot.lock"
+$logFile = Join-Path $RepoRoot 'ci.log'
+$lockTimeout = [TimeSpan]::FromMinutes(20)
+
+function Write-Log {
+    param([string]$Message)
+    $line = ('[{0}] {1}' -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Message)
+    Write-Host $line
+    # Best-effort file log for the "is the tier down?" diagnosis in docs/GatewayTesting.md.
+    try { Add-Content -Path $logFile -Value $line -ErrorAction SilentlyContinue } catch { }
+}
+
+# Run a native command and throw on nonzero exit. Native stderr (git progress) does NOT
+# set $LASTEXITCODE, so it is not treated as failure — only a real nonzero exit is.
+function Invoke-Native {
+    param([Parameter(Mandatory = $true)][string]$What, [Parameter(Mandatory = $true)][scriptblock]$Command)
+    Write-Log "RUN: $What"
+    & $Command
+    if ($LASTEXITCODE -ne 0) {
+        throw "$What failed with exit code $LASTEXITCODE"
+    }
+}
+
+# --- acquire the worktree lock (mkdir is atomic; retry until timeout) ---
+$deadline = (Get-Date).Add($lockTimeout)
+$haveLock = $false
+Write-Log "Acquiring worktree lock $lockDir (mode=$Mode sha=$Sha)"
+while (-not $haveLock) {
+    try {
+        New-Item -Path $lockDir -ItemType Directory -ErrorAction Stop | Out-Null
+        $haveLock = $true
+    }
+    catch {
+        if ((Get-Date) -ge $deadline) {
+            throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockDir. A stuck run may have left it; if no CI job is active, remove it by hand."
+        }
+        Start-Sleep -Seconds 10
+    }
+}
+
+try {
+    Push-Location $RepoRoot
+    try {
+        # Re-fetch + re-checkout under the lock so a concurrent run's bootstrap checkout
+        # (done before we held the lock) cannot leave us on the wrong tree.
+        Invoke-Native 'git fetch origin' { git fetch --prune origin }
+        Invoke-Native "git checkout --force $Sha" { git checkout --force --detach $Sha }
+        # Drop any stray tracked/untracked residue from a prior run so the build is clean.
+        Invoke-Native 'git reset --hard' { git reset --hard $Sha }
+        Invoke-Native 'git clean -fdx (build outputs)' { git clean -fdx -e ci.log }
+
+        $head = (& git rev-parse HEAD).Trim()
+        if ($head -ne $Sha) {
+            throw "Worktree HEAD $head does not match requested SHA $Sha after checkout."
+        }
+        Write-Log "Worktree at $head"
+
+        $workerProj = 'src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj'
+        $workerTests = 'src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj'
+        $integrationTests = 'src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj'
+
+        # build: always. The x86/net48 compile is the floor guard (catches CS0246 on
+        # unregenerated Generated/, x86-only breaks) even when the test step is demoted.
+        Invoke-Native 'x86 Worker build' { dotnet build $workerProj -c Release -p:Platform=x86 }
+
+        if ($Mode -eq 'test' -or $Mode -eq 'live') {
+            Invoke-Native 'x86 Worker.Tests' { dotnet test $workerTests -c Release -p:Platform=x86 }
+        }
+
+        if ($Mode -eq 'live') {
+            # Full-solution build surfaces Windows-only analyzer diagnostics (xUnit1030 etc.)
+            # that never fire on the macOS/Linux NonWindows build.
+            Invoke-Native 'full slnx build' { dotnet build src/ZB.MOM.WW.MxGateway.slnx -c Release }
+
+            $env:MXGATEWAY_RUN_LIVE_MXACCESS_TESTS = '1'
+            try {
+                Invoke-Native 'live-MXAccess smoke' {
+                    dotnet test $integrationTests -c Release --filter 'FullyQualifiedName~WorkerLiveMxAccessSmokeTests'
+                }
+            }
+            finally {
+                Remove-Item Env:\MXGATEWAY_RUN_LIVE_MXACCESS_TESTS -ErrorAction SilentlyContinue
+            }
+        }
+
+        Write-Log "SUCCESS mode=$Mode sha=$Sha"
+    }
+    finally {
+        Pop-Location
+    }
+}
+catch {
+    Write-Log "FAILURE mode=$Mode sha=$Sha : $($_.Exception.Message)"
+    throw
+}
+finally {
+    if ($haveLock) {
+        Remove-Item -Path $lockDir -Recurse -Force -ErrorAction SilentlyContinue
+        Write-Log "Released worktree lock $lockDir"
+    }
+}
@@ -0,0 +1,12 @@
+# Pinned SSH host keys for the windev CI target (10.100.0.48), used by the
+# `windows-x86` / nightly jobs so `StrictHostKeyChecking=yes` can verify the host
+# without an interactive TOFU prompt. Host keys are public data — committing them is
+# intentional (greppable, versioned) per TST-25's design.
+#
+# Regenerate if windev's host keys rotate:
+#   ssh-keyscan -t ed25519,rsa,ecdsa 10.100.0.48 > scripts/ci/windev.known_hosts
+# (then re-add this header). run-windev-ci.sh prefers the WINDEV_SSH_KNOWN_HOSTS
+# secret when set and falls back to this file.
+10.100.0.48 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHRqxpRq/G0BR5vAomHMNc/H7eyTkdqliHNrFlMQ9HCb
+10.100.0.48 ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIvTHAgf7Q+rK2dCyjSUgkYgos4j2eqyJN/AAfs2TXGJgrpp9AfOMnkbPtmMX3CvsrPoZxzBttIYcBYbf9KXb98JAu6fWqeglI/CeXH4oHw6oqMRtM0K2hl2gm3DicKKCQ==
+10.100.0.48 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxKOhjdVawf3ziO9MB4VfLA1zDwnNK028siIzjTHZec4uJumInvSFD5S3tkJvSKGfCzXHtLu9m0zHmvJ4gIvLx4gt60TPABPC/QrdRG8mP4uOlPCciukWNIBhMWFU6oxHpZlQgrjQ3a+6WZxIjZBtaZQt3S5f9zEvOu6NUrLmP9cmrrXtAByMYyjISQf2BswxZ5ugeDHn0rVo3p3qYZ3M8lbXV+CRTwKj0I+VVceffJrYwLT0LyWxi0tDxmHxK1PgkhcIjLDWKhk4UxOA6gdO6MjuHvXZv5+UXJzjgGxom+xqxGhAge0F66p5IEJ/xrJMWoPBtFQ1by7tDZAsRhcbQJVeCbBjEDv7ShCbyh2Gq+PfJhWVVaYnMKcqBWkhPrZH4BE9yhAtSlV8QZsZ1iYUYKNAt9Kcg3ugRYx414KUf2fvUWSuoaRkJnBqQq1Mi0yK1JtTZOoYvN4XEYIr56hXQG1RfA8xWqYzjmbHYTgO8zv6tJJTeXLMqubmDdLP9H+U=