From d769244ac070f028317f780655691132e8fdc422 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 13 Jul 2026 10:02:26 -0400 Subject: [PATCH 1/2] fix(TST-25,TST-26): restore Windows/x86 CI tier via SSH-driven windev job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `windows`/`live-mxaccess` CI jobs were removed in abb0930 because Gitea act_runner host-mode on Windows is broken and a runs-on gate with no runner wedges the queue. This left the entire x86/net48 Worker + Worker.Tests tier (and the live-MXAccess smoke) unguarded by CI. Restore it without a native Windows runner: Linux jobs on ubuntu-latest (which always schedule) SSH to windev (10.100.0.48), fetch+checkout the SHA under test in an isolated clone C:\build\mxaccessgw-ci, run the x86 build/tests there, and propagate the remote exit code back — so a Worker regression turns the job red and an unreachable host fails loud (never stuck). - scripts/ci/run-windev-ci.sh: Linux driver (key/known-hosts, UTF-16LE base64 EncodedCommand bootstrap, ssh, exit-code passthrough). - scripts/ci/windev-worker-ci.ps1: windev stage — worktree lock, re-fetch/ checkout under lock, build|test|live modes; PS 5.1-safe, checks $LASTEXITCODE. - scripts/ci/windev.known_hosts: pinned host keys for StrictHostKeyChecking. - scripts/ci/README.md: operator bring-up + acceptance checklist. - ci.yml: windows-x86 (per-push `test`) + nightly-windev (scheduled `live` + on-failure Gitea issue); drop the removal note; fix header/cron comments. TST-26 (same commit, by rule): correct docs/scripts that still describe the removed jobs — GatewayTesting.md, Contracts.md, check-codegen.ps1 — and reattribute the Generated/-commit guard (primary = check-codegen diff in the portable job; secondary = windows-x86 net48 compile). Mechanism hand-verified on windev: build->0, bogus-SHA->nonzero (lock released), test->356 passed/0 failed in ~50s, ssh exit-code propagation confirmed. Merge requires operator bring-up first (dedicated ci@ key + Gitea secrets) or the per-push job is red every push — see scripts/ci/README.md. --- .gitea/workflows/ci.yml | 68 ++++++-- .../2026-07-12/remediation/00-tracking.md | 5 +- .../remediation/60-testing-docs-gaps.md | 4 +- docs/Contracts.md | 5 +- docs/GatewayTesting.md | 36 ++++- scripts/check-codegen.ps1 | 3 +- scripts/ci/README.md | 58 +++++++ scripts/ci/run-windev-ci.sh | 94 +++++++++++ scripts/ci/windev-worker-ci.ps1 | 147 ++++++++++++++++++ scripts/ci/windev.known_hosts | 12 ++ 10 files changed, 405 insertions(+), 27 deletions(-) create mode 100644 scripts/ci/README.md create mode 100755 scripts/ci/run-windev-ci.sh create mode 100644 scripts/ci/windev-worker-ci.ps1 create mode 100644 scripts/ci/windev.known_hosts diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 3896008..af48fa5 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -2,15 +2,19 @@ # # Scope note: the x86 Worker (src/ZB.MOM.WW.MxGateway.Worker) and Worker.Tests target # .NET Framework 4.8 / x86 and need MXAccess COM installed, so they build ONLY on a Windows host. -# They are out of scope for this Linux `portable` job — the `windows` job below (self-hosted windev -# runner) covers them, and live-MXAccess integration runs are scheduled-only so they never gate a push. +# They are out of scope for this Linux `portable` job — the `windows-x86` job below builds and +# tests them on windev (10.100.0.48) over SSH (see scripts/ci/run-windev-ci.sh), and the nightly +# `nightly-windev` job additionally runs the live-MXAccess smoke, which is scheduled-only so it +# never gates a push. name: ci on: push: pull_request: schedule: - # Nightly live-MXAccess smoke (windev). Push/PR runs skip the live-mxaccess job. + # 06:00 UTC nightly: the `nightly-windev` job re-runs the x86 Worker build + full Worker.Tests on + # windev, then the live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1) and a full-slnx build. + # Push/PR runs skip it — the job is gated `if: github.event_name == 'schedule'`. - cron: '0 6 * * *' jobs: @@ -120,12 +124,52 @@ jobs: - name: Verify generated tree is clean run: git diff --exit-code -- clients/java/src/main/generated - # NOTE: there is intentionally no `windows` or `live-mxaccess` job here. - # The x86 / net48 Worker and live-MXAccess tests need a Windows host, but Gitea's act_runner - # v0.6.1 host-mode on Windows is broken (its hostexecutor writes each step's script to a path it - # then cannot resolve, independent of shell; it also cannot stage JS actions), and Windows - # containers are impractical for the net48/x86/MXAccess Worker. A gated job that no runner can - # satisfy sticks in "queued" forever (it only resolves to "skipped" when a matching runner - # exists), so keeping it here blocks every run from completing. x86 Worker + live-MXAccess - # verification therefore stays on the manual windev worktree process (see CLAUDE.md / - # docs/GatewayTesting.md). Restore these jobs only alongside a Windows runner that actually works. + windows-x86: + # TST-25: x86 / net48 Worker build + Worker.Tests, executed on windev (10.100.0.48) over SSH. + # Runs on a Linux runner so it ALWAYS schedules — act_runner host-mode on Windows is broken (its + # hostexecutor writes each step's script to a path it cannot resolve, and it cannot stage JS + # actions), and a `runs-on` gate with no matching runner wedges the queue in "queued" forever. + # An unreachable windev fails this job RED (loud), never leaves it stuck. A red job can mean the + # Windows tier is down rather than the change — see docs/GatewayTesting.md § Continuous Integration. + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Worker x86 build + tests on windev + env: + WINDEV_SSH_KEY: ${{ secrets.WINDEV_SSH_KEY }} + WINDEV_SSH_KNOWN_HOSTS: ${{ secrets.WINDEV_SSH_KNOWN_HOSTS }} + WINDEV_SSH_USER: ${{ vars.WINDEV_SSH_USER }} + CI_SHA: ${{ github.sha }} + # `test` = x86 build + Worker.Tests (~50s measured on windev). Demote to `build` only if the + # test step ever exceeds ~10 min; the x86 build alone still guards the net48/CS0246/x86 class. + run: ./scripts/ci/run-windev-ci.sh test + + nightly-windev: + # TST-25: scheduled deep run — full Worker.Tests + live-MXAccess smoke + full-slnx build on windev. + # Gated to the schedule event via `if:` (not an unsatisfiable `runs-on`), so push/PR runs mark it + # skipped, not queued. Never gates a push. On failure it opens a Gitea issue so a red nightly is + # visible even though nobody watches the Actions page. + if: github.event_name == 'schedule' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Full Worker.Tests + live-MXAccess smoke on windev + env: + WINDEV_SSH_KEY: ${{ secrets.WINDEV_SSH_KEY }} + WINDEV_SSH_KNOWN_HOSTS: ${{ secrets.WINDEV_SSH_KNOWN_HOSTS }} + WINDEV_SSH_USER: ${{ vars.WINDEV_SSH_USER }} + CI_SHA: ${{ github.sha }} + run: ./scripts/ci/run-windev-ci.sh live + - name: Report a red nightly as a Gitea issue + if: failure() + run: | + curl -sSf -X POST \ + -H "Authorization: token ${{ github.token }}" \ + -H "Content-Type: application/json" \ + "${{ github.server_url }}/api/v1/repos/${{ github.repository }}/issues" \ + -d "{\"title\":\"nightly-windev failed on ${{ github.sha }}\",\"body\":\"The scheduled windev Worker + live-MXAccess run failed: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} . A red nightly may mean the Windows tier is down rather than the change — see docs/GatewayTesting.md (Continuous Integration).\"}" + + # NOTE: there is intentionally no native `windows` runner job. act_runner v0.6.1 host-mode on + # Windows is broken and Windows containers are impractical for the net48/x86/MXAccess Worker, so the + # Windows tier runs via the SSH-driven `windows-x86` (per-push) and `nightly-windev` (scheduled) + # jobs above. Restore a native job only alongside a Windows runner that actually works. diff --git a/archreview/2026-07-12/remediation/00-tracking.md b/archreview/2026-07-12/remediation/00-tracking.md index 9babfef..050f512 100644 --- a/archreview/2026-07-12/remediation/00-tracking.md +++ b/archreview/2026-07-12/remediation/00-tracking.md @@ -126,8 +126,8 @@ Full design + implementation for each row lives in the linked domain doc under i | ID | Sev | Tier | Eff | Dep | Status | Title | |---|---|:-:|:-:|---|---|---| -| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | Not started | Windows/x86 test tier has zero automation — SSH-driven windev CI job | -| TST-26 | Medium | P1 | S | TST-25 (same commit) | Not started | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen | +| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | In progress | Windows/x86 test tier has zero automation — SSH-driven windev CI job | +| TST-26 | Medium | P1 | S | TST-25 (same commit) | In progress | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen | | TST-27 | Medium | P1 | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-28 | Low | P2 | S | relates IPC-02 (old) | Not started | Gateway-side `max_frame_bytes` handshake untested in the CI-run suite | | TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` (fold Phase-5 governance into DesignDecisions.md); delete root artifacts | @@ -156,3 +156,4 @@ Sequence these together rather than piecemeal — several are one change set spa | Date | Change | |---|---| | 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). | +| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. **Pending merge:** operator bring-up (dedicated `ci@` ed25519 key on windev + `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` Gitea secrets), then a green `windows-x86` run; then close old TST-05/TST-24. | diff --git a/archreview/2026-07-12/remediation/60-testing-docs-gaps.md b/archreview/2026-07-12/remediation/60-testing-docs-gaps.md index 3a2359d..7810379 100644 --- a/archreview/2026-07-12/remediation/60-testing-docs-gaps.md +++ b/archreview/2026-07-12/remediation/60-testing-docs-gaps.md @@ -10,8 +10,8 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior | ID | Sev | Tier | Eff | Dep | Status | Title | |----|-----|------|-----|-----|--------|-------| -| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | Not started | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job | -| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | Not started | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs | +| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | In progress | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job | +| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | In progress | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs | | TST-27 | Medium | P1 (doc batch) | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-28 | Low | P2 | S | relates IPC-02 | Not started | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite | | TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts | diff --git a/docs/Contracts.md b/docs/Contracts.md index f0ee3d2..f6ce3db 100644 --- a/docs/Contracts.md +++ b/docs/Contracts.md @@ -112,8 +112,9 @@ dotnet build src/ZB.MOM.WW.MxGateway.Contracts/ZB.MOM.WW.MxGateway.Contracts.csp ``` `scripts/check-codegen.ps1` enforces this in CI (it force-regenerates and fails on any -`git diff` against the committed `Generated/`); the Windows CI job's net48 worker build is the -definitive guard. +`git diff` against the committed `Generated/`) — that regeneration diff in the `portable` +job is the primary guard. The SSH-driven `windows-x86` job's net48 worker build is the +secondary guard (a stale `Generated/` also breaks the x86 build with `CS0246`). Client generation inputs are published through `clients/proto/proto-inputs.json` and the descriptor set under diff --git a/docs/GatewayTesting.md b/docs/GatewayTesting.md index 6743272..89860d3 100644 --- a/docs/GatewayTesting.md +++ b/docs/GatewayTesting.md @@ -423,18 +423,38 @@ runtime because the x86 Worker cannot build on Linux: `MxaccessGateway.java` with spurious protobuf-runtime-version churn on every build, so when no `.proto` changed the job reverts that one file (`git checkout`) before asserting the generated tree is clean. The dev Mac has no JRE, so Java verification is CI-only. -- **`windows`** (self-hosted windev runner) — the only host that builds the **x86 / - net48 Worker and Worker.Tests**; these are Windows-only and out of scope for the Linux - jobs. This job also proves the net48 build against the committed `Generated/` (the - definitive guard for the "regenerate and commit `Generated/`" rule). -- **`live-mxaccess`** (self-hosted, MXAccess-installed) — scheduled only - (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`); never gates a push. +- **`windows-x86`** (Linux runner, per push/PR) — builds the **x86 / net48 Worker and + Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a + Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where + `scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci` + to the SHA under test and runs `scripts/ci/windev-worker-ci.ps1 -Mode test` there; the + remote exit code propagates back, so a Worker regression turns the job red. A native + Windows runner is not used because act_runner host-mode on Windows is broken and a + `runs-on` gate with no runner wedges the queue forever. **A red `windows-x86` can mean + the tier is down, not the change**: an unreachable windev fails the job (never skips), so + if it goes red without a related code change, check the tier — `ssh windev`, read + `C:\build\mxaccessgw-ci\ci.log` — before assuming a real test failure. +- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — re-runs the full x86 build + + Worker.Tests on windev, then the live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`) + and a full `slnx` build (which surfaces Windows-only analyzer diagnostics such as + `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On + failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page. The freshness guard `scripts/check-codegen.ps1` fails the build when the committed client descriptor set or the C# `Generated/` no longer matches the current `.proto` sources — the codegen drift class this repo has hit repeatedly (stale client -descriptors, net48 `CS0246` on unregenerated protos). See -[Client Proto Generation](./ClientProtoGeneration.md) and [Contracts](./Contracts.md). +descriptors, net48 `CS0246` on unregenerated protos). The **primary** guard for the +"regenerate and commit `Generated/`" rule is that regeneration diff in the `portable` job; +the `windows-x86` net48 compile is the **secondary** guard (a stale `Generated/` also breaks +the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGeneration.md) and +[Contracts](./Contracts.md). + +If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI +key/secret rotation in flight), fall back to the manual windev worktree procedure as a +degraded mode: on windev, fast-forward an isolated `origin/main` worktree under `C:\build` +(never the dirty Desktop checkout), then run the x86 Worker build and `Worker.Tests` +(`-p:Platform=x86`) there by hand. Do this per merge for worker-touching changes until the +`windows-x86` job is green again. ## Related Documentation diff --git a/scripts/check-codegen.ps1 b/scripts/check-codegen.ps1 index 1035b24..d7e4177 100644 --- a/scripts/check-codegen.ps1 +++ b/scripts/check-codegen.ps1 @@ -14,7 +14,8 @@ # protos. A drift means a .proto was edited without refreshing the vendored copies, which would # publish a stale wire contract to crate consumers while the in-repo build stays correct. # -# The x86 Worker + Worker.Tests are Windows-only and are guarded by the Windows CI job, not here. +# The x86 Worker + Worker.Tests are Windows-only and are guarded by the SSH-driven `windows-x86` +# CI job (see docs/GatewayTesting.md, Continuous Integration), not here. [CmdletBinding()] param() diff --git a/scripts/ci/README.md b/scripts/ci/README.md new file mode 100644 index 0000000..a331342 --- /dev/null +++ b/scripts/ci/README.md @@ -0,0 +1,58 @@ +# Windows/x86 CI tier (TST-25) + +The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows +`act_runner` is broken (host-mode writes each step's script to a path it can't resolve; a +`runs-on` gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to +windev (`10.100.0.48`) and runs the Worker build/tests there, propagating the remote exit +code back so a regression turns the job red. + +## Pieces + +| File | Runs on | Role | +|---|---|---| +| `run-windev-ci.sh ` | Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. | +| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, runs the x86 build / `Worker.Tests` / live smoke. | +| `windev.known_hosts` | committed | Pinned windev host keys so `StrictHostKeyChecking=yes` has no TOFU prompt. Host keys are public; committing is intentional. | + +Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests`; +`live` = test + full-`slnx` build + live-MXAccess smoke (`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). + +CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR; `nightly-windev` +runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on failure. + +## One-time bring-up (operator — required before this lands on `main`) + +Until these are done, `windows-x86` is **red on every push** (no key → SSH fails). Do them +first, then merge. + +1. **CI clone** — `C:\build\mxaccessgw-ci`, a fresh clone of the Gitea origin (done during + TST-25 verification; recreate with `git clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ci` if missing). Never the dirty Desktop checkout. +2. **Dedicated CI key** — generate a NEW ed25519 keypair for CI (do not reuse the operator's + personal windev key): `ssh-keygen -t ed25519 -f ci_windev -N ''`. Append `ci_windev.pub` + to the windev CI account's `authorized_keys` (optionally with a forced `command=` limiting + it to this bootstrap). +3. **Gitea repo secrets** — store the private key as `WINDEV_SSH_KEY` and, optionally, the + pinned host keys as `WINDEV_SSH_KNOWN_HOSTS` (the committed `windev.known_hosts` is the + fallback). `run-windev-ci.sh` connects as user `ci` by default (`WINDEV_SSH_USER` to + override); ensure that account exists on windev, or set `WINDEV_SSH_USER` to the intended one. +4. **Network precheck** — confirm the Gitea runner container (on the `traefik` docker network, + host `10.100.0.35`) can reach `10.100.0.48:22`: a one-off `nc -z -w5 10.100.0.48 22` step. + The network was created for gitea name resolution, not LAN egress, so verify. +5. **Actions token** — the `nightly-windev` on-failure step creates a Gitea issue with the + built-in token; confirm Actions tokens have issue-write on this repo. + +## Verifying it end-to-end (design's acceptance checks) + +- Push a branch → `portable`, `java`, `windows-x86` all green; windev's clone sits at the pushed SHA. +- Deliberate red: add a failing `[Fact]` to `Worker.Tests`, push, confirm `windows-x86` goes + red; revert. (Proves exit-code propagation — the whole point.) +- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs. +- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock. +- Nightly: trigger the schedule path, confirm `live` runs and a forced failure opens an issue. +- Confirm no key material appears in job logs. + +## Degraded mode + +If the tier is down for infra reasons, fall back to the manual windev worktree procedure +(isolated `C:\build` worktree, `dotnet build`/`test` `-p:Platform=x86` by hand) per merge for +worker-touching changes until `windows-x86` is green again. See `docs/GatewayTesting.md`. diff --git a/scripts/ci/run-windev-ci.sh b/scripts/ci/run-windev-ci.sh new file mode 100755 index 0000000..e899a03 --- /dev/null +++ b/scripts/ci/run-windev-ci.sh @@ -0,0 +1,94 @@ +#!/usr/bin/env bash +# +# TST-25: Linux side of the Windows/x86 Worker CI tier. +# +# Runs in the `windows-x86` / nightly Gitea jobs on a Linux runner (which always schedules, +# unlike a broken Windows act_runner). SSHes to windev (10.100.0.48), where it fetches + +# checks out the SHA under test in the isolated CI clone C:\build\mxaccessgw-ci and runs +# scripts/ci/windev-worker-ci.ps1 there. The remote exit code propagates through +# ssh -> this script -> the CI step, so a Worker regression turns the job red. +# +# Usage: run-windev-ci.sh +# +# Environment: +# CI_SHA commit to test (default: `git rev-parse HEAD` in this checkout) +# WINDEV_SSH_KEY private key PEM (Gitea secret). If empty, falls back to the +# runner's default ssh identity/agent (used for local hand-testing). +# WINDEV_SSH_KNOWN_HOSTS pinned host keys (Gitea secret). If empty, uses the committed +# scripts/ci/windev.known_hosts. +# WINDEV_SSH_USER ssh user on windev (default: ci) +# WINDEV_SSH_HOST windev host/IP (default: 10.100.0.48) +# +# The private key is never echoed (written 0600 to a temp file, removed on exit). git on +# windev writes progress to stderr, so the remote bootstrap does NOT use -ErrorActionPreference +# Stop around git — it checks $LASTEXITCODE instead (known windev gotcha). + +set -euo pipefail + +MODE="${1:-}" +case "$MODE" in + build|test|live) ;; + *) echo "usage: $0 " >&2; exit 2 ;; +esac + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CI_SHA="${CI_SHA:-$(git -C "$SCRIPT_DIR" rev-parse HEAD)}" +WINDEV_SSH_USER="${WINDEV_SSH_USER:-ci}" +WINDEV_SSH_HOST="${WINDEV_SSH_HOST:-10.100.0.48}" +REMOTE_CLONE='C:\build\mxaccessgw-ci' +REMOTE_SCRIPT='scripts\ci\windev-worker-ci.ps1' + +# Validate the SHA shape before it reaches a remote shell command. +if ! printf '%s' "$CI_SHA" | grep -Eq '^[0-9a-fA-F]{7,40}$'; then + echo "run-windev-ci: refusing to run — CI_SHA '$CI_SHA' is not a hex commit id" >&2 + exit 2 +fi + +WORK="$(mktemp -d)" +cleanup() { rm -rf "$WORK"; } +trap cleanup EXIT + +SSH_OPTS=(-o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=15) + +# known_hosts: secret if provided, else the committed pins. +KNOWN_HOSTS="$WORK/known_hosts" +if [ -n "${WINDEV_SSH_KNOWN_HOSTS:-}" ]; then + printf '%s\n' "$WINDEV_SSH_KNOWN_HOSTS" > "$KNOWN_HOSTS" +else + cp "$SCRIPT_DIR/windev.known_hosts" "$KNOWN_HOSTS" +fi +SSH_OPTS+=(-o "UserKnownHostsFile=$KNOWN_HOSTS") + +# Private key: secret if provided, else fall back to the default identity/agent. +if [ -n "${WINDEV_SSH_KEY:-}" ]; then + KEY="$WORK/id_ci" + ( umask 077; printf '%s\n' "$WINDEV_SSH_KEY" > "$KEY" ) + SSH_OPTS+=(-o IdentitiesOnly=yes -i "$KEY") +fi + +echo "run-windev-ci: mode=$MODE sha=$CI_SHA target=${WINDEV_SSH_USER}@${WINDEV_SSH_HOST}" + +# Remote PowerShell bootstrap: fetch + checkout enough to load the versioned CI script, then +# hand off to it. Not Stop around git (stderr progress). windev-worker-ci.ps1 re-fetches and +# re-checks-out under a lock, so this checkout only needs to load the right script version. +read -r -d '' BOOTSTRAP < +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$Sha, + [Parameter(Mandatory = $true)][ValidateSet('build', 'test', 'live')][string]$Mode, + [string]$RepoRoot = 'C:\build\mxaccessgw-ci' +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$lockDir = "$RepoRoot.lock" +$logFile = Join-Path $RepoRoot 'ci.log' +$lockTimeout = [TimeSpan]::FromMinutes(20) + +function Write-Log { + param([string]$Message) + $line = ('[{0}] {1}' -f (Get-Date -Format 'yyyy-MM-dd HH:mm:ss'), $Message) + Write-Host $line + # Best-effort file log for the "is the tier down?" diagnosis in docs/GatewayTesting.md. + try { Add-Content -Path $logFile -Value $line -ErrorAction SilentlyContinue } catch { } +} + +# Run a native command and throw on nonzero exit. Native stderr (git progress) does NOT +# set $LASTEXITCODE, so it is not treated as failure — only a real nonzero exit is. +function Invoke-Native { + param([Parameter(Mandatory = $true)][string]$What, [Parameter(Mandatory = $true)][scriptblock]$Command) + Write-Log "RUN: $What" + & $Command + if ($LASTEXITCODE -ne 0) { + throw "$What failed with exit code $LASTEXITCODE" + } +} + +# --- acquire the worktree lock (mkdir is atomic; retry until timeout) --- +$deadline = (Get-Date).Add($lockTimeout) +$haveLock = $false +Write-Log "Acquiring worktree lock $lockDir (mode=$Mode sha=$Sha)" +while (-not $haveLock) { + try { + New-Item -Path $lockDir -ItemType Directory -ErrorAction Stop | Out-Null + $haveLock = $true + } + catch { + if ((Get-Date) -ge $deadline) { + throw "Timed out after $($lockTimeout.TotalMinutes) min waiting for worktree lock $lockDir. A stuck run may have left it; if no CI job is active, remove it by hand." + } + Start-Sleep -Seconds 10 + } +} + +try { + Push-Location $RepoRoot + try { + # Re-fetch + re-checkout under the lock so a concurrent run's bootstrap checkout + # (done before we held the lock) cannot leave us on the wrong tree. + Invoke-Native 'git fetch origin' { git fetch --prune origin } + Invoke-Native "git checkout --force $Sha" { git checkout --force --detach $Sha } + # Drop any stray tracked/untracked residue from a prior run so the build is clean. + Invoke-Native 'git reset --hard' { git reset --hard $Sha } + Invoke-Native 'git clean -fdx (build outputs)' { git clean -fdx -e ci.log } + + $head = (& git rev-parse HEAD).Trim() + if ($head -ne $Sha) { + throw "Worktree HEAD $head does not match requested SHA $Sha after checkout." + } + Write-Log "Worktree at $head" + + $workerProj = 'src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj' + $workerTests = 'src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj' + $integrationTests = 'src/ZB.MOM.WW.MxGateway.IntegrationTests/ZB.MOM.WW.MxGateway.IntegrationTests.csproj' + + # build: always. The x86/net48 compile is the floor guard (catches CS0246 on + # unregenerated Generated/, x86-only breaks) even when the test step is demoted. + Invoke-Native 'x86 Worker build' { dotnet build $workerProj -c Release -p:Platform=x86 } + + if ($Mode -eq 'test' -or $Mode -eq 'live') { + Invoke-Native 'x86 Worker.Tests' { dotnet test $workerTests -c Release -p:Platform=x86 } + } + + if ($Mode -eq 'live') { + # Full-solution build surfaces Windows-only analyzer diagnostics (xUnit1030 etc.) + # that never fire on the macOS/Linux NonWindows build. + Invoke-Native 'full slnx build' { dotnet build src/ZB.MOM.WW.MxGateway.slnx -c Release } + + $env:MXGATEWAY_RUN_LIVE_MXACCESS_TESTS = '1' + try { + Invoke-Native 'live-MXAccess smoke' { + dotnet test $integrationTests -c Release --filter 'FullyQualifiedName~WorkerLiveMxAccessSmokeTests' + } + } + finally { + Remove-Item Env:\MXGATEWAY_RUN_LIVE_MXACCESS_TESTS -ErrorAction SilentlyContinue + } + } + + Write-Log "SUCCESS mode=$Mode sha=$Sha" + } + finally { + Pop-Location + } +} +catch { + Write-Log "FAILURE mode=$Mode sha=$Sha : $($_.Exception.Message)" + throw +} +finally { + if ($haveLock) { + Remove-Item -Path $lockDir -Recurse -Force -ErrorAction SilentlyContinue + Write-Log "Released worktree lock $lockDir" + } +} diff --git a/scripts/ci/windev.known_hosts b/scripts/ci/windev.known_hosts new file mode 100644 index 0000000..2aa651e --- /dev/null +++ b/scripts/ci/windev.known_hosts @@ -0,0 +1,12 @@ +# Pinned SSH host keys for the windev CI target (10.100.0.48), used by the +# `windows-x86` / nightly jobs so `StrictHostKeyChecking=yes` can verify the host +# without an interactive TOFU prompt. Host keys are public data — committing them is +# intentional (greppable, versioned) per TST-25's design. +# +# Regenerate if windev's host keys rotate: +# ssh-keyscan -t ed25519,rsa,ecdsa 10.100.0.48 > scripts/ci/windev.known_hosts +# (then re-add this header). run-windev-ci.sh prefers the WINDEV_SSH_KNOWN_HOSTS +# secret when set and falls back to this file. +10.100.0.48 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHRqxpRq/G0BR5vAomHMNc/H7eyTkdqliHNrFlMQ9HCb +10.100.0.48 ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIvTHAgf7Q+rK2dCyjSUgkYgos4j2eqyJN/AAfs2TXGJgrpp9AfOMnkbPtmMX3CvsrPoZxzBttIYcBYbf9KXb98JAu6fWqeglI/CeXH4oHw6oqMRtM0K2hl2gm3DicKKCQ== +10.100.0.48 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxKOhjdVawf3ziO9MB4VfLA1zDwnNK028siIzjTHZec4uJumInvSFD5S3tkJvSKGfCzXHtLu9m0zHmvJ4gIvLx4gt60TPABPC/QrdRG8mP4uOlPCciukWNIBhMWFU6oxHpZlQgrjQ3a+6WZxIjZBtaZQt3S5f9zEvOu6NUrLmP9cmrrXtAByMYyjISQf2BswxZ5ugeDHn0rVo3p3qYZ3M8lbXV+CRTwKj0I+VVceffJrYwLT0LyWxi0tDxmHxK1PgkhcIjLDWKhk4UxOA6gdO6MjuHvXZv5+UXJzjgGxom+xqxGhAge0F66p5IEJ/xrJMWoPBtFQ1by7tDZAsRhcbQJVeCbBjEDv7ShCbyh2Gq+PfJhWVVaYnMKcqBWkhPrZH4BE9yhAtSlV8QZsZ1iYUYKNAt9Kcg3ugRYx414KUf2fvUWSuoaRkJnBqQq1Mi0yK1JtTZOoYvN4XEYIr56hXQG1RfA8xWqYzjmbHYTgO8zv6tJJTeXLMqubmDdLP9H+U= From 09f2285957f18b95bd579dda977093fd5416783b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 13 Jul 2026 10:27:47 -0400 Subject: [PATCH 2/2] =?UTF-8?q?docs(TST-25,TST-26):=20mark=20Done=20?= =?UTF-8?q?=E2=80=94=20credentialed=20windows-x86=20green=20on=20d769244?= =?UTF-8?q?=20(run=20#37)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator bring-up complete and the SSH-driven Windows/x86 CI tier passed green end-to-end under Gitea Actions run #37: the Linux runner SSHed windev, checked out the SHA in the isolated C:\build\mxaccessgw-ci clone under lock, built the x86 Worker and ran Worker.Tests (exit 0); nightly-windev correctly skipped on the push event. Flip TST-25/TST-26 status rows to Done and record the bring-up + green-run evidence in the change log. --- archreview/2026-07-12/remediation/00-tracking.md | 7 ++++--- archreview/2026-07-12/remediation/60-testing-docs-gaps.md | 4 ++-- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/archreview/2026-07-12/remediation/00-tracking.md b/archreview/2026-07-12/remediation/00-tracking.md index 050f512..c5dbe44 100644 --- a/archreview/2026-07-12/remediation/00-tracking.md +++ b/archreview/2026-07-12/remediation/00-tracking.md @@ -126,8 +126,8 @@ Full design + implementation for each row lives in the linked domain doc under i | ID | Sev | Tier | Eff | Dep | Status | Title | |---|---|:-:|:-:|---|---|---| -| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | In progress | Windows/x86 test tier has zero automation — SSH-driven windev CI job | -| TST-26 | Medium | P1 | S | TST-25 (same commit) | In progress | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen | +| TST-25 | High | P1 | M | unlocks old TST-05, TST-24 | Done | Windows/x86 test tier has zero automation — SSH-driven windev CI job | +| TST-26 | Medium | P1 | S | TST-25 (same commit) | Done | Docs/scripts describe removed CI jobs; Generated/-guard reattributed to check-codegen | | TST-27 | Medium | P1 | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-28 | Low | P2 | S | relates IPC-02 (old) | Not started | Gateway-side `max_frame_bytes` handshake untested in the CI-run suite | | TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` (fold Phase-5 governance into DesignDecisions.md); delete root artifacts | @@ -156,4 +156,5 @@ Sequence these together rather than piecemeal — several are one change set spa | Date | Change | |---|---| | 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). | -| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. **Pending merge:** operator bring-up (dedicated `ci@` ed25519 key on windev + `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` Gitea secrets), then a green `windows-x86` run; then close old TST-05/TST-24. | +| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. | +| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. Negative-path acceptance checks (deliberate-red, unreachable-host, lock concurrency, forced-failure nightly issue) not yet run — happy-path only. | diff --git a/archreview/2026-07-12/remediation/60-testing-docs-gaps.md b/archreview/2026-07-12/remediation/60-testing-docs-gaps.md index 7810379..3e773cb 100644 --- a/archreview/2026-07-12/remediation/60-testing-docs-gaps.md +++ b/archreview/2026-07-12/remediation/60-testing-docs-gaps.md @@ -10,8 +10,8 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior | ID | Sev | Tier | Eff | Dep | Status | Title | |----|-----|------|-----|-----|--------|-------| -| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | In progress | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job | -| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | In progress | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs | +| TST-25 | High | P1 | M | — (unlocks TST-05, TST-24) | Done | Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job | +| TST-26 | Medium | P1 (folded into TST-25) | S | TST-25 | Done | docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs | | TST-27 | Medium | P1 (doc batch) | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-28 | Low | P2 | S | relates IPC-02 | Not started | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite | | TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |