Files
mxaccessgw/scripts/ci
Joseph Doherty de86f8b80e
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Successful in 1m34s
ci / java (push) Successful in 2m37s
ci / portable (push) Failing after 5m0s
ci: make the windev CI lock exclusive, stop the scheduled double run, isolate test secrets stores (mxaccessgw#150)
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from
the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only):

- Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive.
  New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which
  succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of
  2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same
  second, both bootstraps got in, and the nightly's checkout died on .git/index.lock.
  Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None
  handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same
  harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder
  removes a leftover .git\index.lock, which can only be debris from a killed git.
- windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same
  Worker build + tests first, so the scheduled copy only doubled the load and created the race.
- Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has
  Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23
  until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc,
  csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that
  window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical
  memory, disk and testhost count at start and on failure so the next one is correlatable.
- SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50).
- Dashboard host build (#143, run 817): two host-building test classes failed at the same instant
  in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel
  classes migrating the one per-process secrets store. Tests now build the gateway through
  TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it.
- Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were
  the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6):
  7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and
  burst-coalescing failures (#126/#127/#139) last failed before 50322bac / 7da52b65 (Aug 2026).
- 2026-10-15..20 was the NU1903 package-audit restore failure, fixed in fb20997.

Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 13:36:34 -05:00
..

Windows/x86 CI tier (TST-25)

The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows act_runner is broken (host-mode writes each step's script to a path it can't resolve; a runs-on gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to windev (10.100.0.48) and runs the Worker build/tests there, propagating the remote exit code back so a regression turns the job red.

Pieces

File Runs on Role
run-windev-ci.sh <build|test|live> Linux CI job Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code.
windev-worker-ci.ps1 -Sha -Mode windev Locks the CI clone, checks out the SHA, and runs the requested cumulative Windows tiers.
windev.known_hosts committed Pinned windev host keys so StrictHostKeyChecking=yes has no TOFU prompt. Host keys are public; committing is intentional.

Modes are cumulative: build = x86 Worker build; test = build + Worker.Tests; live = test + full-slnx build + the net10 gateway suite + live-MXAccess smoke (MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1). The live order is x86 Worker build, Worker.Tests, full solution build, gateway tests, then live smoke. One host lock covers checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive (FileShare.None) handle on C:\build\mxaccessgw-ci.ci-lock held by the SSH bootstrap; the OS releases it if the run is killed, so it never goes stale. (It replaced a New-Item -ItemType Directory lock that two runs starting in the same second could both take — mxaccessgw#150.) Runs of SHAs older than that change still use the old directory lock and do not exclude new-style runs; that window closes as branches rebase. each native command is checked immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.

CI jobs (.gitea/workflows/ci.yml): windows-x86 runs test per push/PR (not on the schedule, where it would duplicate the nightly's first two tiers and start a second windev session in the same second); nightly-windev runs live on the 0 6 * * * schedule and opens a Gitea issue on failure.

One-time bring-up (operator — required before this lands on main)

Until these are done, windows-x86 is red on every push (no key → SSH fails). Do them first, then merge.

  1. CI clone — C:\build\mxaccessgw-ci, a fresh clone of the Gitea origin (done during TST-25 verification; recreate with git clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ci if missing). Never the dirty Desktop checkout.
  2. Dedicated CI key — generate a NEW ed25519 keypair for CI (do not reuse the operator's personal windev key): ssh-keygen -t ed25519 -f ci_windev -N ''. Append ci_windev.pub to the windev CI account's authorized_keys (optionally with a forced command= limiting it to this bootstrap).
  3. Gitea repo secret WINDEV_SSH_KEY — store the private key base64-encoded on a single line: base64 -w0 < ci_windev (macOS: base64 < ci_windev | tr -d '\n'). This is required, not cosmetic — Gitea's secret masker is line-oriented, so a raw multiline PEM is not redacted in the step env: echo and leaks in cleartext; the single-line base64 masks to ***. run-windev-ci.sh auto-decodes it. Host keys are public and come from the committed windev.known_hosts pin, so no WINDEV_SSH_KNOWN_HOSTS secret is wired into ci.yml. run-windev-ci.sh connects as user ci by default; set the WINDEV_SSH_USER variable (not a secret — it is public) to the actual account, and ensure that account exists on windev.
  4. Network precheck — confirm the Gitea runner container (on the traefik docker network, host 10.100.0.35) can reach 10.100.0.48:22: a one-off nc -z -w5 10.100.0.48 22 step. The network was created for gitea name resolution, not LAN egress, so verify.
  5. Actions token — the nightly-windev on-failure step creates a Gitea issue with the built-in token; confirm Actions tokens have issue-write on this repo.

Verifying it end-to-end (design's acceptance checks)

  • Push a branch → portable, java, windows-x86 all green; windev's clone sits at the pushed SHA.
  • Deliberate red: add a failing [Fact] to Worker.Tests, push, confirm windows-x86 goes red; revert. (Proves exit-code propagation — the whole point.)
  • Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
  • Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
  • Nightly: trigger the schedule path, confirm live runs and a forced failure opens an issue. Done 2026-08-10 (Check 6). Verified two ways: (a) production — every red nightly since 2026-07-17 has auto-filed an issue (#126–#139) authored by the gitea-actions bot, e.g. run 672 → issue #139, with the built-in token masked to *** in the job log; (b) a forced-failure probe on the throwaway branch test/tst25-check6-nightly-issue (run 677 → issue #140, since closed and the branch deleted), which reproduced the job shape with exit 1 in place of the live step and confirmed the if: failure() step fires, the token carries issue-write, and the payload is well-formed. The probe also caught the one defect: ${{ github.server_url }} is the runner-internal http://gitea:3000, so the run link in the issue body was unreachable from a browser — the body now uses the PUBLIC_SERVER_URL job env instead (the API call still targets github.server_url, which is what the job container can resolve).
  • Confirm no key material appears in job logs.

Degraded mode

If the tier is down for infra reasons, fall back to the manual windev worktree procedure (isolated C:\build worktree, dotnet build/test -p:Platform=x86 by hand) per merge for worker-touching changes until windows-x86 is green again. See docs/GatewayTesting.md.