de86f8b80e
Diagnosis of the retained Windows-nightly failures (119 scheduled nightlies, 28 red; job logs from the Gitea API, C:\build\mxaccessgw-ci\ci.log and the windev System log, read-only): - Checkout (#148, run 827, 2026-09-22): the windev "atomic mkdir" lock was not exclusive. New-Item -ItemType Directory checks for the directory and then calls CreateDirectory, which succeeds on an existing directory; a pwsh race harness had both contenders take it in 1900 of 2000 simultaneous rounds. A scheduled run starts windows-x86 and nightly-windev in the same second, both bootstraps got in, and the nightly's checkout died on .git/index.lock. Fix: the bootstrap (and a standalone windev-worker-ci.ps1) holds an exclusive FileShare.None handle on C:\build\mxaccessgw-ci.ci-lock instead (0 of 1000 double acquisitions in the same harness). The OS drops it if the holder dies, so a killed run leaves no stale lock, and a holder removes a leftover .git\index.lock, which can only be debris from a killed git. - windows-x86 no longer runs on the schedule: nightly-windev's live mode already runs the same Worker build + tests first, so the scheduled copy only doubled the load and created the race. - Host resources (#128/#129/#131-#135/#138, 2026-07-24..08-03): windev's System log has Resource-Exhaustion-Detector event 2004 (low virtual memory) every few minutes from 2026-07-23 until an unexpected shutdown/hard reset on 2026-08-03; OOM in MSBuild/csc/protoc, csc 0xC0000142, PowerShell dying before its first log line and SSH kex resets all fall in that window and stop after the reset. Host-side; windev-worker-ci.ps1 now logs free commit, physical memory, disk and testhost count at start and on failure so the next one is correlatable. - SSH "No route to host" (#144, 2026-09-16): windev was down (unexpected shutdown 00:58, back 05:50). - Dashboard host build (#143, run 817): two host-building test classes failed at the same instant in SqliteSecretsStoreMigrator with "cannot start a transaction within a transaction" -- parallel classes migrating the one per-process secrets store. Tests now build the gateway through TestSupport/TestGatewayHost, which gives each host its own store; TestGatewayHostTests guards it. - Gateway E2E "Worker pipe disconnected" (#142/#145/#146/#147) and handshaking-to-ready (#141) were the fake-worker partial-frame cancellation and readiness-race defects fixed in #153 (d779daf6): 7 gateway-suite failures in 51 windev runs before it, 0 in 28 since. The Worker watchdog and burst-coalescing failures (#126/#127/#139) last failed before50322bac/7da52b65(Aug 2026). - 2026-10-15..20 was the NU1903 package-audit restore failure, fixed infb20997. Docs: GatewayTesting.md (lock, host-resource triage) and scripts/ci/README.md. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
84 lines
5.9 KiB
Markdown
84 lines
5.9 KiB
Markdown
# Windows/x86 CI tier (TST-25)
|
||
|
||
The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows
|
||
`act_runner` is broken (host-mode writes each step's script to a path it can't resolve; a
|
||
`runs-on` gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to
|
||
windev (`10.100.0.48`) and runs the Worker build/tests there, propagating the remote exit
|
||
code back so a regression turns the job red.
|
||
|
||
## Pieces
|
||
|
||
| File | Runs on | Role |
|
||
|---|---|---|
|
||
| `run-windev-ci.sh <build\|test\|live>` | Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. |
|
||
| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, and runs the requested cumulative Windows tiers. |
|
||
| `windev.known_hosts` | committed | Pinned windev host keys so `StrictHostKeyChecking=yes` has no TOFU prompt. Host keys are public; committing is intentional. |
|
||
|
||
Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests`;
|
||
`live` = test + full-`slnx` build + the net10 gateway suite + live-MXAccess smoke
|
||
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The `live` order is x86 Worker build,
|
||
`Worker.Tests`, full solution build, gateway tests, then live smoke. One host lock covers
|
||
checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive
|
||
(`FileShare.None`) handle on `C:\build\mxaccessgw-ci.ci-lock` held by the SSH bootstrap; the OS
|
||
releases it if the run is killed, so it never goes stale. (It replaced a `New-Item -ItemType
|
||
Directory` lock that two runs starting in the same second could both take — mxaccessgw#150.)
|
||
Runs of SHAs older than that change still use the old directory lock and do not exclude new-style
|
||
runs; that window closes as branches rebase. each native command is checked
|
||
immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.
|
||
|
||
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR (not on the schedule,
|
||
where it would duplicate the nightly's first two tiers and start a second windev session in the
|
||
same second); `nightly-windev` runs `live` on the `0 6 * * *` schedule and opens a Gitea issue on
|
||
failure.
|
||
|
||
## One-time bring-up (operator — required before this lands on `main`)
|
||
|
||
Until these are done, `windows-x86` is **red on every push** (no key → SSH fails). Do them
|
||
first, then merge.
|
||
|
||
1. **CI clone** — `C:\build\mxaccessgw-ci`, a fresh clone of the Gitea origin (done during
|
||
TST-25 verification; recreate with `git clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ci` if missing). Never the dirty Desktop checkout.
|
||
2. **Dedicated CI key** — generate a NEW ed25519 keypair for CI (do not reuse the operator's
|
||
personal windev key): `ssh-keygen -t ed25519 -f ci_windev -N ''`. Append `ci_windev.pub`
|
||
to the windev CI account's `authorized_keys` (optionally with a forced `command=` limiting
|
||
it to this bootstrap).
|
||
3. **Gitea repo secret `WINDEV_SSH_KEY`** — store the private key **base64-encoded on a single
|
||
line**: `base64 -w0 < ci_windev` (macOS: `base64 < ci_windev | tr -d '\n'`). This is required,
|
||
not cosmetic — Gitea's secret masker is line-oriented, so a raw multiline PEM is **not**
|
||
redacted in the step `env:` echo and leaks in cleartext; the single-line base64 masks to `***`.
|
||
`run-windev-ci.sh` auto-decodes it. Host keys are public and come from the committed
|
||
`windev.known_hosts` pin, so no `WINDEV_SSH_KNOWN_HOSTS` secret is wired into `ci.yml`.
|
||
`run-windev-ci.sh` connects as user `ci` by default; set the `WINDEV_SSH_USER` **variable**
|
||
(not a secret — it is public) to the actual account, and ensure that account exists on windev.
|
||
4. **Network precheck** — confirm the Gitea runner container (on the `traefik` docker network,
|
||
host `10.100.0.35`) can reach `10.100.0.48:22`: a one-off `nc -z -w5 10.100.0.48 22` step.
|
||
The network was created for gitea name resolution, not LAN egress, so verify.
|
||
5. **Actions token** — the `nightly-windev` on-failure step creates a Gitea issue with the
|
||
built-in token; confirm Actions tokens have issue-write on this repo.
|
||
|
||
## Verifying it end-to-end (design's acceptance checks)
|
||
|
||
- Push a branch → `portable`, `java`, `windows-x86` all green; windev's clone sits at the pushed SHA.
|
||
- Deliberate red: add a failing `[Fact]` to `Worker.Tests`, push, confirm `windows-x86` goes
|
||
red; revert. (Proves exit-code propagation — the whole point.)
|
||
- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
|
||
- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
|
||
- Nightly: trigger the schedule path, confirm `live` runs and a forced failure opens an issue.
|
||
**Done 2026-08-10** (Check 6). Verified two ways: (a) production — every red nightly since
|
||
2026-07-17 has auto-filed an issue (#126–#139) authored by the `gitea-actions` bot, e.g. run 672
|
||
→ issue #139, with the built-in token masked to `***` in the job log; (b) a forced-failure probe
|
||
on the throwaway branch `test/tst25-check6-nightly-issue` (run 677 → issue #140, since closed and
|
||
the branch deleted), which reproduced the job shape with `exit 1` in place of the live step and
|
||
confirmed the `if: failure()` step fires, the token carries issue-write, and the payload is
|
||
well-formed. The probe also caught the one defect: `${{ github.server_url }}` is the
|
||
runner-internal `http://gitea:3000`, so the run link in the issue body was unreachable from a
|
||
browser — the body now uses the `PUBLIC_SERVER_URL` job env instead (the API call still targets
|
||
`github.server_url`, which is what the job container can resolve).
|
||
- Confirm no key material appears in job logs.
|
||
|
||
## Degraded mode
|
||
|
||
If the tier is down for infra reasons, fall back to the manual windev worktree procedure
|
||
(isolated `C:\build` worktree, `dotnet build`/`test` `-p:Platform=x86` by hand) per merge for
|
||
worker-touching changes until `windows-x86` is green again. See `docs/GatewayTesting.md`.
|