Files
mxaccessgw/scripts/ci/README.md
T
Joseph Doherty e74918ccfb
ci / portable (push) Failing after 10m10s
ci / java (push) Failing after 20m12s
ci / windows-x86 (push) Failing after 22m40s
ci: drop the nightly-windev schedule; run the deep windev tier on demand
The scheduled nightly-windev job failed most nights for host reasons and
filed a Gitea issue each time. Remove the cron trigger and the job, and add
a workflow_dispatch trigger whose windev_mode input (test/build/live) drives
the windows-x86 job, so the live tier runs on demand - from the Actions page
or via scripts/ci/run-windev-ci.sh live - the same way clients are packed and
published on demand. Docs updated to match.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-11-15 09:42:13 -05:00

89 lines
6.1 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Windows/x86 CI tier (TST-25)
The x86 / net48 Worker and its tests cannot build on a Linux runner, and a native Windows
`act_runner` is broken (host-mode writes each step's script to a path it can't resolve; a
`runs-on` gate with no runner wedges the queue forever). Instead, a Linux CI job SSHes to
windev (`10.100.0.48`) and runs the Worker build/tests there, propagating the remote exit
code back so a regression turns the job red.
## Pieces
| File | Runs on | Role |
|---|---|---|
| `run-windev-ci.sh <build\|test\|live>` | Linux CI job | Writes the SSH key/known-hosts, base64-encodes a bootstrap, SSHes to windev, returns the remote exit code. |
| `windev-worker-ci.ps1 -Sha -Mode` | windev | Locks the CI clone, checks out the SHA, and runs the requested cumulative Windows tiers. |
| `windev.known_hosts` | committed | Pinned windev host keys so `StrictHostKeyChecking=yes` has no TOFU prompt. Host keys are public; committing is intentional. |
Modes are cumulative: `build` = x86 Worker build; `test` = build + `Worker.Tests`;
`live` = test + full-`slnx` build + the net10 gateway suite + live-MXAccess smoke
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The `live` order is x86 Worker build,
`Worker.Tests`, full solution build, gateway tests, then live smoke. One host lock covers
checkout and every tier, so concurrent runs cannot overlap. The lock is an exclusive
(`FileShare.None`) handle on `C:\build\mxaccessgw-ci.ci-lock` held by the SSH bootstrap; the OS
releases it if the run is killed, so it never goes stale. (It replaced a `New-Item -ItemType
Directory` lock that two runs starting in the same second could both take — mxaccessgw#150.)
Runs of SHAs older than that change still use the old directory lock and do not exclude new-style
runs; that window closes as branches rebase. each native command is checked
immediately and the first nonzero exit aborts the remaining tiers and propagates to CI.
CI jobs (`.gitea/workflows/ci.yml`): `windows-x86` runs `test` per push/PR. There is no
scheduled run. Run the `live` tier on demand, like the client pack/publish scripts:
```bash
# from a machine with SSH access to windev (falls back to the default SSH identity)
WINDEV_SSH_USER=<user> CI_SHA=$(git rev-parse HEAD) scripts/ci/run-windev-ci.sh live
```
or dispatch the `ci` workflow from the Gitea Actions page with `windev_mode: live`. The
`nightly-windev` cron job that used to run `live` every night and file a Gitea issue on failure
was removed on 2026-11-15: it failed most nights for host reasons and the issues were noise.
## One-time bring-up (operator — required before this lands on `main`)
Until these are done, `windows-x86` is **red on every push** (no key → SSH fails). Do them
first, then merge.
1. **CI clone** — `C:\build\mxaccessgw-ci`, a fresh clone of the Gitea origin (done during
TST-25 verification; recreate with `git clone https://gitea.dohertylan.com/dohertj2/mxaccessgw C:\build\mxaccessgw-ci` if missing). Never the dirty Desktop checkout.
2. **Dedicated CI key** — generate a NEW ed25519 keypair for CI (do not reuse the operator's
personal windev key): `ssh-keygen -t ed25519 -f ci_windev -N ''`. Append `ci_windev.pub`
to the windev CI account's `authorized_keys` (optionally with a forced `command=` limiting
it to this bootstrap).
3. **Gitea repo secret `WINDEV_SSH_KEY`** — store the private key **base64-encoded on a single
line**: `base64 -w0 < ci_windev` (macOS: `base64 < ci_windev | tr -d '\n'`). This is required,
not cosmetic — Gitea's secret masker is line-oriented, so a raw multiline PEM is **not**
redacted in the step `env:` echo and leaks in cleartext; the single-line base64 masks to `***`.
`run-windev-ci.sh` auto-decodes it. Host keys are public and come from the committed
`windev.known_hosts` pin, so no `WINDEV_SSH_KNOWN_HOSTS` secret is wired into `ci.yml`.
`run-windev-ci.sh` connects as user `ci` by default; set the `WINDEV_SSH_USER` **variable**
(not a secret — it is public) to the actual account, and ensure that account exists on windev.
4. **Network precheck** — confirm the Gitea runner container (on the `traefik` docker network,
host `10.100.0.35`) can reach `10.100.0.48:22`: a one-off `nc -z -w5 10.100.0.48 22` step.
The network was created for gitea name resolution, not LAN egress, so verify.
## Verifying it end-to-end (design's acceptance checks)
- Push a branch → `portable`, `java`, `windows-x86` all green; windev's clone sits at the pushed SHA.
- Deliberate red: add a failing `[Fact]` to `Worker.Tests`, push, confirm `windows-x86` goes
red; revert. (Proves exit-code propagation — the whole point.)
- Unreachable-host red: point at a bogus port / stop sshd, confirm the job fails fast, not hangs.
- Concurrency: push two branches back-to-back, confirm the second remote run waits on the lock.
- Nightly (historical — the job was removed 2026-11-15): trigger the schedule path, confirm
`live` runs and a forced failure opens an issue. **Done 2026-08-10** (Check 6). Verified two ways: (a) production — every red nightly since
2026-07-17 has auto-filed an issue (#126–#139) authored by the `gitea-actions` bot, e.g. run 672
→ issue #139, with the built-in token masked to `***` in the job log; (b) a forced-failure probe
on the throwaway branch `test/tst25-check6-nightly-issue` (run 677 → issue #140, since closed and
the branch deleted), which reproduced the job shape with `exit 1` in place of the live step and
confirmed the `if: failure()` step fires, the token carries issue-write, and the payload is
well-formed. The probe also caught the one defect: `${{ github.server_url }}` is the
runner-internal `http://gitea:3000`, so the run link in the issue body was unreachable from a
browser — the body now uses the `PUBLIC_SERVER_URL` job env instead (the API call still targets
`github.server_url`, which is what the job container can resolve).
- Confirm no key material appears in job logs.
## Degraded mode
If the tier is down for infra reasons, fall back to the manual windev worktree procedure
(isolated `C:\build` worktree, `dotnet build`/`test` `-p:Platform=x86` by hand) per merge for
worker-touching changes until `windows-x86` is green again. See `docs/GatewayTesting.md`.