ci: drop the nightly-windev schedule; run the deep windev tier on demand
ci / portable (push) Failing after 10m10s
ci / java (push) Failing after 20m12s
ci / windows-x86 (push) Failing after 22m40s

The scheduled nightly-windev job failed most nights for host reasons and
filed a Gitea issue each time. Remove the cron trigger and the job, and add
a workflow_dispatch trigger whose windev_mode input (test/build/live) drives
the windows-x86 job, so the live tier runs on demand - from the Actions page
or via scripts/ci/run-windev-ci.sh live - the same way clients are packed and
published on demand. Docs updated to match.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-11-15 09:42:13 -05:00
parent 26a90883fa
commit e74918ccfb
7 changed files with 65 additions and 85 deletions
+22 -18
View File
@@ -89,10 +89,12 @@ fake-worker tests cannot validate:
when the rig does not drive sample-bearing buffered batches on demand.
All nine tests are gated by the same `MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`
opt-in variable. Opt-in does not mean unscheduled: the `nightly-windev` job runs
this suite on windev every night at 06:00 UTC and files a Gitea issue when it goes
red (see [Continuous Integration](#continuous-integration)), so the smoke no longer
depends on someone remembering to set the variable.
opt-in variable. Nothing runs this suite on a schedule; run it on demand on windev with
the `live` tier — dispatch the `ci` workflow with `windev_mode: live`, or run
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh live` (see
[Continuous Integration](#continuous-integration)). The scheduled `nightly-windev` job
that used to do this was removed on 2026-11-15 because it failed most nights for host
reasons rather than code changes.
Known coverage gap: the suite reaches all six late-added MXAccess **COM** commands
but none of the five **control** commands (`Ping`, `GetSessionState`,
@@ -531,8 +533,8 @@ dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj
## Running the Gateway Suite on windev
The gateway suite (`ZB.MOM.WW.MxGateway.Tests`, net10.0/x64) is not part of the CI
Windows tier — `windows-x86` and `nightly-windev` run only the x86 Worker build and
`Worker.Tests`. It is still run on windev by hand when a change needs Windows
Windows tier on a push — `windows-x86` runs only the x86 Worker build and
`Worker.Tests` unless it is dispatched by hand with `windev_mode: live`. It is still run on windev by hand when a change needs Windows
confirmation, and that run has three Windows-specific characteristics worth knowing
before results are interpreted.
@@ -682,8 +684,7 @@ runtime because the x86 Worker cannot build on Linux:
client (IPC-24 deleted the old unconditional churn-revert step, which masked exactly that
class). The dev Mac has a Homebrew JDK 17, so `gradle generateProto` can be run there to refresh
the Java aggregates when a `.proto` changes.
- **`windows-x86`** (Linux runner, per push/PR; skipped on the schedule, where
`nightly-windev` runs the same tiers) — builds the **x86 / net48 Worker and
- **`windows-x86`** (Linux runner, per push/PR, and on demand via `workflow_dispatch`) — builds the **x86 / net48 Worker and
Worker.Tests**, which are Windows-only and out of scope for the Linux jobs. It runs on a
Linux runner that always schedules and SSHes to windev (`10.100.0.48`), where
`scripts/ci/run-windev-ci.sh` fast-forwards the isolated CI clone `C:\build\mxaccessgw-ci`
@@ -694,25 +695,28 @@ runtime because the x86 Worker cannot build on Linux:
the tier is down, not the change**: an unreachable windev fails the job (never skips), so
if it goes red without a related code change, check the tier — `ssh windev`, read
`C:\build\mxaccessgw-ci\ci.log` — before assuming a real test failure.
- **`nightly-windev`** (Linux runner, scheduled `0 6 * * *`) — runs, in order, the x86 Worker
build, `Worker.Tests`, a full `slnx` build (which surfaces Windows-only analyzer diagnostics
such as `xUnit1030`), the net10 gateway suite, and the live-MXAccess smoke
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). The checkout and all five tiers share the windev
host lock, and the first nonzero native exit stops the sequence and propagates to the CI job.
Gated `if: github.event_name == 'schedule'`, so it never gates a push. On failure it opens a
Gitea issue via the Actions token, since nobody watches the Actions page.
- **Deep windev tier, on demand** — there is no scheduled run. The `live` mode runs, in order,
the x86 Worker build, `Worker.Tests`, a full `slnx` build (which surfaces Windows-only analyzer
diagnostics such as `xUnit1030`), the net10 gateway suite, and the live-MXAccess smoke
(`MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1`). Run it when a change needs it, the same way the clients
are packed and published on demand: dispatch the `ci` workflow from the Gitea Actions page with
`windev_mode: live`, or run `CI_SHA=<sha> scripts/ci/run-windev-ci.sh live` from a machine with
SSH access to windev. The checkout and all five tiers share the windev host lock, and the first
nonzero native exit stops the sequence and propagates back. This replaced the `nightly-windev`
cron job (removed 2026-11-15), which failed most nights for host reasons and filed a Gitea issue
each time.
The windev host lock is an exclusive file handle on `C:\build\mxaccessgw-ci.ci-lock`, held by
the SSH bootstrap for the whole run. It replaced a `New-Item -ItemType Directory` lock that two
runs starting in the same second could both take (PowerShell checks for the directory, then
creates it), which is how the 2026-09-22 nightly collided with the scheduled `windows-x86` run on
creates it), which is how the 2026-09-22 nightly (since removed) collided with the scheduled `windows-x86` run on
`.git/index.lock` (mxaccessgw#150). The OS drops the handle when its holder dies, so a killed run
leaves no stale lock, and a holder may delete a leftover `index.lock` because no other CI git can
be running. Each run also writes a `host (start)` line — and a `host (failure)` line when a tier
fails — to `C:\build\mxaccessgw-ci\ci.log` with free commit, physical memory, disk and the
number of `testhost` processes, so a host-resource failure can be matched to the host's state.
When a nightly goes red, sort the cause before touching code: a test that fails with an
When a `live` run goes red, sort the cause before touching code: a test that fails with an
assertion or test exception is a test or product fault; `OutOfMemoryException` from
MSBuild/csc/protoc, `csc.exe exited with code -1073741502` (0xC0000142, a process that could not
initialize), a remote exit code with no `ci.log` lines at all, or an SSH `kex_exchange_identification`
@@ -743,7 +747,7 @@ second runner.
When queue depth (or the missing-cancel reality) makes waiting impractical, verify a
specific commit out of band instead of waiting behind the queue: run
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` from a machine with SSH access
to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see
to windev (the same script the SSH-driven `windows-x86` job uses — see
`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is
the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from
"tier down" to "runner contended": either way, a stuck or slow shared runner should not
+2 -1
View File
@@ -228,7 +228,8 @@ days. The SQLite DB must also never be opened with a host-side `sqlite3` while t
## 3. Preflight (no changes yet)
1. Decide the target SHA on `origin/main`; confirm CI is green for it (`windows-x86` job runs the
Worker build + `Worker.Tests` on windev; `nightly-windev` runs the live tier,
Worker build + `Worker.Tests` on windev; run the live tier on demand with
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh live` or a `windev_mode: live` dispatch,
`scripts/ci/README.md`).
2. Read the current state of the service (read-only):
```text