dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

docs(TST-30): document shared-runner CI bottleneck + second-runner operator runbook

Browse Source 
Doc half of TST-30 (single shared Gitea runner is a CI throughput/availability
bottleneck): docs/GatewayTesting.md's Continuous Integration section gains a
"Runner capacity is shared and finite" subsection covering the maxParallel=1
instance-level runner shared with dohertj2/lmxopcua, the ~20-30 min queue
latency observed under cross-repo contention, and Gitea 1.26's missing run
cancel/delete API. The existing "windev tier down" degraded-mode paragraph now
also covers "runner contended" as a reason to bypass the queue via
CI_SHA=<sha> scripts/ci/run-windev-ci.sh <mode> or the manual windev worktree
flow, generalizing it per the finding's design note.

New operator runbook docs/runbooks/TST-30-second-ci-runner.md carries the
actual runner registration (option a: second act_runner instance on
10.100.0.35 with the same container.network: traefik config, recommended;
option b: dedicated labelled runner, escalation only; option c: runner on
windev, rejected) plus verification steps and the no-cancel caveat. The
optional workflow-level concurrency group is documented as unverified --
framed as "verify before relying on it" -- and left unimplemented in ci.yml,
since registering the runner and any runs-on gating is operator/infra work
outside this repo's tree.

Tracking: TST-30 -> Done (doc half; runner registration operator-pending) in
both registers + change-log row.
This commit is contained in:
Joseph Doherty
2026-08-07 07:47:37 -04:00
parent 0646c73e48
commit b604fed72b
4 changed files with 151 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/GatewayTesting.md
+33 -5
View File
@@ -452,6 +452,32 @@ runtime because the x86 Worker cannot build on Linux:
`xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.
### Runner capacity is shared and finite
All CI runs on a co-located `gitea-runner` container on docker host `10.100.0.35` with
`maxParallel=1`, and that runner instance is **shared across repos** — it interleaves
`dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs on the single slot rather than being
scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns
`total_count: 0`; the runner is registered at the instance level). Every job in a run
(`portable`, `java`, `windows-x86`) executes serially, so queue latency is additive within
a run and an active `lmxopcua` run blocks `mxaccessgw` entirely — expect ~20–30 minutes of
queue depth under cross-repo contention, not a stuck pipeline. This Gitea version (1.26)
also exposes **no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel`
returns 404, `DELETE .../actions/runs/{id}` returns 400), so a superseded or hung run cannot
be cleared and holds the slot until it finishes or times out. See
`docs/runbooks/TST-30-second-ci-runner.md` for the operator runbook that registers a second
runner to relieve this; until that lands, treat single-runner contention as expected, not a
CI outage.
When queue depth (or the missing-cancel reality) makes waiting impractical, verify a
specific commit out of band instead of waiting behind the queue: run
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` from a machine with SSH access
to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see
`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is
the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from
"tier down" to "runner contended": either way, a stuck or slow shared runner should not
block verifying a commit.
The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the
committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored
protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current
@@ -466,11 +492,13 @@ the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGenerat
[Contracts](./Contracts.md).
If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI
key/secret rotation in flight), fall back to the manual windev worktree procedure as a
degraded mode: on windev, fast-forward an isolated `origin/main` worktree under `C:\build`
(never the dirty Desktop checkout), then run the x86 Worker build and `Worker.Tests`
(`-p:Platform=x86`) there by hand. Do this per merge for worker-touching changes until the
`windows-x86` job is green again.
key/secret rotation in flight) **or** the shared Gitea runner is contended and the queue is
impractical to wait behind (see "Runner capacity is shared and finite" above), fall back to
the manual windev worktree procedure as a degraded mode: on windev, fast-forward an isolated
`origin/main` worktree under `C:\build` (never the dirty Desktop checkout), then run the x86
Worker build and `Worker.Tests` (`-p:Platform=x86`) there by hand. Do this per merge for
worker-touching changes until the `windows-x86` job is green again (tier-down case) or the
queue clears (contention case).
## Related Documentation