docs(TST-30): document shared-runner CI bottleneck + second-runner operator runbook

Doc half of TST-30 (single shared Gitea runner is a CI throughput/availability bottleneck): docs/GatewayTesting.md's Continuous Integration section gains a "Runner capacity is shared and finite" subsection covering the maxParallel=1 instance-level runner shared with dohertj2/lmxopcua, the ~20-30 min queue latency observed under cross-repo contention, and Gitea 1.26's missing run cancel/delete API. The existing "windev tier down" degraded-mode paragraph now also covers "runner contended" as a reason to bypass the queue via CI_SHA=<sha> scripts/ci/run-windev-ci.sh <mode> or the manual windev worktree flow, generalizing it per the finding's design note. New operator runbook docs/runbooks/TST-30-second-ci-runner.md carries the actual runner registration (option a: second act_runner instance on 10.100.0.35 with the same container.network: traefik config, recommended; option b: dedicated labelled runner, escalation only; option c: runner on windev, rejected) plus verification steps and the no-cancel caveat. The optional workflow-level concurrency group is documented as unverified -- framed as "verify before relying on it" -- and left unimplemented in ci.yml, since registering the runner and any runs-on gating is operator/infra work outside this repo's tree. Tracking: TST-30 -> Done (doc half; runner registration operator-pending) in both registers + change-log row.
2026-08-07 07:47:37 -04:00
parent 0646c73e48
commit b604fed72b
4 changed files with 151 additions and 7 deletions
@@ -15,7 +15,7 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior
 | TST-27 | Medium | P1 (doc batch) | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
 | TST-28 | Low | P2 | S | relates IPC-02 | Done | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
 | TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
-| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |
+| TST-30 | Low | P2 | M | — | Done (doc half; runner registration operator-pending per runbook) | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |

 ---

@@ -167,6 +167,8 @@ Independent of the runner count, document the **no-cancel** reality (Gitea 1.26

 **Verification.** Push two branches back-to-back and confirm their runs execute concurrently (not serially) once a second runner exists; `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner list) shows ≥2 runners online; `docs/GatewayTesting.md` describes the shared-runner/no-cancel reality and the bypass. Re-run the TST-25 acceptance push and confirm queue depth is materially lower under a concurrent `lmxopcua` run.

+**Outcome (2026-08-07 — Done, doc half; runner registration operator-pending).** Landed on `fix/tst-30-runner-docs`. Implementation step 2 shipped: `docs/GatewayTesting.md`'s Continuous Integration section gained a "Runner capacity is shared and finite" subsection stating the `maxParallel=1` co-located runner is shared with `dohertj2/lmxopcua` at the instance level (not repo-scoped), the ~20–30 minute queue latency observed under cross-repo contention, and the Gitea 1.26 no-cancel/no-delete API reality; the existing "windev tier down" degraded-mode paragraph now also covers "runner contended" as a reason to use the bypass, generalized per this finding's design note. New operator runbook `docs/runbooks/TST-30-second-ci-runner.md` carries **step 1** (register a second `act_runner` on `10.100.0.35`, option (a) recommended, same `container.network: traefik` config; option (b) dedicated labelled runner as an escalation; option (c) windev-hosted runner rejected) with the verification checklist (concurrent back-to-back pushes, `GET /repos/dohertj2/mxaccessgw/actions/runners` ≥ 2) and a note that the no-cancel reality persists regardless of runner count. **Step 3 (optional workflow-level `concurrency` group)** is documented in the runbook as unverified — explicitly framed as "verify this Gitea deployment honors it before relying on it" — and left unimplemented in `ci.yml`, since it is a `ci.yml` change out of scope for this doc-only pass. **The actual runner registration (step 1) is infrastructure work outside this repo's tree and remains the operator's to execute**, tracked in the runbook. Verification performed: `grep -n 'maxParallel\|shared\|cancel' docs/GatewayTesting.md` shows the new prose; runbook file exists at the path above; no build required (doc-only change).
+
 ---

 ## Cross-domain dependencies
@@ -452,6 +452,32 @@ runtime because the x86 Worker cannot build on Linux:
  `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
  failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.

+### Runner capacity is shared and finite
+
+All CI runs on a co-located `gitea-runner` container on docker host `10.100.0.35` with
+`maxParallel=1`, and that runner instance is **shared across repos** — it interleaves
+`dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs on the single slot rather than being
+scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns
+`total_count: 0`; the runner is registered at the instance level). Every job in a run
+(`portable`, `java`, `windows-x86`) executes serially, so queue latency is additive within
+a run and an active `lmxopcua` run blocks `mxaccessgw` entirely — expect ~20–30 minutes of
+queue depth under cross-repo contention, not a stuck pipeline. This Gitea version (1.26)
+also exposes **no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel`
+returns 404, `DELETE .../actions/runs/{id}` returns 400), so a superseded or hung run cannot
+be cleared and holds the slot until it finishes or times out. See
+`docs/runbooks/TST-30-second-ci-runner.md` for the operator runbook that registers a second
+runner to relieve this; until that lands, treat single-runner contention as expected, not a
+CI outage.
+
+When queue depth (or the missing-cancel reality) makes waiting impractical, verify a
+specific commit out of band instead of waiting behind the queue: run
+`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` from a machine with SSH access
+to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see
+`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is
+the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from
+"tier down" to "runner contended": either way, a stuck or slow shared runner should not
+block verifying a commit.
+
 The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the
 committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored
 protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current
@@ -466,11 +492,13 @@ the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGenerat
 [Contracts](./Contracts.md).

 If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI
-key/secret rotation in flight), fall back to the manual windev worktree procedure as a
-degraded mode: on windev, fast-forward an isolated `origin/main` worktree under `C:\build`
-(never the dirty Desktop checkout), then run the x86 Worker build and `Worker.Tests`
-(`-p:Platform=x86`) there by hand. Do this per merge for worker-touching changes until the
-`windows-x86` job is green again.
+key/secret rotation in flight) **or** the shared Gitea runner is contended and the queue is
+impractical to wait behind (see "Runner capacity is shared and finite" above), fall back to
+the manual windev worktree procedure as a degraded mode: on windev, fast-forward an isolated
+`origin/main` worktree under `C:\build` (never the dirty Desktop checkout), then run the x86
+Worker build and `Worker.Tests` (`-p:Platform=x86`) there by hand. Do this per merge for
+worker-touching changes until the `windows-x86` job is green again (tier-down case) or the
+queue clears (contention case).

 ## Related Documentation

@@ -0,0 +1,113 @@
+# TST-30 — Register A Second CI Runner (Operator Runbook)
+
+Operator steps to relieve the single shared Gitea Actions runner that CI depends on. The
+repo-side half of TST-30 (documenting the shared-runner/no-cancel reality and the
+`run-windev-ci.sh` bypass) is already landed in `docs/GatewayTesting.md`; registering the
+second runner below is infrastructure work outside this repo's tree and is yours to execute.
+
+## Why
+
+All CI for this repo runs on one co-located `gitea-runner` container on docker host
+`10.100.0.35` with `maxParallel=1`. That runner is registered at the **instance** level, not
+scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns
+`total_count: 0`), so it is shared with `dohertj2/lmxopcua` and every job in every run across
+both repos executes serially on the single slot. A `mxaccessgw` push fans out to `portable`,
+`java`, `windows-x86`, and an active `lmxopcua` run blocks all of them — queue depth of
+~20–30 minutes was observed during TST-25 acceptance under cross-repo contention. Gitea 1.26
+also exposes **no run cancel or delete via the API**
+(`POST .../actions/runs/{id}/cancel` → 404, `DELETE .../actions/runs/{id}` → 400), so a
+superseded or hung run cannot be cleared and holds the slot until it finishes or times out.
+This is not a correctness problem — every job still reports accurately — but it undercuts the
+fast-feedback purpose of the TST-25 Windows tier and makes CI fragile to a single host: if
+`10.100.0.35` wedges or goes down, CI for both repos stops with no failover.
+
+## Options (cheapest first)
+
+- **(a) Register a second `act_runner` instance on `10.100.0.35` — recommended.** The host
+  already runs `gitea-runner`; add a second `act_runner` container (or raise the existing
+  runner's `maxParallel` where the docker-in-docker/resource budget allows) so at least two
+  jobs run concurrently. Cheapest change, and it keeps the runner co-located on the
+  `container.network: traefik` network that resolves `gitea:3000` — the property TST-03
+  depended on. **Use the same `container.network: traefik` config as the existing runner.**
+- **(b) Dedicate a labelled runner to `mxaccessgw`.** Cleaner isolation — `lmxopcua` load
+  never blocks this repo — but needs label wiring: register the new runner with a distinct
+  label (e.g. `mxgw`) and change `.gitea/workflows/ci.yml`'s `runs-on:` for this repo's jobs
+  to gate on that label (e.g. `runs-on: [ubuntu-latest, mxgw]`). Only do this if (a) proves
+  insufficient — it is more moving parts for the same throughput gain, and it means `ci.yml`
+  changes, which is out of scope for the doc-only half of TST-30.
+- **(c) Put the runner on windev / a second host — rejected as the primary fix.** windev is
+  the Windows build target (`10.100.0.48`), not a CI host, and co-locating a Linux runner
+  there loses the `gitea:3000` name resolution TST-03 relies on. Only consider if
+  `10.100.0.35` genuinely runs out of capacity for a second instance.
+
+Default to **(a)**. Escalate to (b) only if `lmxopcua` contention persists after a second
+instance is online (i.e., (a) is not sufficient because the two repos' combined load exceeds
+two slots).
+
+## Preconditions
+
+- SSH/docker access to `10.100.0.35`.
+- The existing `gitea-runner` container's compose/run config, to copy its
+  `container.network: traefik` setting and registration token flow (repo memory
+  `project_gitea_ci` records this configuration).
+- Admin access to Gitea (`gitea.dohertylan.com`) to mint a new runner registration token.
+
+## Steps — option (a): second runner instance
+
+1. On `10.100.0.35`, locate the existing `gitea-runner` container/compose definition and copy
+   its configuration for a new instance (same `container.network: traefik`, same Docker
+   socket mount if it uses docker-in-docker, a distinct container name/data volume).
+2. In Gitea, generate a new runner registration token (instance-level, since the existing
+   runner is registered at the instance level too — Admin → Actions → Runners, or
+   `POST /admin/actions/runners/registration-token`).
+3. Register and start the second `act_runner` instance with that token, pointed at the same
+   Gitea origin.
+4. Confirm both runners show online: instance runner list in the Gitea admin UI, or the
+   equivalent API listing.
+
+## Verification
+
+- Push two branches to `mxaccessgw` back-to-back (or trigger one `mxaccessgw` push while an
+  `lmxopcua` run is in flight) and confirm both runs execute **concurrently**, not serially —
+  the second run's jobs should start before the first finishes, not queue behind it.
+- `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner listing) shows
+  **≥2** runners online.
+- Re-run the TST-25 acceptance push (a plain push to a scratch branch) and confirm queue depth
+  is materially lower than the ~20–30 minute baseline observed under a concurrent `lmxopcua`
+  run.
+- Confirm `windows-x86` still resolves `gitea:3000` correctly from a job scheduled on the new
+  runner instance (the `traefik` network property must hold for both instances).
+
+## The no-cancel reality does not go away
+
+A second runner relieves contention; it does not add a cancel/delete API — Gitea 1.26 still
+returns 404/400 for both. A stale or hung run on either runner still holds its slot until it
+finishes or times out. Two runners just means one stale run blocks at most half the capacity
+instead of all of it. Do not treat the second runner as a substitute for the escape hatch: a
+specific commit can still be verified out of band without waiting on either runner via
+`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` (Linux, needs SSH access to
+windev) or the manual windev worktree flow — see the "Runner capacity is shared and finite"
+section in `docs/GatewayTesting.md`.
+
+## Optional: workflow-level `concurrency` group
+
+As belt-and-suspenders against the missing cancel API, `.gitea/workflows/ci.yml` could add a
+top-level `concurrency` group (e.g. keyed on `${{ github.ref }}`) so a newer push to the same
+branch automatically supersedes an in-flight run instead of both running to completion.
+**Verify this Gitea deployment actually honors `concurrency` and cancels the superseded run
+before relying on it** — Gitea Actions' YAML surface does not track GitHub Actions feature
+parity release-for-release, and a `concurrency` block that is silently ignored would look like
+a working safeguard while doing nothing. If verified working, this is a `ci.yml` change (not
+covered by this runbook) and should land as its own small change with its own verification
+(push twice to the same branch quickly, confirm the first run's jobs cancel).
+
+## Done criteria
+
+- A second `act_runner` instance (or raised `maxParallel`) is online on `10.100.0.35` with the
+  same `container.network: traefik` configuration as the existing runner.
+- `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance listing) shows ≥2 runners.
+- Two concurrent runs (one `mxaccessgw`, one `lmxopcua`, or two `mxaccessgw` pushes) execute
+  in parallel rather than serially.
+- `docs/GatewayTesting.md`'s shared-runner/no-cancel prose and the `run-windev-ci.sh` bypass
+  remain accurate (they describe the bypass as still valid, which it is regardless of runner
+  count).