Merge branch 'fix/tst-30-runner-docs'
ci / java (push) Successful in 2m11s
ci / nightly-windev (push) Has been skipped
ci / windows-x86 (push) Failing after 1m38s
ci / portable (push) Failing after 17m43s

# Conflicts:
#	archreview/2026-07-12/remediation/00-tracking.md
This commit is contained in:
Joseph Doherty
2026-08-07 07:50:21 -04:00
4 changed files with 151 additions and 7 deletions
File diff suppressed because one or more lines are too long
@@ -15,7 +15,7 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior
| TST-27 | Medium | P1 (doc batch) | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-27 | Medium | P1 (doc batch) | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
| TST-28 | Low | P2 | S | relates IPC-02 | Done | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite | | TST-28 | Low | P2 | S | relates IPC-02 | Done | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
| TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts | | TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) | | TST-30 | Low | P2 | M | — | Done (doc half; runner registration operator-pending per runbook) | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |
--- ---
@@ -167,6 +167,8 @@ Independent of the runner count, document the **no-cancel** reality (Gitea 1.26
**Verification.** Push two branches back-to-back and confirm their runs execute concurrently (not serially) once a second runner exists; `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner list) shows ≥2 runners online; `docs/GatewayTesting.md` describes the shared-runner/no-cancel reality and the bypass. Re-run the TST-25 acceptance push and confirm queue depth is materially lower under a concurrent `lmxopcua` run. **Verification.** Push two branches back-to-back and confirm their runs execute concurrently (not serially) once a second runner exists; `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner list) shows ≥2 runners online; `docs/GatewayTesting.md` describes the shared-runner/no-cancel reality and the bypass. Re-run the TST-25 acceptance push and confirm queue depth is materially lower under a concurrent `lmxopcua` run.
**Outcome (2026-08-07 — Done, doc half; runner registration operator-pending).** Landed on `fix/tst-30-runner-docs`. Implementation step 2 shipped: `docs/GatewayTesting.md`'s Continuous Integration section gained a "Runner capacity is shared and finite" subsection stating the `maxParallel=1` co-located runner is shared with `dohertj2/lmxopcua` at the instance level (not repo-scoped), the ~2030 minute queue latency observed under cross-repo contention, and the Gitea 1.26 no-cancel/no-delete API reality; the existing "windev tier down" degraded-mode paragraph now also covers "runner contended" as a reason to use the bypass, generalized per this finding's design note. New operator runbook `docs/runbooks/TST-30-second-ci-runner.md` carries **step 1** (register a second `act_runner` on `10.100.0.35`, option (a) recommended, same `container.network: traefik` config; option (b) dedicated labelled runner as an escalation; option (c) windev-hosted runner rejected) with the verification checklist (concurrent back-to-back pushes, `GET /repos/dohertj2/mxaccessgw/actions/runners` ≥ 2) and a note that the no-cancel reality persists regardless of runner count. **Step 3 (optional workflow-level `concurrency` group)** is documented in the runbook as unverified — explicitly framed as "verify this Gitea deployment honors it before relying on it" — and left unimplemented in `ci.yml`, since it is a `ci.yml` change out of scope for this doc-only pass. **The actual runner registration (step 1) is infrastructure work outside this repo's tree and remains the operator's to execute**, tracked in the runbook. Verification performed: `grep -n 'maxParallel\|shared\|cancel' docs/GatewayTesting.md` shows the new prose; runbook file exists at the path above; no build required (doc-only change).
--- ---
## Cross-domain dependencies ## Cross-domain dependencies
+33 -5
View File
@@ -453,6 +453,32 @@ runtime because the x86 Worker cannot build on Linux:
`xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On
failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page. failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page.
### Runner capacity is shared and finite
All CI runs on a co-located `gitea-runner` container on docker host `10.100.0.35` with
`maxParallel=1`, and that runner instance is **shared across repos** — it interleaves
`dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs on the single slot rather than being
scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns
`total_count: 0`; the runner is registered at the instance level). Every job in a run
(`portable`, `java`, `windows-x86`) executes serially, so queue latency is additive within
a run and an active `lmxopcua` run blocks `mxaccessgw` entirely — expect ~2030 minutes of
queue depth under cross-repo contention, not a stuck pipeline. This Gitea version (1.26)
also exposes **no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel`
returns 404, `DELETE .../actions/runs/{id}` returns 400), so a superseded or hung run cannot
be cleared and holds the slot until it finishes or times out. See
`docs/runbooks/TST-30-second-ci-runner.md` for the operator runbook that registers a second
runner to relieve this; until that lands, treat single-runner contention as expected, not a
CI outage.
When queue depth (or the missing-cancel reality) makes waiting impractical, verify a
specific commit out of band instead of waiting behind the queue: run
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` from a machine with SSH access
to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see
`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is
the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from
"tier down" to "runner contended": either way, a stuck or slow shared runner should not
block verifying a commit.
The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the
committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored
protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current
@@ -467,11 +493,13 @@ the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGenerat
[Contracts](./Contracts.md). [Contracts](./Contracts.md).
If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI
key/secret rotation in flight), fall back to the manual windev worktree procedure as a key/secret rotation in flight) **or** the shared Gitea runner is contended and the queue is
degraded mode: on windev, fast-forward an isolated `origin/main` worktree under `C:\build` impractical to wait behind (see "Runner capacity is shared and finite" above), fall back to
(never the dirty Desktop checkout), then run the x86 Worker build and `Worker.Tests` the manual windev worktree procedure as a degraded mode: on windev, fast-forward an isolated
(`-p:Platform=x86`) there by hand. Do this per merge for worker-touching changes until the `origin/main` worktree under `C:\build` (never the dirty Desktop checkout), then run the x86
`windows-x86` job is green again. Worker build and `Worker.Tests` (`-p:Platform=x86`) there by hand. Do this per merge for
worker-touching changes until the `windows-x86` job is green again (tier-down case) or the
queue clears (contention case).
## Related Documentation ## Related Documentation
+113
View File
@@ -0,0 +1,113 @@
# TST-30 — Register A Second CI Runner (Operator Runbook)
Operator steps to relieve the single shared Gitea Actions runner that CI depends on. The
repo-side half of TST-30 (documenting the shared-runner/no-cancel reality and the
`run-windev-ci.sh` bypass) is already landed in `docs/GatewayTesting.md`; registering the
second runner below is infrastructure work outside this repo's tree and is yours to execute.
## Why
All CI for this repo runs on one co-located `gitea-runner` container on docker host
`10.100.0.35` with `maxParallel=1`. That runner is registered at the **instance** level, not
scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns
`total_count: 0`), so it is shared with `dohertj2/lmxopcua` and every job in every run across
both repos executes serially on the single slot. A `mxaccessgw` push fans out to `portable`,
`java`, `windows-x86`, and an active `lmxopcua` run blocks all of them — queue depth of
~2030 minutes was observed during TST-25 acceptance under cross-repo contention. Gitea 1.26
also exposes **no run cancel or delete via the API**
(`POST .../actions/runs/{id}/cancel` → 404, `DELETE .../actions/runs/{id}` → 400), so a
superseded or hung run cannot be cleared and holds the slot until it finishes or times out.
This is not a correctness problem — every job still reports accurately — but it undercuts the
fast-feedback purpose of the TST-25 Windows tier and makes CI fragile to a single host: if
`10.100.0.35` wedges or goes down, CI for both repos stops with no failover.
## Options (cheapest first)
- **(a) Register a second `act_runner` instance on `10.100.0.35` — recommended.** The host
already runs `gitea-runner`; add a second `act_runner` container (or raise the existing
runner's `maxParallel` where the docker-in-docker/resource budget allows) so at least two
jobs run concurrently. Cheapest change, and it keeps the runner co-located on the
`container.network: traefik` network that resolves `gitea:3000` — the property TST-03
depended on. **Use the same `container.network: traefik` config as the existing runner.**
- **(b) Dedicate a labelled runner to `mxaccessgw`.** Cleaner isolation — `lmxopcua` load
never blocks this repo — but needs label wiring: register the new runner with a distinct
label (e.g. `mxgw`) and change `.gitea/workflows/ci.yml`'s `runs-on:` for this repo's jobs
to gate on that label (e.g. `runs-on: [ubuntu-latest, mxgw]`). Only do this if (a) proves
insufficient — it is more moving parts for the same throughput gain, and it means `ci.yml`
changes, which is out of scope for the doc-only half of TST-30.
- **(c) Put the runner on windev / a second host — rejected as the primary fix.** windev is
the Windows build target (`10.100.0.48`), not a CI host, and co-locating a Linux runner
there loses the `gitea:3000` name resolution TST-03 relies on. Only consider if
`10.100.0.35` genuinely runs out of capacity for a second instance.
Default to **(a)**. Escalate to (b) only if `lmxopcua` contention persists after a second
instance is online (i.e., (a) is not sufficient because the two repos' combined load exceeds
two slots).
## Preconditions
- SSH/docker access to `10.100.0.35`.
- The existing `gitea-runner` container's compose/run config, to copy its
`container.network: traefik` setting and registration token flow (repo memory
`project_gitea_ci` records this configuration).
- Admin access to Gitea (`gitea.dohertylan.com`) to mint a new runner registration token.
## Steps — option (a): second runner instance
1. On `10.100.0.35`, locate the existing `gitea-runner` container/compose definition and copy
its configuration for a new instance (same `container.network: traefik`, same Docker
socket mount if it uses docker-in-docker, a distinct container name/data volume).
2. In Gitea, generate a new runner registration token (instance-level, since the existing
runner is registered at the instance level too — Admin → Actions → Runners, or
`POST /admin/actions/runners/registration-token`).
3. Register and start the second `act_runner` instance with that token, pointed at the same
Gitea origin.
4. Confirm both runners show online: instance runner list in the Gitea admin UI, or the
equivalent API listing.
## Verification
- Push two branches to `mxaccessgw` back-to-back (or trigger one `mxaccessgw` push while an
`lmxopcua` run is in flight) and confirm both runs execute **concurrently**, not serially —
the second run's jobs should start before the first finishes, not queue behind it.
- `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner listing) shows
**≥2** runners online.
- Re-run the TST-25 acceptance push (a plain push to a scratch branch) and confirm queue depth
is materially lower than the ~2030 minute baseline observed under a concurrent `lmxopcua`
run.
- Confirm `windows-x86` still resolves `gitea:3000` correctly from a job scheduled on the new
runner instance (the `traefik` network property must hold for both instances).
## The no-cancel reality does not go away
A second runner relieves contention; it does not add a cancel/delete API — Gitea 1.26 still
returns 404/400 for both. A stale or hung run on either runner still holds its slot until it
finishes or times out. Two runners just means one stale run blocks at most half the capacity
instead of all of it. Do not treat the second runner as a substitute for the escape hatch: a
specific commit can still be verified out of band without waiting on either runner via
`CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live>` (Linux, needs SSH access to
windev) or the manual windev worktree flow — see the "Runner capacity is shared and finite"
section in `docs/GatewayTesting.md`.
## Optional: workflow-level `concurrency` group
As belt-and-suspenders against the missing cancel API, `.gitea/workflows/ci.yml` could add a
top-level `concurrency` group (e.g. keyed on `${{ github.ref }}`) so a newer push to the same
branch automatically supersedes an in-flight run instead of both running to completion.
**Verify this Gitea deployment actually honors `concurrency` and cancels the superseded run
before relying on it** — Gitea Actions' YAML surface does not track GitHub Actions feature
parity release-for-release, and a `concurrency` block that is silently ignored would look like
a working safeguard while doing nothing. If verified working, this is a `ci.yml` change (not
covered by this runbook) and should land as its own small change with its own verification
(push twice to the same branch quickly, confirm the first run's jobs cancel).
## Done criteria
- A second `act_runner` instance (or raised `maxParallel`) is online on `10.100.0.35` with the
same `container.network: traefik` configuration as the existing runner.
- `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance listing) shows ≥2 runners.
- Two concurrent runs (one `mxaccessgw`, one `lmxopcua`, or two `mxaccessgw` pushes) execute
in parallel rather than serially.
- `docs/GatewayTesting.md`'s shared-runner/no-cancel prose and the `run-windev-ci.sh` bypass
remain accurate (they describe the bypass as still valid, which it is regardless of runner
count).