mxaccessgw/docs/runbooks/TST-30-second-ci-runner.md

TST-30 — Register A Second CI Runner (Operator Runbook)

Executed 2026-08-07 — option (a) shipped; this runbook is now history plus the one correction below. gitea-runner-2 (runner id 5, capacity 2, labels ubuntu-latest/ ubuntu-22.04) runs on 10.100.0.35 from the /opt/gitea compose stack with the same container.network: traefik setting as the original; its registration token is mounted from a 0600 file rather than inlined in compose. The existing gitea-runner (id 1, capacity 4) was left untouched, so capacity went 4 → 6 by addition and the change reverts by removing one container. Concurrency was verified by pushing HEAD to two scratch branches while an unrelated run was in flight: jobs from three runs ran simultaneously across both runners, and a gitea-runner-2 job cloned successfully from http://gitea:3000 (the property option (c) was rejected for losing).

Correction to the Verification and Done-criteria sections below: they expect GET /repos/dohertj2/mxaccessgw/actions/runners to show ≥2 runners. It does not — it still returns total_count: 0, correctly, because both runners are registered at the instance level, which is the very condition the "Why" section describes. Use GET /api/v1/admin/actions/runners instead (it now lists only id 1 and id 5; id 4 was removed 2026-08-07 — a local macOS act_runner mislabelled ubuntu-latest/ubuntu-22.04/ubuntu-20.04, so it captured Linux-labelled jobs it had no Docker daemon to run and failed them. Its config and registration are kept disabled at ~/gitea-act-runner.disabled-2026-08-07 (launchd plist at ~/Library/LaunchAgents/com.dohertj2.gitea-act-runner.plist.disabled-2026-08-07) so it can be re-registered with mac-specific labels if a mac-only job ever needs one). For per-job runner attribution, GET /repos/{owner}/{repo}/actions/runs/{id}/jobs exposes runner_id/runner_name on each job; the actions/tasks listing does not.

Follow-up 2026-08-07 — token hygiene on the host. gitea-runner (id 1) now takes its registration token from the same 0600 file mount runner-2 uses instead of an inline plaintext value in compose, and /opt/gitea/docker-compose.yml plus both .bak copies are 0600 root:root; runner-1 was recreated alone and kept its identity (.runner byte-identical). Both runners share one instance-scope registration token, which was world-readable for roughly five months and is still live — a probe registered runner id 6 with it, then deleted it. Gitea 1.26.4 cannot rotate that token from the CLI or the API (both endpoints are get-or-create and return the same value), so the reset is a pending operator action in the admin web UI ("Reset registration token"). After the reset, refresh /opt/gitea/runner_token with the new value and shred the two token-bearing compose backups, which are the last copies of the old one.

Operator steps to relieve the single shared Gitea Actions runner that CI depends on. The repo-side half of TST-30 (documenting the shared-runner/no-cancel reality and the run-windev-ci.sh bypass) is already landed in docs/GatewayTesting.md; registering the second runner below is infrastructure work outside this repo's tree and is yours to execute.

Why

All CI for this repo runs on one co-located gitea-runner container on docker host 10.100.0.35 with maxParallel=1. That runner is registered at the instance level, not scoped to this repo (GET /repos/dohertj2/mxaccessgw/actions/runners returns total_count: 0), so it is shared with dohertj2/lmxopcua and every job in every run across both repos executes serially on the single slot. A mxaccessgw push fans out to portable, java, windows-x86, and an active lmxopcua run blocks all of them — queue depth of ~20–30 minutes was observed during TST-25 acceptance under cross-repo contention. Gitea 1.26 also exposes no run cancel or delete via the API (POST .../actions/runs/{id}/cancel → 404, DELETE .../actions/runs/{id} → 400), so a superseded or hung run cannot be cleared and holds the slot until it finishes or times out. This is not a correctness problem — every job still reports accurately — but it undercuts the fast-feedback purpose of the TST-25 Windows tier and makes CI fragile to a single host: if 10.100.0.35 wedges or goes down, CI for both repos stops with no failover.

Options (cheapest first)

• (a) Register a second act_runner instance on 10.100.0.35 — recommended. The host already runs gitea-runner; add a second act_runner container (or raise the existing runner's maxParallel where the docker-in-docker/resource budget allows) so at least two jobs run concurrently. Cheapest change, and it keeps the runner co-located on the container.network: traefik network that resolves gitea:3000 — the property TST-03 depended on. Use the same container.network: traefik config as the existing runner.
• (b) Dedicate a labelled runner to mxaccessgw. Cleaner isolation — lmxopcua load never blocks this repo — but needs label wiring: register the new runner with a distinct label (e.g. mxgw) and change .gitea/workflows/ci.yml's runs-on: for this repo's jobs to gate on that label (e.g. runs-on: [ubuntu-latest, mxgw]). Only do this if (a) proves insufficient — it is more moving parts for the same throughput gain, and it means ci.yml changes, which is out of scope for the doc-only half of TST-30.
• (c) Put the runner on windev / a second host — rejected as the primary fix. windev is the Windows build target (10.100.0.48), not a CI host, and co-locating a Linux runner there loses the gitea:3000 name resolution TST-03 relies on. Only consider if 10.100.0.35 genuinely runs out of capacity for a second instance.

Default to (a). Escalate to (b) only if lmxopcua contention persists after a second instance is online (i.e., (a) is not sufficient because the two repos' combined load exceeds two slots).

Preconditions

• SSH/docker access to 10.100.0.35.
• The existing gitea-runner container's compose/run config, to copy its container.network: traefik setting and registration token flow (repo memory project_gitea_ci records this configuration).
• Admin access to Gitea (gitea.dohertylan.com) to mint a new runner registration token.

Steps — option (a): second runner instance

1. On 10.100.0.35, locate the existing gitea-runner container/compose definition and copy its configuration for a new instance (same container.network: traefik, same Docker socket mount if it uses docker-in-docker, a distinct container name/data volume).
2. In Gitea, generate a new runner registration token (instance-level, since the existing runner is registered at the instance level too — Admin → Actions → Runners, or POST /admin/actions/runners/registration-token).
3. Register and start the second act_runner instance with that token, pointed at the same Gitea origin.
4. Confirm both runners show online: instance runner list in the Gitea admin UI, or the equivalent API listing.

Verification

• Push two branches to mxaccessgw back-to-back (or trigger one mxaccessgw push while an lmxopcua run is in flight) and confirm both runs execute concurrently, not serially — the second run's jobs should start before the first finishes, not queue behind it.
• GET /repos/dohertj2/mxaccessgw/actions/runners (or the instance runner listing) shows ≥2 runners online.
• Re-run the TST-25 acceptance push (a plain push to a scratch branch) and confirm queue depth is materially lower than the ~20–30 minute baseline observed under a concurrent lmxopcua run.
• Confirm windows-x86 still resolves gitea:3000 correctly from a job scheduled on the new runner instance (the traefik network property must hold for both instances).

The no-cancel reality does not go away

A second runner relieves contention; it does not add a cancel/delete API — Gitea 1.26 still returns 404/400 for both. A stale or hung run on either runner still holds its slot until it finishes or times out. Two runners just means one stale run blocks at most half the capacity instead of all of it. Do not treat the second runner as a substitute for the escape hatch: a specific commit can still be verified out of band without waiting on either runner via CI_SHA=<sha> scripts/ci/run-windev-ci.sh <build|test|live> (Linux, needs SSH access to windev) or the manual windev worktree flow — see the "Runner capacity is shared and finite" section in docs/GatewayTesting.md.

Optional: workflow-level concurrency group

As belt-and-suspenders against the missing cancel API, .gitea/workflows/ci.yml could add a top-level concurrency group (e.g. keyed on ${{ github.ref }}) so a newer push to the same branch automatically supersedes an in-flight run instead of both running to completion. Verify this Gitea deployment actually honors concurrency and cancels the superseded run before relying on it — Gitea Actions' YAML surface does not track GitHub Actions feature parity release-for-release, and a concurrency block that is silently ignored would look like a working safeguard while doing nothing. If verified working, this is a ci.yml change (not covered by this runbook) and should land as its own small change with its own verification (push twice to the same branch quickly, confirm the first run's jobs cancel).

Done criteria

• A second act_runner instance (or raised maxParallel) is online on 10.100.0.35 with the same container.network: traefik configuration as the existing runner.
• GET /repos/dohertj2/mxaccessgw/actions/runners (or the instance listing) shows ≥2 runners.
• Two concurrent runs (one mxaccessgw, one lmxopcua, or two mxaccessgw pushes) execute in parallel rather than serially.
• docs/GatewayTesting.md's shared-runner/no-cancel prose and the run-windev-ci.sh bypass remain accurate (they describe the bypass as still valid, which it is regardless of runner count).