docs(tst-30): second runner gitea-runner-2 live; close operator action

2026-08-07 09:12:00 -04:00
parent a346d514dd
commit 41e86481e2
4 changed files with 25 additions and 3 deletions
@@ -15,7 +15,7 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior
 | TST-27 | Medium | P1 (doc batch) | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
 | TST-28 | Low | P2 | S | relates IPC-02 | Done | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
 | TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
-| TST-30 | Low | P2 | M | — | Done (doc half; runner registration operator-pending per runbook) | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |
+| TST-30 | Low | P2 | M | — | Done | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |
 ---
@@ -169,6 +169,8 @@ Independent of the runner count, document the **no-cancel** reality (Gitea 1.26
 **Outcome (2026-08-07 — Done, doc half; runner registration operator-pending).** Landed on `fix/tst-30-runner-docs`. Implementation step 2 shipped: `docs/GatewayTesting.md`'s Continuous Integration section gained a "Runner capacity is shared and finite" subsection stating the `maxParallel=1` co-located runner is shared with `dohertj2/lmxopcua` at the instance level (not repo-scoped), the ~20–30 minute queue latency observed under cross-repo contention, and the Gitea 1.26 no-cancel/no-delete API reality; the existing "windev tier down" degraded-mode paragraph now also covers "runner contended" as a reason to use the bypass, generalized per this finding's design note. New operator runbook `docs/runbooks/TST-30-second-ci-runner.md` carries **step 1** (register a second `act_runner` on `10.100.0.35`, option (a) recommended, same `container.network: traefik` config; option (b) dedicated labelled runner as an escalation; option (c) windev-hosted runner rejected) with the verification checklist (concurrent back-to-back pushes, `GET /repos/dohertj2/mxaccessgw/actions/runners` ≥ 2) and a note that the no-cancel reality persists regardless of runner count. **Step 3 (optional workflow-level `concurrency` group)** is documented in the runbook as unverified — explicitly framed as "verify this Gitea deployment honors it before relying on it" — and left unimplemented in `ci.yml`, since it is a `ci.yml` change out of scope for this doc-only pass. **The actual runner registration (step 1) is infrastructure work outside this repo's tree and remains the operator's to execute**, tracked in the runbook. Verification performed: `grep -n 'maxParallel\|shared\|cancel' docs/GatewayTesting.md` shows the new prose; runbook file exists at the path above; no build required (doc-only change).
 **Outcome (2026-08-07 — operator half executed; finding now fully `Done`).** The runner registration left open above was executed per `docs/runbooks/TST-30-second-ci-runner.md` option (a). A second instance-level `act_runner` container, `gitea-runner-2` (runner id 5, capacity 2, labels `ubuntu-latest`/`ubuntu-22.04`), now runs on `10.100.0.35` from the `/opt/gitea` compose stack with the same `container.network: traefik` setting as the original; its registration token is mounted from a `0600` file rather than inlined in compose. The existing `gitea-runner` (id 1, capacity 4) was **not** modified — capacity went 4 → 6 by addition, so the change is reversible by removing one container. Concurrency verified live by pushing HEAD (`a346d51`) to two scratch branches, `scratch/tst30-a` (run 661) and `scratch/tst30-b` (run 662), while an unrelated run (660) was already in flight: at 13:07:53Z jobs from **three** runs were `in_progress` simultaneously — run 660 `portable` and run 662 `portable`/`java` on runner 1, run 661 `portable`/`java` on `gitea-runner-2` — which the pre-change single-runner topology could not have produced. `gitea:3000` resolution holds on the new instance: run 661's `portable` job (task 1145, scheduled on `gitea-runner-2`) logged `git remote add origin http://gitea:3000/dohertj2/mxaccessgw` followed by a successful `fetch … From http://gitea:3000/dohertj2/mxaccessgw`, and its job container's workspace was confirmed checked out at `a346d514dd24e775640e5667aa7cd8e561fec68a`. **Runbook correction:** its verification checklist said `GET /repos/dohertj2/mxaccessgw/actions/runners` should show ≥2 — that endpoint still returns `total_count: 0` because both runners are registered at the **instance** level, exactly as this finding documented; the correct check is `GET /api/v1/admin/actions/runners`, which lists ids 1, 4 (an unrelated local macOS runner), and 5. Recorded as a dated "Executed" note at the top of the runbook. The no-cancel reality is unchanged and the `run-windev-ci.sh` bypass remains valid, so `docs/GatewayTesting.md`'s prose needed no edit.
 ---
 ## Cross-domain dependencies
@@ -15,5 +15,5 @@ These were discovered while remediating the 2026-07-12 backlog but were **out of
 These are **live-infrastructure actions the operator must execute** — the repo-side work is complete and merged:
 - **SEC-36** — rotate the dev LDAP service-account credential per `docs/runbooks/SEC-36-ldap-credential-rotation.md` (generate new secret in `scadaproj/infra/glauth`, pre-stage the NSSM env var on deployed hosts, rotate GLAuth on `10.100.0.35`, verify dashboard login). The committed literal is gone from the working tree but remains recoverable from git history until rotation completes — **rotation is the load-bearing half.**
- **TST-30** — register a second Gitea `act_runner` on `10.100.0.35` per `docs/runbooks/TST-30-second-ci-runner.md` to relieve the single-shared-runner bottleneck.
+- ~~**TST-30** — register a second Gitea `act_runner` on `10.100.0.35` per `docs/runbooks/TST-30-second-ci-runner.md` to relieve the single-shared-runner bottleneck.~~ **Executed 2026-08-07**: `gitea-runner-2` (id 5, capacity 2) is online on `10.100.0.35` via the `/opt/gitea` compose stack, same `container.network: traefik`, token from a `0600` file mount; the existing runner (id 1, capacity 4) was untouched. Concurrency verified — jobs from three runs ran simultaneously across both runners, and a `gitea-runner-2` job cloned successfully from `http://gitea:3000`. TST-30 is now fully `Done`.
 - **TST-25 follow-ups** — old **TST-05** (scheduled live-MXAccess smoke) is now covered by the `nightly-windev` job; old **TST-24** (client wire tests in CI) is unblocked by the working Windows tier.
@@ -1,5 +1,24 @@
 # TST-30 — Register A Second CI Runner (Operator Runbook)
 > **Executed 2026-08-07 — option (a) shipped; this runbook is now history plus the one
 > correction below.** `gitea-runner-2` (runner id 5, capacity 2, labels `ubuntu-latest`/
 > `ubuntu-22.04`) runs on `10.100.0.35` from the `/opt/gitea` compose stack with the same
 > `container.network: traefik` setting as the original; its registration token is mounted from a
 > `0600` file rather than inlined in compose. The existing `gitea-runner` (id 1, capacity 4) was
 > left untouched, so capacity went 4 → 6 by addition and the change reverts by removing one
 > container. Concurrency was verified by pushing HEAD to two scratch branches while an unrelated
 > run was in flight: jobs from three runs ran simultaneously across both runners, and a
 > `gitea-runner-2` job cloned successfully from `http://gitea:3000` (the property option (c) was
 > rejected for losing).
 >
 > **Correction to the Verification and Done-criteria sections below:** they expect
 > `GET /repos/dohertj2/mxaccessgw/actions/runners` to show ≥2 runners. It does not — it still
 > returns `total_count: 0`, correctly, because both runners are registered at the **instance**
 > level, which is the very condition the "Why" section describes. Use
 > `GET /api/v1/admin/actions/runners` instead (it lists id 1, id 4 — an unrelated local macOS
 > runner — and id 5). For per-job runner attribution, `GET /repos/{owner}/{repo}/actions/runs/{id}/jobs`
 > exposes `runner_id`/`runner_name` on each job; the `actions/tasks` listing does not.
 Operator steps to relieve the single shared Gitea Actions runner that CI depends on. The
 repo-side half of TST-30 (documenting the shared-runner/no-cancel reality and the
 `run-windev-ci.sh` bypass) is already landed in `docs/GatewayTesting.md`; registering the