From b604fed72b724fbc4ad68cf42c7948affc44dfba Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 7 Aug 2026 07:47:37 -0400 Subject: [PATCH] docs(TST-30): document shared-runner CI bottleneck + second-runner operator runbook Doc half of TST-30 (single shared Gitea runner is a CI throughput/availability bottleneck): docs/GatewayTesting.md's Continuous Integration section gains a "Runner capacity is shared and finite" subsection covering the maxParallel=1 instance-level runner shared with dohertj2/lmxopcua, the ~20-30 min queue latency observed under cross-repo contention, and Gitea 1.26's missing run cancel/delete API. The existing "windev tier down" degraded-mode paragraph now also covers "runner contended" as a reason to bypass the queue via CI_SHA= scripts/ci/run-windev-ci.sh or the manual windev worktree flow, generalizing it per the finding's design note. New operator runbook docs/runbooks/TST-30-second-ci-runner.md carries the actual runner registration (option a: second act_runner instance on 10.100.0.35 with the same container.network: traefik config, recommended; option b: dedicated labelled runner, escalation only; option c: runner on windev, rejected) plus verification steps and the no-cancel caveat. The optional workflow-level concurrency group is documented as unverified -- framed as "verify before relying on it" -- and left unimplemented in ci.yml, since registering the runner and any runs-on gating is operator/infra work outside this repo's tree. Tracking: TST-30 -> Done (doc half; runner registration operator-pending) in both registers + change-log row. --- .../2026-07-12/remediation/00-tracking.md | 3 +- .../remediation/60-testing-docs-gaps.md | 4 +- docs/GatewayTesting.md | 38 +++++- docs/runbooks/TST-30-second-ci-runner.md | 113 ++++++++++++++++++ 4 files changed, 151 insertions(+), 7 deletions(-) create mode 100644 docs/runbooks/TST-30-second-ci-runner.md diff --git a/archreview/2026-07-12/remediation/00-tracking.md b/archreview/2026-07-12/remediation/00-tracking.md index aeb4c3a..6a5a6cf 100644 --- a/archreview/2026-07-12/remediation/00-tracking.md +++ b/archreview/2026-07-12/remediation/00-tracking.md @@ -131,7 +131,7 @@ Full design + implementation for each row lives in the linked domain doc under i | TST-27 | Medium | P1 | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-28 | Low | P2 | S | relates IPC-02 (old) | Done | Gateway-side `max_frame_bytes` handshake untested in the CI-run suite | | TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` (fold Phase-5 governance into DesignDecisions.md); delete root artifacts | -| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete API) | +| TST-30 | Low | P2 | M | — | Done (doc half; runner registration operator-pending per runbook) | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete API) | ## Cross-cutting clusters @@ -176,3 +176,4 @@ Sequence these together rather than piecemeal — several are one change set spa | 2026-08-07 | Code-review follow-ups on the same branch (commit `6bc3f9b`). (1) **Important** — `ResolveDrainReplyByteBudget` was a step, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (exactly 1024 at the validator floor `MaxMessageBytes = 1024 + 64 KiB`), so a byte-heavy `DrainEvents` truncated on every call and the drain-until-empty loop never terminated. Now `Math.Max(frameMax - reserve, frameMax / 2)` — monotonic, never below half the frame max. New test `WorkerPipeSessionTests.DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates` drives a byte-heavy queue at the exact validator floor and asserts drain-to-empty with no head reported oversized. (2) **Hardening** — the reply-too-large fallback write is now itself size-guarded (`WriteReplyTooLargeFallbackAsync`, shared by the control and STA reply seams) so a pathologically tiny negotiated max below the gateway floor (the WRK-24 gap) cannot make even the backstop session-fatal; log-and-swallow, comment points at WRK-24. (3) **Comment** — corrected the `RepeatedFieldOverheadBytes` docs: `WorkerEvent.CalculateSize()` already includes the event's tag+length, so the 8 bytes is pure slack, not wrapper compensation. **Evidence** — macOS build 0/0, validator filter 4/4. windev (`windev-worker-ci.ps1 -Sha 6bc3f9b -Mode test`, 07:07): x86 Worker build 0/0, `Worker.Tests` **368 passed / 0 failed / 11 skipped**, script exit 0. (An earlier run of the same SHA flaked on the pre-existing `RunAsync_WhenStaActivityIsStale_WritesWatchdogFault` — a 5 s CTS timeout under first-run load, untouched by this change; it passed on the clean re-run and in both prior full runs.) | | 2026-08-07 | **P1 doc-drift batch → `Done`: TST-27, WRK-26 (discharges IPC-29), CLI-42, CLI-43, IPC-28** (branch `fix/doc-drift-batch`). Doc-only; no source, proto, or test changes — cross-checked against HEAD in this worktree. **TST-27**: `docs/GatewayConfiguration.md`'s `ShowTagValues` row no longer says "Reserved" — it now states what `false` (default) does (`DashboardEventBroadcaster` blanks tag values from a deep-cloned `MxEvent` before the SignalR events-hub mirror, metadata still renders), the security relevance (the per-session hub ACL, SEC-25 roadmap item 12, still does not exist, so this redaction is the only thing between a low-trust Viewer and other sessions' tag values), and the honest scope limit (the flag does **not** cover `/browse`). **WRK-26** (discharges **IPC-29**): `docs/MxAccessWorkerInstanceDesign.md`'s "Outbound Queues" section rewritten from the stale five-level priority list to the two-class `Control`/`Event` scheduler actually shipped (`WorkerFrameWriter`/`WorkerFrameWritePriority.cs`), with the collapsed-decision rationale recorded, and the overflow paragraph rewritten to the implemented fail-fast (`WorkerFault` category `QueueOverflow` → fault frame written → `RunAsync` unwinds → generic `WorkerExitCode.UnexpectedFailure`, dedicated code still open). `docs/WorkerFrameProtocol.md` gained a new "Write Scheduling And Sequencing" section: the two priority classes, enqueue-then-contend/single-lock-holder-drains-all, write-time peek-stamp-commit sequencing, per-frame-rejection vs. stream-failure semantics, and flush coalescing — stated truthfully as landed (WRK-23's peek-stamp-commit is live at HEAD) or not (the drain loop still awaits each event `WriteAsync` individually, so WRK-25's N-events-one-flush batching has **not** landed and the section says so explicitly). Cancellation is deliberately **not** documented as a firm contract — a one-paragraph placeholder notes it is pending WRK-22, which has not landed (confirmed by reading `WorkerFrameWriter.cs`: no `Claimed`/tombstone machinery exists yet). `gateway.md:328-330` was cross-checked and left unchanged — its sequence prose (both sides stamp at write, per GWC-28) already reads true. **CLI-42**: `clients/rust/README.md` and `docs/ClientPackaging.md`'s Rust section now document the vendored proto layout matching `clients/rust/build.rs` exactly — repo-path-first resolution (`../../src/ZB.MOM.WW.MxGateway.Contracts/Protos`) falling back to `clients/rust/protos/` when the canonical path is absent (published-tarball case), the same-commit refresh rule enforced by `scripts/check-codegen.ps1` Check 3, and why `cargo package`/`cargo publish` run without `--no-verify` (matches `scripts/pack-clients.ps1:190-192`). **CLI-43**: `docs/style-guides/JavaStyleGuide.md` line 8 now says "Target Java 17 (the Ignition 8.3 baseline...)" mirroring the CLI-12 wording, matching the shipped `clients/java/build.gradle` toolchain-17 build. **IPC-28**: `docs/Grpc.md`'s exception-mapping prose gained `CommandTooLarge` → `ResourceExhausted` (verified against the live `switch` in `Grpc/MxAccessGatewayService.cs:950-960`), and the `Invoke` section gained one sentence on the oversized-payload path (`WorkerClient.InvokeAsync` rejects at the enqueue boundary per-correlation, session not faulted — verified against `WorkerClient.cs:220-234`), cross-referencing the headroom rule already documented in `docs/GatewayConfiguration.md:120-129`. Did not touch the DrainEvents-truncation row or the proto/`Generated/` trees — those belong to a parallel codegen task per the handoff note. **Source files cross-read for accuracy** (no edits): `src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs`, `.../WorkerFrameWritePriority.cs`, `.../WorkerPipeSession.cs` (confirmed two-class scheduler, WRK-21/23/28/30 landed, WRK-25/WRK-22 not landed), `src/ZB.MOM.WW.MxGateway.Worker/WorkerApplication.cs` (exit-code mapping), `src/ZB.MOM.WW.MxGateway.Worker/MxAccess/MxAccessEventQueue.cs` (overflow fault path), `src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardEventBroadcaster.cs` + `Configuration/DashboardOptions.cs` + `docs/GatewayDashboardDesign.md:170` (ShowTagValues), `src/ZB.MOM.WW.MxGateway.Server/Grpc/MxAccessGatewayService.cs:940-963` + `Workers/WorkerClient.cs:205-244` + `Workers/WorkerClientErrorCode.cs` (CommandTooLarge mapping), `clients/rust/build.rs`, `clients/rust/Cargo.toml`, `scripts/check-codegen.ps1`, `scripts/pack-clients.ps1` (Rust vendoring), `gateway.md:326-360` (sequence-prose cross-check). Verification (greps, doc-only — no build required): `grep -n 'Reserved' docs/GatewayConfiguration.md` no longer matches the `ShowTagValues` row; `grep -n 'faults' docs/MxAccessWorkerInstanceDesign.md` shows no remaining five-level list; `grep -n 'scheduling' docs/WorkerFrameProtocol.md` finds the new section; `grep -rn 'Java 21' docs/style-guides/` empty; `grep -i vendored docs/ClientPackaging.md clients/rust/README.md` non-empty in both; `grep -n 'CommandTooLarge' docs/Grpc.md` shows the mapping. | | 2026-08-07 | **IPC-23 + IPC-24 + IPC-25 + IPC-32 → `Done`; GWC-25 deferred proto-comment resolved** (branch `fix/ipc-24-25-codegen`). The proto-comment/doc + regen wave for the P0 codegen-freshness cluster. **Proto comments (comment-only):** `mxaccess_worker.proto` `GatewayHello.max_frame_bytes` gained the every-frame-must-fit / reply-builders-truncate sentence (IPC-23); `mxaccess_gateway.proto` `DrainEventsReply` gained the count-cap+byte-cap / drain-until-empty comment (IPC-23); `mxaccess_gateway.proto` `ReplayGap.oldest_available_sequence` gained the empty-ring value definition (`highest observed + 1`, `oldest − 1` formula stays valid) resolving GWC-25's deferred amendment. **Regen wave:** `Contracts/Generated/{MxaccessGateway,MxaccessWorker}.cs` (XML-doc only, no descriptor delta), `clients/rust/protos/{mxaccess_gateway,mxaccess_worker}.proto` byte-copied, Go `internal/generated/{mxaccess_gateway,mxaccess_worker}.pb.go` (worker binding was genuinely stale — it lacked the whole `MaxFrameBytes` field/accessor/rawDesc, now refreshed; gateway.pb.go got the new doc comments), Python `generated/mxaccess_worker_pb2.py` (real descriptor delta — `max_frame_bytes` field + cascaded offsets), Java aggregates `MxaccessGateway.java`/`MxaccessWorker.java` (javadoc from the new proto comments — **zero** protobuf-version churn under the pinned toolchain), and the client descriptor set `mxaccessgw-client-v1.protoset`. **IPC-24:** regenerating Java on the pinned toolchain (grpc 1.76.0 / protobuf 4.33.1, Homebrew JDK 17) produced only the legitimate javadoc delta with no `validateProtobufGencodeVersion`/`major=/minor=/patch=` churn — so the preferred path was taken: the unconditional churn-revert step and its comment were **deleted** from `.gitea/workflows/ci.yml` (the `git diff --exit-code` gate is now a true message-level drift gate for the single-file Java aggregates). **IPC-25:** pinned the Go generators in `clients/go/generate-proto.ps1` (`protoc-gen-go` v1.36.11 assert, `protoc-gen-go-grpc` 1.6.2 assert, protoc warn-only; also fixed a latent `(if …)`→`$(if …)` parse bug that broke the script under pwsh 7); added **Check 4** to `scripts/check-codegen.ps1` (regenerate Go+Python bindings, fail on any diff, tool-missing FAILS not skips) and relabeled the banners `1/4`…`4/4` + header comment (**IPC-32** folded in); added the pinned-generator installs to the `portable` CI job before the codegen step. **Docs same commit:** `ClientProtoGeneration.md` (pinned-versions table + Go/Python Check-4 guard + Java-revert-deleted note), `Contracts.md` (four-check enumeration), `GatewayTesting.md` (java job + four-check prose), `clients/java/zb-mom-ww-mxgateway-client/build.gradle` checkGeneratedClean caveat. **IPC-23 DrainEvents-truncation doc deviation:** `docs/Grpc.md` has no DrainEvents row (it documents only the seven public RPCs; DrainEvents is a worker diagnostic), and `docs/WorkerFrameProtocol.md` (owned by the parallel doc batch) already carries the byte-cap/drain-until-empty prose via WRK-21 — so no Grpc.md edit was made. **Evidence (macOS):** `pwsh scripts/check-codegen.ps1` all four checks green (banners `1/4`…`4/4`); `dotnet build …NonWindows.slnx` 0 warnings/0 errors; `dotnet test …Tests --filter ~ClientProtoInputTests` green; `grep -c max_frame_bytes` → Go worker 2, Python worker 1 (both non-zero, were 0); `clients/go` gofmt clean + `go build` + `go test` ok; `clients/python` `pytest` 163 passed/1 skipped; Java `gradle generateProto` clean under JDK 17. **Negative-path proof:** added a scratch field to `mxaccess_worker.proto`, regenerated only `Contracts/Generated/`, reran `check-codegen.ps1` and confirmed **Check 4 fails naming both `clients/go/internal/generated` and `clients/python/…/generated`**, then reverted. **Not published** (CLI-39 gates Go/Python republish). | +| 2026-08-07 | **TST-30 → `Done` (doc half); runner registration operator-pending** (branch `fix/tst-30-runner-docs`). Doc + operator-runbook task only — no source/tests, per the finding's P2 scope. `docs/GatewayTesting.md`'s Continuous Integration section gained a "Runner capacity is shared and finite" subsection: the co-located `gitea-runner` on `10.100.0.35` runs `maxParallel=1` and is registered at the Gitea **instance** level (`GET /repos/dohertj2/mxaccessgw/actions/runners` → `total_count: 0`), so it interleaves with `dohertj2/lmxopcua` and every job in a run serializes — ~20–30 minute queue depth was observed under cross-repo contention during TST-25 acceptance. Documented that Gitea 1.26 exposes no run cancel/delete (`POST .../cancel` → 404, `DELETE` → 400), so a superseded or hung run holds its slot to completion. The existing "windev tier down" degraded-mode paragraph now also names "runner contended" as a reason to use the bypass — `CI_SHA= scripts/ci/run-windev-ci.sh ` or the manual windev worktree flow — generalizing it per the finding's design note (was tier-down-only). New operator runbook `docs/runbooks/TST-30-second-ci-runner.md`: implementation step 1 (register a second `act_runner` on `10.100.0.35`), preference-ordered per the design — (a) second instance with the same `container.network: traefik` config, **recommended**; (b) dedicated labelled runner + `ci.yml` `runs-on` gate, escalation only; (c) runner on windev, **rejected** (loses `gitea:3000` resolution TST-03 depends on) — plus the verification checklist (concurrent back-to-back pushes, `GET .../actions/runners` shows ≥2) and a reminder the no-cancel reality persists regardless of runner count. The optional workflow-level `concurrency` group (design step 3) is documented in the runbook as unverified, framed explicitly as "verify this Gitea deployment honors it before relying on it," and left **unimplemented** in `.gitea/workflows/ci.yml` — no `runs-on` or workflow edits were made, consistent with the task scoping the actual runner registration as operator/infra work outside this repo's tree. Tracking: both registers' TST-30 rows and this row. Verification (doc-only, no build required): `grep -n 'maxParallel\|shared\|cancel' docs/GatewayTesting.md` shows the new prose (the "Runner capacity is shared and finite" subsection, the no-cancel API detail, and the degraded-mode "runner contended" generalization — plus pre-existing unrelated `shared` hits in the LDAP section); `docs/runbooks/TST-30-second-ci-runner.md` exists. | diff --git a/archreview/2026-07-12/remediation/60-testing-docs-gaps.md b/archreview/2026-07-12/remediation/60-testing-docs-gaps.md index 9db9bd5..2b2fbdf 100644 --- a/archreview/2026-07-12/remediation/60-testing-docs-gaps.md +++ b/archreview/2026-07-12/remediation/60-testing-docs-gaps.md @@ -15,7 +15,7 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior | TST-27 | Medium | P1 (doc batch) | S | — | Done | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live | | TST-28 | Low | P2 | S | relates IPC-02 | Done | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite | | TST-29 | Low | P2 | S | — | Done | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts | -| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) | +| TST-30 | Low | P2 | M | — | Done (doc half; runner registration operator-pending per runbook) | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) | --- @@ -167,6 +167,8 @@ Independent of the runner count, document the **no-cancel** reality (Gitea 1.26 **Verification.** Push two branches back-to-back and confirm their runs execute concurrently (not serially) once a second runner exists; `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner list) shows ≥2 runners online; `docs/GatewayTesting.md` describes the shared-runner/no-cancel reality and the bypass. Re-run the TST-25 acceptance push and confirm queue depth is materially lower under a concurrent `lmxopcua` run. +**Outcome (2026-08-07 — Done, doc half; runner registration operator-pending).** Landed on `fix/tst-30-runner-docs`. Implementation step 2 shipped: `docs/GatewayTesting.md`'s Continuous Integration section gained a "Runner capacity is shared and finite" subsection stating the `maxParallel=1` co-located runner is shared with `dohertj2/lmxopcua` at the instance level (not repo-scoped), the ~20–30 minute queue latency observed under cross-repo contention, and the Gitea 1.26 no-cancel/no-delete API reality; the existing "windev tier down" degraded-mode paragraph now also covers "runner contended" as a reason to use the bypass, generalized per this finding's design note. New operator runbook `docs/runbooks/TST-30-second-ci-runner.md` carries **step 1** (register a second `act_runner` on `10.100.0.35`, option (a) recommended, same `container.network: traefik` config; option (b) dedicated labelled runner as an escalation; option (c) windev-hosted runner rejected) with the verification checklist (concurrent back-to-back pushes, `GET /repos/dohertj2/mxaccessgw/actions/runners` ≥ 2) and a note that the no-cancel reality persists regardless of runner count. **Step 3 (optional workflow-level `concurrency` group)** is documented in the runbook as unverified — explicitly framed as "verify this Gitea deployment honors it before relying on it" — and left unimplemented in `ci.yml`, since it is a `ci.yml` change out of scope for this doc-only pass. **The actual runner registration (step 1) is infrastructure work outside this repo's tree and remains the operator's to execute**, tracked in the runbook. Verification performed: `grep -n 'maxParallel\|shared\|cancel' docs/GatewayTesting.md` shows the new prose; runbook file exists at the path above; no build required (doc-only change). + --- ## Cross-domain dependencies diff --git a/docs/GatewayTesting.md b/docs/GatewayTesting.md index f55b9aa..63a4c38 100644 --- a/docs/GatewayTesting.md +++ b/docs/GatewayTesting.md @@ -452,6 +452,32 @@ runtime because the x86 Worker cannot build on Linux: `xUnit1030`). Gated `if: github.event_name == 'schedule'`, so it never gates a push. On failure it opens a Gitea issue via the Actions token, since nobody watches the Actions page. +### Runner capacity is shared and finite + +All CI runs on a co-located `gitea-runner` container on docker host `10.100.0.35` with +`maxParallel=1`, and that runner instance is **shared across repos** — it interleaves +`dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs on the single slot rather than being +scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns +`total_count: 0`; the runner is registered at the instance level). Every job in a run +(`portable`, `java`, `windows-x86`) executes serially, so queue latency is additive within +a run and an active `lmxopcua` run blocks `mxaccessgw` entirely — expect ~20–30 minutes of +queue depth under cross-repo contention, not a stuck pipeline. This Gitea version (1.26) +also exposes **no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel` +returns 404, `DELETE .../actions/runs/{id}` returns 400), so a superseded or hung run cannot +be cleared and holds the slot until it finishes or times out. See +`docs/runbooks/TST-30-second-ci-runner.md` for the operator runbook that registers a second +runner to relieve this; until that lands, treat single-runner contention as expected, not a +CI outage. + +When queue depth (or the missing-cancel reality) makes waiting impractical, verify a +specific commit out of band instead of waiting behind the queue: run +`CI_SHA= scripts/ci/run-windev-ci.sh ` from a machine with SSH access +to windev (the same script the SSH-driven `windows-x86`/`nightly-windev` jobs use — see +`scripts/ci/README.md`), or fall back to the manual windev worktree procedure below. This is +the same escape hatch used when the windev tier itself is down — TST-30 generalizes it from +"tier down" to "runner contended": either way, a stuck or slow shared runner should not +block verifying a commit. + The freshness guard `scripts/check-codegen.ps1` runs four checks and fails the build when the committed client descriptor set (Check 1), the C# `Generated/` (Check 2), the Rust vendored protos (Check 3), or the Go/Python client bindings (Check 4, IPC-25) no longer match the current @@ -466,11 +492,13 @@ the x86 build with `CS0246`). See [Client Proto Generation](./ClientProtoGenerat [Contracts](./Contracts.md). If the SSH-driven Windows tier is unavailable for infrastructure reasons (windev down, CI -key/secret rotation in flight), fall back to the manual windev worktree procedure as a -degraded mode: on windev, fast-forward an isolated `origin/main` worktree under `C:\build` -(never the dirty Desktop checkout), then run the x86 Worker build and `Worker.Tests` -(`-p:Platform=x86`) there by hand. Do this per merge for worker-touching changes until the -`windows-x86` job is green again. +key/secret rotation in flight) **or** the shared Gitea runner is contended and the queue is +impractical to wait behind (see "Runner capacity is shared and finite" above), fall back to +the manual windev worktree procedure as a degraded mode: on windev, fast-forward an isolated +`origin/main` worktree under `C:\build` (never the dirty Desktop checkout), then run the x86 +Worker build and `Worker.Tests` (`-p:Platform=x86`) there by hand. Do this per merge for +worker-touching changes until the `windows-x86` job is green again (tier-down case) or the +queue clears (contention case). ## Related Documentation diff --git a/docs/runbooks/TST-30-second-ci-runner.md b/docs/runbooks/TST-30-second-ci-runner.md new file mode 100644 index 0000000..e955fdf --- /dev/null +++ b/docs/runbooks/TST-30-second-ci-runner.md @@ -0,0 +1,113 @@ +# TST-30 — Register A Second CI Runner (Operator Runbook) + +Operator steps to relieve the single shared Gitea Actions runner that CI depends on. The +repo-side half of TST-30 (documenting the shared-runner/no-cancel reality and the +`run-windev-ci.sh` bypass) is already landed in `docs/GatewayTesting.md`; registering the +second runner below is infrastructure work outside this repo's tree and is yours to execute. + +## Why + +All CI for this repo runs on one co-located `gitea-runner` container on docker host +`10.100.0.35` with `maxParallel=1`. That runner is registered at the **instance** level, not +scoped to this repo (`GET /repos/dohertj2/mxaccessgw/actions/runners` returns +`total_count: 0`), so it is shared with `dohertj2/lmxopcua` and every job in every run across +both repos executes serially on the single slot. A `mxaccessgw` push fans out to `portable`, +`java`, `windows-x86`, and an active `lmxopcua` run blocks all of them — queue depth of +~20–30 minutes was observed during TST-25 acceptance under cross-repo contention. Gitea 1.26 +also exposes **no run cancel or delete via the API** +(`POST .../actions/runs/{id}/cancel` → 404, `DELETE .../actions/runs/{id}` → 400), so a +superseded or hung run cannot be cleared and holds the slot until it finishes or times out. +This is not a correctness problem — every job still reports accurately — but it undercuts the +fast-feedback purpose of the TST-25 Windows tier and makes CI fragile to a single host: if +`10.100.0.35` wedges or goes down, CI for both repos stops with no failover. + +## Options (cheapest first) + +- **(a) Register a second `act_runner` instance on `10.100.0.35` — recommended.** The host + already runs `gitea-runner`; add a second `act_runner` container (or raise the existing + runner's `maxParallel` where the docker-in-docker/resource budget allows) so at least two + jobs run concurrently. Cheapest change, and it keeps the runner co-located on the + `container.network: traefik` network that resolves `gitea:3000` — the property TST-03 + depended on. **Use the same `container.network: traefik` config as the existing runner.** +- **(b) Dedicate a labelled runner to `mxaccessgw`.** Cleaner isolation — `lmxopcua` load + never blocks this repo — but needs label wiring: register the new runner with a distinct + label (e.g. `mxgw`) and change `.gitea/workflows/ci.yml`'s `runs-on:` for this repo's jobs + to gate on that label (e.g. `runs-on: [ubuntu-latest, mxgw]`). Only do this if (a) proves + insufficient — it is more moving parts for the same throughput gain, and it means `ci.yml` + changes, which is out of scope for the doc-only half of TST-30. +- **(c) Put the runner on windev / a second host — rejected as the primary fix.** windev is + the Windows build target (`10.100.0.48`), not a CI host, and co-locating a Linux runner + there loses the `gitea:3000` name resolution TST-03 relies on. Only consider if + `10.100.0.35` genuinely runs out of capacity for a second instance. + +Default to **(a)**. Escalate to (b) only if `lmxopcua` contention persists after a second +instance is online (i.e., (a) is not sufficient because the two repos' combined load exceeds +two slots). + +## Preconditions + +- SSH/docker access to `10.100.0.35`. +- The existing `gitea-runner` container's compose/run config, to copy its + `container.network: traefik` setting and registration token flow (repo memory + `project_gitea_ci` records this configuration). +- Admin access to Gitea (`gitea.dohertylan.com`) to mint a new runner registration token. + +## Steps — option (a): second runner instance + +1. On `10.100.0.35`, locate the existing `gitea-runner` container/compose definition and copy + its configuration for a new instance (same `container.network: traefik`, same Docker + socket mount if it uses docker-in-docker, a distinct container name/data volume). +2. In Gitea, generate a new runner registration token (instance-level, since the existing + runner is registered at the instance level too — Admin → Actions → Runners, or + `POST /admin/actions/runners/registration-token`). +3. Register and start the second `act_runner` instance with that token, pointed at the same + Gitea origin. +4. Confirm both runners show online: instance runner list in the Gitea admin UI, or the + equivalent API listing. + +## Verification + +- Push two branches to `mxaccessgw` back-to-back (or trigger one `mxaccessgw` push while an + `lmxopcua` run is in flight) and confirm both runs execute **concurrently**, not serially — + the second run's jobs should start before the first finishes, not queue behind it. +- `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner listing) shows + **≥2** runners online. +- Re-run the TST-25 acceptance push (a plain push to a scratch branch) and confirm queue depth + is materially lower than the ~20–30 minute baseline observed under a concurrent `lmxopcua` + run. +- Confirm `windows-x86` still resolves `gitea:3000` correctly from a job scheduled on the new + runner instance (the `traefik` network property must hold for both instances). + +## The no-cancel reality does not go away + +A second runner relieves contention; it does not add a cancel/delete API — Gitea 1.26 still +returns 404/400 for both. A stale or hung run on either runner still holds its slot until it +finishes or times out. Two runners just means one stale run blocks at most half the capacity +instead of all of it. Do not treat the second runner as a substitute for the escape hatch: a +specific commit can still be verified out of band without waiting on either runner via +`CI_SHA= scripts/ci/run-windev-ci.sh ` (Linux, needs SSH access to +windev) or the manual windev worktree flow — see the "Runner capacity is shared and finite" +section in `docs/GatewayTesting.md`. + +## Optional: workflow-level `concurrency` group + +As belt-and-suspenders against the missing cancel API, `.gitea/workflows/ci.yml` could add a +top-level `concurrency` group (e.g. keyed on `${{ github.ref }}`) so a newer push to the same +branch automatically supersedes an in-flight run instead of both running to completion. +**Verify this Gitea deployment actually honors `concurrency` and cancels the superseded run +before relying on it** — Gitea Actions' YAML surface does not track GitHub Actions feature +parity release-for-release, and a `concurrency` block that is silently ignored would look like +a working safeguard while doing nothing. If verified working, this is a `ci.yml` change (not +covered by this runbook) and should land as its own small change with its own verification +(push twice to the same branch quickly, confirm the first run's jobs cancel). + +## Done criteria + +- A second `act_runner` instance (or raised `maxParallel`) is online on `10.100.0.35` with the + same `container.network: traefik` configuration as the existing runner. +- `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance listing) shows ≥2 runners. +- Two concurrent runs (one `mxaccessgw`, one `lmxopcua`, or two `mxaccessgw` pushes) execute + in parallel rather than serially. +- `docs/GatewayTesting.md`'s shared-runner/no-cancel prose and the `run-windev-ci.sh` bypass + remain accurate (they describe the bypass as still valid, which it is regardless of runner + count).