docs(TST-30): new finding — single shared Gitea runner is a CI throughput/availability bottleneck
Surfaced during TST-25 acceptance verification: all CI runs on one shared gitea-runner (maxParallel=1, co-located 10.100.0.35) interleaved with dohertj2/lmxopcua, and Gitea 1.26 has no run cancel/delete API, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner; document the no-cancel reality and the run-windev-ci.sh queue-bypass. Low/P2. Roll-ups updated.
This commit is contained in:
Joseph Doherty
@@ -22,8 +22,8 @@ Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation
|
||||
| Contracts & IPC | [30-contracts-ipc.md](30-contracts-ipc.md) | — | 3 | 5 | 2 | 10 |
|
||||
| Security & dashboard | [40-security-dashboard.md](40-security-dashboard.md) | — | 1 | 4 | 1 | 6 |
|
||||
| Clients | [50-clients.md](50-clients.md) | — | 5 | 6 | — | 11 |
|
||||
| Testing, docs & gaps | [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | 1 | 2 | 2 | — | 5 |
|
||||
| **Total** | | **1** | **14** | **28** | **4** | **47** |
|
||||
| Testing, docs & gaps | [60-testing-docs-gaps.md](60-testing-docs-gaps.md) | 1 | 2 | 3 | — | 6 |
|
||||
| **Total** | | **1** | **14** | **29** | **4** | **48** |
|
||||
|
||||
## Roadmap-tier roll-up
|
||||
|
||||
@@ -31,7 +31,7 @@ Source review: [`../00-overall.md`](../00-overall.md) · Per-domain remediation
|
||||
|---|---|:-:|---|
|
||||
| **P0** | Correctness & safety — all small-to-medium | 10 | GWC-25, WRK-21, IPC-23, IPC-24, IPC-25, IPC-30, SEC-31, SEC-32, CLI-35, CLI-36 |
|
||||
| **P1** | Process & hardening | 12 | GWC-24, WRK-26, SEC-33, SEC-36, CLI-37, CLI-38, CLI-39, CLI-42, CLI-45, TST-25, TST-26, TST-27 |
|
||||
| **P2** | Completeness & polish | 9 | GWC-26, GWC-27, GWC-28, WRK-25, IPC-26, IPC-27, SEC-34, TST-28, TST-29 |
|
||||
| **P2** | Completeness & polish | 10 | GWC-26, GWC-27, GWC-28, WRK-25, IPC-26, IPC-27, SEC-34, TST-28, TST-29, TST-30 |
|
||||
| **—** | Not individually in the roadmap (Lows/Infos) | 16 | GWC-29/30, WRK-22/23/24/27/28, IPC-28/29/31/32, SEC-35, CLI-40/41/43/44 |
|
||||
|
||||
### P0 — do first (10)
|
||||
@@ -131,6 +131,7 @@ Full design + implementation for each row lives in the linked domain doc under i
|
||||
| TST-27 | Medium | P1 | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
|
||||
| TST-28 | Low | P2 | S | relates IPC-02 (old) | Not started | Gateway-side `max_frame_bytes` handshake untested in the CI-run suite |
|
||||
| TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` (fold Phase-5 governance into DesignDecisions.md); delete root artifacts |
|
||||
| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete API) |
|
||||
|
||||
## Cross-cutting clusters
|
||||
|
||||
@@ -158,4 +159,5 @@ Sequence these together rather than piecemeal — several are one change set spa
|
||||
| 2026-07-13 | Initial tracking doc generated from the six domain remediation designs. All 47 findings `Not started` (IPC-31, SEC-35 `N/A`). |
|
||||
| 2026-07-13 | TST-25/TST-26 → `In progress` (branch `fix/tst-25-windev-ci`). Added `scripts/ci/{windev-worker-ci.ps1,run-windev-ci.sh,windev.known_hosts}`, `windows-x86` (per-push) + `nightly-windev` (scheduled) jobs in `ci.yml`, and the TST-26 doc/comment fixes (GatewayTesting.md, Contracts.md, check-codegen.ps1). Mechanism hand-verified on windev: `build`→0, bogus-SHA→nonzero (lock released), `test`→356 passed/0 failed in ~50s (per-push stays `test`, no demotion), and run-windev-ci.sh SSH+EncodedCommand exit-code propagation confirmed. |
|
||||
| 2026-07-13 | Operator bring-up complete: dedicated CI ed25519 key installed in windev `administrators_authorized_keys` (authorized into `dohertj2`, which owns the working MXAccess/toolchain env — a fresh OS account would break the build; the key is independently revocable), Gitea secrets `WINDEV_SSH_KEY`/`WINDEV_SSH_KNOWN_HOSTS` + variable `WINDEV_SSH_USER=dohertj2` stored, runner→`10.100.0.48:22` egress verified on the `traefik` net, issue-write confirmed. **TST-25/TST-26 → `Done`:** credentialed `windows-x86` ran GREEN on `d769244` (Gitea run #37) — Linux runner SSHed windev, checked out the SHA in `C:\build\mxaccessgw-ci` under lock, ran the x86 Worker build + `Worker.Tests`, exit 0; `nightly-windev` correctly skipped on the push event. Branch merged to `main`. Follow-ups (old tracker): revisit **TST-05** (scheduled live smoke — now covered by `nightly-windev`) and **TST-24** (client wire tests) which this unlocks. |
|
||||
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Pending: forced-failure nightly issue. Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests). |
|
||||
| 2026-07-13 | Ran the TST-25 acceptance checks (scripts/ci/README.md) — they caught **two real CI defects, both fixed** on `fix/tst-25-ci-key-log-leak`: (1) **CI SSH key leaked in cleartext** in the `windows-x86` step env echo (Gitea's line-oriented masker missed the multiline PEM) — rotated the CI key on windev (old pubkey revoked), stored the key **base64-encoded** so the masker redacts it to `***` (confirmed on run #38), taught `run-windev-ci.sh` to decode, dropped the redundant public known-hosts secret from the job env; (2) **bootstrap lock race** — `run-windev-ci.sh`'s pre-hand-off `git fetch`/`checkout` ran outside the worktree lock, so concurrent runs collided on `.git/index.lock`; the bootstrap now holds the lock (ps1 re-uses it via `MXGW_CI_LOCK_HELD`), retest confirmed clean serialization. Also **deflaked** `SessionManagerTests` fail-fast timing assertions (absolute `<100ms` wall-clock bound flaked under CI load; now anchored to the configured timeout / dropped for the zero-timeout case). Checks passed: unreachable-host fast-fail (exit 255/15s), deliberate-red propagation (Worker.Tests failure → exit 1), lock concurrency (2nd run waits), no-key-in-logs (masked). Merge target `df7e20d` verified GREEN via the local windev path (Worker build + 356 tests); merged to `main` `19cbf7b`. Check 6 (forced-failure nightly issue): issue endpoint+token proven live at bring-up (#124); in-CI forced-failure probe abandoned to shared-runner congestion (residual `if: failure()` gating is standard Actions). |
|
||||
| 2026-07-13 | New finding **TST-30** (`Low`/`P2`) added — surfaced during TST-25 acceptance: CI runs on a single shared `gitea-runner` (`maxParallel=1`, co-located `10.100.0.35`) interleaved with `dohertj2/lmxopcua`, and Gitea 1.26 exposes no run cancel/delete, so queue latency is unbounded under cross-repo contention and the runner is a single point of failure. Design: add a second/labelled runner + document the no-cancel reality and the `run-windev-ci.sh` bypass. Roll-ups updated (Testing Low 2→3, total 47→48; P2 9→10). |
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Source review: [60-testing-docs-gaps.md](../60-testing-docs-gaps.md) · Baseline: `main` @ `4f5371f` · Generated: 2026-07-13
|
||||
|
||||
This cycle's new findings (TST-25..29) are all fallout or residue of the prior remediation: the Windows CI tier was removed after act_runner proved broken (TST-25), the docs describing that tier were not updated in the same commit (TST-26), a flag SEC-25 made live is still documented as dead (TST-27), a handshake field is asserted only on the side CI cannot run (TST-28), and the root-artifact triage from TST-14 needs its retirement decision refreshed (TST-29). Every citation below was re-verified against the working tree on 2026-07-13.
|
||||
This cycle's new findings (TST-25..29) are all fallout or residue of the prior remediation: the Windows CI tier was removed after act_runner proved broken (TST-25), the docs describing that tier were not updated in the same commit (TST-26), a flag SEC-25 made live is still documented as dead (TST-27), a handshake field is asserted only on the side CI cannot run (TST-28), and the root-artifact triage from TST-14 needs its retirement decision refreshed (TST-29). Every citation below was re-verified against the working tree on 2026-07-13. **TST-30** was added later on 2026-07-13, surfaced while running TST-25's acceptance checks — the CI infrastructure itself (a single shared runner) is a throughput/availability bottleneck.
|
||||
|
||||
Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior design doc, `archreview/remediation/60-testing-docs-gaps.md`; this doc covers only the new findings. Note TST-25 is the unlock for two of those: a working Windows tier is where the scheduled live-MXAccess smoke (old TST-05) and CI-run client wire tests (old TST-24) land.
|
||||
|
||||
@@ -15,6 +15,7 @@ Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior
|
||||
| TST-27 | Medium | P1 (doc batch) | S | — | Not started | `ShowTagValues` config row still says "Reserved" after SEC-25 made the flag live |
|
||||
| TST-28 | Low | P2 | S | relates IPC-02 | Not started | Gateway-side `max_frame_bytes` handshake field untested in the CI-run suite |
|
||||
| TST-29 | Low | P2 | S | — | Not started | Retire `oldtasks.md` after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts |
|
||||
| TST-30 | Low | P2 | M | — | Not started | Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete) |
|
||||
|
||||
---
|
||||
|
||||
@@ -145,6 +146,29 @@ Design details for (b):
|
||||
|
||||
---
|
||||
|
||||
## TST-30 — Single shared Gitea runner is a CI throughput/availability bottleneck `Low` · `P2` · Effort M
|
||||
|
||||
**Finding.** All CI for this repo runs on **one** co-located `gitea-runner` container on docker host `10.100.0.35` with `maxParallel=1` (observed in the runner logs during TST-25 acceptance: `Running job with maxParallel=1`). That runner is **shared across repos** — it interleaves `dohertj2/mxaccessgw` and `dohertj2/lmxopcua` jobs on the single slot — and Gitea's per-repo runner list is empty (`GET /repos/dohertj2/mxaccessgw/actions/runners` → `total_count: 0`), i.e. it is an instance/shared runner, not repo-scoped. Every job in every run executes serially: a single `mxaccessgw` push fans out to `portable` (~5 min), `java`, `windows-x86`, so the wall time for the queue behind it is additive, and an active `lmxopcua` run blocks `mxaccessgw` entirely. Compounding it, this Gitea version (1.26.4) exposes **no run cancel or delete via the API** (`POST .../actions/runs/{id}/cancel` → 404; `DELETE .../actions/runs/{id}` → 400), so superseded or stale runs cannot be cleared and consume the slot in full before newer runs start.
|
||||
|
||||
**Impact.** Low severity, real drag. CI feedback latency is unbounded under cross-repo contention (observed ~20–30 min queue depth during TST-25 verification, forcing direct local `run-windev-ci.sh` verification to bypass the queue). Because the runner is the *only* one, it is also a single point of failure — if it wedges or its host is down, **all** CI (both repos) stops, with no failover; and there is no way to abort a run that is hung or was obsoleted by a newer push, so a stuck job can hold the slot until its internal timeout. This does not threaten correctness (jobs still run and report accurately) but it undercuts the fast-feedback purpose of the TST-25 tier and makes the CI system fragile in exactly the way act_runner history (TST-25) warns about.
|
||||
|
||||
**Design.** Add a **second runner** so the two repos (and the fan-out jobs) stop contending on one slot, and prefer repo/label scoping so `mxaccessgw`'s Windows-adjacent jobs aren't starved by an unrelated repo. Options, cheapest first:
|
||||
|
||||
- **(a) Raise `maxParallel` / register a second runner instance on `10.100.0.35` — recommended.** The host already runs `gitea-runner`; add a second `act_runner` instance (or bump the existing runner's capacity where the docker-in-docker/resource budget allows) so at least two jobs run concurrently. Cheapest change, keeps the co-located-with-`gitea:3000` network property that TST-03 depended on (`container.network: traefik`).
|
||||
- **(b) Dedicate a runner to `mxaccessgw`** with a distinct label and gate the CI jobs on it, so `lmxopcua` load never blocks this repo. Cleaner isolation than (a) but needs label wiring in `ci.yml` (`runs-on: [ubuntu-latest, mxgw]` or similar) and a labelled runner registration.
|
||||
- **(c) Put the runner on windev / a second host** — rejected as the primary fix: windev is the Windows build target, not a CI host, and co-locating the Linux runner there loses the `gitea:3000` resolution TST-03 relies on. Only consider if `10.100.0.35` runs out of capacity.
|
||||
|
||||
Independent of the runner count, document the **no-cancel** reality (Gitea 1.26 API) and the escape hatch: a specific SHA can be verified out-of-band via `scripts/ci/run-windev-ci.sh` (Linux) or the manual windev worktree flow, bypassing the queue — this is already the degraded-mode path in `scripts/ci/README.md`; TST-30 generalizes it from "windev tier down" to "runner contended".
|
||||
|
||||
**Implementation.**
|
||||
1. Register a second `act_runner` on `10.100.0.35` (option a) with the same `container.network: traefik` config (repo memory `project_gitea_ci`), or a labelled dedicated runner (option b) and add the label to `ci.yml`'s `runs-on`. Confirm both runners appear and pick up jobs concurrently (two pushes should no longer serialize end-to-end).
|
||||
2. `docs/GatewayTesting.md` (Continuous Integration section) — note that the runner is shared/finite, that queue latency under cross-repo contention is expected, that this Gitea version has no run cancel/delete, and that `scripts/ci/run-windev-ci.sh <sha>` / the manual worktree flow verify a commit without waiting for the queue.
|
||||
3. Optional: add a workflow-level `concurrency` group to `ci.yml` so a newer push to the same branch supersedes an in-flight run *if* the deployed Gitea honors it (belt-and-suspenders against the missing cancel API; verify it actually cancels before relying on it).
|
||||
|
||||
**Verification.** Push two branches back-to-back and confirm their runs execute concurrently (not serially) once a second runner exists; `GET /repos/dohertj2/mxaccessgw/actions/runners` (or the instance runner list) shows ≥2 runners online; `docs/GatewayTesting.md` describes the shared-runner/no-cancel reality and the bypass. Re-run the TST-25 acceptance push and confirm queue depth is materially lower under a concurrent `lmxopcua` run.
|
||||
|
||||
---
|
||||
|
||||
## Cross-domain dependencies
|
||||
|
||||
- **TST-25 → old TST-05 / old TST-24:** the SSH-driven nightly is where the scheduled live-MXAccess smoke lands (closes old TST-05), and a working CI Windows tier unblocks wiring client wire-behavior tests into CI (old TST-24).
|
||||
|
||||
Reference in New Issue
Block a user