Files
mxaccessgw/archreview/2026-07-12/remediation/60-testing-docs-gaps.md
T
Joseph Doherty 4b8ba7a5bd
ci / java (push) Successful in 1m58s
ci / windows-x86 (push) Successful in 1m6s
ci / nightly-windev (push) Has been skipped
ci / portable (push) Successful in 8m7s
docs(TST-30): new finding — single shared Gitea runner is a CI throughput/availability bottleneck
Surfaced during TST-25 acceptance verification: all CI runs on one shared
gitea-runner (maxParallel=1, co-located 10.100.0.35) interleaved with
dohertj2/lmxopcua, and Gitea 1.26 has no run cancel/delete API, so queue
latency is unbounded under cross-repo contention and the runner is a single
point of failure. Design: add a second/labelled runner; document the no-cancel
reality and the run-windev-ci.sh queue-bypass. Low/P2. Roll-ups updated.
2026-07-13 12:37:10 -04:00

31 KiB
Raw Blame History

Testing, Documentation & Underdeveloped Areas — Remediation Design & Implementation (2026-07-12 cycle)

Source review: 60-testing-docs-gaps.md · Baseline: main @ 4f5371f · Generated: 2026-07-13

This cycle's new findings (TST-25..29) are all fallout or residue of the prior remediation: the Windows CI tier was removed after act_runner proved broken (TST-25), the docs describing that tier were not updated in the same commit (TST-26), a flag SEC-25 made live is still documented as dead (TST-27), a handshake field is asserted only on the side CI cannot run (TST-28), and the root-artifact triage from TST-14 needs its retirement decision refreshed (TST-29). Every citation below was re-verified against the working tree on 2026-07-13. TST-30 was added later on 2026-07-13, surfaced while running TST-25's acceptance checks — the CI infrastructure itself (a single shared runner) is a throughput/availability bottleneck.

Prior-cycle open findings (TST-05..24 where still open) are tracked in the prior design doc, archreview/remediation/60-testing-docs-gaps.md; this doc covers only the new findings. Note TST-25 is the unlock for two of those: a working Windows tier is where the scheduled live-MXAccess smoke (old TST-05) and CI-run client wire tests (old TST-24) land.

Finding register

ID Sev Tier Eff Dep Status Title
TST-25 High P1 M — (unlocks TST-05, TST-24) Done Windows/x86 test tier has zero automation — restore via SSH-driven windev CI job
TST-26 Medium P1 (folded into TST-25) S TST-25 Done docs/GatewayTesting.md, check-codegen.ps1, and ci.yml comments describe removed CI jobs
TST-27 Medium P1 (doc batch) S Not started ShowTagValues config row still says "Reserved" after SEC-25 made the flag live
TST-28 Low P2 S relates IPC-02 Not started Gateway-side max_frame_bytes handshake field untested in the CI-run suite
TST-29 Low P2 S Not started Retire oldtasks.md after folding the Phase-5 governance record into DesignDecisions.md; delete root docs-review artifacts
TST-30 Low P2 M Not started Single shared Gitea runner is a CI throughput/availability bottleneck (cross-repo contention, no run cancel/delete)

TST-25 — Windows/x86 test tier has zero automation High · P1 · Effort M

Finding. The windows and live-mxaccess CI jobs were removed in abb0930 because Gitea act_runner v0.6.1 host-mode is broken on Windows and a runs-on-gated job with no matching runner wedges every run in "queued" (removal note: .gitea/workflows/ci.yml:123-131). Consequence: the x86/net48 Worker build, all of src/ZB.MOM.WW.MxGateway.Worker.Tests — including the tests for remediation-added machinery that exists nowhere else (WorkerFrameWritePriority ordering at Worker.Tests/Ipc/WorkerFrameProtocolTests.cs:353-395, worker-side max_frame_bytes adoption, WnWrap/subtag alarm consumers, StaRuntime changes) — and the 8-fact live suite (src/ZB.MOM.WW.MxGateway.IntegrationTests/WorkerLiveMxAccessSmokeTests.cs) run only when someone remembers the manual windev worktree process. The portable job's check-codegen.ps1 Generated-diff check substitutes for the net48 CS0246 guard; nothing substitutes for actually compiling and running the worker tests.

Impact. A whole tier of unit tests — not just live smoke — is unguarded. A worker-side regression (frame priority inversion, frame-max adoption break, alarm-consumer conversion bug, STA loop change) ships green through CI and is caught, if at all, by hand on windev or in production. This regressed the prior cycle's TST-03/TST-05 posture and blocks old TST-24.

Design. Three candidates, weighed against the operational facts (Gitea Actions runs on a co-located gitea-runner container on docker host 10.100.0.35, job containers on network traefik resolving gitea:3000; windev is Windows 10 at 10.100.0.48, ssh alias windev, passwordless ssh from the Mac, MXAccess installed, and a proven isolated-worktree flow already exists in C:\build driven by remote PowerShell via base64 -EncodedCommand):

  • (a) Retry a Windows act_runner in host mode with the right labels — rejected. The failure is not a label problem: act_runner v0.6.1's hostexecutor writes each step's script to a path it then cannot resolve (independent of shell) and cannot stage JS actions, so no label configuration fixes it; Windows containers are impractical for a net48/x86/MXAccess build. And the structural hazard remains: any runs-on gate with no live runner sticks queued forever, so a flaky/offline Windows runner would block every run. Revisit only if a fixed act_runner release ships — it would then replace the SSH job with a native one.
  • (b) SSH-driven job — recommended. A Linux CI job (same ubuntu-latest runner class as portable, so it always schedules) ssh's to windev, fast-forwards an isolated CI clone under C:\build to the exact SHA under test, runs the x86 build + Worker.Tests there, and propagates the remote exit code through ssh → step → job. This reuses the already-proven windev worktree + -EncodedCommand recipe, requires no new runner software on Windows, and cannot wedge the queue (the Linux runner always exists; an unreachable windev fails the job red — loud, not stuck).
  • (c) Trigger split — adopted as part of (b). Worker tests have nontrivial wall time on windev, and the live suite needs provider state and must never gate a push. Split: per-push/PR job runs dotnet build Worker.csproj -p:Platform=x86 + dotnet test Worker.Tests -p:Platform=x86 (unit suite — expected minutes; if measured wall time exceeds ~10 min, demote the test step to nightly and keep the per-push x86 build, which alone catches the net48/CS0246/x86 compile class); the nightly scheduled job (the existing cron: '0 6 * * *', currently pointless) re-runs the x86 build + full Worker.Tests on main, then runs the live-MXAccess smoke with MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1, and optionally a full src/ZB.MOM.WW.MxGateway.slnx build (catches Windows-only analyzer diagnostics like xUnit1030 that never surface on macOS/Linux).

Design details for (b):

  • Isolation. A dedicated CI clone C:\build\mxaccessgw-ci — never the Desktop checkout (dirty feature branch, per the established windev rule). The job's inline bootstrap (base64 -EncodedCommand) does git fetch origin && git checkout --force <sha> in that clone, then invokes the checked-out scripts/ci/windev-worker-ci.ps1 -Sha <sha> -Mode <build|test|live> so the CI logic itself is versioned with the code under test; only the tiny bootstrap lives in ci.yml.
  • Serialization. One clone, potentially overlapping runs: windev-worker-ci.ps1 acquires a lock (New-Item C:\build\mxaccessgw-ci.lock -ItemType Directory, retry loop with ~20 min timeout, always released in finally) so concurrent pushes queue instead of stomping the worktree. Add a workflow-level concurrency group too if the deployed Gitea version honors it — the lock is the guarantee, concurrency the nicety.
  • Secrets. Generate a dedicated ed25519 keypair for CI (do not reuse the operator's personal windev key). Private key → Gitea Actions repo secret WINDEV_SSH_KEY; pinned host key → secret WINDEV_SSH_KNOWN_HOSTS (or a committed scripts/ci/windev.known_hosts — host keys are public data; committing is acceptable and greppable). The job writes the key to ~/.ssh/id_ed25519 mode 0600 and connects with -o BatchMode=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=.... On windev, append the pub key to authorized_keys for the CI account; optionally prefix with a forced command= restricting it to the bootstrap script. Per CLAUDE.md, the key must never be echoed into logs.
  • Network precheck. The job container is on the traefik docker network; confirm it has egress to 10.100.0.48:22 (a one-off nc -z -w5 10.100.0.48 22 step during bring-up; docker bridge networks normally route to the LAN, but the network was created for gitea resolution, not egress, so verify rather than assume).
  • Failure visibility. Per-push runs are first-class CI jobs — a red windev run is a red commit/PR check exactly like portable/java, and an unreachable windev fails (never skips) so a down Windows tier is loud. Nightly runs surface on the repo's Actions page against main; because nobody watches that page, add a final if: failure() step to the nightly job that opens (or comments on) a Gitea issue via the API using the built-in actions token, so a red nightly produces a notification artifact. Document in docs/GatewayTesting.md that a red windows-x86 job means the Windows tier — not the change — may be down, and how to check (ssh windev, read C:\build\mxaccessgw-ci\ci.log).

Implementation.

  1. Windev prep (one-time, manual): create C:\build\mxaccessgw-ci as a fresh clone of the Gitea origin (read-only deploy token or the existing credential store); verify dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86 and dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 succeed there by hand; record wall time (drives the per-push vs nightly split for the test step). Generate the CI keypair, install the pub key on windev, store WINDEV_SSH_KEY (+ known-hosts) in Gitea repo secrets.

  2. New scripts/ci/windev-worker-ci.ps1 (runs on windev, invoked at the tested SHA): parameters -Sha, -Mode build|test|live; acquires/releases the lock dir; build = x86 Worker build; test = build + dotnet test Worker.Tests -p:Platform=x86; live = test + MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1 dotnet test src/ZB.MOM.WW.MxGateway.IntegrationTests/... --filter FullyQualifiedName~WorkerLiveMxAccessSmokeTests (+ optional full-slnx build); exits nonzero on any failure.

  3. New scripts/ci/run-windev-ci.sh (runs in the Linux job): writes the key/known-hosts from env, builds the base64 -EncodedCommand bootstrap (fetch + checkout $CI_SHA in C:\build\mxaccessgw-ci, then powershell -File scripts\ci\windev-worker-ci.ps1 -Sha $CI_SHA -Mode $1), runs ssh -o BatchMode=yes ci@10.100.0.48 ..., exits with the ssh exit code. Do not wrap the remote git calls in $ErrorActionPreference='Stop' (git writes progress to stderr — known windev gotcha).

  4. ci.yml — add the per-push job (same commit as steps 56):

    windows-x86:
      # x86/net48 Worker build + Worker.Tests, executed on windev (10.100.0.48) over SSH.
      # Runs on a Linux runner so it always schedules (act_runner host-mode on Windows is
      # broken; a runs-on gate with no runner wedges the queue — see docs/GatewayTesting.md).
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v4
        - name: Worker x86 build + tests on windev
          env:
            WINDEV_SSH_KEY: ${{ secrets.WINDEV_SSH_KEY }}
            WINDEV_SSH_KNOWN_HOSTS: ${{ secrets.WINDEV_SSH_KNOWN_HOSTS }}
            CI_SHA: ${{ github.sha }}
          run: ./scripts/ci/run-windev-ci.sh test   # demote to `build` if measured wall time > ~10 min
    
  5. ci.yml — add the nightly job gated if: github.event_name == 'schedule' (safe: the gate is an if, not an unsatisfiable runs-on, so push runs mark it skipped, not queued), runs-on: ubuntu-latest, calling run-windev-ci.sh live; final if: failure() step posts a Gitea issue/comment via the API token. Fix the cron comment at ci.yml:12-14 to describe what the schedule now actually runs.

  6. ci.yml — replace the removal note at :123-131 with a short pointer: host-mode act_runner remains broken; the Windows tier runs via the SSH-driven windows-x86/nightly jobs; restore native jobs only with a proven runner.

  7. TST-26 doc edits in the same commit — see TST-26 below.

  8. After the first green nightly, mark old TST-05 addressed (live smoke is scheduled again) and unblock old TST-24 (client wire tests can join CI).

Verification.

  • Push a branch: portable, java, and windows-x86 all green; confirm on windev that C:\build\mxaccessgw-ci sits at the pushed SHA.
  • Deliberate red: on a scratch branch, add a failing [Fact] to Worker.Tests (or an intentional x86-only compile error), push, confirm windows-x86 turns red and the run fails; revert. This proves exit-code propagation end-to-end, the whole point of the SSH design.
  • Unreachable-host red: temporarily point the script at a bogus port (or stop sshd), confirm the job fails fast with a clear message rather than hanging or passing.
  • Concurrency: push two branches back-to-back; confirm the second run's remote step waits on the lock and both complete.
  • Nightly: temporarily set the cron a few minutes ahead (or trigger the schedule path manually), confirm the live job runs with MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1 and, with a forced failure, that the on-failure issue step fires.
  • Confirm no secret material appears in the job logs.

TST-26 — Docs describe CI jobs that no longer exist Medium · P1 (folded into TST-25) · Effort S

Finding. docs/GatewayTesting.md:426-431 still documents the windows job ("the only host that builds the x86/net48 Worker… the definitive guard for the 'regenerate and commit Generated/' rule") and the live-mxaccess scheduled job — both deleted from .gitea/workflows/ci.yml in abb0930, which touched only the workflow file (a violation of the repo's docs-change-in-same-commit rule). Same stale claim at scripts/check-codegen.ps1:17 ("guarded by the Windows CI job, not here") and in the ci.yml:12-14 cron comment ("Nightly live-MXAccess smoke (windev)") though the schedule currently re-runs only portable+java.

Impact. Actively misleading in the worst direction: a reader concludes the worker build is CI-guarded when it is not, and the "definitive guard" claim is doubly wrong — the surviving guard for the Generated-commit rule is the portable job's check-codegen.ps1 diff check, not a Windows compile.

Design. No standalone commit: these edits land in the same commit as the TST-25 ci.yml change (that is the same-commit rule the removal violated). Content rules: describe the jobs that exist (portable, java, windows-x86, nightly), name check-codegen.ps1's Generated-diff check as the primary guard for the regenerate-and-commit rule with the windows-x86 net48 compile as the second line, and document the manual windev fallback (isolated C:\build worktree) as the degraded-mode procedure with its trigger condition (SSH job red for infra reasons). If TST-25 is somehow delayed, do not hold these edits hostage — an interim commit correcting the docs to the two-job reality plus a mandatory manual cadence (per-merge for worker-touching changes, weekly otherwise) is strictly better than the current false claim; rewrite again when the SSH job lands.

Implementation.

  1. docs/GatewayTesting.md:426-431 — replace the windows and live-mxaccess bullets with: windows-x86 (SSH-driven, per-push, x86 build + Worker.Tests on windev; failure semantics — red job = tier or change broken, how to distinguish) and the nightly scheduled job (full Worker.Tests + live-MXAccess smoke, MXGATEWAY_RUN_LIVE_MXACCESS_TESTS=1, never gates a push, on-failure issue notification). Reword the paragraph at :433-437 so the Generated/ guard attribution is: primary = check-codegen.ps1 regeneration diff (portable job), secondary = net48 compile (windows-x86).
  2. scripts/check-codegen.ps1:17 — change "are guarded by the Windows CI job, not here" to "are guarded by the SSH-driven windows-x86 CI job (see docs/GatewayTesting.md § Continuous Integration), not here".
  3. .gitea/workflows/ci.yml:12-14 — cron comment now describes the nightly windev run (covered by TST-25 step 5); :123-131 removal note replaced (TST-25 step 6).

Verification. Greps prove the drift is gone: grep -n '"windows"' docs/GatewayTesting.md and grep -n 'live-mxaccess' docs/GatewayTesting.md return only descriptions of jobs that exist in ci.yml; grep -n 'Windows CI job' scripts/check-codegen.ps1 returns the corrected text; grep -rn 'definitive guard' docs/ attributes the Generated/ rule to check-codegen.ps1. Read the CI section back against the final ci.yml job list — every named job must exist, and every job in ci.yml must be named.


TST-27 — ShowTagValues config row still says "Reserved" Medium · P1 (doc batch) · Effort S

Finding. docs/GatewayConfiguration.md:185 reads "Reserved display control for tag values. The dashboard does not show full tag values by default." — but SEC-25 wired the flag live: src/ZB.MOM.WW.MxGateway.Server/Dashboard/Hubs/DashboardEventBroadcaster.cs:16-29 blanks tag values from the deep-cloned event before mirroring to SignalR when Dashboard:ShowTagValues is false (the default). docs/GatewayDashboardDesign.md:170 was updated in the SEC-25 change; the authoritative config reference was not — remediation-created drift, same-commit rule violated.

Impact. An operator consulting the options table concludes toggling the flag does nothing, when it actually controls whether tag values reach every dashboard hub subscriber — security-relevant precisely because the per-session hub ACL (TST-15) does not exist yet: the redaction is currently the only thing between a low-trust Viewer and other sessions' tag values.

Design. Documentation-only correction of the one stale row; no code change. State three things in the row: (1) what false (default) does — tag values are blanked from the MxEvent copy mirrored to the SignalR events hub; metadata (tag reference, quality, status, timestamps) still renders; (2) the security relevance — with no per-session hub ACL yet (TST-15), true exposes all sessions' tag values to every authenticated dashboard viewer; (3) the honest scope limit — the flag gates only the hub mirror, not the /browse live-value display (the TST-16 residual, tracked separately; do not paper over it by implying full coverage). Ship in the cycle's P1 doc-drift batch (with WRK-26, CLI-42, SEC residuals per the 00-overall roadmap) — one commit for the batch is fine since each edit is independently verifiable.

Implementation.

  1. docs/GatewayConfiguration.md:185 — replace the row description with the three-part text above, cross-referencing docs/GatewayDashboardDesign.md for the mechanism.
  2. While in the file, confirm the shape block's "ShowTagValues": false at :60 needs no change (it doesn't — value and default are correct; only the prose row lies).
  3. No edit to DashboardEventBroadcaster.cs or GatewayDashboardDesign.md (both already correct).

Verification. grep -n 'Reserved' docs/GatewayConfiguration.md no longer matches the ShowTagValues row; read the new row back against DashboardEventBroadcaster.cs:16-29 (behavior), Configuration/DashboardOptions.cs (default false), and GatewayDashboardDesign.md:170 (consistency). Confirm the row does not claim /browse coverage.


TST-28 — Gateway-side max_frame_bytes untested in the CI-run suite Low · P2 · Effort S · relates IPC-02

Finding. src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs:988 populates MaxFrameBytes = (uint)_connection.FrameOptions.MaxMessageBytes in the GatewayHello (wired from Worker.MaxMessageBytes via Sessions/SessionWorkerClientFactory.cs:73-76), but no test under src/ZB.MOM.WW.MxGateway.Tests references MaxFrameBytes at all. The adoption side is asserted only in Worker.Tests — Windows-only, i.e. inside the TST-25 gap.

Impact. A regression to sending 0 (which the worker deliberately treats as "older gateway, use default" per the IPC-02 negotiation contract) would pass every automated test while silently reverting the negotiated frame limit — an invisible downgrade of shipped IPC-02 behavior.

Design. One cheap fake-worker assertion; no new infrastructure needed. FakeWorkerHarness.CompleteStartupAsync already reads and returns the GatewayHello envelope (Tests/Gateway/Workers/Fakes/FakeWorkerHarness.cs:149-165), and CreateConnectedPairAsync already accepts a maxMessageBytes override that flows into the WorkerFrameProtocolOptions shared with the WorkerClient the harness creates — so the test is: start the client, complete the handshake, assert on the returned envelope. Cover both the default (WorkerFrameProtocolOptions.DefaultMaxMessageBytes) and an overridden value (e.g. 2 * 1024 * 1024) as a [Theory], which also pins that the field tracks configuration rather than a constant. Rejected alternative: asserting via the worker-side adoption tests only — that is exactly the Windows-only blind spot this finding names.

Implementation.

  1. Add a [Theory] to src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs (natural home — its CompleteHandshakeAsync helper at :26 already drives this path), e.g. StartAsync_SendsGatewayHelloWithConfiguredMaxFrameBytes: create the harness with CreateConnectedPairAsync(maxMessageBytes: value) for value ∈ {default, 2 MiB}, create/start the WorkerClient, capture the envelope from CompleteStartupAsync, assert envelope.GatewayHello.MaxFrameBytes == (uint)value (and != 0 for readability of intent).
  2. No production code change; no doc change (docs/WorkerFrameProtocol.md:25-30 already documents the negotiation).

Verification. dotnet test src/ZB.MOM.WW.MxGateway.Tests/ZB.MOM.WW.MxGateway.Tests.csproj --filter FullyQualifiedName~WorkerClientTests green on macOS/Linux (i.e. it runs in the portable CI job — the point of the finding). Mutation check while developing: temporarily hard-code MaxFrameBytes = 0 in WorkerClient.cs:988 and confirm the new test fails, then revert.


TST-29 — Retire oldtasks.md; delete root docs-review artifacts Low · P2 · Effort S

Finding. TST-14's plan was "keep oldtasks.md only until the epic resumes, then delete", but TST-04 made it the durable governance record: oldtasks.md:27-35,75-88 is now the only tracked statement that Phase 5 (orphan-worker reattach) is deferred, not planned and that EnableOrphanReattach (Task 26) "does not exist and must not be referenced". CLAUDE.md:79 links to it for exactly that statement. Meanwhile the mechanical half of TST-14 — deleting the five untracked, gitignored root working files (MxAccessGateway-docs-{issues,fixed,final}.md, MxGatewayClient-docs-{issues,fixed}.md, ~15k lines) — remains undone and keeps tripping greps.

Impact. Low, but real: deleting oldtasks.md naively would orphan a load-bearing governance record; keeping it forever contradicts its own "just a human-readable mirror" charter (oldtasks.md:12-14) and leaves two sources that can drift from docs/plans/2026-06-15-session-resilience.md.tasks.json. The untracked artifacts are pure grep noise.

Design. Move the durable decisions to where invariant-adjacent decisions live, then delete the mirror. docs/DesignDecisions.md gains a "Session-resilience epic scope" entry recording: Phase 5 deferred-not-planned with the rationale (reverses the "gateway restart does not reattach orphan workers" invariant; adoption-manifest store + phone-home protocol; deferred unless a concrete requirement appears); the explicit rule that EnableOrphanReattach does not exist and must not be referenced until its task lands; and the settled Phase-4 Viewer-default decision (admin-sees-all, Viewer strictly scoped to owned/granted sessions — oldtasks.md:71-72) so the TST-15 implementation doesn't have to rediscover it. The tasks.json remains the sole resume state for the still-pending Phase 4 tasks — one authority, no mirror. Do this now rather than waiting for the epic to close: the mirror's residual value (Task 14 Java remainder, Phase 4 checklist) is fully carried by tasks.json + the archreview register. Alternative rejected: keep oldtasks.md as the governance record — a file named "oldtasks" that must never be deleted is a trap for future cleanup passes.

Implementation.

  1. docs/DesignDecisions.md — add the "Session-resilience epic scope" decision entry (content above, dated, citing the epic plan and TST-04).
  2. Update the two tracked references before deletion: CLAUDE.md:79 ("see oldtasks.md (session-resilience epic Phase 5)" → point at docs/DesignDecisions.md); stillpending.md:7,165 ("remaining epic tasks tracked in oldtasks.md" → point at docs/plans/2026-06-15-session-resilience.md.tasks.json). Leave docs/plans/2026-06-16-stillpending-section8-design.md (historical plan; a stale pointer in a dated plan is acceptable — optionally annotate).
  3. git rm oldtasks.md — same commit as steps 12 (the record must never be absent from main).
  4. rm the five untracked root files: MxAccessGateway-docs-{issues,fixed,final}.md, MxGatewayClient-docs-{issues,fixed}.md (gitignored at .gitignore:152-154 — deletion is local-only, zero repo impact; no commit involved).
  5. Do not touch stillpending.md's own promote-or-migrate question, REVIEW-PROCESS.md, or A2-galaxyrepository-adoption-handoff.md — those remain TST-14's open triage in the prior-cycle doc.

Verification. grep -rn 'oldtasks' --include='*.md' . (excluding archreview/, which is a historical record) returns no tracked references; grep -n 'EnableOrphanReattach\|deferred, not planned' docs/DesignDecisions.md finds the migrated statements; ls *-docs-*.md at the repo root returns nothing; git status clean. Read CLAUDE.md:79 back to confirm the invariant sentence still holds and its pointer resolves.


TST-30 — Single shared Gitea runner is a CI throughput/availability bottleneck Low · P2 · Effort M

Finding. All CI for this repo runs on one co-located gitea-runner container on docker host 10.100.0.35 with maxParallel=1 (observed in the runner logs during TST-25 acceptance: Running job with maxParallel=1). That runner is shared across repos — it interleaves dohertj2/mxaccessgw and dohertj2/lmxopcua jobs on the single slot — and Gitea's per-repo runner list is empty (GET /repos/dohertj2/mxaccessgw/actions/runnerstotal_count: 0), i.e. it is an instance/shared runner, not repo-scoped. Every job in every run executes serially: a single mxaccessgw push fans out to portable (~5 min), java, windows-x86, so the wall time for the queue behind it is additive, and an active lmxopcua run blocks mxaccessgw entirely. Compounding it, this Gitea version (1.26.4) exposes no run cancel or delete via the API (POST .../actions/runs/{id}/cancel → 404; DELETE .../actions/runs/{id} → 400), so superseded or stale runs cannot be cleared and consume the slot in full before newer runs start.

Impact. Low severity, real drag. CI feedback latency is unbounded under cross-repo contention (observed ~2030 min queue depth during TST-25 verification, forcing direct local run-windev-ci.sh verification to bypass the queue). Because the runner is the only one, it is also a single point of failure — if it wedges or its host is down, all CI (both repos) stops, with no failover; and there is no way to abort a run that is hung or was obsoleted by a newer push, so a stuck job can hold the slot until its internal timeout. This does not threaten correctness (jobs still run and report accurately) but it undercuts the fast-feedback purpose of the TST-25 tier and makes the CI system fragile in exactly the way act_runner history (TST-25) warns about.

Design. Add a second runner so the two repos (and the fan-out jobs) stop contending on one slot, and prefer repo/label scoping so mxaccessgw's Windows-adjacent jobs aren't starved by an unrelated repo. Options, cheapest first:

  • (a) Raise maxParallel / register a second runner instance on 10.100.0.35 — recommended. The host already runs gitea-runner; add a second act_runner instance (or bump the existing runner's capacity where the docker-in-docker/resource budget allows) so at least two jobs run concurrently. Cheapest change, keeps the co-located-with-gitea:3000 network property that TST-03 depended on (container.network: traefik).
  • (b) Dedicate a runner to mxaccessgw with a distinct label and gate the CI jobs on it, so lmxopcua load never blocks this repo. Cleaner isolation than (a) but needs label wiring in ci.yml (runs-on: [ubuntu-latest, mxgw] or similar) and a labelled runner registration.
  • (c) Put the runner on windev / a second host — rejected as the primary fix: windev is the Windows build target, not a CI host, and co-locating the Linux runner there loses the gitea:3000 resolution TST-03 relies on. Only consider if 10.100.0.35 runs out of capacity.

Independent of the runner count, document the no-cancel reality (Gitea 1.26 API) and the escape hatch: a specific SHA can be verified out-of-band via scripts/ci/run-windev-ci.sh (Linux) or the manual windev worktree flow, bypassing the queue — this is already the degraded-mode path in scripts/ci/README.md; TST-30 generalizes it from "windev tier down" to "runner contended".

Implementation.

  1. Register a second act_runner on 10.100.0.35 (option a) with the same container.network: traefik config (repo memory project_gitea_ci), or a labelled dedicated runner (option b) and add the label to ci.yml's runs-on. Confirm both runners appear and pick up jobs concurrently (two pushes should no longer serialize end-to-end).
  2. docs/GatewayTesting.md (Continuous Integration section) — note that the runner is shared/finite, that queue latency under cross-repo contention is expected, that this Gitea version has no run cancel/delete, and that scripts/ci/run-windev-ci.sh <sha> / the manual worktree flow verify a commit without waiting for the queue.
  3. Optional: add a workflow-level concurrency group to ci.yml so a newer push to the same branch supersedes an in-flight run if the deployed Gitea honors it (belt-and-suspenders against the missing cancel API; verify it actually cancels before relying on it).

Verification. Push two branches back-to-back and confirm their runs execute concurrently (not serially) once a second runner exists; GET /repos/dohertj2/mxaccessgw/actions/runners (or the instance runner list) shows ≥2 runners online; docs/GatewayTesting.md describes the shared-runner/no-cancel reality and the bypass. Re-run the TST-25 acceptance push and confirm queue depth is materially lower under a concurrent lmxopcua run.


Cross-domain dependencies

  • TST-25 → old TST-05 / old TST-24: the SSH-driven nightly is where the scheduled live-MXAccess smoke lands (closes old TST-05), and a working CI Windows tier unblocks wiring client wire-behavior tests into CI (old TST-24).
  • TST-25 ↔ IPC-24/IPC-25: the nightly windev job is also the natural home for any Windows-side codegen verification the contracts/IPC remediation adds; coordinate job naming so both plans extend the same windows-x86/nightly jobs rather than adding parallel ones.
  • TST-26 ⊂ TST-25: same commit, by rule.
  • TST-27: ships in the cycle's P1 doc-drift batch alongside WRK-26 and CLI-42 (roadmap item 8); its /browse residual stays with TST-16 (prior cycle).
  • TST-28 ← IPC-02: the test pins the gateway half of the IPC-02 negotiation; the worker half's tests become CI-run once TST-25 lands.