Legacy CI run 959 failed after retained log silence; root cause and runner cleanup evidence missing #718

New Issue

2026-10-13T12:26:21-04:00

dohertj2 commented

2026-10-13 12:26:21 -04:00

Current terminal evidence — 2026-10-13 16:51 UTC

Fresh Actions API inspection now reports job 3552 completed / failure, with completion at 16:31:07 UTC. Its whole-solution test step, floor step and upload step all report failure. The retained log still ends at the successful Core.Abstractions suite at 13:39:04 UTC; it contains no later root-cause or timeout explanation. State reconciliation to a terminal job outcome is now established. Actual orphan-process absence, effective timeout limits and the cause remain unverified; no cancellation/deletion/rerun or host command was performed.

The historical observation below is retained as context, not the current job state. Current run 977 uses runner 1 according to its latest API metadata; earlier runner-5 observations are historical, not current placement.

Observed state

At 2026-10-13 16:25 UTC, legacy PR run 959 still reports job 3552 in_progress. It started 13:17:17 UTC on runner 1 (73f8c09b0d35). Its last available log timestamp is 13:39:04 UTC, a successful Core.Abstractions suite. The whole-solution test step remains in_progress; the executed-floor and TRX upload steps are queued.

This is an authoritative API/log observation, not proof of a hung process, a configured timeout violation or a common contention cause. Head f794dc50 predates the bounded CI workflow. Run 975 at 76bdca90 passed and #707's concurrency change is merged; current master run 977 is progressing on runner 5 and PR #714 run 979 waits under repository concurrency. Neither later run proves this older execution was cleaned up.

The old workflow has no explicit job timeout. Effective server/runner limits and actual live process/container state have not been established. Logs and job metadata were retained locally; no cancellation, deletion or rerun was performed.

Required follow-up

Reconcile runner process/container liveness with Gitea's reported job state using authorized runner/host evidence.
Preserve available logs/TRX and record effective job limits plus a terminal outcome or documented reconciliation action.
Determine whether a process stall, stale job reporting, or another cause explains the retained state; do not infer cause solely from log silence or duration.
If operational cleanup is needed, use an evidence-preserving supported cancellation route. The installed tea cancellation command aliases deletion; do not delete the run/log evidence merely to clear the display.
File a concrete software/configuration follow-up only if the investigation establishes one. Do not weaken test assertions or restart active verification runs.

Related #707 (completed concurrency change), #712 / PR #714 (independent current verification). No new runtime or workflow change is proposed without a reproduced cause.

## Current terminal evidence — 2026-10-13 16:51 UTC Fresh Actions API inspection now reports job 3552 **completed / failure**, with completion at 16:31:07 UTC. Its whole-solution test step, floor step and upload step all report failure. The retained log still ends at the successful Core.Abstractions suite at 13:39:04 UTC; it contains no later root-cause or timeout explanation. State reconciliation to a terminal job outcome is now established. Actual orphan-process absence, effective timeout limits and the cause remain unverified; no cancellation/deletion/rerun or host command was performed. The historical observation below is retained as context, not the current job state. Current run 977 uses runner 1 according to its latest API metadata; earlier runner-5 observations are historical, not current placement. ## Observed state At 2026-10-13 16:25 UTC, [legacy PR run 959](https://gitea.dohertylan.com/dohertj2/lmxopcua/actions/runs/959) still reports job 3552 in_progress. It started 13:17:17 UTC on runner 1 (73f8c09b0d35). Its last available log timestamp is 13:39:04 UTC, a successful Core.Abstractions suite. The whole-solution test step remains in_progress; the executed-floor and TRX upload steps are queued. This is an authoritative API/log observation, **not proof of a hung process, a configured timeout violation or a common contention cause**. Head f794dc50 predates the bounded CI workflow. Run 975 at 76bdca90 passed and #707's concurrency change is merged; current master run 977 is progressing on runner 5 and PR #714 run 979 waits under repository concurrency. Neither later run proves this older execution was cleaned up. The old workflow has no explicit job timeout. Effective server/runner limits and actual live process/container state have not been established. Logs and job metadata were retained locally; no cancellation, deletion or rerun was performed. ## Required follow-up - Reconcile runner process/container liveness with Gitea's reported job state using authorized runner/host evidence. - Preserve available logs/TRX and record effective job limits plus a terminal outcome or documented reconciliation action. - Determine whether a process stall, stale job reporting, or another cause explains the retained state; do not infer cause solely from log silence or duration. - If operational cleanup is needed, use an evidence-preserving supported cancellation route. The installed tea cancellation command aliases deletion; do not delete the run/log evidence merely to clear the display. - File a concrete software/configuration follow-up only if the investigation establishes one. Do not weaken test assertions or restart active verification runs. Related #707 (completed concurrency change), #712 / PR #714 (independent current verification). No new runtime or workflow change is proposed without a reproduced cause.

dohertj2 changed title from ~~Reconcile retained legacy CI run 959 process and in-progress state~~ to Legacy CI run 959 failed after retained log silence; root cause and runner cleanup evidence missing

2026-10-13 12:51:32 -04:00

dohertj2 commented

2026-10-13 14:02:40 -04:00

Additional read-only evidence: the configured local contexts are orbstack/default Unix sockets and desktop-linux. Orbstack/default list the project development containers but no running or stopped Gitea/act-runner container matching the historical runner; desktop-linux socket is unavailable. No daemon/context configuration was changed, no container exec/stop or remote-host command was performed.

Fresh job metadata still records run959 build3551 success on runner5/gitea-runner-2, and test3552 failure on runner1/73f8c09b0d35, started13:17:17UTC and completed16:31:07UTC. The retained job log gives no terminal cause. Available local contexts do not provide the runner-side logs/configuration needed to explain that failure. This is not evidence that the historical runner process/container was cleaned up elsewhere, or that any particular timeout/resource cause occurred. #718 remains open for actual runner logs, effective limits and cleanup proof.

Additional read-only evidence: the configured local contexts are orbstack/default Unix sockets and desktop-linux. Orbstack/default list the project development containers but no running or stopped Gitea/act-runner container matching the historical runner; desktop-linux socket is unavailable. No daemon/context configuration was changed, no container exec/stop or remote-host command was performed. Fresh job metadata still records run959 build3551 success on runner5/gitea-runner-2, and test3552 failure on runner1/73f8c09b0d35, started13:17:17UTC and completed16:31:07UTC. The retained job log gives no terminal cause. Available local contexts do not provide the runner-side logs/configuration needed to explain that failure. This is not evidence that the historical runner process/container was cleaned up elsewhere, or that any particular timeout/resource cause occurred. #718 remains open for actual runner logs, effective limits and cleanup proof.

dohertj2 commented

2026-10-13 14:33:52 -04:00

Read-only follow-up through the running server OpenAPI1.26.4 endpoints: runner1, which executed failed test job3552, is currently named73f8c09b0d35, online, not busy, enabled and non-ephemeral; it advertises ubuntu-latest and ubuntu-22.04 labels. Run959 has no retained artifacts (total_count0).

This does not establish the historical failure cause or prove absence of orphan child processes. It does rule out treating its current runner registration as offline or missing. Existing checked project infra paths contain no Gitea/runner deployment file naming its host; local Docker contexts do not expose this runner. Runner-host logs and process evidence remain unavailable. No runner edits, registration/token calls, job cancellation/deletion/reruns, remote commands or configuration changes were performed. Issue remains open.

Read-only follow-up through the running server OpenAPI1.26.4 endpoints: runner1, which executed failed test job3552, is currently named73f8c09b0d35, online, not busy, enabled and non-ephemeral; it advertises ubuntu-latest and ubuntu-22.04 labels. Run959 has no retained artifacts (total_count0). This does not establish the historical failure cause or prove absence of orphan child processes. It does rule out treating its current runner registration as offline or missing. Existing checked project infra paths contain no Gitea/runner deployment file naming its host; local Docker contexts do not expose this runner. Runner-host logs and process evidence remain unavailable. No runner edits, registration/token calls, job cancellation/deletion/reruns, remote commands or configuration changes were performed. Issue remains open.

dohertj2 referenced this issue

2026-10-13 14:43:09 -04:00

CI runbook retains obsolete job and worker model and test floor #723

dohertj2 referenced this issue

2026-10-13 16:03:28 -04:00

Offline CI retains 417 distinct test-project build warnings #725

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#718