CI v2-integration: hosted runner never reaches tests — TIMEOUT waiting for questdb (fixture network reachability) #860
Assignees
dohertj2 (dohertj2)
Clear assignees
dohertj2 (dohertj2)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: dohertj2/lmxopcua#860
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hosted
v2-integration.ymlhas never reached its test step. Both nightly runs on record fail instart throwaway fixtures:TIMEOUT waiting for questdbb36e688f):ready: postgres,ready: postgres query,seeded: postgres,ready: sql server, thenTIMEOUT waiting for questdbafter 120 sso
assert not-all-skippedthen fails withtrx not found.Probably not QuestDB-specific (hypothesis, not verified on the runner). Postgres and SQL Server readiness both go through
docker exec. The QuestDB wait is the first network probe:curl http://$(fixture_host):$QDB_HTTP_PORT/..., wherefixture_host(scripts/ci/integration-fixtures.sh) is the job container's default-gateway IP (ip route) when/.dockerenvexists, else 127.0.0.1. If the act_runner job container can't reach host-published ports through that gateway (separate job network, host firewall, or noipbinary so it falls back to 127.0.0.1), every later network wait (opc-plc, opc-plc NAT) and every test endpoint would fail the same way.Next step: in a dispatched run, print
fixture_host,command -v ip,ip route,docker network ls, and the job container's networks; trycurlagainst the QuestDB container IP (docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') and against the gateway. Possible fixes: attach the fixture containers to the job container's network and address them by name, or setFIXTURE_HOSTexplicitly.Blocks #818's floor raise:
MIN_EXECUTED440 → 448 (v2-integration.yml) waits on a hosted TRX showingDriver.OpcUaClient.trxexecuted=9 andDriver.OpcUaClient.Browser.trxexecuted=12. Locally, with the same CI filter and the CI-style throwaway NAT container plus its exported cert, those counts are 9 and 12 with 0 skipped (2026-11-11).Likely root cause fixed (
4b68729a), Pushed to master ine857135e. the job imagecatthehacker/ubuntu:act-latestships noip, sofixture_host()fell back to 127.0.0.1 inside the job container (networktraefik) and every published-port probe timed out — QuestDB is just the first one probed over the network. The script now reads the default gateway from /proc/net/route (plain bash) and logs the chosen host. Verified locally in that exact image on a user-defined network (gateway parsed, published port answered 200). Keeping open until the next nightlyv2-integrationrun shows the fixtures ready; if the runner host firewalls bridge→published-port traffic it will still time out and needs a host-side fix.Progress. The first hosted run since
4b68729awas a manual dispatch, run 1224 on88698ca0. It got past the old timeout but failed one step earlier, at container start:Bind for :::19000 failed: port is already allocated. The runner's docker daemon is shared, so a fixed host port can already be taken.Fix:
b114779bonfix/860-ci-fixture-ports.downcannot remove them.logssubcommand replaces the workflow's hard-coded container list.Local check: with 19000 already taken, QuestDB moved to 18813/19001 and every fixture reached ready.
logsanddownboth found the suffixed containers.A hosted run has been dispatched on the branch to confirm before merging.
Fixed and hosted-verified. Integration run 1225 on
b114779bpassed, the firstv2-integrationrun ever to complete on the runner. Port 19000 was busy again on the runner, and the fixture script moved QuestDB to 18813/19001 as designed. 525 tests executed, 40 skipped, with every project green. Merged to master in43f8b8e8.