Merge #860 #818 CI integration fixtures: busy-port retry, per-run names; floor 448

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk

.github/workflows/v2-integration.yml

@@ -76,15 +76,12 @@ jobs:
         env:
           # #816/#818 added 8 executing OpcUaClient tests; raise to 448 once a hosted run confirms them (blocked on #860)
           # (scripts/ci/README.md §"Fixture-backed integration").
-          MIN_EXECUTED: "440"
+          MIN_EXECUTED: "448"
         run: bash scripts/ci/assert-not-all-skipped.sh artifacts/trx/*.trx
       - name: fixture logs on failure
         if: failure()
         continue-on-error: true
-        run: |
-          for c in otopcua-ci-mssql otopcua-ci-pg otopcua-ci-questdb otopcua-ci-opcplc otopcua-ci-opcplc-nat; do
-            echo "=== $c ==="; docker logs --tail 40 "$c" 2>&1 || true
-          done
+        run: bash scripts/ci/integration-fixtures.sh logs
       - name: stop fixtures
         if: always()
         continue-on-error: true

infra/README.md

@@ -297,11 +297,11 @@ Things that are not obvious (#606):
   step also prints an individual-skip diagnostic (`Results/UnitTestResult` records with outcome
   `NotExecuted`); it does not affect the floor. Raise the floor as the suite grows.
 
-### CI legs beyond `unit-tests` (#757; unverified on a hosted runner until first run)
+### CI legs beyond `unit-tests` (#757; `v2-integration` hosted-verified 2026-11-11, run 1225)
 
 `v2-integration.yml` runs the fixture-backed Sql / Historian (SQL Server, QuestDB) / OpcUaClient integration
 projects against throwaway containers the job starts itself (`scripts/ci/integration-fixtures.sh`; not the
-shared 10.100.0.35 host), floor `MIN_EXECUTED=440`; that includes a port-NAT opc-plc (`otopcua-ci-opcplc-nat`,
+shared 10.100.0.35 host), floor `MIN_EXECUTED=448` (first hosted pass: run 1225, 2026-11-11, 525 executed); that includes a port-NAT opc-plc (`otopcua-ci-opcplc-nat`,
 host port 50110, exported as `OPCUA_NAT_ENDPOINT`) for the NAT browse tests (#813). `v2-e2e.yml` is manual/weekly and
 skips cleanly unless 10.100.0.35 is reachable from the runner; it sets `OPCUA_NAT_ENDPOINT=opc.tcp://10.100.0.35:50010`
 (the shared NAT stack, not a probe-required port — its tests skip if it is down). Each has its own concurrency group; see

scripts/ci/README.md

@@ -68,7 +68,7 @@ Neither touches `v2-ci.yml`; each has its own concurrency group (`lmxopcua-v2-in
 - **`v2-integration.yml`** (push to `master`, path-filtered PRs, manual; `timeout-minutes: 60`).
   `integration-fixtures.sh up` starts throwaway SQL Server 2022, PostgreSQL 17 (seeded from `seed-pg.sql`),
   QuestDB and opc-plc — plus a port-NAT opc-plc (`otopcua-ci-opcplc-nat`, host 50110 → 50000, private `.invalid`
-  advertised host, `OPCUA_NAT_ENDPOINT`, #813) — (`otopcua-ci-*`, non-default host ports), exports the env vars the fixtures read, and
+  advertised host, `OPCUA_NAT_ENDPOINT`, #813) — (`otopcua-ci-*-<run id>`, non-default host ports; a port already taken on the shared runner host is retried one higher and the chosen ports are exported), exports the env vars the fixtures read, and
   `down` removes them. It uses plain `docker run`, not `services:` (act_runner support is undocumented and
   Postgres needs a seed step); the only runner requirement is a docker daemon.
   **QuestDB is a throwaway container by decision (#841), not the shared `/opt/otopcua-questdb` fixture (#822):** CI stays
@@ -79,7 +79,7 @@ Neither touches `v2-ci.yml`; each has its own concurrency group (`lmxopcua-v2-in
   one, and fails `up` if that line is missing. Upgrade QuestDB by bumping the compose file (and the docker-dev overlay);
   `CI_QDB_IMAGE` overrides the image for a one-off trial. Both run QuestDB's defaults (incl.
   `cairo.ttl.use.wall.clock=true`, which the historian relies on). Runs the Sql, Historian.SqlServer,
-  Historian.QuestDb, OpcUaClient and OpcUaClient.Browser `*.IntegrationTests` with `MIN_EXECUTED=440`.
+  Historian.QuestDb, OpcUaClient and OpcUaClient.Browser `*.IntegrationTests` with `MIN_EXECUTED=448`.
   Local run: 468 executed / 8 skipped (Oracle x2, the blackhole `docker pause` gate, one QuestDB stress test,
   4 others), about 4.5 min of test time, QuestDB the longest (~3.3 min). Not covered (no pullable image or
   needs credentials): Modbus/S7/AB/FOCAS/TwinCAT sims, Mqtt, MTConnect, Oracle, Host real-infra mode.
@@ -89,9 +89,9 @@ Neither touches `v2-ci.yml`; each has its own concurrency group (`lmxopcua-v2-in
   `OPCUA_NAT_SERVER_CERT`). Locally (2026-11-11, same CI filter) the two OpcUaClient projects went 13 → 21 executed
   with 0 skipped (`Driver.OpcUaClient` 9, `Driver.OpcUaClient.Browser` 12), and the NAT class passed 7/7 against a
   CI-style throwaway `docker run` of the NAT container plus the `docker exec` certificate hand-off.
-  Once a hosted run's TRX shows `Driver.OpcUaClient.trx` executed=9 and `Driver.OpcUaClient.Browser.trx` executed=12,
-  raise `MIN_EXECUTED` in `v2-integration.yml` **440 → 448** (and `v2-e2e.yml` 35 → 39). Hosted runs have not yet
-  reached the test step (fixture start times out on the first network probe, #860), so this is still open (#818).
+  **Hosted-confirmed 2026-11-11 (run 1225):** `Driver.OpcUaClient.trx` executed=9, `Driver.OpcUaClient.Browser.trx`
+  executed=12, 525 executed in all, so `MIN_EXECUTED` in `v2-integration.yml` is now **448** (#818). The `v2-e2e.yml`
+  35 → 39 raise still waits for a hosted e2e run.
   If the NAT three show as skipped there, the certificate
   hand-off (`docker exec … /app/pki/own/certs/*.der`) did not work on the runner — fix that, do not lower the bump.
 - **`v2-e2e.yml`** (manual + weekly cron only; `timeout-minutes: 90`). `probe-e2e-fixtures.sh` TCP-probes

scripts/ci/integration-fixtures.sh

@@ -6,6 +6,7 @@
 #         otopcua-ci-*, wait until each is usable, and print the env exports the
 #         *.IntegrationTests fixtures read (also appended to $GITHUB_ENV when set).
 #   down  remove those containers (idempotent; touches nothing else).
+#   logs  print the tail of each fixture container's log (the workflow's on-failure step).
 #
 # Deliberately NOT the shared 10.100.0.35 host: CI must not depend on that box, and these
 # containers are created and destroyed inside the job. Host ports are non-default so a
@@ -27,7 +28,15 @@ OPC_NAT_PORT="${CI_OPC_NAT_PORT:-50110}"
 # Throwaway-container-only credentials (never used against anything else).
 SA_PASSWORD="${CI_SA_PASSWORD:-Ci_throwaway_Pw1}"
 PG_PASSWORD="${CI_PG_PASSWORD:-Ci_throwaway_pg1}"
-NAMES=(otopcua-ci-mssql otopcua-ci-pg otopcua-ci-questdb otopcua-ci-opcplc otopcua-ci-opcplc-nat)
+# Per-run container names: the runner's docker daemon is shared, so fixed names let a concurrent run's `down` remove
+# this run's containers. GITHUB_RUN_ID is set in every step of a run, so `up`, `logs` and `down` agree.
+SUFFIX="${CI_FIXTURE_SUFFIX-${GITHUB_RUN_ID:+-$GITHUB_RUN_ID}}"
+C_MSSQL="otopcua-ci-mssql$SUFFIX"
+C_PG="otopcua-ci-pg$SUFFIX"
+C_QDB="otopcua-ci-questdb$SUFFIX"
+C_OPC="otopcua-ci-opcplc$SUFFIX"
+C_NAT="otopcua-ci-opcplc-nat$SUFFIX"
+NAMES=("$C_MSSQL" "$C_PG" "$C_QDB" "$C_OPC" "$C_NAT")
 ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
 # #841: CI keeps its OWN throwaway QuestDB (hermetic — never the shared /opt/otopcua-questdb fixture), but runs the
 # SAME image the shared fixture's compose file pins, read from that file so the two cannot drift. CI_QDB_IMAGE
@@ -72,6 +81,35 @@ fixture_host() {
 
 down() { docker rm -f "${NAMES[@]}" >/dev/null 2>&1 || true; }
 
+logs() { for c in "${NAMES[@]}"; do echo "=== $c ==="; docker logs --tail 40 "$c" 2>&1 || true; done; }
+
+# start_on_free_port <container> <VAR[,VAR2]> <docker run args, with @P0@ / @P1@ standing for the ports>
+# The runner's docker host is shared, so a default host port can already be taken (#860 follow-up: run 1224 failed
+# "Bind for :::19000 failed: port is already allocated"). On that error, retry with every port bumped by one; the
+# chosen values are written back to the named variables, which the waits and the exports read afterwards.
+start_on_free_port() {
+  local name="$1" vars="$2"; shift 2
+  local -a names; IFS=, read -r -a names <<< "$vars"
+  local -a ports=(); local v; for v in "${names[@]}"; do ports+=("${!v}"); done
+  local attempt out i
+  for ((attempt = 0; attempt < 20; attempt++)); do
+    local -a args=("$@")
+    for i in "${!ports[@]}"; do args=("${args[@]//@P${i}@/${ports[$i]}}"); done
+    if out="$(docker run -d --name "$name" "${args[@]}" 2>&1)"; then
+      for i in "${!names[@]}"; do printf -v "${names[$i]}" '%s' "${ports[$i]}"; done
+      return 0
+    fi
+    docker rm -f "$name" >/dev/null 2>&1 || true
+    if [[ "$out" == *"port is already allocated"* || "$out" == *"address already in use"* ]]; then
+      echo "host port(s) ${ports[*]} busy for $name; retrying one higher" >&2
+      for i in "${!ports[@]}"; do ports[$i]=$((ports[$i] + 1)); done
+      continue
+    fi
+    echo "$out" >&2; return 1
+  done
+  echo "no free host port for $name after 20 attempts" >&2; return 1
+}
+
 wait_for() { # <description> <timeout-seconds> <command...>
   local what="$1" t="$2"; shift 2
   local end=$((SECONDS + t))
@@ -86,31 +124,31 @@ up() {
   command -v docker >/dev/null || { echo "docker not available" >&2; exit 1; }
   local qdb_image; qdb_image="$(questdb_image)"
   down
-  docker run -d --name otopcua-ci-mssql -e ACCEPT_EULA=Y -e MSSQL_PID=Developer \
-    -e "MSSQL_SA_PASSWORD=$SA_PASSWORD" -p "$SQL_PORT:1433" \
-    mcr.microsoft.com/mssql/server:2022-latest >/dev/null
-  docker run -d --name otopcua-ci-pg -e "POSTGRES_PASSWORD=$PG_PASSWORD" -e POSTGRES_DB=sqlpollfixture \
-    -p "$PG_PORT:5432" postgres:17-alpine >/dev/null
-  docker run -d --name otopcua-ci-questdb -p "$QDB_PORT:8812" -p "$QDB_HTTP_PORT:9000" \
-    "$qdb_image" >/dev/null
-  docker run -d --name otopcua-ci-opcplc -p "$OPC_PORT:$OPC_PORT" \
-    mcr.microsoft.com/iotedge/opc-plc:2.14.10 --pn="$OPC_PORT" --ut --aa --alm >/dev/null
+  start_on_free_port "$C_MSSQL" SQL_PORT -e ACCEPT_EULA=Y -e MSSQL_PID=Developer \
+    -e "MSSQL_SA_PASSWORD=$SA_PASSWORD" -p "@P0@:1433" mcr.microsoft.com/mssql/server:2022-latest
+  start_on_free_port "$C_PG" PG_PORT -e "POSTGRES_PASSWORD=$PG_PASSWORD" -e POSTGRES_DB=sqlpollfixture \
+    -p "@P0@:5432" postgres:17-alpine
+  start_on_free_port "$C_QDB" QDB_PORT,QDB_HTTP_PORT -p "@P0@:8812" -p "@P1@:9000" "$qdb_image"
+  # opc-plc advertises the port it listens on, so it must be the SAME inside and out (--pn follows the host port).
+  start_on_free_port "$C_OPC" OPC_PORT -p "@P0@:@P0@" \
+    mcr.microsoft.com/iotedge/opc-plc:2.14.10 --pn="@P0@" --ut --aa --alm
   # Mirrors tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/Docker/nat/docker-compose.yml.
-  docker run -d --name otopcua-ci-opcplc-nat --hostname otopcua-ci-opcplc-nat-private -p "$OPC_NAT_PORT:50000" \
-    mcr.microsoft.com/iotedge/opc-plc:2.14.10 --pn=50000 --ph=otopcua-ci-opcplc-nat-private.invalid --ut --aa >/dev/null
+  start_on_free_port "$C_NAT" OPC_NAT_PORT --hostname otopcua-ci-opcplc-nat-private -p "@P0@:50000" \
+    mcr.microsoft.com/iotedge/opc-plc:2.14.10 --pn=50000 --ph=otopcua-ci-opcplc-nat-private.invalid --ut --aa
+  echo "fixture ports: sql=$SQL_PORT pg=$PG_PORT questdb=$QDB_PORT/$QDB_HTTP_PORT opc=$OPC_PORT opc-nat=$OPC_NAT_PORT (containers *$SUFFIX)"
 
-  wait_for "postgres" 120 docker exec otopcua-ci-pg pg_isready -U postgres -d sqlpollfixture
+  wait_for "postgres" 120 docker exec "$C_PG" pg_isready -U postgres -d sqlpollfixture
   # pg_isready can answer during initdb's temp server; require a real query on the final server.
-  wait_for "postgres query" 60 docker exec -e "PGPASSWORD=$PG_PASSWORD" otopcua-ci-pg \
+  wait_for "postgres query" 60 docker exec -e "PGPASSWORD=$PG_PASSWORD" "$C_PG" \
     psql -h 127.0.0.1 -U postgres -d sqlpollfixture -c 'select 1'
-  docker exec -i -e "PGPASSWORD=$PG_PASSWORD" otopcua-ci-pg \
+  docker exec -i -e "PGPASSWORD=$PG_PASSWORD" "$C_PG" \
     psql -v ON_ERROR_STOP=1 -v "app_password=$PG_PASSWORD" -h 127.0.0.1 -U postgres -d sqlpollfixture \
     < "$ROOT/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Sql.IntegrationTests/Docker/postgres/seed-pg.sql" >/dev/null
   echo "seeded: postgres"
 
   echo "fixture host: $(fixture_host) (in container: $([ -f /.dockerenv ] && echo yes || echo no))"
   local sqlcmd=/opt/mssql-tools18/bin/sqlcmd
-  wait_for "sql server" 300 docker exec otopcua-ci-mssql "$sqlcmd" -C -S localhost -U sa -P "$SA_PASSWORD" -Q 'SELECT 1'
+  wait_for "sql server" 300 docker exec "$C_MSSQL" "$sqlcmd" -C -S localhost -U sa -P "$SA_PASSWORD" -Q 'SELECT 1'
   wait_for "questdb" 120 curl -fsS "http://$(fixture_host):$QDB_HTTP_PORT/exec?query=select%201"
   wait_for "opc-plc" 120 bash -c "exec 3<>/dev/tcp/$(fixture_host)/$OPC_PORT"
   wait_for "opc-plc (NAT)" 120 bash -c "exec 3<>/dev/tcp/$(fixture_host)/$OPC_NAT_PORT"
@@ -121,7 +159,7 @@ up() {
   local nat_cert="${RUNNER_TEMP:-${TMPDIR:-/tmp}}/otopcua-ci-fixtures/opc-plc-nat-server.der"
   mkdir -p "$(dirname "$nat_cert")"
   wait_for "opc-plc (NAT) certificate" 60 bash -c \
-    "docker exec otopcua-ci-opcplc-nat bash -c 'cat /app/pki/own/certs/*.der' > '$nat_cert' && [ -s '$nat_cert' ]"
+    "docker exec '$C_NAT' bash -c 'cat /app/pki/own/certs/*.der' > '$nat_cert' && [ -s '$nat_cert' ]"
 
   local h; h="$(fixture_host)"
   local -a exports=(
@@ -145,5 +183,6 @@ up() {
 case "${1:-}" in
   up) up ;;
   down) down ;;
-  *) echo "usage: $0 up|down" >&2; exit 2 ;;
+  logs) logs ;;
+  *) echo "usage: $0 up|down|logs" >&2; exit 2 ;;
 esac