ci(v2): whole-solution unit leg + fail-on-skip gate + deflake CLI sleeps (arch-review 07/S-1,S-2,S-4)

S-1: replace the hand-maintained 5-project unit-tests matrix (which silently dropped
Client's 388 tests, Analyzers' 31, and every driver + most Core suite) with ONE
whole-solution leg — dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter
"Category!=E2E&Category!=LiveIntegration". Self-maintaining: a new *.Tests project is
covered automatically, matching CLAUDE.md's own guidance.

S-2: emit trx + add scripts/ci/assert-not-all-skipped.sh, a fail-on-skip gate that
turns 'green CI == everything skipped' into a red build. Wired strict (MIN_EXECUTED=1)
on the unit leg; report-only (MIN_EXECUTED=0) on the fixtureless integration leg so
its skip tally is VISIBLE without a false red — with a documented follow-up to start
the one public-image fixture (opc-plc) as a service and flip it strict.

S-4 (paired): the newly-CI'd Client.CLI.Tests had fixed-sleep startup races
(await Task.Delay(100/150) before cancelling a background command) that would flake
under CI load. Added SubscribeInvoked / SubscribeAlarmsInvoked readiness signals
(TaskCompletionSource) to FakeOpcUaClientService and replaced the 11 sleeps across
AlarmsCommandTests / SubscribeCommandTests / EventHandlerLifecycleTests with a
deterministic await-the-signal (10s timeout guard).

Verified: workflow YAML parses; skip-gate proven locally on a real trx (executed=31
=> OK), a synthetic all-skipped trx (executed=0 => exit 1 with diagnostic),
report-only mode (never fails), multi-file sum, and missing-file (exit 2);
Client.CLI.Tests 104/104 green after the deflake. (CI job execution itself is
verifiable only on push — nothing pushed.)
This commit is contained in:
Joseph Doherty
2026-07-08 17:58:42 -04:00
parent 9cad9ed0fc
commit 10b898305f
6 changed files with 127 additions and 27 deletions
+54
View File
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
#
# assert-not-all-skipped.sh — arch-review 07/S-2 fail-on-skip guard.
#
# A CI test leg that ran but executed ZERO tests reports success, so a whole tier
# silently skipping (fixtures unreachable, a --filter that matched nothing, a probe
# that Assert.Skip'd every case) is indistinguishable from a real pass. This gate
# reads the trx result summary and FAILS when fewer than MIN_EXECUTED tests actually
# ran across the supplied trx files — turning "green means skipped" into a red build.
#
# Usage: scripts/ci/assert-not-all-skipped.sh <trx> [<trx> ...]
# Env: MIN_EXECUTED minimum executed tests required (default 1).
# Set MIN_EXECUTED=0 for a report-only pass (never fails) —
# used by the fixtureless integration leg until its fixtures
# run as workflow services (S-2 option #2 follow-up).
#
set -euo pipefail
min_executed="${MIN_EXECUTED:-1}"
if [ "$#" -eq 0 ]; then
echo "assert-not-all-skipped: no trx files given" >&2
exit 2
fi
total_executed=0
total_skipped=0
seen=0
for f in "$@"; do
if [ ! -f "$f" ]; then
echo "assert-not-all-skipped: trx not found: $f" >&2
exit 2
fi
seen=$((seen + 1))
# The trx <Counters .../> element in <ResultSummary> carries executed + notExecuted counts.
counters="$(grep -oE '<Counters[^>]*/?>' "$f" | head -1)"
ex="$(printf '%s' "$counters" | grep -oE 'executed="[0-9]+"' | grep -oE '[0-9]+' || true)"
ne="$(printf '%s' "$counters" | grep -oE 'notExecuted="[0-9]+"' | grep -oE '[0-9]+' || true)"
ex="${ex:-0}"
ne="${ne:-0}"
total_executed=$((total_executed + ex))
total_skipped=$((total_skipped + ne))
echo " $(basename "$f"): executed=${ex} skipped=${ne}"
done
echo "assert-not-all-skipped: files=${seen} executed=${total_executed} skipped=${total_skipped} (min required=${min_executed})"
if [ "$total_executed" -lt "$min_executed" ]; then
echo "::error::CI executed ${total_executed} test(s) (< ${min_executed}) — the tier ran nothing (all skipped?). Treating as FAILURE, not a pass." >&2
exit 1
fi
echo "assert-not-all-skipped: OK"