ci(v2): whole-solution unit leg + fail-on-skip gate + deflake CLI sleeps (arch-review 07/S-1,S-2,S-4)

S-1: replace the hand-maintained 5-project unit-tests matrix (which silently dropped
Client's 388 tests, Analyzers' 31, and every driver + most Core suite) with ONE
whole-solution leg — dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter
"Category!=E2E&Category!=LiveIntegration". Self-maintaining: a new *.Tests project is
covered automatically, matching CLAUDE.md's own guidance.

S-2: emit trx + add scripts/ci/assert-not-all-skipped.sh, a fail-on-skip gate that
turns 'green CI == everything skipped' into a red build. Wired strict (MIN_EXECUTED=1)
on the unit leg; report-only (MIN_EXECUTED=0) on the fixtureless integration leg so
its skip tally is VISIBLE without a false red — with a documented follow-up to start
the one public-image fixture (opc-plc) as a service and flip it strict.

S-4 (paired): the newly-CI'd Client.CLI.Tests had fixed-sleep startup races
(await Task.Delay(100/150) before cancelling a background command) that would flake
under CI load. Added SubscribeInvoked / SubscribeAlarmsInvoked readiness signals
(TaskCompletionSource) to FakeOpcUaClientService and replaced the 11 sleeps across
AlarmsCommandTests / SubscribeCommandTests / EventHandlerLifecycleTests with a
deterministic await-the-signal (10s timeout guard).

Verified: workflow YAML parses; skip-gate proven locally on a real trx (executed=31
=> OK), a synthetic all-skipped trx (executed=0 => exit 1 with diagnostic),
report-only mode (never fails), multi-file sum, and missing-file (exit 2);
Client.CLI.Tests 104/104 green after the deflake. (CI job execution itself is
verifiable only on push — nothing pushed.)
This commit is contained in:
Joseph Doherty
2026-07-08 17:58:42 -04:00
parent 9cad9ed0fc
commit 10b898305f
6 changed files with 127 additions and 27 deletions
@@ -23,7 +23,7 @@ public class AlarmsCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -48,7 +48,7 @@ public class AlarmsCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -73,7 +73,7 @@ public class AlarmsCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -101,7 +101,7 @@ public class AlarmsCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -124,7 +124,7 @@ public class AlarmsCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -146,7 +146,7 @@ public class AlarmsCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -51,7 +51,7 @@ public class EventHandlerLifecycleTests
var task = Task.Run(async () => await command.ExecuteAsync(console));
await Task.Delay(150);
await fakeService.SubscribeAlarmsInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -45,6 +45,20 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService
/// <summary>Gets a value indicating whether UnsubscribeAlarmsAsync was called.</summary>
public bool UnsubscribeAlarmsCalled { get; private set; }
// Readiness signals (arch-review 07/S-4): tests that start a long-running command on a
// background task previously slept a fixed `await Task.Delay(100)` to let it reach its
// subscribe call before cancelling — a race that flakes under CI load. These TCS complete
// the instant the command actually invokes the corresponding subscribe, so tests can await
// the real signal (with a generous timeout) instead of guessing a delay.
private readonly TaskCompletionSource _subscribeInvoked = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly TaskCompletionSource _subscribeAlarmsInvoked = new(TaskCreationOptions.RunContinuationsAsynchronously);
/// <summary>Completes when the command under test first calls <see cref="SubscribeAsync"/>.</summary>
public Task SubscribeInvoked => _subscribeInvoked.Task;
/// <summary>Completes when the command under test first calls <see cref="SubscribeAlarmsAsync"/>.</summary>
public Task SubscribeAlarmsInvoked => _subscribeAlarmsInvoked.Task;
/// <summary>Gets a value indicating whether RequestConditionRefreshAsync was called.</summary>
public bool RequestConditionRefreshCalled { get; private set; }
@@ -187,6 +201,7 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService
public Task SubscribeAsync(NodeId nodeId, int intervalMs = 1000, CancellationToken ct = default)
{
SubscribeCalls.Add((nodeId, intervalMs));
_subscribeInvoked.TrySetResult();
if (SubscribeException != null) throw SubscribeException;
return Task.CompletedTask;
}
@@ -202,6 +217,7 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService
public Task SubscribeAlarmsAsync(NodeId? sourceNodeId = null, int intervalMs = 1000, CancellationToken ct = default)
{
SubscribeAlarmsCalls.Add((sourceNodeId, intervalMs));
_subscribeAlarmsInvoked.TrySetResult();
return Task.CompletedTask;
}
@@ -26,8 +26,8 @@ public class SubscribeCommandTests
// Use the console's cancellation to trigger stop.
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
// Give it a moment to subscribe, then cancel
await Task.Delay(100);
// Wait until the command has actually subscribed (deterministic — no fixed-sleep race), then cancel
await fakeService.SubscribeInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -52,7 +52,7 @@ public class SubscribeCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -75,7 +75,7 @@ public class SubscribeCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;
@@ -100,7 +100,7 @@ public class SubscribeCommandTests
var task = Task.Run(async () => { await command.ExecuteAsync(console); });
await Task.Delay(100);
await fakeService.SubscribeInvoked.WaitAsync(TimeSpan.FromSeconds(10));
console.RequestCancellation();
await task;