CI: two unit-test races failed hosted v2-ci runs 1218–1220 (MeshPsk refresh, credential-lockout burst) #871

New Issue

2026-11-11T15:09:59-05:00

dohertj2 commented

2026-11-11 15:09:59 -05:00

Hosted v2-ci unit-tests failed on master for runs 1218, 1219 and 1220 (d1b6cad2, 03e51457, e857135e). Each failure was a test race, not a product defect:

MeshPskProviderRefreshTests.A_rotated_key_is_swapped_in_atomically_with_the_old_key_served_until_then (runs 1218, 1219): resolver.Calls expected 1, was 2. GatedResolver.Entered was set by the priming refresh and never reset, so Wait() returned before the held refresh reached the store and the call-count snapshot raced its increment.
FailedCredentialLockoutTests.An_all_bad_parallel_burst_authenticates_only_max_attempts_and_refuses_the_waiters_as_locked_out (runs 1218, 1220): authenticated expected 5, was 0. The 15 waiters could queue before a slot holder ran the increment after BeginAsync.
ConditionRefreshDeferralWireTests.Concurrent_reports_…(enforced: True) (run 1219): the stress precondition failed (deferredSeen was 1). Already fixed by 284358a2 (presence wait with a 90 s ceiling), which landed after that run.

Fix: Hold() resets Entered; the lockout test waits for both the queued count and the slot holders' increments. Security assertions unchanged. Both tests looped 20 times locally with 0 failures.

Hosted `v2-ci` unit-tests failed on master for runs 1218, 1219 and 1220 (`d1b6cad2`, `03e51457`, `e857135e`). Each failure was a test race, not a product defect: - `MeshPskProviderRefreshTests.A_rotated_key_is_swapped_in_atomically_with_the_old_key_served_until_then` (runs 1218, 1219): `resolver.Calls` expected 1, was 2. `GatedResolver.Entered` was set by the priming refresh and never reset, so `Wait()` returned before the held refresh reached the store and the call-count snapshot raced its increment. - `FailedCredentialLockoutTests.An_all_bad_parallel_burst_authenticates_only_max_attempts_and_refuses_the_waiters_as_locked_out` (runs 1218, 1220): `authenticated` expected 5, was 0. The 15 waiters could queue before a slot holder ran the increment after `BeginAsync`. - `ConditionRefreshDeferralWireTests.Concurrent_reports_…(enforced: True)` (run 1219): the stress precondition failed (`deferredSeen` was 1). Already fixed by `284358a2` (presence wait with a 90 s ceiling), which landed after that run. Fix: `Hold()` resets `Entered`; the lockout test waits for both the queued count and the slot holders' increments. Security assertions unchanged. Both tests looped 20 times locally with 0 failures.

dohertj2 added the type:test priority:p3 labels 2026-11-11 15:09:59 -05:00

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

test: #871 fix two hosted-CI races that failed master runs 1218-1220

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

test(adminui): #871 reaper not-a-guid database name unique per run

dohertj2 referenced this issue from a commit

2026-11-11 15:38:12 -05:00

Merge #866 #870 #871 #872: MxStatus detail vocabulary + quality grading, ring event queue, CI races

dohertj2 commented

2026-11-11 15:38:33 -05:00

Fixed, pushed to master in 363f0d81 (9aaab9ee). Both tests looped 20 times locally with 0 failures.

Also fixed (11ea972c): SqlTestDatabaseReaperTests.Reap_leaves_an_aged_name_whose_suffix_is_not_a_guid_alone failed once in the local pre-push suite and passed 3 of 3 on re-run. It used a fixed database name on the SQL Server that every checkout and CI share, so the name is now unique per run. That cause is plausible but not proven.

Fixed, pushed to master in `363f0d81` (`9aaab9ee`). Both tests looped 20 times locally with 0 failures. Also fixed (`11ea972c`): `SqlTestDatabaseReaperTests.Reap_leaves_an_aged_name_whose_suffix_is_not_a_guid_alone` failed once in the local pre-push suite and passed 3 of 3 on re-run. It used a fixed database name on the SQL Server that every checkout and CI share, so the name is now unique per run. That cause is plausible but not proven.

dohertj2 closed this issue

2026-11-11 15:38:33 -05:00

dohertj2 commented

2026-11-11 17:52:01 -05:00

Confirmed on the hosted runner: v2-ci run 1222 (363f0d81, which includes these fixes) is green. Runs 1218–1221 had failed on these races.

Confirmed on the hosted runner: v2-ci run 1222 (`363f0d81`, which includes these fixes) is **green**. Runs 1218–1221 had failed on these races.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#871