fix(test): order three more unsynchronized assertions behind the observables they follow

Deferred flake-pattern sweep of tests/ for the class fixed in c4caebe9 and cfa6acbf — a bounded wait on observable A followed by a bare assert on an observable B that the product only reaches strictly after A. Three clear instances, each reproduced deterministically by delaying only the later step and each re-verified green with that same delay still injected. AlarmOnTriggerRuns_ShedAtTheSameCap_WithAnAlarmScopedSiteEvent gated on the rate-limited shed site event and then asserted the shed COUNT bare. AlarmActor.ShedAlarmRun increments the counter and only then emits the event, and the event fires on the first shed only — so the gate observed Flap(4)'s shed and ordered nothing with respect to Flap(5)'s, which is a separate mailbox message with no observable of its own (an alarm on-trigger run has no Ask caller to reply to, unlike ScriptActor.ShedRun, whose sibling test is correctly ordered by its ScriptCallResult and is left alone). Deferring Flap(5) by 2 s failed it with "Expected: 2 / Actual: 1". The count is now ACCUMULATED across polls rather than re-read, because SiteHealthCollector.CollectReport DRAINS the interval counters — a poll loop that simply re-read it would consume the first shed and never reach 2. EndToEnd_GrpcStubError_RowStays_Pending_NextTick_Succeeds gated on the central row arriving and then asserted bare that the site SQLite row had left Pending. SiteAuditTelemetryActor pushes via IngestAuditEventsAsync (which is what writes the central row) and calls MarkForwardedAsync only after parsing the ack. Delaying just that post-push step failed it with "Assert.DoesNotContain() Failure: Filter matched in collection". PreSnapshotBuffer_IsCapped_DropsOldest_AndCountsTheDrops gated on "Count >= cap" and then asserted "Count == cap + 1" bare — a gate strictly weaker than the assertion it guards, so it ordered nothing with respect to the last event of a FlushBuffer loop that delivers one at a time. Parking that loop after its 19,999th delivery failed it with "Expected: 20001 / Actual: 20000". Also hardens GrpcCentralTransportTests.WaitUntil, which returned silently on timeout; today's single caller re-asserts immediately, so this only sharpens the message rather than fixing a live flake. Cleared with evidence, not guessed: SiteAlarmLiveCacheService's LingerStop removes the site entry inside one lock, so IsLive and GetCurrentAlarms flip atomically; and SiteReconciliationActor walks response.Gap with a sequential foreach in which the asserted "Gone" log precedes the awaited "Good" row, the inverse of this class. Test-only; every ordering named above is correct as written.
2026-08-15 03:37:35 -04:00
parent ca30d17f94
commit 9fb52153fd
4 changed files with 74 additions and 15 deletions
@@ -213,14 +213,32 @@ public class SyncCallEmissionEndToEndTests : TestKit, IClassFixture<MsSqlMigrati
            Assert.Equal(evt.EventId, rows[0].EventId);
        }, TimeSpan.FromSeconds(15));

+        // Safe as a bare assertion: DirectActorSiteStreamAuditClient increments
+        // CallCount as its FIRST statement, before the Ask that writes the
+        // central row — so the row existing already implies the second call.
        Assert.True(stubClient.CallCount >= 2,
            $"Expected at least one failed push + one successful push; saw {stubClient.CallCount} total client calls.");

        // The site SQLite row must have flipped to Forwarded after the
        // successful retry. ReadPendingAsync only returns Pending rows; the
        // row should NOT show up there anymore.
-        var stillPending = await sqliteWriter.ReadPendingAsync(64);
-        Assert.DoesNotContain(stillPending, p => p.EventId == evt.EventId);
+        //
+        // AwaitAssert, not a bare Assert: the drain marks rows forwarded only
+        // AFTER the push returns and its ack is parsed — SiteAuditTelemetryActor
+        // pushes via IngestAuditEventsAsync (which is what writes the central
+        // row) and only then calls MarkForwardedAsync. Observing the central row
+        // above therefore establishes no happens-before edge with the site-side
+        // state flip; on a loaded run the post-push continuation can be scheduled
+        // after the poll that saw the row, leaving it still Pending. Reproduced
+        // deterministically by delaying only that post-push step, which fails
+        // exactly this test with "Assert.DoesNotContain() Failure: Filter matched
+        // in collection". The bounded wait removes the ordering assumption only —
+        // the row must still actually leave Pending or the test fails as before.
+        await AwaitAssertAsync(async () =>
+        {
+            var stillPending = await sqliteWriter.ReadPendingAsync(64);
+            Assert.DoesNotContain(stillPending, p => p.EventId == evt.EventId);
+        }, TimeSpan.FromSeconds(15));
    }

    [SkippableFact]