P8-8: replace the remaining wall-clock sleep-then-assert instances outside
Communication.Tests' Medium-wave sweep with the same row-40 house pattern —
continuous polling (AssertHoldsFor) for ABSENCE assertions instead of a single
end-of-window sample, which cannot distinguish "stayed quiet throughout" from
"hadn't happened yet when the one sample landed" and cannot catch a value that
flips wrong and back before that sample.
- DataConnectionActorBatchTests: QualityCounters_ArePushedOnTransitionOnly_AndCoalesced
(:206/:214 class) converted to AssertHoldsFor; ResolutionProbes_BackOff...
(:181/:184, a genuine real-time backoff-RATE assertion, not a sync-primitive
stand-in) converted to AssertRoundsWithinWindow, which enforces the ceiling
continuously (fails fast on a no-backoff regression) while the floor still
necessarily waits out the window.
- SiteAlarmLiveCacheServiceTests (:452-454, :470-472): converted to the
project's own AssertHoldsFor (mirrors SiteAlarmAggregatorActorTests', same
assembly).
- StoreAndForwardServiceTests (:891-892): async AssertHoldsForAsync variant
(the class runs on the async test thread).
- KpiHistoryRecorderActorTests (:1043-1049): two skip-path FoldCount==0
absence checks converted.
Each helper is a private per-file copy, matching the Medium-wave T17
precedent (SiteAlarmAggregatorActorTests.AssertHoldsFor) rather than a shared
TestSupport utility. Proof of the mechanism: a standalone (non-repo) harness
run 100 trials of a real background thread flipping a boolean true then back
to false inside a 300ms window — Thread.Sleep(300)+one sample missed the
violation 100/100 times (false pass), continuous 20ms polling caught it
100/100 times. Most of the actual conversions here guard monotonic counters
(their single-sample variant would eventually see the final value too), so
each converted test carries an inline honesty note on what specifically the
conversion buys for that assertion (fail-fast vs. true flip-back detection).
P8-9: ScadaBridgeTelemetryTests.EmitHelpers_DoNotThrow called
SetQueueDepthProvider(() => 5) against the process-global row-32 provider
slot with no clear and no [Collection] — a landmine for any future
same-process reader of scadabridge.store_and_forward.queue.depth. Fixed with
compare-and-clear in a finally via the existing (production, untouched)
ClearQueueDepthProvider. Searched this assembly and the whole test tree for a
second same-process reader that asserts on the value: none exists today
(MetricsEndpointTests scrapes /metrics but only asserts on the response
shape, and the only value-asserting test, QueueDepthGaugeTests, lives in
StoreAndForward.Tests — a separate test-host process the leak cannot reach),
so no [Collection] was added; the test's doc comment records the search and
flags it for re-check if that changes.
Test-only change, zero production edits. Full suites green: DCL.Tests
342/342, Communication.Tests 729/729, StoreAndForward.Tests 135/135,
KpiHistory.Tests 54/54, Host.Tests 522/522. Each converted test additionally
looped 20x in isolation with zero flakes.
Six sleeps across four test files stood in for an observable edge. Each preceded an
ABSENCE assertion (or "settled" before one), so they failed toward false-pass rather than
flake: the assertion could not distinguish "the mechanism ran and correctly stayed quiet"
from "the mechanism had not run yet".
ScriptDeadlineAtEnqueueTests (two sleeps, the worst offender)
- :105 Thread.Sleep(2s) — now two explicit edges: the run must be observably QUEUED
(scheduler.QueueDepth >= 1, never checked before), then the wait polls until the 1 s
budget has elapsed, bailing early if the run leaves the queue, and asserts it is STILL
queued afterwards. The test now proves the design claim it was named for — queue wait
burns the run's budget — instead of assuming it.
- :128 Thread.Sleep(2s) + Assert.Equal(0, DetachedThreadCount) — replaced by a positive
control: a second script that genuinely wedges a worker, on the same scheduler, same
1 s timeout + 1 s grace, armed strictly later. Its stuck report is the edge; once it
lands, every watchdog armed at or before it has provably finished its grace delay. The
absence is then asserted per script name, and the detach counter is pinned at 1 (the
wedged run) rather than 0 — which a counter that never moves also satisfied.
DeploymentWarmThenGateTests (:117 plus two sibling Task.Delay(500) startup waits)
- The manager calls SetInstanceCounts unconditionally when its startup config load
completes, so a report counter on the test collector is the startup edge. The
empty-site zero is now the manager's own observation instead of an untouched field.
SiteAlarmLiveCacheServiceTests (:195, :216)
- Both sampled IsLive once after a fixed sleep. Now they poll continuously across three
linger windows on a new teardown observable (the mock client counts subscription
cancellations), so a stop at any instant inside the window is caught, and each closes
with a positive control proving that counter moves.
DataConnectionActorTests (:435)
- A Task.Delay(200) "settle" after an AwaitCondition on a connect counter. The counter
only proves ConnectAsync was CALLED; the actor is still Connecting until it processes
the piped result. Now waits for the Connected health publication that BecomeConnected
emits, so the subscribe under test provably runs against Connected rather than being
silently stashed. (The stash path keeps its own coverage in
WP6_ConnectingState_StashesSubscribeRequests.)
Negative controls (product broken in the working tree, old vs new test compared):
- Stuck-script watchdog disabled entirely: OLD test passed 2/2, NEW test failed. The
whole watchdog could be deleted and the old assertions were satisfied.
- Startup reporting a phantom instance count and taking longer than 500 ms: OLD passed
2/2, NEW failed (expected 0, actual 1).
- Linger cancellation disabled + the remaining-viewer guard dropped, so the stream is
torn down under a live viewer while IsLive keeps reporting true:
Resubscribe_Within_Linger OLD passed 2/2, NEW failed. (Last_Viewer_Leaving caught this
particular break in the old form too, via a later assertion — its strengthening is the
teardown observable and the positive control, not a false-pass fix.)
- DataConnectionActorTests :435 has no false-pass demo: the sleep guarded no assertion
there, and the change is determinism plus proving the claimed state.
Also fixes a latent race these timing changes exposed in
InstanceActorChildAttributeRaceTests: it resolved child actors single-shot immediately
after construction, but ResolveOne throws until the child's cell is published on the
parent AND the cell's Actor property stays null until the child's constructor has run on
its own mailbox thread. Neither is ordered against the test thread, so it failed both ways
under full-suite load (~1 run in 6 once the sleeps above stopped padding the schedule;
14/14 green on the unmodified tree, so it was latent, not new). Resolution now retries to
a bounded deadline; the assertion is unchanged.
Verification: SiteRuntime.Tests 8/8 green (605 tests), Communication.Tests 3/3 (717),
DataConnectionLayer.Tests 3/3 (319). Test-only change — no src/ modifications.
Extend CommunicationOptionsValidator with eager bounds for the four T4
live-alarm-cache options (linger >= 0, reconcile > 0, seed concurrency 1..64,
subscribers-per-site >= 1). Enforce the per-site viewer cap fail-safe in
SiteAlarmLiveCacheService.Subscribe (reject excess viewers with a no-op
disposable rather than growing the list or throwing into the Blazor render
path). Surface two telemetry instruments on the existing ScadaBridgeTelemetry
meter: an active-aggregator observable gauge and a reconnect counter, wired from
the aggregator actor's PreStart/PostStop and its NodeA<->NodeB flip /
reconcile-driven reopen.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.
- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
(AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
into an in-memory dict. Placeholder rows seeded from the snapshot and never
expected on the stream; a real-alarm delta (distinct key) never wipes them.
NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
linger-delayed last-out stop (version + TryRemove(ref) race guards), the
snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
bounded start-retry self-heal on transient start failure, and the immutable
published-snapshot store the page reads. Cache mutated only on the actor thread;
viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
(60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
(200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
(shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
both Important findings addressed.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
Joseph Doherty