Six sleeps across four test files stood in for an observable edge. Each preceded an
ABSENCE assertion (or "settled" before one), so they failed toward false-pass rather than
flake: the assertion could not distinguish "the mechanism ran and correctly stayed quiet"
from "the mechanism had not run yet".
ScriptDeadlineAtEnqueueTests (two sleeps, the worst offender)
- :105 Thread.Sleep(2s) — now two explicit edges: the run must be observably QUEUED
(scheduler.QueueDepth >= 1, never checked before), then the wait polls until the 1 s
budget has elapsed, bailing early if the run leaves the queue, and asserts it is STILL
queued afterwards. The test now proves the design claim it was named for — queue wait
burns the run's budget — instead of assuming it.
- :128 Thread.Sleep(2s) + Assert.Equal(0, DetachedThreadCount) — replaced by a positive
control: a second script that genuinely wedges a worker, on the same scheduler, same
1 s timeout + 1 s grace, armed strictly later. Its stuck report is the edge; once it
lands, every watchdog armed at or before it has provably finished its grace delay. The
absence is then asserted per script name, and the detach counter is pinned at 1 (the
wedged run) rather than 0 — which a counter that never moves also satisfied.
DeploymentWarmThenGateTests (:117 plus two sibling Task.Delay(500) startup waits)
- The manager calls SetInstanceCounts unconditionally when its startup config load
completes, so a report counter on the test collector is the startup edge. The
empty-site zero is now the manager's own observation instead of an untouched field.
SiteAlarmLiveCacheServiceTests (:195, :216)
- Both sampled IsLive once after a fixed sleep. Now they poll continuously across three
linger windows on a new teardown observable (the mock client counts subscription
cancellations), so a stop at any instant inside the window is caught, and each closes
with a positive control proving that counter moves.
DataConnectionActorTests (:435)
- A Task.Delay(200) "settle" after an AwaitCondition on a connect counter. The counter
only proves ConnectAsync was CALLED; the actor is still Connecting until it processes
the piped result. Now waits for the Connected health publication that BecomeConnected
emits, so the subscribe under test provably runs against Connected rather than being
silently stashed. (The stash path keeps its own coverage in
WP6_ConnectingState_StashesSubscribeRequests.)
Negative controls (product broken in the working tree, old vs new test compared):
- Stuck-script watchdog disabled entirely: OLD test passed 2/2, NEW test failed. The
whole watchdog could be deleted and the old assertions were satisfied.
- Startup reporting a phantom instance count and taking longer than 500 ms: OLD passed
2/2, NEW failed (expected 0, actual 1).
- Linger cancellation disabled + the remaining-viewer guard dropped, so the stream is
torn down under a live viewer while IsLive keeps reporting true:
Resubscribe_Within_Linger OLD passed 2/2, NEW failed. (Last_Viewer_Leaving caught this
particular break in the old form too, via a later assertion — its strengthening is the
teardown observable and the positive control, not a false-pass fix.)
- DataConnectionActorTests :435 has no false-pass demo: the sleep guarded no assertion
there, and the change is determinism plus proving the claimed state.
Also fixes a latent race these timing changes exposed in
InstanceActorChildAttributeRaceTests: it resolved child actors single-shot immediately
after construction, but ResolveOne throws until the child's cell is published on the
parent AND the cell's Actor property stays null until the child's constructor has run on
its own mailbox thread. Neither is ordered against the test thread, so it failed both ways
under full-suite load (~1 run in 6 once the sleeps above stopped padding the schedule;
14/14 green on the unmodified tree, so it was latent, not new). Resolution now retries to
a bounded deadline; the assertion is unchanged.
Verification: SiteRuntime.Tests 8/8 green (605 tests), Communication.Tests 3/3 (717),
DataConnectionLayer.Tests 3/3 (319). Test-only change — no src/ modifications.
A site health report showed tagResolutionCounts 0/0 for a connection whose
dataConnectionTagQuality on the SAME report read good:2 — reproduced live on
the docker rig against site-a's rig-probe-1 instance (2 Good OPC UA tags on
"OPC PLC Simulator"), on the ACTIVE node's own report.
Not a standby-shadowing or keying/removal problem, and not a serialization
problem: both dictionaries live on one collector instance on one node, and the
CentralControlDtoMapper round-trip is correct in both directions. The actor's
own derived counts (TotalSubscribedTagCount / ResolvedTagCount) were right the
whole time — GetHealthReport replied 2/2.
Root cause: HandleSubscribeCompleted — the path that registers tags on a first,
healthy subscribe — mutated _instancesByTag and _subscriptionIds but never
pushed to ISiteHealthCollector. The only pushes were BecomeConnected's (fires
before any instance subscribes, so its 0/0 is legitimate), the batch apply
(reconnect re-subscribe / resolution probe) and unsubscribe (teardown). A
connection whose tags all resolve first time reaches none of those, and
ScheduleTagResolutionRetry cancels the retry timer outright when nothing is
unresolved — so the connect-time 0/0 was never superseded and stood forever.
The blind spot was therefore worst on a perfectly healthy connection, which is
exactly when the "device still booting / tag path unresolved" diagnostic should
read clean rather than empty. Any reconnect masked it (the batch path pushes),
which is why it presented only on a clean first deploy.
Fix: push the derived counts at the end of the first-subscribe apply, mirroring
the existing push site in HandleBatchSubscribeCompleted. No semantic change to
the null/no-data convention — the map entry already existed, its values were
simply stale zeros.
Tests: fail-before regression test pins the exact mechanism (NSubstitute records
a single UpdateTagResolution(name, 0, 0) pre-fix while the actor reports 2/2).
DCL 320, HealthMonitoring 97, SiteRuntime 605, Communication 717 all green;
slnx builds clean.
Docs: Component-DataConnectionLayer.md said the counts were "derived at report
time", which implies a pull that cannot go stale and papered over this bug.
Corrected to state they are pushed, name the four push sites, and record why a
healthy connection was the worst case.
STATIC tags (no further OnDataChange after advise) depend entirely on the
seed read. Pre-fix HandleSubscribe seeded only on Success && Value != null,
silently dropping a seed that raced the just-created advise (VT_EMPTY) — so a
static tag stayed Uncertain forever while the source read Good. ReSubscribeAll
did no seeding at all, so a static tag could not self-heal across reconnect.
- New SeedTagsAsync helper: per-tag ReadAsync (not a bulk read — some gateways
time out on large batches) with round-based bounded retry
(SeedReadMaxAttempts/SeedReadRetryDelay), logging any tag that never yields a
value (named — previously zero log trace).
- HandleSubscribe seed loop delegates to SeedTagsAsync.
- ReSubscribeAll re-seeds re-advised tags after reconnect via the
generation-guarded TagValueReceived path (fan-out keys off
_subscriptionsByInstance, preserved across reconnect).
Diagnosed live on wonder-app-vd03 2026-06-17 (see scadabridge-dcl-static-tag-false-bad).
Mechanism #2 (single transient-bad push) left as a follow-up.
DataConnectionActor seeded a tag's initial value by Tell-ing TagValueReceived
from HandleSubscribe's background task, which runs BEFORE HandleSubscribeCompleted
registers the instance's tags in _subscriptionsByInstance. HandleTagValueReceived's
fan-out then found no subscriber and dropped the value. A tag that soon gets a
data-change notification recovers, but a STATIC tag (e.g. an idle MES field that
never changes) was left Uncertain forever — the dropped seed was its only value.
Seeds now ride back on SubscribeCompleted and are delivered after registration,
reusing HandleTagValueReceived's generation guard, fan-out and quality accounting.
+1 regression test (DCL026).
Joseph Doherty