The driver-side half of the same mistake the address-space fix corrected.
ReconcileDrivers and PushDesiredSubscriptionsFromArtifact both read the
artifact as `...FirstOrDefault() ?? Array.Empty<byte>()`. Their THROW paths
already returned early, but empty bytes flowed onwards as a real answer: zero
driver specs, so DriverSpawnPlanner planned every running child for StopChild,
and then an empty desired set dropped each surviving driver's live
subscription handle. A node lost its entire field I/O and still ACKed Applied.
Same rule as the address space: no bytes is no answer, not "a configuration
with no drivers". Nothing legitimate produces a zero-length blob — deleting
the last driver still deploys a JSON document with empty arrays. Both sites
now skip and keep what is running. The two guards are independent because
PushDesiredSubscriptions does its OWN ConfigDb read, so the row can go missing
between them.
Also corrects the ReconcileDrivers doc comment, which claimed an empty blob
made it "effectively a no-op" — with children running it was the opposite.
Tests: DriverHostActorUnreadableArtifactTests, RED-first (verified failing —
after the empty dispatch the driver list was empty). A zero-length ArtifactBlob
reproduces byte-for-byte what the missing-row case delivers, so the race does
not have to be constructed.
Two controls, both load-bearing:
- a READABLE driverless artifact still stops the driver, so the guard keys on
"the read gave us nothing", not on "fewer drivers than before";
- dropping a driver's LAST TAG still clears its subscription. This one was
added after the absence assertion was caught passing on a race: the
unsubscribe is an async self-tell, so with the second guard deleted the test
still went green. The control observes that same unsubscribe arriving well
inside the settle window, which is what makes the absence meaningful — with
the guard deleted the suite now correctly goes RED.
SubscribableStubDriver gains an UnsubscribeCount for that observation.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A transient ConfigDb error while loading a deployment's artifact emptied the
served address space. LoadArtifact caught the exception, logged "rebuild
becomes no-op" and returned zero bytes; those parsed to an EMPTY composition,
which the planner diffed against the live one as a PureRemove and the applier
faithfully executed — 16 nodes gone, while the log claimed nothing happened.
An artifact we could not obtain is not an empty configuration. Nothing
legitimate produces a zero-length blob (an operator who really deletes
everything still deploys a JSON document with empty arrays), so "no bytes" can
only mean "no answer". HandleRebuild now abandons the rebuild on one, leaving
both the materialised nodes and _lastApplied intact so the retry diffs against
what is actually being served. The loader's own log line is now true.
The two cases are logged differently: warn when there is a live address space
being protected, debug when nothing has been deployed to this node yet.
Covered by a RED-first actor test (verified failing: the load error tore down
eq-2's subtree), plus a positive control proving a READABLE artifact that
genuinely drops an equipment still removes it — the guard suppresses teardown
only when the read failed.
Found on the LocalDb Phase 1 follow-up live gate, which induced it by flapping
SQL; that gate's log is the production evidence for the seam.
Closes#485
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The gate closed both follow-ups on the docker-dev rig and found a third defect
that offline tests could not have found first: it lived behind the one being
fixed. Once 0.1.2 made back-fill work, a rebuilt node could be observed writing
for the first time — and its writes went nowhere, because the healthy node kept
the old peer's seq watermark. Library 0.1.1 -> 0.1.2 -> 0.1.3 over the course
of the gate; both consumers now pinned to 0.1.3.
Check 4 (was PARTIAL, now PASS): with SQL stopped and site-a-2's LocalDb volume
destroyed, a-1 logged "Snapshot sent (as_of_seq 0, 4 rows)" — a line that could
not appear on 0.1.1 — and a-2 came back with a row_version dump byte-identical
to a-1's, carrying a-1's origin node ids and its ORIGINAL applied_at_utc. It
then booted from cache and served 17 ns=2 nodes, diff-identical to its peer,
from a configuration it never applied and could not have fetched.
Check 8 (PASS): two restarts with an unchanged artifact left the oplog at 7 and
the pointer timestamp frozen — the re-cache was skipped outright. Positive
control: a real config change still wrote, 7 -> 10 -> 13.
Also records one unrelated pre-existing finding the gate's SQL flapping
surfaced: a transient ConfigDb error during artifact load empties the served
address space while logging "rebuild becomes no-op". The cache layer correctly
refused to store the bad artifact and the node recovered on restart, but the
two logs disagree and it is the same class as the Phase 1 gate's check-3
defect. Untouched by this work; wants its own issue.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty