The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.
Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
__localdb_row_version HLC and origin node id, and the standby logged zero
config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
errors.
Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.
Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.
Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.
Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.
Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
Re-scraped via a network-sharing sidecar: the metrics are present.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty