docs(localdb): live-gate the two Phase 1 follow-ups; pin LocalDb 0.1.3

The gate closed both follow-ups on the docker-dev rig and found a third defect
that offline tests could not have found first: it lived behind the one being
fixed. Once 0.1.2 made back-fill work, a rebuilt node could be observed writing
for the first time — and its writes went nowhere, because the healthy node kept
the old peer's seq watermark. Library 0.1.1 -> 0.1.2 -> 0.1.3 over the course
of the gate; both consumers now pinned to 0.1.3.

Check 4 (was PARTIAL, now PASS): with SQL stopped and site-a-2's LocalDb volume
destroyed, a-1 logged "Snapshot sent (as_of_seq 0, 4 rows)" — a line that could
not appear on 0.1.1 — and a-2 came back with a row_version dump byte-identical
to a-1's, carrying a-1's origin node ids and its ORIGINAL applied_at_utc. It
then booted from cache and served 17 ns=2 nodes, diff-identical to its peer,
from a configuration it never applied and could not have fetched.

Check 8 (PASS): two restarts with an unchanged artifact left the oplog at 7 and
the pointer timestamp frozen — the re-cache was skipped outright. Positive
control: a real config change still wrote, 7 -> 10 -> 13.

Also records one unrelated pre-existing finding the gate's SQL flapping
surfaced: a transient ConfigDb error during artifact load empties the served
address space while logging "rebuild becomes no-op". The cache layer correctly
refused to store the bad artifact and the node recovered on restart, but the
two logs disagree and it is the same class as the Phase 1 gate's check-3
defect. Untouched by this work; wants its own issue.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 02:14:00 -04:00
parent 232bff5df7
commit f587e7972e
4 changed files with 85 additions and 8 deletions
@@ -75,9 +75,12 @@ The replication library's passive endpoint verifies **no** authentication — th
- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
cached configuration back to it, with no new deploy and no central SQL. Two things are worth
knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.2** (on `0.1.1` a converged pair has pruned every
oplog row on ack, and the wiped node was never snapshotted — it stayed empty until the next
deploy), and it heals the *cache*, so the node regains boot-from-cache for future outages. A node
knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
applies a deploy.) A node
wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
peer replication configured self-heals only from central on its next successful apply.