From f587e7972eeb63de08bf4d74226f2a3def042950 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 21 Jul 2026 02:14:00 -0400 Subject: [PATCH] docs(localdb): live-gate the two Phase 1 follow-ups; pin LocalDb 0.1.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gate closed both follow-ups on the docker-dev rig and found a third defect that offline tests could not have found first: it lived behind the one being fixed. Once 0.1.2 made back-fill work, a rebuilt node could be observed writing for the first time — and its writes went nowhere, because the healthy node kept the old peer's seq watermark. Library 0.1.1 -> 0.1.2 -> 0.1.3 over the course of the gate; both consumers now pinned to 0.1.3. Check 4 (was PARTIAL, now PASS): with SQL stopped and site-a-2's LocalDb volume destroyed, a-1 logged "Snapshot sent (as_of_seq 0, 4 rows)" — a line that could not appear on 0.1.1 — and a-2 came back with a row_version dump byte-identical to a-1's, carrying a-1's origin node ids and its ORIGINAL applied_at_utc. It then booted from cache and served 17 ns=2 nodes, diff-identical to its peer, from a configuration it never applied and could not have fetched. Check 8 (PASS): two restarts with an unchanged artifact left the oplog at 7 and the pointer timestamp frozen — the re-cache was skipped outright. Positive control: a real config change still wrote, 7 -> 10 -> 13. Also records one unrelated pre-existing finding the gate's SQL flapping surfaced: a transient ConfigDb error during artifact load empties the served address space while logging "rebuild becomes no-op". The cache layer correctly refused to store the bad artifact and the node recovered on restart, but the two logs disagree and it is the same class as the Phase 1 gate's check-3 defect. Untouched by this work; wants its own issue. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- Directory.Packages.props | 4 +- .../2026-07-20-localdb-pair-replication.md | 9 ++- .../2026-07-20-localdb-phase1-live-gate.md | 12 +++- .../2026-07-21-localdb-followups-live-gate.md | 68 +++++++++++++++++++ 4 files changed, 85 insertions(+), 8 deletions(-) create mode 100644 docs/plans/2026-07-21-localdb-followups-live-gate.md diff --git a/Directory.Packages.props b/Directory.Packages.props index c569cf82..82221614 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -127,8 +127,8 @@ package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the CVE-2025-6965-fixed native binary without needing the surgical pin below. --> - - + + diff --git a/docs/operations/2026-07-20-localdb-pair-replication.md b/docs/operations/2026-07-20-localdb-pair-replication.md index c8d979cb..78422482 100644 --- a/docs/operations/2026-07-20-localdb-pair-replication.md +++ b/docs/operations/2026-07-20-localdb-pair-replication.md @@ -75,9 +75,12 @@ The replication library's passive endpoint verifies **no** authentication — th - **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume, a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the cached configuration back to it, with no new deploy and no central SQL. Two things are worth - knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.2** (on `0.1.1` a converged pair has pruned every - oplog row on ack, and the wiped node was never snapshotted — it stayed empty until the next - deploy), and it heals the *cache*, so the node regains boot-from-cache for future outages. A node + knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains + boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack + and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is + back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the + peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node + applies a deploy.) A node wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no** peer replication configured self-heals only from central on its next successful apply. diff --git a/docs/plans/2026-07-20-localdb-phase1-live-gate.md b/docs/plans/2026-07-20-localdb-phase1-live-gate.md index 2008438d..6bd03ec3 100644 --- a/docs/plans/2026-07-20-localdb-phase1-live-gate.md +++ b/docs/plans/2026-07-20-localdb-phase1-live-gate.md @@ -61,7 +61,8 @@ missed**, each now fixed on-branch with a regression test: - **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is not healed by its peer — it self-heals from central on next boot instead. - → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2** (scadaproj `cad3bcb`). The diagnosis in this doc was + → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in + `2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the @@ -69,7 +70,11 @@ missed**, each now fixed on-branch with a regression test: an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels: `SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library, and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here — - the latter verified RED against the pinned 0.1.1 and green on 0.1.2. + the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second, + deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were + silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a + rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could + not have, since the scenario had no test while it was still a documented limitation. - **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap; a periodic pruner or a skip-if-unchanged cache write would close it. @@ -84,4 +89,5 @@ missed**, each now fixed on-branch with a regression test: the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows. Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on -`fix/localdb-phase1-followups` (2026-07-21). +`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see +`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process. diff --git a/docs/plans/2026-07-21-localdb-followups-live-gate.md b/docs/plans/2026-07-21-localdb-followups-live-gate.md new file mode 100644 index 00000000..ae028c2e --- /dev/null +++ b/docs/plans/2026-07-21-localdb-followups-live-gate.md @@ -0,0 +1,68 @@ +# LocalDb Phase 1 follow-ups — live gate (docker-dev rig) + +> Closes the two limitations left open by `2026-07-20-localdb-phase1-live-gate.md` (checks 4 and 8). +> Rig: local `docker-dev/docker-compose.yml`, 6 nodes rebuilt from `fix/localdb-phase1-followups`. +> site-a pair replicates; central + site-b stay default-OFF. +> +> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the +> `db`/`-wal`/`-shm` triplet, querying the copy — never host `sqlite3` on the live WAL file. + +## Result + +**Both follow-ups closed, and the gate found a third defect** — one that was *unreachable* until the +first fix landed. Library `ZB.MOM.WW.LocalDb` went `0.1.1` → `0.1.2` → `0.1.3` in the course of this +gate; both consumers (OtOpcUa, ScadaBridge) are pinned to `0.1.3`. + +| # | Check | Result | Evidence | +|---|---|---|---| +| 8 | Oplog stops growing on a default-OFF node | ✅ PASS | site-b-1 restarted twice with an unchanged cached artifact: `oplog_depth` **7 → 7 → 7**, `deployment_pointer.applied_at_utc` frozen at `05:34:58` across both — the re-cache was skipped outright. Repeated on the `0.1.3` build after a real deploy: **13 → 13**. Pre-fix this grew ~2 rows per restart forever, because a default-OFF node has no peer to ack and prune them. | +| 8b | …but a genuinely new deployment still writes (positive control) | ✅ PASS | A real config change (renamed tag `SITEA-tag`) produced rev `822bdf34`, then `002c16b3`. site-b-1's oplog **7 → 10 → 13** and its pointer advanced each time. The skip suppresses *identical* re-writes only; it is not a dead cache-write path. | +| 4 | A wiped node is back-filled by its peer, no new deploy, central down | ✅ PASS (was ⚠️ PARTIAL) | Precondition asserted first — site-a-1's oplog **0**, `needs_snapshot 0`: the exact state that used to make back-fill impossible. Then SQL stopped, site-a-2's LocalDb **volume destroyed**, container recreated. a-1 logged `Snapshot sent (as_of_seq 0, 4 rows)` — a line that could not appear on `0.1.1`. a-2 came back with pointer `SITE-A / 475df87a @ 03:09:12` (a-1's **original** timestamp, not a re-derived one), chunks 2, and a `__localdb_row_version` dump **byte-identical to a-1's**, carrying origin node ids `6812ca9c` and `935659a5` — the rebuilt node's own id appears nowhere, so the rows were replicated, not recreated. | +| 4b | …and the back-filled node then boots from cache | ✅ PASS | With SQL still down, site-a-2 logged `RUNNING FROM CACHE — … booted deployment 475df87a… cached at 2026-07-21T03:09:12`, `raw subtree materialised (containers=16, tags=1)`. OPC UA browse of both nodes: **17 ns=2 nodes each, diff-identical** — the rebuilt node serves exactly what its healthy peer does, from a configuration it never applied itself and could not have fetched. | +| 4c | The rebuilt node's OWN writes reach its peer | ✅ PASS **after a third fix** (see below) | On `0.1.2` this failed: a-1 held `last_applied_remote = 10` while the rebuilt a-2's oplog was seqs **1,2,3** and its `last_acked` stayed **0** — never accepted. On `0.1.3`, a-1 logs `Replication peer identity changed (ebb10854 -> c85d8b1d): the peer was rebuilt … Resetting the inbound watermark`, and after the next deploy a-1 shows `last_applied_remote = 3` / a-2 shows `last_acked = 3` with a drained oplog and identical row_versions. | + +## The third defect (the reason this gate was worth running) + +`0.1.2` made back-fill work, which for the first time let a rebuilt node be observed *writing*. It +turned out its writes went nowhere. + +`last_applied_remote_seq` is a watermark in the **peer's** seq space, and a rebuilt peer is a new +database whose oplog numbers from 1. Two independent bugs stacked, and fixing either alone leaves the +data stuck: + +1. The healthy node kept advertising the **old** peer's watermark. The *sending* side seeds its pump + directly from that number (`_sentThruSeq`), so the rebuilt node skipped its entire oplog. + → the watermark (and the observed peer clock) is now reset when the peer's node id changes. + `last_acked_seq` is deliberately **not** reset: it describes our own oplog's pruned horizon and is + exactly what `0.1.2` keys the snapshot decision off — clearing it would switch back-fill back off. +2. A peer claiming to have applied more of our stream than we have ever produced can only be + remembering a previous incarnation of *us*. → that claim is now rejected in favour of pumping from + the start; LWW makes the re-send harmless. This mirrors the clamp `RecordPeerAckAsync` already + applies to acks. + +Both were reproduced offline as a RED test before fixing +(`SnapshotResyncTests.RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark`). + +**Why offline tests could not have found it first:** the wiped-peer scenario had no test because the +scenario was believed impossible to recover from — it was a *documented limitation*, not a bug. The +gate is what turned the limitation into a reproducible case, and the second defect only existed +behind the first. + +## Unrelated finding (NOT a follow-up of this work — pre-existing, worth its own issue) + +While SQL was being stopped and started to drive check 4, site-b-1 took a deploy whose artifact load +hit a transient ConfigDb error, and **emptied its served address space**: + +``` +[06:11:20 ERR] An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'. +[06:11:20 WRN] OpcUaPublish: failed to load artifact for deployment 43d54397…; rebuild becomes no-op +[06:11:20 INF] AddressSpaceApplier: applied plan (kind=PureRemove, added=0, removed=16, rebuild=True) +``` + +The log says "rebuild becomes no-op" and the applier then removed all 16 nodes — those two disagree. +The cache layer behaved correctly and refused to store the bad artifact ("not caching deployment … its +artifact was empty or could not be loaded, so it is not a usable last-known-good configuration"), and +the node recovered fully on restart (`containers=16`). But a transient ConfigDb blip emptying the +served address space instead of holding last-known-good is the same failure *class* as the Phase 1 +gate's check-3 defect. Untouched by this work — site-b-1 has replication off entirely — and induced by +this gate's own SQL flapping.