From 41c36064dd6de6eb73966d5669bcd6b00d4d4b8c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 23:10:44 -0400 Subject: [PATCH] docs(localdb): phase-1 live-gate evidence (8/8 pass; 4 defects fixed, 2 limitations) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the docker-dev live gate: 8/8 checks pass (check 4 with a documented limitation). The gate caught four real defects every offline test missed — the NU1101 packageSourceMapping gap, the ASPNETCORE_HTTP_PORTS re-bind, the empty address space on boot-from-cache, and the cache not repopulating on RestoreApplied — all fixed on-branch with regression tests. Two limitations documented as follow-ups: no replication back-fill of a fully-wiped node, and oplog growth on default-OFF nodes. --- .../2026-07-20-localdb-phase1-live-gate.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/plans/2026-07-20-localdb-phase1-live-gate.md diff --git a/docs/plans/2026-07-20-localdb-phase1-live-gate.md b/docs/plans/2026-07-20-localdb-phase1-live-gate.md new file mode 100644 index 00000000..379ca4be --- /dev/null +++ b/docs/plans/2026-07-20-localdb-phase1-live-gate.md @@ -0,0 +1,68 @@ +# LocalDb Phase 1 — live gate (docker-dev rig) + +> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks. +> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates; +> central + site-b are default-OFF. +> +> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out +> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a +> `curlimages/curl` sidecar with `--network container:` (`aspnet:10.0` has no curl). + +## Rig build/up + +Two real defects surfaced by the rebuild (both fixed on-branch, re-verified): + +1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the + `dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local + dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified + with a temp-packages-dir restore. +2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only + `urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all + interfaces) as the container default instead. A driver node that never sets `URLS` fell through to + `localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine + (they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS → + 5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) / + `[::]:8080` only (site-b). + +Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up. + +## Checks + +| # | Check | Result | Evidence | +|---|---|---|---| +| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 | +| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 | +| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added | +| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 limitation documented, 1 defect fixed | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** | +| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) | +| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk | +| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart | +| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up | + +## Summary + +**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built +from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real +`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had +missed**, each now fixed on-branch with a regression test: + +1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea + feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache. +2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the + aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000. +3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the + down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the + healthy peer's 16**. The single most important find: boot-from-cache was half-working. +4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its + served state from central but stayed cache-less, unable to boot-from-cache on the next outage. + +**Two documented limitations (follow-ups, not blockers):** + +- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are + pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is + not healed by its peer — it self-heals from central on next boot instead. +- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so + a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap; + a periodic pruner or a skip-if-unchanged cache write would close it. + +Not merged, per the plan. The two limitations are recorded as Phase 1 follow-ups.