docs(localdb): phase-1 live-gate evidence (8/8 pass; 4 defects fixed, 2 limitations)
Records the docker-dev live gate: 8/8 checks pass (check 4 with a documented limitation). The gate caught four real defects every offline test missed — the NU1101 packageSourceMapping gap, the ASPNETCORE_HTTP_PORTS re-bind, the empty address space on boot-from-cache, and the cache not repopulating on RestoreApplied — all fixed on-branch with regression tests. Two limitations documented as follow-ups: no replication back-fill of a fully-wiped node, and oplog growth on default-OFF nodes.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,68 @@
|
||||
# LocalDb Phase 1 — live gate (docker-dev rig)
|
||||
|
||||
> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
|
||||
> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
|
||||
> central + site-b are default-OFF.
|
||||
>
|
||||
> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
|
||||
> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
|
||||
> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).
|
||||
|
||||
## Rig build/up
|
||||
|
||||
Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):
|
||||
|
||||
1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
|
||||
`dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
|
||||
dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
|
||||
with a temp-packages-dir restore.
|
||||
2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
|
||||
`urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
|
||||
interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
|
||||
`localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
|
||||
(they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
|
||||
5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
|
||||
`[::]:8080` only (site-b).
|
||||
|
||||
Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
|
||||
|
||||
## Checks
|
||||
|
||||
| # | Check | Result | Evidence |
|
||||
|---|---|---|---|
|
||||
| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
|
||||
| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
|
||||
| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
|
||||
| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 limitation documented, 1 defect fixed | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
|
||||
| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
|
||||
| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
|
||||
| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
|
||||
| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
|
||||
|
||||
## Summary
|
||||
|
||||
**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
|
||||
from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
|
||||
`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
|
||||
missed**, each now fixed on-branch with a regression test:
|
||||
|
||||
1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
|
||||
feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
|
||||
2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
|
||||
aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
|
||||
3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
|
||||
down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
|
||||
healthy peer's 16**. The single most important find: boot-from-cache was half-working.
|
||||
4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
|
||||
served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
|
||||
|
||||
**Two documented limitations (follow-ups, not blockers):**
|
||||
|
||||
- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
|
||||
pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
|
||||
not healed by its peer — it self-heals from central on next boot instead.
|
||||
- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
|
||||
a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
|
||||
a periodic pruner or a skip-if-unchanged cache write would close it.
|
||||
|
||||
Not merged, per the plan. The two limitations are recorded as Phase 1 follow-ups.
|
||||
Reference in New Issue
Block a user