# LocalDb Phase 1 — live gate (docker-dev rig)

> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
> central + site-b are default-OFF.
>
> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).

## Rig build/up

Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):

1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
   `dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
   dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
   with a temp-packages-dir restore.
2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
   `urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
   interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
   `localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
   (they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
   5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
   `[::]:8080` only (site-b).

Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.

## Checks

| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |

## Summary

**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
missed**, each now fixed on-branch with a regression test:

1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
   feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
   aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
   down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
   healthy peer's 16**. The single most important find: boot-from-cache was half-working.
4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
   served state from central but stayed cache-less, unable to boot-from-cache on the next outage.

**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**

- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
  pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
  not healed by its peer — it self-heals from central on next boot instead.
  → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in
  `2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was
  slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
  against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
  empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
  peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
  an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
  `SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
  and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
  the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second,
  deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were
  silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a
  rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could
  not have, since the scenario had no test while it was still a documented limitation.
- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
  a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
  a periodic pruner or a skip-if-unchanged cache write would close it.
  → **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
  the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
  chunk count is present — so re-caching an artifact the node already holds (every restart's
  boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
  pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
  missing chunks), both verified RED against a naive pointer-only skip.
  Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
  is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
  the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.

Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see
`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process.
