lmxopcua/docs/plans/2026-07-20-localdb-phase1-live-gate.md

LocalDb Phase 1 — live gate (docker-dev rig)

Task 16 of 2026-07-20-localdb-adoption-phase1.md. Evidence log for the 8 live-rig checks. Rig: local docker-dev/docker-compose.yml (6 host nodes, one shared SQL). site-a pair replicates; central + site-b are default-OFF.

Safety rules followed: all DB inspection via docker cp of the db/-wal/-shm triplet out of the container, querying the copy — never host sqlite3 on the live WAL file. Metrics via a curlimages/curl sidecar with --network container:<node> (aspnet:10.0 has no curl).

Rig build/up

Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):

1. NU1101 — packageSourceMapping gap (4b2f0e6e). ZB.MOM.WW.LocalDb* was not mapped to the dohertj2-gitea feed, so any clean restore (Docker image build, CI, fresh clone) failed. Local dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified with a temp-packages-dir restore.
2. ASPNETCORE_HTTP_PORTS re-bind gap (ce9fa07f). The Kestrel re-bind read only urls/ASPNETCORE_URLS; the aspnet:8.0+ base image sets ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the container default instead. A driver node that never sets URLS fell through to localhost:5000, silently moving its health/metrics surface to loopback. Central nodes were fine (they set ASPNETCORE_URLS=9000 explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS → 5000; 4 new unit tests. After the fix, driver nodes bind [::]:8080 + [::]:9001 (site-a) / [::]:8080 only (site-b).

Rig: 6 nodes rebuilt from branch feat/localdb-phase1 @ ce9fa07f, all up.

Checks

#	Check	Result	Evidence
1	Rig up, site-a pair healthy, /metrics shows localdb_ series	✅ PASS	Both site-a nodes /healthz 200 Healthy; /metrics shows localdb_oplog_depth + localdb_sync_reconnects_total on meter ZB.MOM.WW.LocalDb.Replication (allowlist works); site-a-2 served /localdb_sync.v1.LocalDbSync/Sync with routing match_status=success (interceptor passed, session established); both oplogs depth 0
2	Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin	✅ PASS	Deploy 6e687451… (rev efb04c79…) via POST :9200/api/deployments → both site-a-1 and site-a-2 deployment_pointer = SITE-A / 6e687451… / efb04c79… / sha EFB04C79… (identical); artifact 1 chunk each; pointer __localdb_row_version identical on both: hlc 116955620225646592, origin node_id a0576df6-…, tombstone 0 — one row replicated, not two derivations; both oplogs depth 0
3	Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal	✅ PASS (found + fixed a defect)	With SQL stopped, site-a-2 restart logged RUNNING FROM CACHE — … booted deployment 6e687451…; address space materialised from the cached blob: AddressSpaceApplier: applied plan (added=18) / OpcUaPublish: applied rebuild (added=18). Browse confirms 16 nodes on site-a-2 = 16 on site-a-1. Defect the gate caught (9137cb41): pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served 1 node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to RebuildAddressSpace; regression test added
4	Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot	⚠️ PARTIAL — 1 defect fixed; the limitation is since CLOSED (LocalDb 0.1.2 — see Summary)	Wiped a-2's LocalDb volume, restarted with SQL down: sync session re-established (a-2 served a Sync call) but a-2's cache stayed 0/0/0 and a-1's oplog was 0 — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and no snapshot-resync fired. ⇒ Limitation: pure replication does not back-fill a fully-wiped, already-converged node (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a defect (c6a9f93a): recovery via RestoreApplied never wrote the cache (only fresh ApplyAndAck did), so a wiped node stayed cache-less. Fixed: RestoreApplied now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: restored served state … on bootstrap → cache repopulated SITE-A / 6e687451, chunks=1. Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.
5	Transport kill: pause a-2, oplog rises; unpause, drains to 0	✅ PASS	docker pause on a-2 severed the sync transport; during the window a-2 accumulated oplog=2 unacked local writes (its re-cache) while a-1 held 0. docker unpause → a-2's oplog drained to 0 within 5s, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario WritesWhileTransportDown_SurviveRejoin (proven red without RegisterReplicated)
6	Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes	✅ PASS	Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (475df87a, 962962dc); the oldest 6e687451's chunks are 0 on both (pruned); 1 artifact tombstone on both — the prune replicated as a tombstone, not left as a live chunk
7	Both-nodes-together restart: clean rejoin, identical counts, no I/O errors	✅ PASS	Restarted site-a-1 + site-a-2 together: 0 disk I/O error/SQLITE_IOERR/corrupt lines in either log; both re-cached and converged to identical counts (ptr 962962dc…, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart
8	site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001	✅ PASS (+1 finding, since CLOSED by the skip-if-unchanged cache write — see Summary)	site-b-1 cache works locally (SITE-B / 6e687451, chunks=1, from its own apply); listeners [::]:8080 only — no 9001; port 9001 not bound (/proc/net/tcp); /healthz 200; no sync/reconnect metric series. Finding (follow-up, not a blocker): localdb_oplog_depth=5 — because OnReady registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library MaxOplogRows cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up

Summary

8/8 checks pass (check 4 with a documented limitation). The gate exercised the real image built from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real docker cp-triplet DB inspection — and caught four real defects that every offline test had missed, each now fixed on-branch with a regression test:

1. NU1101 packageSourceMapping gap (4b2f0e6e) — ZB.MOM.WW.LocalDb* unmapped to the gitea feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
2. ASPNETCORE_HTTP_PORTS re-bind gap (ce9fa07f) — the sync-listener re-bind ignored the aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
3. Empty address space on boot-from-cache (9137cb41) — the rebuild re-read the artifact from the down ConfigDb and no-op'd, so a cache-booted node logged success but served clients 1 node vs the healthy peer's 16. The single most important find: boot-from-cache was half-working.
4. Cache not repopulated on RestoreApplied (c6a9f93a) — a wiped/fresh-volume node recovered its served state from central but stayed cache-less, unable to boot-from-cache on the next outage.

Two documented limitations (follow-ups, not blockers) — both since CLOSED:

• No replication back-fill of a fully-wiped node (check 4): a peer's already-acked cache rows are pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is not healed by its peer — it self-heals from central on next boot instead. → FIXED in ZB.MOM.WW.LocalDb 0.1.2 + 0.1.3 (scadaproj cad3bcb + 3b7489a), live-gated in 2026-07-21-localdb-followups-live-gate.md. The diagnosis in this doc was slightly off: the cap was not the gate. ComputeSnapshotRequiredAsync measured the peer's gap against the oldest surviving oplog row and read an empty oplog as "no gap possible" — and an empty oplog is the steady state of a converged pair, since ack-pruning deletes everything the peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With an empty oplog the gap is now measured against last_acked_seq. Covered at both levels: SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot in the library, and LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy here — the latter verified RED against the pinned 0.1.1 and green on 0.1.2. 0.1.3 closes a second, deeper half that only became reachable once back-fill worked: the rebuilt node's OWN writes were silently dropped, because last_applied_remote_seq is a watermark in the peer's seq space and a rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could not have, since the scenario had no test while it was still a documented limitation.
• Oplog growth on default-OFF nodes (check 8): OnReady registers replication unconditionally, so a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap; a periodic pruner or a skip-if-unchanged cache write would close it. → FIXED at the source: LocalDbDeploymentArtifactCache.StoreAsync now skips entirely when the pointer already names this deployment/revision, the SHA matches the bytes, and the expected chunk count is present — so re-caching an artifact the node already holds (every restart's boot-from-cache, every RestoreApplied) writes nothing and mints no oplog rows. Two guard tests pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over missing chunks), both verified RED against a naive pointer-only skip. Also correcting this row's bound: the growth was never headed for the 1M cap. AddZbLocalDbReplication is registered unconditionally, so MaintenanceBackgroundService runs on default-OFF nodes too and the 7-day MaxOplogAge cap prunes — the true bound was ~7 days of writes, not 1M rows.

Merged to master as c957db52 (2026-07-21) and pushed. Both follow-ups closed on fix/localdb-phase1-followups (2026-07-21) and live-gated — see 2026-07-21-localdb-followups-live-gate.md, which found a third defect in the process.