ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md

LocalDb Phase 2 — live gate evidence (2026-07-20)

Gate for Task 20. Status: all 10 checks captured and passing, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's third table was empty).

Build under test: feat/localdb-phase2 @ 166f07fa, LocalDb 0.1.1 — confirmed live, not assumed: the metrics scrape reports otel_scope_name="ZB.MOM.WW.LocalDb.Replication" otel_scope_version="0.1.1". Rig redeployed with docker/deploy.sh, exit 0, all 8 nodes up.

Method notes — two corrections to the plan

1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs. macOS↔container locks do not cross virtiofs, so a host sqlite3 open recovers the WAL out from under the container and triggers a permanent disk I/O error (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of the 2026-07-20 incident. Every read below used a helper that copies the db/-wal/-shm triplet and queries the copy; cp is a pure byte read with no SQLite involvement.

2. docker exec … curl cannot scrape the metrics — the aspnet:10.0 image has no curl, and with 2>/dev/null the failure is silent and looks exactly like "no metrics exported". An earlier pass nearly recorded that as a finding. Use a network-sharing sidecar:

docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics

3. The probe harness must be delivered by deploy artifacts, not instance deploy. External systems reach a site only through ArtifactDeploymentService (FetchGlobalArtifactsAsync → GetAllExternalSystemsAsync), which the per-instance deploy path does not invoke. A script that names an external system that never arrived fails silently — no rows, no error.

---

1. Migration ran — PASS

Node	.migrated markers	legacy sf_messages	consolidated sf_messages
site-a-node-a	scadabridge.db.migrated, store-and-forward.db.migrated	11804	11804
site-a-node-b	both	11804	11804

Both nodes migrated their own legacy files independently and the identical-content rows then LWW-converged — expected per the plan, not an anomaly.

Caveat: the migration itself ran on an earlier deploy of this build (markers dated 05:04), so this is after-the-fact evidence. deployed_configurations was 0 in both legacy files, so that table's migration path is untested here; check 3 exercises it fresh instead.

2. No SMTP/notification rows migrated — PASS

smtp_configurations and notification_lists are 0 on both nodes.

That count alone is vacuous — the legacy source also held 0 SMTP rows, so a broken exclusion would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for exactly these ten tables on both nodes —

OperationTracking, data_connection_definitions, database_connections,
deployed_configurations, external_systems, native_alarm_state,
sf_messages, shared_scripts, site_events, static_attribute_overrides

— and smtp_configurations / notification_lists have no triggers at all, so no replication channel exists for them regardless of content. That is the property the security decision rests on.

3. Config converges byte-identical — PASS

Deployed gate-sensor-1 (template 3) to site-a. Both nodes:

• whole-row md5 identical: 0ec0f3cf790bfd65c45edf6c4afae50f
• deployment_id 20b81607…, revision_hash sha256:253e1a56…, deployed_at 2026-07-20T09:06:11.2087221+00:00
• __localdb_row_version: HLC 116951506695487488, origin node cf61c688-a57f-4c5c-ba76-feff6e5679fe, is_tombstone 0 — identical on both

Identical HLC and origin id is the strong form: node B holds the row A wrote, not an independently-derived copy of it.

4. The standby made no config fetch — PASS

Deploy window from 2026-07-20T09:05:52Z:

• site-a-a (active, deploy target): 1 fetch — Fetching config for deployment 20b81607… (notify-and-fetch). Legitimate: the active node pulling its own artifact.
• site-a-b (standby): 0 fetches.

Under the old architecture SiteReplicationActor would have notified the standby and it would have fetched independently. It now receives the config purely by CDC — corroborated from the other direction by check 3's shared origin id.

5. Store-and-forward converges — PASS

Harness: external system GateDeadTarget → http://127.0.0.1:9 with method Ping, plus an interval script calling ExternalSystem.CachedCall (only CachedCall buffers; plain ExternalSystem.Call does not). Delivered via deploy artifacts.

• Buffered rows appear on both nodes with an identical rowset md5 a484e369f6e0f00e7debff9b64464566.
• __localdb_row_version for sf_messages: 2509 rows from 1 origin on both nodes.
• The Pending→Parked transition of the pre-existing backlog also replicated identically (10280 Pending / 1524 Parked on both), so UPDATE replicates, not just INSERT.

6. Site failover — PASS

Stopped the active node site-a-a at 09:18:37. Ten seconds later site-a-b logged InstanceActor started for gate-churn-1 and ScriptActor GateBufferProbe started — the standby picked up the workload and kept buffering. While partitioned, node B's oplog rose to 4 unacked entries.

Restarted site-a-a at 09:18:57. After rejoin:

	node-a	node-b
probe rows	48	48
total sf_messages	11852	11852
oplog	0	0
dead letters	0	0
duplicate ids	0	0
rowset md5	54ded633c28134222bf034f3d0cec680	54ded633c28134222bf034f3d0cec680

Buffered messages survived the flip, the backlog drained on rejoin, and zero duplicate ids on either node is the exactly-once evidence.

7. Cascade delete — PASS

Seeded gate-sensor-1 with a static_attribute_overrides row (via an Instance.SetAttribute script — the attribute-override deploy path writes into the config JSON, not this table), confirmed it replicated, then deleted the instance.

	pre-delete	post-delete
deployed_configurations	1 / 1	0 / 0
static_attribute_overrides	1 / 1	0 / 0
native_alarm_state	0 / 0	0 / 0

Crucially, both nodes hold explicit tombstones — deployed_configurations=1, static_attribute_overrides=1 in __localdb_row_version WHERE is_tombstone=1. The rows are not merely absent on B; B applied the deletes.

Caveat: native_alarm_state was empty for this instance, so the third leg of the cascade is untested live. It is covered offline by LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables, which was verified non-vacuous.

8. Dead letters, oplog, metrics — PASS

• __localdb_oplog = 0 and __localdb_dead_letter = 0 on both nodes at every sampling point.
• localdb_oplog_depth 0; localdb_sync_applied_total 11808 (a) / 224 (b); localdb_sync_reconnects_total 1.

9. Both nodes stopped and started together (D5) — PASS

Stopped site-a-a and site-a-b together, started them together.

	node-a	node-b
sf_messages	11845	11845
probe rows	41	41
oplog	0	0
dead letters	0	0

Zero occurrences of disk I/O error / database disk image / SQLITE_IOERR / corrupt in either node's log after restart. Clean rejoin.

10. Drain under churn — PASS

Sampled every 8 s under sustained probe load:

t=8s   oplog_a=0 oplog_b=0  probe_a=23 probe_b=23
t=16s  oplog_a=0 oplog_b=0  probe_a=25 probe_b=25
t=24s  oplog_a=0 oplog_b=0  probe_a=26 probe_b=26
t=32s  oplog_a=0 oplog_b=0  probe_a=28 probe_b=28
t=40s  oplog_a=0 oplog_b=0  probe_a=30 probe_b=30
t=48s  oplog_a=0 oplog_b=0  probe_a=31 probe_b=31
t=56s  oplog_a=0 oplog_b=0  probe_a=33 probe_b=33
t=64s  oplog_a=0 oplog_b=0  probe_a=35 probe_b=35

Writes climb steadily and the two nodes stay in lockstep at every sample, with no oplog accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the plan cared about (monotonic growth that never drains) does not occur.

On its own this sampling is weak — an 8 s interval is far coarser than the flush interval, so a transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer was down and back to 0 after rejoin.

---

Observations outside the gate

• A central external-system delete does not remove the row from sites. After deleting GateDeadTarget centrally and re-running deploy artifacts, both site nodes still carry it. Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans. Update 2026-08-07 (truth sweep): RESOLVED by 2d03f2d5 ("fix(site-runtime): reconcile artifact deletions on apply", 2026-08-01) — the artifact apply is now upsert-then-reconcile: after storing the incoming complete set, SiteStorageService.DeleteRowsExceptAsync removes any stored row absent from it, per artifact table, explicitly including external systems (DeleteExternalSystemsExceptAsync on external_systems) plus shared scripts, database connections and data-connection definitions; a null (not-shipped) list still touches nothing, and runtime cleanup rides along (shared-script unregistration, DCL eviction of the removed connection).
• Deleting an instance orphans its buffered messages. Removing the soakgen-* instances left their 11,804 sf_messages with no tracking snapshot, producing a continuous Cached-telemetry drain: no tracking snapshot for … warning flood. Also pre-existing, and worth a cleanup path. RESOLVED by 62cddcfa ("let an unresolvable cached row leave the drain queue", merged 663d03e8) — an unresolvable row is now dropped from the drain queue instead of being retried forever, so the hot loop and the warning flood are gone.

Rig state as left

• Owed cleanup done: SoakGenerator (2021) and soakgen-1..4 deleted; ExternalSystemDefinitions id 1 restored to http://scadabridge-restapi:5200 and confirmed propagated to both site nodes.
• Gate harness removed: gate-sensor-1, gate-churn-1, template-3 scripts GateBufferProbe / GateSetStatic, and the GateDeadTarget external system are deleted centrally. The site-side external_systems row for GateDeadTarget remains, per the observation above.
• Remaining instances: soak-motor-1..4 (template 4, not deployable) — untouched, as found.
• The ~11.8k orphaned sf_messages remain on both site-a nodes.