ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md

LocalDb Phase 2 — live gate evidence (INCOMPLETE, 2026-07-20)

Gate for Task 20. Status: 5 of 10 checks captured and passing, 1 blocked, 4 not run. Do NOT proceed to Task 21 on this evidence.

Build under test: feat/localdb-phase2 @ 166f07fa (LocalDb 0.1.1, confirmed live — the metrics scrape reports otel_scope_version="0.1.1"). Rig redeployed via docker/deploy.sh, exit 0, all 8 nodes up, central healthy.

Method note — reading the databases

Every SQLite read below went through a snapshot helper that copies the db/-wal/-shm triplet and queries the COPY. The plan's instruction to "run DB checks host-side against the bind mounts" is wrong and must not be followed. macOS↔container locks do not cross virtiofs, so a host sqlite3 open recovers the WAL out from under the container and triggers a permanent disk I/O error (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of the 2026-07-20 incident. cp is a pure byte read with no SQLite involvement, so copying first is safe.

---

Passing

1. Migration ran — PASS

Both markers present on both site-a nodes, and row counts match the legacy source:

Node	.migrated markers	legacy sf_messages	consolidated sf_messages
site-a-node-a	scadabridge.db.migrated, store-and-forward.db.migrated	11804	11804
site-a-node-b	both	11804	11804

deployed_configurations was 0 in both legacy files, so that table's migration is untested by this check — it is covered instead by check 3, which deploys fresh.

Note the migration itself ran on an earlier deploy of this build (markers dated 05:04), so this is after-the-fact evidence, not an observed run. The markers and counts are exactly what the plan asks for, but a from-scratch re-run was not performed.

2. No SMTP/notification rows migrated — PASS

smtp_configurations and notification_lists are both 0 on both nodes.

That count alone is vacuous — the legacy source also had 0 SMTP rows, so an exclusion that did not work would look identical. The non-vacuous evidence is the registered-table list: the CDC capture triggers exist for exactly these ten tables on both nodes —

OperationTracking, data_connection_definitions, database_connections,
deployed_configurations, external_systems, native_alarm_state,
sf_messages, shared_scripts, site_events, static_attribute_overrides

— and smtp_configurations / notification_lists have no triggers, so no replication channel exists for them at all. That is the property the security decision actually rests on.

3. Config converges byte-identical — PASS

Deployed gate-sensor-1 (template 3, site 1) to site-a. Both nodes:

• whole-row md5 identical: 0ec0f3cf790bfd65c45edf6c4afae50f
• deployment_id 20b81607…, revision_hash sha256:253e1a56…, deployed_at 2026-07-20T09:06:11.2087221+00:00 — identical
• __localdb_row_version: HLC 116951506695487488, originating node cf61c688-a57f-4c5c-ba76-feff6e5679fe, is_tombstone 0 — identical on both

Identical HLC and originating node id is the strong form: node B holds the row A wrote, not an independently-derived copy.

4. The standby made no config fetch — PASS

Deploy window from 2026-07-20T09:05:52Z:

• scadabridge-site-a-a (active, deploy target): 1 fetch — Fetching config for deployment 20b81607… instance gate-sensor-1 (notify-and-fetch). Legitimate: this is the active node pulling its own artifact.
• scadabridge-site-a-b (standby): 0 fetches.

Under the old architecture the standby was notified by SiteReplicationActor and fetched independently. It now receives the config purely by CDC replication — which check 3 corroborates from the other direction (the row on B carries A's node id).

8 (partial). Oplog drained, no dead letters, localdb_* scraped — PASS

• __localdb_oplog = 0 rows on both nodes; localdb_oplog_depth = 0 on both.
• __localdb_dead_letter = 0 on both nodes.
• localdb_sync_applied_total = 11808 (node a) / 224 (node b); localdb_sync_reconnects_total = 1.

Correction worth recording: an earlier scrape appeared to show zero localdb_* metrics. That was an artifact — docker exec … curl fails because the aspnet:10.0 image has no curl, and the error was being swallowed by 2>/dev/null. The correct method is a network-sharing sidecar:

docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics

I nearly recorded "metrics not scraped" as a finding on the strength of a missing binary.

---

Blocked

5. Store-and-forward convergence — NOT DEMONSTRATED

Not a product failure — the rig could not be driven into the required state. What happened:

1. Removing the owed soakgen-1..4 instances left their 11,804 buffered sf_messages orphaned (11,780 Pending / 24 Parked). The site now logs a continuous flood of Cached-telemetry drain: no tracking snapshot for … warnings for them.
2. A purpose-built probe (external system GateDeadTarget → http://127.0.0.1:9, method Ping, plus an interval script on template 3 calling ExternalSystem.CachedCall) deployed cleanly and its ScriptActor GateBufferProbe started, but produced zero buffered rows and zero errors.
3. Root cause identified: the site nodes' external_systems table still holds only Test REST API -> http://127.0.0.1:9. GateDeadTarget never reached the site, so the script's call cannot resolve a target. Site config tables are populated from deployment artifacts, and the artifact for gate-sensor-1 evidently does not carry an external system that is referenced only by name inside script code.

Consequence: the central-side restore of ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 has not propagated to the site nodes, which still target the dead 127.0.0.1:9 — visible in the log as POST http://127.0.0.1:9/api/Add → Connection refused. The backlog therefore cannot drain either.

To finish check 5, someone needs a probe whose external system is genuinely delivered to the site (most likely bound to an instance, the way the retired SoakGenerator template did it), and the orphaned backlog should be cleared first so the signal is not buried.

Not run

• 6. Site failover — buffered messages survive a site-a flip and deliver exactly once.
• 7. Cascade delete — no orphan static_attribute_overrides / native_alarm_state on the peer. (The offline suite covers this well: LocalDbPhase2ConvergenceTests.RemovingAnInstance_ ConvergesAllThreeCascadeTables, verified non-vacuous. The live run is still owed.)
• 9. Both nodes stopped and started together (D5) — clean rejoin.
• 10. Post-cutover drain under churn — oplog rises and drains between bursts. Blocked by the same missing load generator as check 5.

Rig state as left

• gate-sensor-1 (instance 9, template 3) deployed on site-a, with a GateBufferProbe interval script (template 3, script 1015) that currently does nothing.
• External system GateDeadTarget (id 2) with method Ping (id 2) — created for the probe.
• ExternalSystemDefinitions id 1 restored centrally to http://scadabridge-restapi:5200; the site copies are still stale at http://127.0.0.1:9.
• SoakGenerator template (2021) and soakgen-1..4 deleted, per the owed cleanup — but their 11,804 orphaned sf_messages remain and are generating a warning flood.
• soak-motor-1..4 (template 4) untouched; template 4 is not deployable.