diff --git a/docs/plans/2026-07-19-localdb-phase2-live-gate.md b/docs/plans/2026-07-19-localdb-phase2-live-gate.md new file mode 100644 index 00000000..5f120fa0 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-live-gate.md @@ -0,0 +1,146 @@ +# LocalDb Phase 2 — live gate evidence (INCOMPLETE, 2026-07-20) + +Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: 5 of 10 checks captured +and passing, 1 blocked, 4 not run.** Do NOT proceed to Task 21 on this evidence. + +Build under test: `feat/localdb-phase2` @ `166f07fa` (LocalDb `0.1.1`, confirmed live — the +metrics scrape reports `otel_scope_version="0.1.1"`). Rig redeployed via `docker/deploy.sh`, +exit 0, all 8 nodes up, central healthy. + +## Method note — reading the databases + +Every SQLite read below went through a snapshot helper that copies the `db`/`-wal`/`-shm` +triplet and queries the COPY. **The plan's instruction to "run DB checks host-side against the +bind mounts" is wrong and must not be followed.** macOS↔container locks do not cross virtiofs, +so a host `sqlite3` open recovers the WAL out from under the container and triggers a permanent +`disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of the 2026-07-20 incident. +`cp` is a pure byte read with no SQLite involvement, so copying first is safe. + +--- + +## Passing + +### 1. Migration ran — PASS + +Both markers present on both site-a nodes, and row counts match the legacy source: + +| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` | +|---|---|---|---| +| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 | +| site-a-node-b | both | 11804 | 11804 | + +`deployed_configurations` was 0 in both legacy files, so that table's migration is untested by +this check — it is covered instead by check 3, which deploys fresh. + +Note the migration itself ran on an **earlier** deploy of this build (markers dated 05:04), so +this is after-the-fact evidence, not an observed run. The markers and counts are exactly what the +plan asks for, but a from-scratch re-run was not performed. + +### 2. No SMTP/notification rows migrated — PASS + +`smtp_configurations` and `notification_lists` are both 0 on both nodes. + +**That count alone is vacuous** — the legacy source also had 0 SMTP rows, so an exclusion that +did not work would look identical. The non-vacuous evidence is the registered-table list: the +CDC capture triggers exist for exactly these ten tables on both nodes — + +``` +OperationTracking, data_connection_definitions, database_connections, +deployed_configurations, external_systems, native_alarm_state, +sf_messages, shared_scripts, site_events, static_attribute_overrides +``` + +— and `smtp_configurations` / `notification_lists` have **no triggers**, so no replication +channel exists for them at all. That is the property the security decision actually rests on. + +### 3. Config converges byte-identical — PASS + +Deployed `gate-sensor-1` (template 3, site 1) to site-a. Both nodes: + +- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f` +- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`, `deployed_at` + `2026-07-20T09:06:11.2087221+00:00` — identical +- `__localdb_row_version`: HLC `116951506695487488`, originating node + `cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both** + +Identical HLC and originating node id is the strong form: node B holds the row A wrote, not an +independently-derived copy. + +### 4. The standby made no config fetch — PASS + +Deploy window from `2026-07-20T09:05:52Z`: + +- `scadabridge-site-a-a` (active, deploy target): **1** fetch — + `Fetching config for deployment 20b81607… instance gate-sensor-1 (notify-and-fetch)`. + Legitimate: this is the active node pulling its own artifact. +- `scadabridge-site-a-b` (standby): **0** fetches. + +Under the old architecture the standby was notified by `SiteReplicationActor` and fetched +independently. It now receives the config purely by CDC replication — which check 3 corroborates +from the other direction (the row on B carries A's node id). + +### 8 (partial). Oplog drained, no dead letters, `localdb_*` scraped — PASS + +- `__localdb_oplog` = **0** rows on both nodes; `localdb_oplog_depth` = 0 on both. +- `__localdb_dead_letter` = **0** on both nodes. +- `localdb_sync_applied_total` = 11808 (node a) / 224 (node b); `localdb_sync_reconnects_total` = 1. + +**Correction worth recording:** an earlier scrape appeared to show zero `localdb_*` metrics. That +was an artifact — `docker exec … curl` fails because the `aspnet:10.0` image has no `curl`, and +the error was being swallowed by `2>/dev/null`. The correct method is a network-sharing sidecar: + +```bash +docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics +``` + +I nearly recorded "metrics not scraped" as a finding on the strength of a missing binary. + +--- + +## Blocked + +### 5. Store-and-forward convergence — NOT DEMONSTRATED + +Not a product failure — the rig could not be driven into the required state. What happened: + +1. Removing the owed `soakgen-1..4` instances left their **11,804 buffered `sf_messages` + orphaned** (11,780 Pending / 24 Parked). The site now logs a continuous flood of + `Cached-telemetry drain: no tracking snapshot for …` warnings for them. +2. A purpose-built probe (external system `GateDeadTarget` → `http://127.0.0.1:9`, method `Ping`, + plus an interval script on template 3 calling `ExternalSystem.CachedCall`) deployed cleanly and + its `ScriptActor GateBufferProbe started`, but produced **zero** buffered rows and zero errors. +3. Root cause identified: the site nodes' `external_systems` table still holds only + `Test REST API -> http://127.0.0.1:9`. `GateDeadTarget` **never reached the site**, so the + script's call cannot resolve a target. Site config tables are populated from deployment + artifacts, and the artifact for `gate-sensor-1` evidently does not carry an external system + that is referenced only by name inside script code. + +Consequence: the central-side restore of `ExternalSystemDefinitions` id 1 to +`http://scadabridge-restapi:5200` **has not propagated to the site nodes**, which still target the +dead `127.0.0.1:9` — visible in the log as `POST http://127.0.0.1:9/api/Add` → +`Connection refused`. The backlog therefore cannot drain either. + +To finish check 5, someone needs a probe whose external system is genuinely delivered to the site +(most likely bound to an instance, the way the retired `SoakGenerator` template did it), and the +orphaned backlog should be cleared first so the signal is not buried. + +## Not run + +- **6. Site failover** — buffered messages survive a site-a flip and deliver exactly once. +- **7. Cascade delete** — no orphan `static_attribute_overrides` / `native_alarm_state` on the peer. + (The offline suite covers this well: `LocalDbPhase2ConvergenceTests.RemovingAnInstance_ + ConvergesAllThreeCascadeTables`, verified non-vacuous. The live run is still owed.) +- **9. Both nodes stopped and started together** (D5) — clean rejoin. +- **10. Post-cutover drain under churn** — oplog rises and drains between bursts. Blocked by the + same missing load generator as check 5. + +## Rig state as left + +- `gate-sensor-1` (instance 9, template 3) deployed on site-a, with a `GateBufferProbe` interval + script (template 3, script 1015) that currently does nothing. +- External system `GateDeadTarget` (id 2) with method `Ping` (id 2) — created for the probe. +- `ExternalSystemDefinitions` id 1 restored centrally to `http://scadabridge-restapi:5200`; + **the site copies are still stale at `http://127.0.0.1:9`.** +- `SoakGenerator` template (2021) and `soakgen-1..4` deleted, per the owed cleanup — but their + 11,804 orphaned `sf_messages` remain and are generating a warning flood. +- `soak-motor-1..4` (template 4) untouched; template 4 is not deployable.