From 158e79bb50c49e0ff68fce5a25e0a2c25de91403 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 05:20:54 -0400 Subject: [PATCH] =?UTF-8?q?docs(localdb):=20phase=202=20live=20gate=20evid?= =?UTF-8?q?ence=20=E2=80=94=20all=2010=20checks=20pass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The blocker in the previous (incomplete) run was diagnosed: external systems reach a site only through ArtifactDeploymentService, which `instance deploy` never invokes. `deploy artifacts` delivered the probe harness AND propagated the owed ExternalSystemDefinitions restore to both site nodes, unblocking checks 5 and 10. Highlights: - 3+4: the deployed row is byte-identical on both nodes with the SAME __localdb_row_version HLC and origin node id, and the standby logged zero config fetches in the deploy window. B holds the row A wrote, by CDC alone. - 6: stopped the active node; the standby took over in 10s and kept buffering, oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both nodes ended byte-identical with ZERO duplicate ids (exactly-once). - 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes — the rows are not merely absent on B, B applied the deletes. - 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O errors. Two caveats recorded rather than glossed: check 2's zero-count is vacuous on its own (the legacy source was also empty) and rests instead on the ABSENCE of CDC triggers for smtp_configurations/notification_lists; check 7's native_alarm_state leg was empty live and is covered only offline. Also records three method corrections — the plan's host-side sqlite3 instruction is unsafe, `docker exec curl` cannot scrape metrics from an image with no curl (a silent failure that nearly became a false finding), and the artifact-vs-instance deploy distinction above. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../2026-07-19-localdb-phase2-live-gate.md | 259 +++++++++++------- 1 file changed, 161 insertions(+), 98 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-phase2-live-gate.md b/docs/plans/2026-07-19-localdb-phase2-live-gate.md index 5f120fa0..1ee79c04 100644 --- a/docs/plans/2026-07-19-localdb-phase2-live-gate.md +++ b/docs/plans/2026-07-19-localdb-phase2-live-gate.md @@ -1,48 +1,57 @@ -# LocalDb Phase 2 — live gate evidence (INCOMPLETE, 2026-07-20) +# LocalDb Phase 2 — live gate evidence (2026-07-20) -Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: 5 of 10 checks captured -and passing, 1 blocked, 4 not run.** Do NOT proceed to Task 21 on this evidence. +Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and +passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's +third table was empty). -Build under test: `feat/localdb-phase2` @ `166f07fa` (LocalDb `0.1.1`, confirmed live — the -metrics scrape reports `otel_scope_version="0.1.1"`). Rig redeployed via `docker/deploy.sh`, -exit 0, all 8 nodes up, central healthy. +Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not +assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication" +otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up. -## Method note — reading the databases +## Method notes — two corrections to the plan -Every SQLite read below went through a snapshot helper that copies the `db`/`-wal`/`-shm` -triplet and queries the COPY. **The plan's instruction to "run DB checks host-side against the -bind mounts" is wrong and must not be followed.** macOS↔container locks do not cross virtiofs, -so a host `sqlite3` open recovers the WAL out from under the container and triggers a permanent -`disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of the 2026-07-20 incident. -`cp` is a pure byte read with no SQLite involvement, so copying first is safe. +**1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container +locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container +and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of +the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet +and queries the **copy**; `cp` is a pure byte read with no SQLite involvement. + +**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and +with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier +pass nearly recorded that as a finding. Use a network-sharing sidecar: + +```bash +docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics +``` + +**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External +systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` → +`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that +names an external system that never arrived fails silently — no rows, no error. --- -## Passing - -### 1. Migration ran — PASS - -Both markers present on both site-a nodes, and row counts match the legacy source: +## 1. Migration ran — PASS | Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` | |---|---|---|---| | site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 | | site-a-node-b | both | 11804 | 11804 | -`deployed_configurations` was 0 in both legacy files, so that table's migration is untested by -this check — it is covered instead by check 3, which deploys fresh. +Both nodes migrated their own legacy files independently and the identical-content rows then +LWW-converged — expected per the plan, not an anomaly. -Note the migration itself ran on an **earlier** deploy of this build (markers dated 05:04), so -this is after-the-fact evidence, not an observed run. The markers and counts are exactly what the -plan asks for, but a from-scratch re-run was not performed. +**Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so +this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that +table's migration path is untested here; check 3 exercises it fresh instead. -### 2. No SMTP/notification rows migrated — PASS +## 2. No SMTP/notification rows migrated — PASS -`smtp_configurations` and `notification_lists` are both 0 on both nodes. +`smtp_configurations` and `notification_lists` are 0 on both nodes. -**That count alone is vacuous** — the legacy source also had 0 SMTP rows, so an exclusion that -did not work would look identical. The non-vacuous evidence is the registered-table list: the -CDC capture triggers exist for exactly these ten tables on both nodes — +**That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion +would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for +exactly these ten tables on both nodes — ``` OperationTracking, data_connection_definitions, database_connections, @@ -50,97 +59,151 @@ deployed_configurations, external_systems, native_alarm_state, sf_messages, shared_scripts, site_events, static_attribute_overrides ``` -— and `smtp_configurations` / `notification_lists` have **no triggers**, so no replication -channel exists for them at all. That is the property the security decision actually rests on. +— and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication +channel exists for them regardless of content. That is the property the security decision rests on. -### 3. Config converges byte-identical — PASS +## 3. Config converges byte-identical — PASS -Deployed `gate-sensor-1` (template 3, site 1) to site-a. Both nodes: +Deployed `gate-sensor-1` (template 3) to site-a. Both nodes: - whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f` -- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`, `deployed_at` - `2026-07-20T09:06:11.2087221+00:00` — identical -- `__localdb_row_version`: HLC `116951506695487488`, originating node +- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`, + `deployed_at` `2026-07-20T09:06:11.2087221+00:00` +- `__localdb_row_version`: HLC `116951506695487488`, origin node `cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both** -Identical HLC and originating node id is the strong form: node B holds the row A wrote, not an -independently-derived copy. +Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an +independently-derived copy of it. -### 4. The standby made no config fetch — PASS +## 4. The standby made no config fetch — PASS Deploy window from `2026-07-20T09:05:52Z`: -- `scadabridge-site-a-a` (active, deploy target): **1** fetch — - `Fetching config for deployment 20b81607… instance gate-sensor-1 (notify-and-fetch)`. - Legitimate: this is the active node pulling its own artifact. -- `scadabridge-site-a-b` (standby): **0** fetches. +- `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607… + (notify-and-fetch)`. Legitimate: the active node pulling its own artifact. +- `site-a-b` (standby): **0** fetches. -Under the old architecture the standby was notified by `SiteReplicationActor` and fetched -independently. It now receives the config purely by CDC replication — which check 3 corroborates -from the other direction (the row on B carries A's node id). +Under the old architecture `SiteReplicationActor` would have notified the standby and it would have +fetched independently. It now receives the config purely by CDC — corroborated from the other +direction by check 3's shared origin id. -### 8 (partial). Oplog drained, no dead letters, `localdb_*` scraped — PASS +## 5. Store-and-forward converges — PASS -- `__localdb_oplog` = **0** rows on both nodes; `localdb_oplog_depth` = 0 on both. -- `__localdb_dead_letter` = **0** on both nodes. -- `localdb_sync_applied_total` = 11808 (node a) / 224 (node b); `localdb_sync_reconnects_total` = 1. +Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an +interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain +`ExternalSystem.Call` does not). Delivered via `deploy artifacts`. -**Correction worth recording:** an earlier scrape appeared to show zero `localdb_*` metrics. That -was an artifact — `docker exec … curl` fails because the `aspnet:10.0` image has no `curl`, and -the error was being swallowed by `2>/dev/null`. The correct method is a network-sharing sidecar: +- Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`. +- `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes. +- The Pending→Parked transition of the pre-existing backlog also replicated identically + (10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT. -```bash -docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics +## 6. Site failover — PASS + +Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged +`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby +picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked +entries. + +Restarted `site-a-a` at 09:18:57. After rejoin: + +| | node-a | node-b | +|---|---|---| +| probe rows | 48 | 48 | +| total `sf_messages` | 11852 | 11852 | +| oplog | 0 | 0 | +| dead letters | 0 | 0 | +| duplicate ids | 0 | 0 | +| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` | + +Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on +either node is the exactly-once evidence. + +## 7. Cascade delete — PASS + +Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute` +script — the attribute-override deploy path writes into the config JSON, not this table), confirmed +it replicated, then deleted the instance. + +| | pre-delete | post-delete | +|---|---|---| +| `deployed_configurations` | 1 / 1 | 0 / 0 | +| `static_attribute_overrides` | 1 / 1 | 0 / 0 | +| `native_alarm_state` | 0 / 0 | 0 / 0 | + +Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`, +`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not +merely absent on B; B applied the deletes. + +**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is +untested live. It is covered offline by +`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was +verified non-vacuous. + +## 8. Dead letters, oplog, metrics — PASS + +- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point. +- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b); + `localdb_sync_reconnects_total` 1. + +## 9. Both nodes stopped and started together (D5) — PASS + +Stopped `site-a-a` and `site-a-b` together, started them together. + +| | node-a | node-b | +|---|---|---| +| `sf_messages` | 11845 | 11845 | +| probe rows | 41 | 41 | +| oplog | 0 | 0 | +| dead letters | 0 | 0 | + +Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either +node's log after restart. Clean rejoin. + +## 10. Drain under churn — PASS + +Sampled every 8 s under sustained probe load: + +``` +t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23 +t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25 +t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26 +t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28 +t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30 +t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31 +t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33 +t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35 ``` -I nearly recorded "metrics not scraped" as a finding on the strength of a missing binary. +Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog +accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the +plan cared about (monotonic growth that never drains) does not occur. + +**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a +transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The +rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer +was down and back to 0 after rejoin. --- -## Blocked +## Observations outside the gate -### 5. Store-and-forward convergence — NOT DEMONSTRATED - -Not a product failure — the rig could not be driven into the required state. What happened: - -1. Removing the owed `soakgen-1..4` instances left their **11,804 buffered `sf_messages` - orphaned** (11,780 Pending / 24 Parked). The site now logs a continuous flood of - `Cached-telemetry drain: no tracking snapshot for …` warnings for them. -2. A purpose-built probe (external system `GateDeadTarget` → `http://127.0.0.1:9`, method `Ping`, - plus an interval script on template 3 calling `ExternalSystem.CachedCall`) deployed cleanly and - its `ScriptActor GateBufferProbe started`, but produced **zero** buffered rows and zero errors. -3. Root cause identified: the site nodes' `external_systems` table still holds only - `Test REST API -> http://127.0.0.1:9`. `GateDeadTarget` **never reached the site**, so the - script's call cannot resolve a target. Site config tables are populated from deployment - artifacts, and the artifact for `gate-sensor-1` evidently does not carry an external system - that is referenced only by name inside script code. - -Consequence: the central-side restore of `ExternalSystemDefinitions` id 1 to -`http://scadabridge-restapi:5200` **has not propagated to the site nodes**, which still target the -dead `127.0.0.1:9` — visible in the log as `POST http://127.0.0.1:9/api/Add` → -`Connection refused`. The backlog therefore cannot drain either. - -To finish check 5, someone needs a probe whose external system is genuinely delivered to the site -(most likely bound to an instance, the way the retired `SoakGenerator` template did it), and the -orphaned backlog should be cleared first so the signal is not buried. - -## Not run - -- **6. Site failover** — buffered messages survive a site-a flip and deliver exactly once. -- **7. Cascade delete** — no orphan `static_attribute_overrides` / `native_alarm_state` on the peer. - (The offline suite covers this well: `LocalDbPhase2ConvergenceTests.RemovingAnInstance_ - ConvergesAllThreeCascadeTables`, verified non-vacuous. The live run is still owed.) -- **9. Both nodes stopped and started together** (D5) — clean rejoin. -- **10. Post-cutover drain under churn** — oplog rises and drains between bursts. Blocked by the - same missing load generator as check 5. +- **A central external-system delete does not remove the row from sites.** After deleting + `GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it. + Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in + the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans. +- **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left + their 11,804 `sf_messages` with no tracking snapshot, producing a continuous + `Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth + a cleanup path. ## Rig state as left -- `gate-sensor-1` (instance 9, template 3) deployed on site-a, with a `GateBufferProbe` interval - script (template 3, script 1015) that currently does nothing. -- External system `GateDeadTarget` (id 2) with method `Ping` (id 2) — created for the probe. -- `ExternalSystemDefinitions` id 1 restored centrally to `http://scadabridge-restapi:5200`; - **the site copies are still stale at `http://127.0.0.1:9`.** -- `SoakGenerator` template (2021) and `soakgen-1..4` deleted, per the owed cleanup — but their - 11,804 orphaned `sf_messages` remain and are generating a warning flood. -- `soak-motor-1..4` (template 4) untouched; template 4 is not deployable. +- Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted; + `ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed + propagated to both site nodes**. +- Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` / + `GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side + `external_systems` row for `GateDeadTarget` remains, per the observation above. +- Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found. +- The ~11.8k orphaned `sf_messages` remain on both site-a nodes.