dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(localdb): phase 2 live gate evidence — all 10 checks pass

Browse Source 
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.

Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
  __localdb_row_version HLC and origin node id, and the standby logged zero
  config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
  oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
  nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
  the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
  errors.

Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.

Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 05:20:54 -04:00
parent 3c87b11bcf
commit 158e79bb50
1 changed files with 161 additions and 98 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-19-localdb-phase2-live-gate.md
+161 -98
View File
@@ -1,48 +1,57 @@
# LocalDb Phase 2 — live gate evidence (INCOMPLETE, 2026-07-20) # LocalDb Phase 2 — live gate evidence (2026-07-20)
Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: 5 of 10 checks captured Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and
and passing, 1 blocked, 4 not run.** Do NOT proceed to Task 21 on this evidence. passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's
third table was empty).
Build under test: `feat/localdb-phase2` @ `166f07fa` (LocalDb `0.1.1`, confirmed live — the Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not
metrics scrape reports `otel_scope_version="0.1.1"`). Rig redeployed via `docker/deploy.sh`, assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication"
exit 0, all 8 nodes up, central healthy. otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up.
## Method note — reading the databases ## Method notes — two corrections to the plan
Every SQLite read below went through a snapshot helper that copies the `db`/`-wal`/`-shm` **1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container
triplet and queries the COPY. **The plan's instruction to "run DB checks host-side against the locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container
bind mounts" is wrong and must not be followed.** macOS↔container locks do not cross virtiofs, and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of
so a host `sqlite3` open recovers the WAL out from under the container and triggers a permanent the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet
`disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of the 2026-07-20 incident. and queries the **copy**; `cp` is a pure byte read with no SQLite involvement.
`cp` is a pure byte read with no SQLite involvement, so copying first is safe.
**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and
with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier
pass nearly recorded that as a finding. Use a network-sharing sidecar:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
```
**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External
systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` →
`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that
names an external system that never arrived fails silently — no rows, no error.
--- ---
## Passing ## 1. Migration ran — PASS
### 1. Migration ran — PASS
Both markers present on both site-a nodes, and row counts match the legacy source:
| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` | | Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` |
|---|---|---|---| |---|---|---|---|
| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 | | site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 |
| site-a-node-b | both | 11804 | 11804 | | site-a-node-b | both | 11804 | 11804 |
`deployed_configurations` was 0 in both legacy files, so that table's migration is untested by Both nodes migrated their own legacy files independently and the identical-content rows then
this check — it is covered instead by check 3, which deploys fresh. LWW-converged — expected per the plan, not an anomaly.
Note the migration itself ran on an **earlier** deploy of this build (markers dated 05:04), so **Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so
this is after-the-fact evidence, not an observed run. The markers and counts are exactly what the this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that
plan asks for, but a from-scratch re-run was not performed. table's migration path is untested here; check 3 exercises it fresh instead.
### 2. No SMTP/notification rows migrated — PASS ## 2. No SMTP/notification rows migrated — PASS
`smtp_configurations` and `notification_lists` are both 0 on both nodes. `smtp_configurations` and `notification_lists` are 0 on both nodes.
**That count alone is vacuous** — the legacy source also had 0 SMTP rows, so an exclusion that **That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion
did not work would look identical. The non-vacuous evidence is the registered-table list: the would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for
CDC capture triggers exist for exactly these ten tables on both nodes — exactly these ten tables on both nodes —
``` ```
OperationTracking, data_connection_definitions, database_connections, OperationTracking, data_connection_definitions, database_connections,
@@ -50,97 +59,151 @@ deployed_configurations, external_systems, native_alarm_state,
sf_messages, shared_scripts, site_events, static_attribute_overrides sf_messages, shared_scripts, site_events, static_attribute_overrides
``` ```
— and `smtp_configurations` / `notification_lists` have **no triggers**, so no replication — and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication
channel exists for them at all. That is the property the security decision actually rests on. channel exists for them regardless of content. That is the property the security decision rests on.
### 3. Config converges byte-identical — PASS ## 3. Config converges byte-identical — PASS
Deployed `gate-sensor-1` (template 3, site 1) to site-a. Both nodes: Deployed `gate-sensor-1` (template 3) to site-a. Both nodes:
- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f` - whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f`
- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`, `deployed_at` - `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`,
`2026-07-20T09:06:11.2087221+00:00` — identical `deployed_at` `2026-07-20T09:06:11.2087221+00:00`
- `__localdb_row_version`: HLC `116951506695487488`, originating node - `__localdb_row_version`: HLC `116951506695487488`, origin node
`cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both** `cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both**
Identical HLC and originating node id is the strong form: node B holds the row A wrote, not an Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an
independently-derived copy. independently-derived copy of it.
### 4. The standby made no config fetch — PASS ## 4. The standby made no config fetch — PASS
Deploy window from `2026-07-20T09:05:52Z`: Deploy window from `2026-07-20T09:05:52Z`:
- `scadabridge-site-a-a` (active, deploy target): **1** fetch — - `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607…
`Fetching config for deployment 20b81607… instance gate-sensor-1 (notify-and-fetch)`. (notify-and-fetch)`. Legitimate: the active node pulling its own artifact.
Legitimate: this is the active node pulling its own artifact. - `site-a-b` (standby): **0** fetches.
- `scadabridge-site-a-b` (standby): **0** fetches.
Under the old architecture the standby was notified by `SiteReplicationActor` and fetched Under the old architecture `SiteReplicationActor` would have notified the standby and it would have
independently. It now receives the config purely by CDC replication — which check 3 corroborates fetched independently. It now receives the config purely by CDC — corroborated from the other
from the other direction (the row on B carries A's node id). direction by check 3's shared origin id.
### 8 (partial). Oplog drained, no dead letters, `localdb_*` scraped — PASS ## 5. Store-and-forward converges — PASS
- `__localdb_oplog` = **0** rows on both nodes; `localdb_oplog_depth` = 0 on both. Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an
- `__localdb_dead_letter` = **0** on both nodes. interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain
- `localdb_sync_applied_total` = 11808 (node a) / 224 (node b); `localdb_sync_reconnects_total` = 1. `ExternalSystem.Call` does not). Delivered via `deploy artifacts`.
**Correction worth recording:** an earlier scrape appeared to show zero `localdb_*` metrics. That - Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`.
was an artifact — `docker exec … curl` fails because the `aspnet:10.0` image has no `curl`, and - `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes.
the error was being swallowed by `2>/dev/null`. The correct method is a network-sharing sidecar: - The Pending→Parked transition of the pre-existing backlog also replicated identically
(10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT.
```bash ## 6. Site failover — PASS
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged
`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby
picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked
entries.
Restarted `site-a-a` at 09:18:57. After rejoin:
| | node-a | node-b |
|---|---|---|
| probe rows | 48 | 48 |
| total `sf_messages` | 11852 | 11852 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
| duplicate ids | 0 | 0 |
| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` |
Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on
either node is the exactly-once evidence.
## 7. Cascade delete — PASS
Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute`
script — the attribute-override deploy path writes into the config JSON, not this table), confirmed
it replicated, then deleted the instance.
| | pre-delete | post-delete |
|---|---|---|
| `deployed_configurations` | 1 / 1 | 0 / 0 |
| `static_attribute_overrides` | 1 / 1 | 0 / 0 |
| `native_alarm_state` | 0 / 0 | 0 / 0 |
Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`,
`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not
merely absent on B; B applied the deletes.
**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is
untested live. It is covered offline by
`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was
verified non-vacuous.
## 8. Dead letters, oplog, metrics — PASS
- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point.
- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b);
`localdb_sync_reconnects_total` 1.
## 9. Both nodes stopped and started together (D5) — PASS
Stopped `site-a-a` and `site-a-b` together, started them together.
| | node-a | node-b |
|---|---|---|
| `sf_messages` | 11845 | 11845 |
| probe rows | 41 | 41 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either
node's log after restart. Clean rejoin.
## 10. Drain under churn — PASS
Sampled every 8 s under sustained probe load:
```
t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23
t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25
t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26
t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28
t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30
t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31
t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33
t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35
``` ```
I nearly recorded "metrics not scraped" as a finding on the strength of a missing binary. Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog
accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the
plan cared about (monotonic growth that never drains) does not occur.
**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a
transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The
rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer
was down and back to 0 after rejoin.
--- ---
## Blocked ## Observations outside the gate
### 5. Store-and-forward convergence — NOT DEMONSTRATED - **A central external-system delete does not remove the row from sites.** After deleting
`GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it.
Not a product failure — the rig could not be driven into the required state. What happened: Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in
the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans.
1. Removing the owed `soakgen-1..4` instances left their **11,804 buffered `sf_messages` - **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left
orphaned** (11,780 Pending / 24 Parked). The site now logs a continuous flood of their 11,804 `sf_messages` with no tracking snapshot, producing a continuous
`Cached-telemetry drain: no tracking snapshot for …` warnings for them. `Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth
2. A purpose-built probe (external system `GateDeadTarget` → `http://127.0.0.1:9`, method `Ping`, a cleanup path.
plus an interval script on template 3 calling `ExternalSystem.CachedCall`) deployed cleanly and
its `ScriptActor GateBufferProbe started`, but produced **zero** buffered rows and zero errors.
3. Root cause identified: the site nodes' `external_systems` table still holds only
`Test REST API -> http://127.0.0.1:9`. `GateDeadTarget` **never reached the site**, so the
script's call cannot resolve a target. Site config tables are populated from deployment
artifacts, and the artifact for `gate-sensor-1` evidently does not carry an external system
that is referenced only by name inside script code.
Consequence: the central-side restore of `ExternalSystemDefinitions` id 1 to
`http://scadabridge-restapi:5200` **has not propagated to the site nodes**, which still target the
dead `127.0.0.1:9` — visible in the log as `POST http://127.0.0.1:9/api/Add` →
`Connection refused`. The backlog therefore cannot drain either.
To finish check 5, someone needs a probe whose external system is genuinely delivered to the site
(most likely bound to an instance, the way the retired `SoakGenerator` template did it), and the
orphaned backlog should be cleared first so the signal is not buried.
## Not run
- **6. Site failover** — buffered messages survive a site-a flip and deliver exactly once.
- **7. Cascade delete** — no orphan `static_attribute_overrides` / `native_alarm_state` on the peer.
(The offline suite covers this well: `LocalDbPhase2ConvergenceTests.RemovingAnInstance_
ConvergesAllThreeCascadeTables`, verified non-vacuous. The live run is still owed.)
- **9. Both nodes stopped and started together** (D5) — clean rejoin.
- **10. Post-cutover drain under churn** — oplog rises and drains between bursts. Blocked by the
same missing load generator as check 5.
## Rig state as left ## Rig state as left
- `gate-sensor-1` (instance 9, template 3) deployed on site-a, with a `GateBufferProbe` interval - Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted;
script (template 3, script 1015) that currently does nothing. `ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed
- External system `GateDeadTarget` (id 2) with method `Ping` (id 2) — created for the probe. propagated to both site nodes**.
- `ExternalSystemDefinitions` id 1 restored centrally to `http://scadabridge-restapi:5200`; - Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` /
**the site copies are still stale at `http://127.0.0.1:9`.** `GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side
- `SoakGenerator` template (2021) and `soakgen-1..4` deleted, per the owed cleanup — but their `external_systems` row for `GateDeadTarget` remains, per the observation above.
11,804 orphaned `sf_messages` remain and are generating a warning flood. - Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found.
- `soak-motor-1..4` (template 4) untouched; template 4 is not deployable. - The ~11.8k orphaned `sf_messages` remain on both site-a nodes.