dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
228ff8b428329035199b7dcb5d9187b579249f9c
ScadaBridge/docs/plans/2026-07-19-localdb-phase2-live-gate.md
T
Joseph Doherty 158e79bb50 docs(localdb): phase 2 live gate evidence — all 10 checks pass 
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.

Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
  __localdb_row_version HLC and origin node id, and the standby logged zero
  config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
  oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
  nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
  the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
  errors.

Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.

Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 05:20:54 -04:00

210 lines
9.7 KiB
Markdown
Raw Blame History

# LocalDb Phase 2 — live gate evidence (2026-07-20)
Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and
passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's
third table was empty).
Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not
assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication"
otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up.
## Method notes — two corrections to the plan
**1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container
locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container
and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of
the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet
and queries the **copy**; `cp` is a pure byte read with no SQLite involvement.
**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and
with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier
pass nearly recorded that as a finding. Use a network-sharing sidecar:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
```
**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External
systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` →
`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that
names an external system that never arrived fails silently — no rows, no error.
---
## 1. Migration ran — PASS
| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` |
|---|---|---|---|
| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 |
| site-a-node-b | both | 11804 | 11804 |
Both nodes migrated their own legacy files independently and the identical-content rows then
LWW-converged — expected per the plan, not an anomaly.
**Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so
this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that
table's migration path is untested here; check 3 exercises it fresh instead.
## 2. No SMTP/notification rows migrated — PASS
`smtp_configurations` and `notification_lists` are 0 on both nodes.
**That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion
would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for
exactly these ten tables on both nodes —
```
OperationTracking, data_connection_definitions, database_connections,
deployed_configurations, external_systems, native_alarm_state,
sf_messages, shared_scripts, site_events, static_attribute_overrides
```
— and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication
channel exists for them regardless of content. That is the property the security decision rests on.
## 3. Config converges byte-identical — PASS
Deployed `gate-sensor-1` (template 3) to site-a. Both nodes:
- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f`
- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`,
`deployed_at` `2026-07-20T09:06:11.2087221+00:00`
- `__localdb_row_version`: HLC `116951506695487488`, origin node
`cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both**
Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an
independently-derived copy of it.
## 4. The standby made no config fetch — PASS
Deploy window from `2026-07-20T09:05:52Z`:
- `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607…
(notify-and-fetch)`. Legitimate: the active node pulling its own artifact.
- `site-a-b` (standby): **0** fetches.
Under the old architecture `SiteReplicationActor` would have notified the standby and it would have
fetched independently. It now receives the config purely by CDC — corroborated from the other
direction by check 3's shared origin id.
## 5. Store-and-forward converges — PASS
Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an
interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain
`ExternalSystem.Call` does not). Delivered via `deploy artifacts`.
- Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`.
- `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes.
- The Pending→Parked transition of the pre-existing backlog also replicated identically
(10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT.
## 6. Site failover — PASS
Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged
`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby
picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked
entries.
Restarted `site-a-a` at 09:18:57. After rejoin:
| | node-a | node-b |
|---|---|---|
| probe rows | 48 | 48 |
| total `sf_messages` | 11852 | 11852 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
| duplicate ids | 0 | 0 |
| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` |
Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on
either node is the exactly-once evidence.
## 7. Cascade delete — PASS
Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute`
script — the attribute-override deploy path writes into the config JSON, not this table), confirmed
it replicated, then deleted the instance.
| | pre-delete | post-delete |
|---|---|---|
| `deployed_configurations` | 1 / 1 | 0 / 0 |
| `static_attribute_overrides` | 1 / 1 | 0 / 0 |
| `native_alarm_state` | 0 / 0 | 0 / 0 |
Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`,
`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not
merely absent on B; B applied the deletes.
**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is
untested live. It is covered offline by
`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was
verified non-vacuous.
## 8. Dead letters, oplog, metrics — PASS
- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point.
- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b);
`localdb_sync_reconnects_total` 1.
## 9. Both nodes stopped and started together (D5) — PASS
Stopped `site-a-a` and `site-a-b` together, started them together.
| | node-a | node-b |
|---|---|---|
| `sf_messages` | 11845 | 11845 |
| probe rows | 41 | 41 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either
node's log after restart. Clean rejoin.
## 10. Drain under churn — PASS
Sampled every 8 s under sustained probe load:
```
t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23
t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25
t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26
t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28
t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30
t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31
t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33
t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35
```
Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog
accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the
plan cared about (monotonic growth that never drains) does not occur.
**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a
transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The
rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer
was down and back to 0 after rejoin.
---
## Observations outside the gate
- **A central external-system delete does not remove the row from sites.** After deleting
`GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it.
Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in
the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans.
- **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left
their 11,804 `sf_messages` with no tracking snapshot, producing a continuous
`Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth
a cleanup path.
## Rig state as left
- Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted;
`ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed
propagated to both site nodes**.
- Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` /
`GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side
`external_systems` row for `GateDeadTarget` remains, per the observation above.
- Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found.
- The ~11.8k orphaned `sf_messages` remain on both site-a nodes.
Reference in New Issue View Git Blame Copy Permalink