docs(localdb): close the phase 2 gate — questions answered, plan written

Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 02:15:53 -04:00
parent d512572dac
commit 12eb97e07a
3 changed files with 182 additions and 10 deletions
+58 -3
View File
@@ -19,6 +19,14 @@
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
>
> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.**
>
> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based
> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages`
> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop
> condition from any of D1D6 is met. The one binding number Phase 2 must honour is
> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets.
---
@@ -268,12 +276,59 @@ rate that this run could not measure.
---
## 1b. Clean re-run (2026-07-20, post-root-cause)
The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after
both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side
`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load
unchanged (`SoakGenerator` ×4 against a refusing endpoint):
| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` |
|---|---|---|---|---|---|
| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 |
| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 |
| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 |
| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 |
| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 |
| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 |
| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 |
**Measured:**
- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six
intervals (+288 rows over 360 s, zero variance).
- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached
`retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`,
status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only.
- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json`
max **721 B** over 4 rows (rig config is trivially small — see the caveat below).
- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded.
- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load,
which is the expected result and not a measurement failure (Phase 2's tables are not yet
registered, so this load cannot reach the oplog by construction).
- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window
(`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that
Finding 1 was observer-induced: identical load, identical library, nothing host-reading the
files, no errors.
- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under
the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC.
**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural
ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never
exercised at all. This re-run therefore confirms *steady-state health and the absence of the
Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
representative — D6's sizing rests on the documented ~6070 KB production `config_json`, not on
this number.
## 2. What still owes measurement
Carry into the re-run once Finding 1 is fixed:
Carry forward (items 12 partially discharged by §1b above):
1. Sustained `sf_messages` rows/sec under a saturating generator (needs many more instances or a
shorter interval to approach the 50/s ceiling).
1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still
unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s
ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap.
2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an
A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does*
answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions