docs(localdb): close the phase 2 gate — questions answered, plan written
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short sampling that Task 2 was waiting on. Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered inline. The questions are kept, not deleted — several of the answers overturn a premise the gate itself stated, and that reasoning is the value: - Oplog sizing is not the binding constraint. Cap overrun prunes and flags needs_snapshot (graceful resync), so the caps need no defensive sizing; the real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes. - LWW-vs-in-flight-send cannot arise on a correct pair (only the active node sweeps). The honest cost is D2's semantic change: the standby is convergent, no longer byte-identical. - Migration-under-load is designed out, not managed — both mechanisms are deleted in the same commit and D5 forecloses rolling upgrades. - Rollback is genuinely "revert the commit": the migration is additive and leaves the legacy files intact. Bounded cost is the post-cutover delta. - One DB per process stays; every write axis is bounded. §2's requirement to state CDC's duplicate-delivery bound is routed explicitly to Task 21 rather than left implicit. Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Independently confirms the disk-I/O storm was observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s ceiling and the retry path was never exercised — acceptable only because that ceiling is structural rather than empirical, and the rig's 721 B config rows are explicitly NOT representative. GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
@@ -19,6 +19,14 @@
|
||||
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
|
||||
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
|
||||
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
|
||||
>
|
||||
> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.**
|
||||
>
|
||||
> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based
|
||||
> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages`
|
||||
> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop
|
||||
> condition from any of D1–D6 is met. The one binding number Phase 2 must honour is
|
||||
> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets.
|
||||
|
||||
---
|
||||
|
||||
@@ -268,12 +276,59 @@ rate that this run could not measure.
|
||||
|
||||
---
|
||||
|
||||
## 1b. Clean re-run (2026-07-20, post-root-cause)
|
||||
|
||||
The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after
|
||||
both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side
|
||||
`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load
|
||||
unchanged (`SoakGenerator` ×4 against a refusing endpoint):
|
||||
|
||||
| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` |
|
||||
|---|---|---|---|---|---|
|
||||
| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 |
|
||||
| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 |
|
||||
| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 |
|
||||
| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 |
|
||||
| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 |
|
||||
| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 |
|
||||
| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 |
|
||||
|
||||
**Measured:**
|
||||
|
||||
- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six
|
||||
intervals (+288 rows over 360 s, zero variance).
|
||||
- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached
|
||||
`retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`,
|
||||
status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only.
|
||||
- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json`
|
||||
max **721 B** over 4 rows (rig config is trivially small — see the caveat below).
|
||||
- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded.
|
||||
- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load,
|
||||
which is the expected result and not a measurement failure (Phase 2's tables are not yet
|
||||
registered, so this load cannot reach the oplog by construction).
|
||||
- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window
|
||||
(`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that
|
||||
Finding 1 was observer-induced: identical load, identical library, nothing host-reading the
|
||||
files, no errors.
|
||||
- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under
|
||||
the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC.
|
||||
|
||||
**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural
|
||||
ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never
|
||||
exercised at all. This re-run therefore confirms *steady-state health and the absence of the
|
||||
Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
|
||||
**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
|
||||
sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
|
||||
representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on
|
||||
this number.
|
||||
|
||||
## 2. What still owes measurement
|
||||
|
||||
Carry into the re-run once Finding 1 is fixed:
|
||||
Carry forward (items 1–2 partially discharged by §1b above):
|
||||
|
||||
1. Sustained `sf_messages` rows/sec under a saturating generator (needs many more instances or a
|
||||
shorter interval to approach the 50/s ceiling).
|
||||
1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still
|
||||
unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s
|
||||
ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap.
|
||||
2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an
|
||||
A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does*
|
||||
answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions
|
||||
|
||||
Reference in New Issue
Block a user