docs(localdb): close the phase 2 gate — questions answered, plan written

Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 02:15:53 -04:00
parent d512572dac
commit 12eb97e07a
3 changed files with 182 additions and 10 deletions
@@ -69,8 +69,8 @@
"MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
],
"tasks": [
{"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Phase 2 tables are NOT in the Phase 1 oplog - measure at the legacy DBs host-side, size caps arithmetically; empirical drain check moves to Task 20 evidence 10. Stop conditions: on-paper oplog overflow OR any config_json near 4 MB (D6)."},
{"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "pending", "classification": "trivial", "blockedBy": [1]},
{"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."},
{"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."},
{"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]},
{"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]},
{"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]},
@@ -101,6 +101,6 @@
"note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating."
}
],
"lastUpdated": "2026-07-19",
"phase2Status": "BLOCKED at the Task 1 gate (2026-07-20). Task 1 ran and its verdict is STOP - see docs/plans/2026-07-19-localdb-phase2-soak.md. The blocker is NOT oplog sizing (fine) or D6 (resolved, no stop condition): the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite Error 10 'disk I/O error' on essentially every write on the ACTIVE node under sustained load, affecting site_events + OperationTracking + audit telemetry, and silently dropping site events. Isolated to load (not node, not host-side observation) by failing over between nodes, and to LocalDb specifically (legacy store-and-forward.db/scadabridge.db in the same bind mount take ZERO errors under identical load). Phase 2 would register 8 more tables into that DB - incl. the two highest-volume - while deleting the bespoke fallbacks, so this Phase 1 defect must be root-caused and fixed before Task 3. Tasks 2-21 remain pending and untouched; no plan code has been written. Secondary defect in the same path: SiteAuditTelemetryActor closes over ActorContext across an await (NotSupportedException). Plan-premise corrections now recorded in the soak doc: D6 (largest known production config_json ~60-70 KB not >128 KB, but MaxBatchSize must drop 500 -> 16), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush), sf_messages hard ceiling 50 rows/sec (SweepBatchLimit/RetryTimerInterval), and oplog cap overrun = graceful snapshot resync, not failure. Rig caveats: ExternalSystemDefinitions id 1 still repointed to http://127.0.0.1:9 and SoakGenerator instances 5-8 still generating load - clean up before Task 20."
"lastUpdated": "2026-07-20",
"phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load."
}
+120 -3
View File
@@ -1,8 +1,11 @@
# LocalDb Phase 2 — Gate Document
> **Status: NOT STARTED.** This is the gate, not the plan. Phase 2 work must not begin
> until an implementation plan is written against the facts below and the open questions
> in §5 are answered.
> **Status: CLOSED (2026-07-20).** The implementation plan is
> [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md); its
> decisions D1D6 answer this gate and its Task 1 supplied the measurements §5 demanded
> (soak record: [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md)).
> Each §5 question is answered inline below. **The questions are deliberately not deleted —
> the reasoning is the value, and several of the answers overturn a premise stated above.**
**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence.
@@ -50,6 +53,11 @@ Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-d
race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC
inherits the same bound, a tighter one, or a different failure shape.
> **Routed (2026-07-20).** The failure shape changes, so N5's bound does not simply carry
> over: CDC has no chunked anti-entropy hop to duplicate across, but LWW admits re-delivery
> of a row whose park loses to a later write (see the LWW answer in §5). **Task 21 states the
> new duplicate-delivery bound explicitly** and rewrites the N5 note rather than deleting it.
## 3. Substantially harder than Phase 1
Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no
@@ -84,16 +92,125 @@ class:
thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that
has not been run.* The live gate proved correctness, not steady-state behaviour over
time.
**ANSWERED — the soak ran (Task 1); the caps are not the binding constraint, batch
*bytes* are.** Two of this question's own premises turned out to be wrong, and both
corrections relax it:
1. **The stop condition was much weaker than written.** Exceeding `MaxOplogRows` /
`MaxOplogAge` does not wedge or lose data — `OplogStore` prunes and flags the peer
`needs_snapshot` (`OplogStore.cs:109-138`, `MaintenanceBackgroundService.cs:57`), which
degrades to a **graceful snapshot resync**. Overrun is a performance event, not a
correctness event, so these caps do not need to be sized defensively.
2. **`sf_messages` has a hard ceiling, not an estimate.** `SweepBatchLimit` (500) ÷
`RetryTimerInterval` (10 s) = **≤50 row-writes/sec**, structurally. Measured rig rate
under the purpose-built `SoakGenerator` load was far below that (see the soak record).
Rows are small — max `payload_json` **76 bytes** on the rig.
`native_alarm_state` is answered under the last question in this section, not here (and
the plan's D4, whose premise it corrects). The **real** ceiling this
question was groping toward is D6's 4 MB gRPC message cap, and it binds on
`MaxBatchSize × max-row-bytes`, not on oplog depth — **Task 19 sets
`LocalDb:Replication:MaxBatchSize = 16`**, the one firmly evidence-backed number the soak
produced. The keyed-instances escape hatch is **not** needed; this plan proceeds.
Independently, the soak retired a scare: a `disk I/O error` storm initially read as a
Phase 1 library defect and STOP-gated this plan was root-caused as **observer-induced**
(host-side `sqlite3` against live bind-mounted WAL files) — see
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md)
§0. The unmodified library sustains the full soak load with **zero** SQLite errors.
- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do
when one node parks a message the other is mid-delivery on?
**ANSWERED — the scenario cannot arise on a correctly-behaving pair, and D2 covers what
happens if it does.** Only the **active** node sweeps and delivers; the standby holds a
convergent copy and sends nothing. So "one node mid-delivery while the other parks" is
a split-brain symptom, not steady-state behaviour, and store-and-forward is not where it
should be defended against.
If it does happen, LWW resolves per row by HLC: the later write wins, and the losing
node's view converges to it. The concrete risk is **at-least-once delivery** — a park
that loses to a stale in-flight update can be re-swept and re-sent. That is not new;
store-and-forward is already at-least-once by construction (a send that succeeds but
whose ack is lost is retried). What **is** new is D2's semantic change, recorded here
because it is the honest cost of this gate: the standby is no longer guaranteed
**byte-identical** to the active buffer, only **convergent**. The bespoke replicator's
`ReplaceAllAsync` bought identity by wiping — and the N1 directional guard existed
precisely because that wipe was dangerous. The library's snapshot resync **merges per
row and never wipes** (`SnapshotApplier`, `LwwApplier.cs:69-78`), so the failure the
guard prevented is structurally impossible rather than merely tested against. Task 10
ports `SfBufferResyncPredicateTests` as a **convergence** assertion, not a
directional-authority one.
- **Migration-under-load.** How does the one-time copy of an actively-replicating
`scadabridge.db` interact with the bespoke replicator still running during cutover?
**ANSWERED — the interaction is designed out, not managed.** The two mechanisms never run
concurrently. `SiteReplicationActor` and StoreAndForward's `ReplicationService` are
**deleted in the same commit** that registers the Phase 2 tables (Tasks 14/15), and D5
forecloses rolling upgrades: **both nodes of a site stop and start together**. A process
that boots with the new code has no bespoke replicator to race, and one running the old
code has no CDC triggers. There is no window in which a row is written by one mechanism
and read by the other.
Within a single booting process the ordering is the load-bearing part, and it is the same
invariant Phase 1 established: **DDL → `RegisterReplicated` → migrate → writes**
(`SiteLocalDbSetup.OnReady`). Rows written *before* registration are invisible to the peer
forever, silently — which is why Tasks 8/9 run the migrator strictly **after**
registration, so every migrated row is captured by the CDC triggers and replicates
normally. Both nodes migrating independently is fine: they migrate the same source rows,
and LWW converges them.
- **Rollback story.** With no dual-mechanism period, what is the recovery path if the
cutover fails in production — beyond "revert the commit"?
**ANSWERED — "revert the commit" is genuinely the path, and it is safe because the
migration is additive and the legacy files are left intact.** Tasks 8/9 **copy** rows out
of `scadabridge.db` and `store-and-forward.db` into the consolidated LocalDb file; they do
not drop, truncate, or delete the source databases. Rolling back is therefore: stop both
nodes of the site, deploy the previous build, start both together (D5). The old code
reopens the legacy files and finds them exactly as it left them.
The bounded, honest cost of a rollback is **the delta** — rows written into the
consolidated DB after cutover do not flow back into the legacy files. For config tables
that self-heals: `SiteReconciliationActor` reports local inventory to central at startup
and fetches whatever it lacks, so a rolled-back node re-converges to central's truth
without operator action. For `sf_messages` the delta is **lost undelivered buffer** — the
practical mitigation is to drain the buffer before cutting over, which Task 20's live gate
and Task 21's runbook both call for.
What has **no** rollback is a site pair split across versions — hence D5. That is a
deployment-procedure constraint, and Task 21 puts it in the runbook rather than leaving it
as tribal knowledge.
- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly
because Phase 1's tables were small. Adding config + S&F changes the size and write
profile of that single file.
**ANSWERED — yes, and the soak is the evidence.** The write profile the consolidated file
must absorb is bounded on every axis:
| Table | Rate bound | Row size | Basis |
|---|---|---|---|
| `sf_messages` | **≤50 writes/sec** (hard) | 76 B max on the rig | `SweepBatchLimit` ÷ `RetryTimerInterval` |
| `native_alarm_state` | `distinct_source_refs × 10/sec` | small | per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`) |
| config tables | deploy-driven, effectively idle | ~721 B max on the rig | Task 1 measurement |
These are unremarkable for SQLite in WAL mode, and the soak ran the full generator load
against the Phase 1 consolidated file for 30 minutes with **zero** SQLite errors and no
oplog growth pathology. **One DB per process stays.**
Two caveats carried into the plan rather than hidden here. First, the rig's config rows
are tiny (max 721 B) and **cannot** be treated as representative — the size ceiling that
matters comes from the documented ~6070 KB production `config_json`, which is what
motivates D6 and `MaxBatchSize = 16`. Second, **D4's premise was wrong**:
`native_alarm_state` is *not* "unbounded by design" and *not* "by a wide margin the
highest-volume table" — the coalescing flush bounds it, and this rig has no alarm
generator at all (measured 0 rows), so its bound is analytic rather than observed. If a
production site ever shows alarm churn that swamps the shared oplog, the keyed-instances
hatch named in D4 remains the escape — it is simply not needed to start.
## 6. Not in scope (unchanged from the design)
`auditlog.db` (diverges per node by design — central pulls the union; replicating it would
+58 -3
View File
@@ -19,6 +19,14 @@
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
>
> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.**
>
> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based
> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages`
> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop
> condition from any of D1D6 is met. The one binding number Phase 2 must honour is
> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets.
---
@@ -268,12 +276,59 @@ rate that this run could not measure.
---
## 1b. Clean re-run (2026-07-20, post-root-cause)
The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after
both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side
`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load
unchanged (`SoakGenerator` ×4 against a refusing endpoint):
| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` |
|---|---|---|---|---|---|
| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 |
| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 |
| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 |
| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 |
| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 |
| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 |
| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 |
**Measured:**
- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six
intervals (+288 rows over 360 s, zero variance).
- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached
`retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`,
status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only.
- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json`
max **721 B** over 4 rows (rig config is trivially small — see the caveat below).
- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded.
- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load,
which is the expected result and not a measurement failure (Phase 2's tables are not yet
registered, so this load cannot reach the oplog by construction).
- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window
(`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that
Finding 1 was observer-induced: identical load, identical library, nothing host-reading the
files, no errors.
- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under
the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC.
**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural
ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never
exercised at all. This re-run therefore confirms *steady-state health and the absence of the
Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
representative — D6's sizing rests on the documented ~6070 KB production `config_json`, not on
this number.
## 2. What still owes measurement
Carry into the re-run once Finding 1 is fixed:
Carry forward (items 12 partially discharged by §1b above):
1. Sustained `sf_messages` rows/sec under a saturating generator (needs many more instances or a
shorter interval to approach the 50/s ceiling).
1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still
unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s
ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap.
2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an
A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does*
answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions