docs(localdb): close the phase 2 gate — questions answered, plan written

Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts

docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json

@@ -69,8 +69,8 @@
     "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
   ],
   "tasks": [
-    {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Phase 2 tables are NOT in the Phase 1 oplog - measure at the legacy DBs host-side, size caps arithmetically; empirical drain check moves to Task 20 evidence 10. Stop conditions: on-paper oplog overflow OR any config_json near 4 MB (D6)."},
-    {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "pending", "classification": "trivial", "blockedBy": [1]},
+    {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."},
+    {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."},
     {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]},
     {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]},
     {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]},
@@ -101,6 +101,6 @@
       "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating."
     }
   ],
-  "lastUpdated": "2026-07-19",
-  "phase2Status": "BLOCKED at the Task 1 gate (2026-07-20). Task 1 ran and its verdict is STOP - see docs/plans/2026-07-19-localdb-phase2-soak.md. The blocker is NOT oplog sizing (fine) or D6 (resolved, no stop condition): the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite Error 10 'disk I/O error' on essentially every write on the ACTIVE node under sustained load, affecting site_events + OperationTracking + audit telemetry, and silently dropping site events. Isolated to load (not node, not host-side observation) by failing over between nodes, and to LocalDb specifically (legacy store-and-forward.db/scadabridge.db in the same bind mount take ZERO errors under identical load). Phase 2 would register 8 more tables into that DB - incl. the two highest-volume - while deleting the bespoke fallbacks, so this Phase 1 defect must be root-caused and fixed before Task 3. Tasks 2-21 remain pending and untouched; no plan code has been written. Secondary defect in the same path: SiteAuditTelemetryActor closes over ActorContext across an await (NotSupportedException). Plan-premise corrections now recorded in the soak doc: D6 (largest known production config_json ~60-70 KB not >128 KB, but MaxBatchSize must drop 500 -> 16), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush), sf_messages hard ceiling 50 rows/sec (SweepBatchLimit/RetryTimerInterval), and oplog cap overrun = graceful snapshot resync, not failure. Rig caveats: ExternalSystemDefinitions id 1 still repointed to http://127.0.0.1:9 and SoakGenerator instances 5-8 still generating load - clean up before Task 20."
+  "lastUpdated": "2026-07-20",
+  "phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load."
 }