Files

T

Joseph Doherty d512572dac docs(localdb): phase 2 resume state — where the plan stands and what carries forward

Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts

2026-07-20 02:01:42 -04:00

5.6 KiB

Raw Blame History

LocalDb Phase 2 — resume state (written 2026-07-20, pre-compaction)

Scratch handoff for picking the plan back up. Delete once Phase 2 lands.

Plan: 2026-07-19-localdb-adoption-phase2.md (execute with superpowers-extended-cc:executing-plans; task state in …-phase2.md.tasks.json) Branch: feat/localdb-phase2 (from feat/localdb-phase1, which is not merged/pushed) Tree: clean @ 8652eab9. Build 0 warnings.

Where we are

Task 1 (the gate) is DONE. Its STOP verdict has been superseded — Phase 2 is unblocked.

Task 1 appeared to fail on a Phase 1 blocker (LocalDb throwing disk I/O error under load). That was root-caused the same day as observer-induced and not a product defect: host-side sqlite3 reads of the live bind-mounted WAL files reset the WAL under the container. The unmodified library sustains the full soak load indefinitely. Full write-up in docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md §0 (cause) and §11a (fixes shipped).

Tasks 2–21 are untouched. No plan code has been written.

Before starting Task 3

Task 2 (trivial, ~3 min) — the decision record. Blocked on Task 1's numbers, which are partly missing (below).
Re-run the cut-short sampling. The soak died before producing sustained sf_messages / native_alarm_state write rates. Use the snap helper now in the plan (Task 1 step 2, ~line 217) — copies the .db/-wal/-shm triplet and queries the copy. Never run host-side sqlite3 against a live file. Both site-a nodes must have been restarted since any host-side read (they have been — see rig state).
Then Wave 1 (Tasks 3 + 4) can be dispatched in parallel.

Findings that change the plan (carry these forward)

From 2026-07-19-localdb-phase2-soak.md:

D6's premise is wrong, its risk is real. The plan says config_json is "documented to exceed 128 KB per row." The source actually says the escaped Akka envelope blew the 128 KB frame and the raw JSON "only needs to be ~60-70 KB". So the largest known production row is ~60–70 KB → no stop condition. But MaxBatchSize=500 is unsafe (500 × 70 KB ≈ 35 MB vs a 4 MB gRPC cap) → Task 19 must set LocalDb:Replication:MaxBatchSize = 16. This is the one firmly evidence-backed number.
D4 is wrong. native_alarm_state is not "unbounded by design" — NativeAlarmActor coalesces per SourceReference on a 100 ms flush (NativeAlarmActor.cs:473-502), so worst case is distinct_source_refs × 10/sec.
sf_messages has a hard 50 rows/sec ceiling — SweepBatchLimit (500) ÷ RetryTimerInterval (10 s). A ceiling, not an estimate.
Task 1 step 7's stop condition is much weaker than written. Exceeding the oplog caps prunes and flags needs_snapshot → graceful snapshot resync (OplogStore.cs:109-138, MaintenanceBackgroundService.cs:57). Not data loss, not a wedge.

The plan's own Task-1 method also needed four corrections (metrics sidecar, safe sampling, no alarm generator on this rig, template 4 undeployable) — all recorded in the soak doc §1.

Rig operational gotchas (hard-won; not obvious from the repo)

ExternalSystem.Call does NOT buffer to store-and-forward — only ExternalSystem.CachedCall does. Using Call gives HTTP traffic and zero sf_messages rows. This is the single most important detail for generating S&F load.
Never query the bind-mounted SQLite files from the macOS host while containers run. Use the plan's snap helper, or docker run --rm -v <dir>:/d alpine/sqlite3 ….
No curl in the aspnet:10.0 image, and metrics port 8084 is not published. Use a sidecar: docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
Seeded template 4 (Motor Controller) is not deployable — 34 pre-deployment validation errors. The soak used a purpose-built SoakGenerator template instead.
CLI: template script update requires --name and --trigger-type even to change only --code. In zsh, don't put auth flags in an unquoted variable (no word-splitting).
Rig auth: --url http://localhost:9000 --username multi-role --password password. LdapGroupMappings role strings must be canonical Designer/Deployer; central caches them at startup, so restart central after seeding them.

Rig state as left

Reseeded and healthy; both site-a nodes restarted after the poisoning incident.

Needs cleanup before the Task 20 live gate:

ExternalSystemDefinitions id 1 is still repointed to http://127.0.0.1:9 — restore to http://scadabridge-restapi:5200.
Template SoakGenerator (id 2021) and instances soakgen-1..4 (ids 5–8) are still deployed on site-a and still generating load.

Commits this session

Commit	What
`25463d52`	plan review pass — corrections from code verification (D1/D3/D6, Task 1 method, Task 14)
`cf46e596`	fix(rig): `seed-sites.sh` canonical `Designer`/`Deployer` role names
`fc553bd9`	soak findings (gate verdict, since superseded)
`82c869a1`	task record: gate verdict
`e9e11d63`	root-cause brief for the disk-I/O issue
`8652eab9`	root cause + hardening: extended SQLite error codes, `SiteAuditTelemetryActor` async-context fix, `reseed.sh` init-script fix

Two known flakes, do not chase: InstanceActorChildAttributeRaceTests (intermittent under full-suite load) and ParentExecutionIdCorrelationTests (~91 s / times out on a cold MSSQL fixture, ~1 s warm).

5.6 KiB Raw Blame History Unescape Escape