docs(localdb): phase 2 resume state — where the plan stands and what carries forward
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its STOP verdict superseded, what must happen before Task 3, the four plan-premise corrections, the rig gotchas that are not discoverable from the repo, and the rig cleanup the Task 20 live gate needs. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,106 @@
|
||||
# LocalDb Phase 2 — resume state (written 2026-07-20, pre-compaction)
|
||||
|
||||
Scratch handoff for picking the plan back up. Delete once Phase 2 lands.
|
||||
|
||||
**Plan:** [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
|
||||
(execute with `superpowers-extended-cc:executing-plans`; task state in
|
||||
`…-phase2.md.tasks.json`)
|
||||
**Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged/pushed)
|
||||
**Tree:** clean @ `8652eab9`. Build 0 warnings.
|
||||
|
||||
---
|
||||
|
||||
## Where we are
|
||||
|
||||
**Task 1 (the gate) is DONE. Its STOP verdict has been superseded — Phase 2 is unblocked.**
|
||||
|
||||
Task 1 appeared to fail on a Phase 1 blocker (LocalDb throwing `disk I/O error` under load).
|
||||
That was **root-caused the same day as observer-induced and not a product defect**: host-side
|
||||
`sqlite3` reads of the live bind-mounted WAL files reset the WAL under the container. The
|
||||
unmodified library sustains the full soak load indefinitely. Full write-up in
|
||||
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md)
|
||||
§0 (cause) and §11a (fixes shipped).
|
||||
|
||||
**Tasks 2–21 are untouched. No plan code has been written.**
|
||||
|
||||
### Before starting Task 3
|
||||
|
||||
1. **Task 2** (trivial, ~3 min) — the decision record. Blocked on Task 1's numbers, which are
|
||||
partly missing (below).
|
||||
2. **Re-run the cut-short sampling.** The soak died before producing sustained
|
||||
`sf_messages` / `native_alarm_state` write rates. Use the **`snap` helper now in the plan**
|
||||
(Task 1 step 2, ~line 217) — copies the `.db`/`-wal`/`-shm` triplet and queries the copy.
|
||||
**Never** run host-side `sqlite3` against a live file. Both site-a nodes must have been
|
||||
restarted since any host-side read (they have been — see rig state).
|
||||
3. Then Wave 1 (Tasks 3 + 4) can be dispatched in parallel.
|
||||
|
||||
---
|
||||
|
||||
## Findings that change the plan (carry these forward)
|
||||
|
||||
From [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md):
|
||||
|
||||
- **D6's premise is wrong, its risk is real.** The plan says `config_json` is "documented to
|
||||
exceed 128 KB per row." The source actually says the *escaped Akka envelope* blew the 128 KB
|
||||
frame and the raw JSON "only needs to be ~60-70 KB". So the largest known production row is
|
||||
~60–70 KB → **no stop condition**. But `MaxBatchSize=500` is unsafe (500 × 70 KB ≈ 35 MB vs a
|
||||
4 MB gRPC cap) → **Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** This is the one
|
||||
firmly evidence-backed number.
|
||||
- **D4 is wrong.** `native_alarm_state` is not "unbounded by design" — `NativeAlarmActor`
|
||||
coalesces per `SourceReference` on a 100 ms flush (`NativeAlarmActor.cs:473-502`), so
|
||||
worst case is `distinct_source_refs × 10/sec`.
|
||||
- **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷
|
||||
`RetryTimerInterval` (10 s). A ceiling, not an estimate.
|
||||
- **Task 1 step 7's stop condition is much weaker than written.** Exceeding the oplog caps
|
||||
prunes and flags `needs_snapshot` → graceful snapshot resync (`OplogStore.cs:109-138`,
|
||||
`MaintenanceBackgroundService.cs:57`). Not data loss, not a wedge.
|
||||
|
||||
The plan's own Task-1 method also needed four corrections (metrics sidecar, safe sampling,
|
||||
no alarm generator on this rig, template 4 undeployable) — all recorded in the soak doc §1.
|
||||
|
||||
---
|
||||
|
||||
## Rig operational gotchas (hard-won; not obvious from the repo)
|
||||
|
||||
- **`ExternalSystem.Call` does NOT buffer to store-and-forward — only `ExternalSystem.CachedCall`
|
||||
does.** Using `Call` gives HTTP traffic and zero `sf_messages` rows. This is the single most
|
||||
important detail for generating S&F load.
|
||||
- **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use
|
||||
the plan's `snap` helper, or `docker run --rm -v <dir>:/d alpine/sqlite3 …`.
|
||||
- **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a
|
||||
sidecar: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics`
|
||||
- **Seeded template 4 (`Motor Controller`) is not deployable** — 34 pre-deployment validation
|
||||
errors. The soak used a purpose-built `SoakGenerator` template instead.
|
||||
- CLI: `template script update` requires `--name` and `--trigger-type` even to change only
|
||||
`--code`. In zsh, don't put auth flags in an unquoted variable (no word-splitting).
|
||||
- Rig auth: `--url http://localhost:9000 --username multi-role --password password`.
|
||||
`LdapGroupMappings` role strings must be canonical `Designer`/`Deployer`; central caches them
|
||||
at startup, so restart central after seeding them.
|
||||
|
||||
---
|
||||
|
||||
## Rig state as left
|
||||
|
||||
Reseeded and healthy; both site-a nodes restarted after the poisoning incident.
|
||||
|
||||
**Needs cleanup before the Task 20 live gate:**
|
||||
- `ExternalSystemDefinitions` id 1 is still repointed to `http://127.0.0.1:9` — restore to
|
||||
`http://scadabridge-restapi:5200`.
|
||||
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed
|
||||
on site-a and **still generating load**.
|
||||
|
||||
---
|
||||
|
||||
## Commits this session
|
||||
|
||||
| Commit | What |
|
||||
|---|---|
|
||||
| `25463d52` | plan review pass — corrections from code verification (D1/D3/D6, Task 1 method, Task 14) |
|
||||
| `cf46e596` | fix(rig): `seed-sites.sh` canonical `Designer`/`Deployer` role names |
|
||||
| `fc553bd9` | soak findings (gate verdict, since superseded) |
|
||||
| `82c869a1` | task record: gate verdict |
|
||||
| `e9e11d63` | root-cause brief for the disk-I/O issue |
|
||||
| `8652eab9` | root cause + hardening: extended SQLite error codes, `SiteAuditTelemetryActor` async-context fix, `reseed.sh` init-script fix |
|
||||
|
||||
Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under full-suite
|
||||
load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL fixture, ~1 s warm).
|
||||
Reference in New Issue
Block a user