# LocalDb Phase 2 — resume state (updated 2026-07-20, pre-compaction #2) Scratch handoff for picking the plan back up. Delete once Phase 2 lands. **Plan:** [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) (execute with `superpowers-extended-cc:executing-plans`; task state in `…-phase2.md.tasks.json`, which carries a per-task deviation note) **Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged) **Tree:** clean @ `0ad11d6b`. Build 0 warnings. 25 commits ahead of `origin/feat/localdb-phase2`. --- ## Where we are **Tasks 1–16 are DONE. The cutover has landed. Next up: Task 17.** | Task | State | |---|---| | 1–6 | done (waves 0–2) — see the git log | | 7 `SiteLocalDbSetup` DDL | done | | 8 `sf_messages` migrator | done | | 9 config-table migrator | done | | 10 S&F CDC convergence specs | done | | 11 resync + config convergence specs | done | | 12 active-node purge pin | done | | 13 guarded-write scope check | done | | **14 + 15 + 16 the CUTOVER** | **done — landed as ONE commit `037798b3`** | | **17–21** | **not started** | **Verification at the cutover boundary:** build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 — **all pass, 0 failures.** (SiteRuntime/StoreAndForward fell from 533/153 as 6 obsolete test files were deleted.) ### The bespoke replicators are gone `SiteReplicationActor`, `ReplicationMessages.cs`, `ReplicationService` and `StoreAndForwardStorage.ReplaceAllAsync` are deleted. Ten tables now replicate via LocalDb CDC: the Phase 1 pair plus `sf_messages` and the 7 site config tables. `notification_lists` and `smtp_configurations` are created but **never registered** — permanently empty by design, and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. That decision is pinned by its own security-named test and stated in `OnReady`. --- ## Next: Task 17 (config-key cleanup) Partly **required**, not optional: `SiteRuntimeOptions.cs:66` still documents `ConfigFetchRetryCount` as "Consumed by `SiteReplicationActor`", which no longer exists. Per the plan: relax `StartupValidator.cs:113` (`SiteDbPath`) and `StoreAndForwardOptionsValidator.cs:19-21` (`SqliteDbPath`) to migration-only; delete `ReplicationEnabled` and `ConfigFetchRetryCount` outright with their validator rules and config entries; **leave the two path keys present in configs** with a comment — the legacy migrator reads them, and removing them would strand un-migrated data on a node that has not yet started once. **10** `appsettings.Site.json` files, including `deploy/wonder-app-vd03/`. Then 18 (two-node convergence suite), 19 (rig config — `MaxBatchSize = 16`), 20 (live gate), 21 (docs truth pass). --- ## Findings that change the plan (carry these forward) ### From Task 1 (unchanged) - **D6's premise is wrong, its risk is real.** Largest known production `config_json` is ~60–70 KB, not >128 KB → no stop condition. But `MaxBatchSize=500` × 70 KB ≈ 35 MB vs a 4 MB gRPC cap → **Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** The one firmly evidence-backed number. - **D4 is wrong.** `native_alarm_state` is bounded by per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`). - **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷ `RetryTimerInterval` (10 s). - **Task 1 step 7's stop condition is weaker than written.** Exceeding the oplog caps prunes and flags `needs_snapshot` → graceful resync, not data loss. Measured (six 60 s intervals, copy-based `snap()`): `sf_messages` **0.80 rows/sec** insert, **0/sec** retry-UPDATE, oplog 0, max `payload_json` 76 B, max `config_json` 721 B (rig — *not* representative), zero SQLite errors in 30 min. ### From waves 1–2 - **LATENT PHASE-1 DEFECT, fixed in-app only.** The LocalDb library never creates the parent directory of `LocalDb:Path`, and `SqliteLocalDb` opens the file **eagerly in its constructor**, so a missing directory is a **hard boot failure** (`SQLite Error 14`). The default site config uses the **relative** `./data/site-localdb.db`; the docker rig escapes only because its volume mount creates `/app/data`. Fixed via `SiteLocalDbDirectory.Ensure(config)` before `AddZbLocalDb`, pinned by `Host.Tests/SiteLocalDbDirectoryTests`. **OPEN DECISION FOR THE USER:** the real fix arguably belongs in the `ZB.MOM.WW.LocalDb` library — **every other consumer still has this gap.** Not done here: a library change means a version bump and is outside this plan's scope. - **`SiteStorageService.CreateConnection()` contract flipped** — returns an **already-open** connection; calling `Open`/`OpenAsync` on one **throws**. - **LocalDb has NO in-memory mode.** Every `Mode=Memory;Cache=Shared` fixture became a real temp file. - **A store constructor change's test blast radius is ~7× the plan's estimate** (Task 5: 40 files, 7 projects). - **`tests/ZB.MOM.WW.ScadaBridge.TestSupport`** holds `TestLocalDb` (dispose **before** deleting — the master connection anchors the WAL). Use it; don't copy fixtures. - **Where behaviour moved owner, retarget tests — don't delete them with the code.** WAL is genuinely LocalDb's; directory creation is **not**. Both directions occurred. ### From waves 3–5 (new) - **PLAN DEFECT — Tasks 14/15/16 cannot compile separately.** `SiteReplicationActor` takes a `ReplicationService` and calls `ReplaceAllAsync`; `DeploymentManagerActor` Tells types from `ReplicationMessages.cs`; `AkkaHostedService` constructs the actor. Landed as one commit, which also strengthens Task 14's own invariant (never both mechanisms, never neither). - **The legacy-copy column intersection.** The plan said "migrate all 16 columns"; that would be a data-loss bug. An older legacy file lacks `execution_id`/`parent_execution_id`/ `last_attempt_at_ms`, naming a missing column throws `no such column`, and the existing reader treats that as an unrecognised shape and **silently discards every row**. The copy intersects the legacy and current column sets, with a required-column (PK) guard so the tolerance cannot degrade into copying NULL-keyed rows. - **`MigratorColumnLists_MatchTheLiveSchema`** (added beyond the plan). A column-list mismatch is invisible at runtime in BOTH directions — a typo is silently dropped by the intersection, a missing column silently leaves data behind. `SiteStorageTables`/`LegacyTable` are `internal` for it. - **Absence assertions need a positive control.** `MessageAddedThenRemoved_NeverConvergesToPresent` PASSED with `sf_messages` unregistered entirely — an absent row is also what a pair replicating nothing looks like. Fixed with a control row that must converge in the same window. - **The positional-argument hazard was REAL.** Removing `DeploymentManagerActor`'s optional `IActorRef? replicationActor` shifted 6 trailing optionals and 4 test call sites bound the wrong arguments, some with no compile error. **`Props.Create` builds an expression tree, which rejects OUT-OF-POSITION named arguments** — so named args work only if in position, and the rest must be padded positionally. - **`ReplaceAllAsync` was unsafe to keep, not merely unused.** A mass DELETE on a now-replicated table would be captured and shipped to the peer. - **`LocalDbSitePairHarness`** is the shared two-node fixture (Phase 1 + both Phase 2 suites; Task 18 makes four). Its `Phase2ReplicatedTables` list is **literal, not derived from production code**, so a registration that drifts fails the tests instead of agreeing with itself. - **Ordinal sort gotcha:** `_` (0x5F) sorts before `b` (0x62), so `data_connection_definitions` precedes `database_connections`. - **Never `git checkout ` to undo a non-vacuity experiment** — it reverts to HEAD and takes in-progress work with it (cost one redo of Task 8). Use `cp` backups. --- ## Rig operational gotchas (hard-won; not obvious from the repo) - **`ExternalSystem.Call` does NOT buffer to store-and-forward — only `ExternalSystem.CachedCall` does.** The single most important detail for generating S&F load. - **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use the plan's `snap` helper, or `docker run --rm -v :/d alpine/sqlite3 …`. - **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a sidecar: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` - **Seeded template 4 (`Motor Controller`) is not deployable** — 34 validation errors. The soak used a purpose-built `SoakGenerator` template. - CLI: `template script update` requires `--name` and `--trigger-type` even to change only `--code`. In zsh, don't put auth flags in an unquoted variable. - Rig auth: `--url http://localhost:9000 --username multi-role --password password`. `LdapGroupMappings` roles must be canonical `Designer`/`Deployer`; central caches them at startup. ## Rig state as left Reseeded and healthy; both site-a nodes restarted after the poisoning incident. **Needs cleanup before the Task 20 live gate:** - `ExternalSystemDefinitions` id 1 is still repointed to `http://127.0.0.1:9` — restore to `http://scadabridge-restapi:5200`. - Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed on site-a and **still generating load**. --- ## Commits (this branch, newest last) | Commit | What | |---|---| | `25463d52` … `d512572d` | plan review, soak, root-cause + hardening, first resume doc | | `12eb97e0` | **Task 2** — gate CLOSED | | `9e3239c5` | **Task 3** — `StoreAndForwardSchema.Apply` | | `ac5eb12c` | **Task 4** — `SiteStorageSchema.Apply` | | `f2efeb37` | **Tasks 5+6** — both stores take `ILocalDb`; latent directory defect fixed; `TestSupport` added | | `fefbbb31` | resume state through wave 2 | | `f8aa02e2` | **Task 7** — Phase 2 DDL in the consolidated DB (not yet registered) | | `bdc0dffe` | **Task 8** — migrate legacy `store-and-forward.db` | | `5ddc7eed` | **Task 9** — migrate legacy `scadabridge.db` config tables | | `2bbe6631` | **Task 10** — S&F CDC convergence specs (+ `LocalDbSitePairHarness` extraction) | | `c56bf4ae` | **Task 11** — resync + config convergence specs | | `79ce5161` | **Tasks 12+13** — purge pin; guarded-write scope | | `037798b3` | **Tasks 14+15+16 — THE CUTOVER** | | `0ad11d6b` | task state for the cutover | Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under full-suite load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL fixture, ~1 s warm). Neither reproduced during waves 1–5. ## Execution notes - Waves ran **sequentially in the main worktree**, not as the plan's parallel worktree agents — the tasks were small enough that worktree setup + merge cost more than it saved, and sequential execution removes the git race the isolation rule exists to prevent. - Subagents were used only in waves 1–2, for bulk mechanical fixture rewiring. **Their reported results were re-verified independently** — one agent's partial run had missed 2 real failures that a full-suite run caught. Re-run the suites yourself. - Every new test was checked for **non-vacuity** by breaking the thing it claims to pin (removing DDL, removing the `Migrate` call, unregistering a table, commenting out the purge). This caught one genuinely vacuous test. Keep doing it.