ScadaBridge/docs/plans/2026-07-20-phase2-resume-state.md

# LocalDb Phase 2 — resume state (updated 2026-07-20, pre-compaction #2)

Scratch handoff for picking the plan back up. Delete once Phase 2 lands.

**Plan:** [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
(execute with `superpowers-extended-cc:executing-plans`; task state in
`…-phase2.md.tasks.json`, which carries a per-task deviation note)
**Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged)
**Tree:** clean @ `0ad11d6b`. Build 0 warnings. 25 commits ahead of `origin/feat/localdb-phase2`.

---

## Where we are

**Tasks 1–16 are DONE. The cutover has landed. Next up: Task 17.**

| Task | State |
|---|---|
| 1–6 | done (waves 0–2) — see the git log |
| 7 `SiteLocalDbSetup` DDL | done |
| 8 `sf_messages` migrator | done |
| 9 config-table migrator | done |
| 10 S&F CDC convergence specs | done |
| 11 resync + config convergence specs | done |
| 12 active-node purge pin | done |
| 13 guarded-write scope check | done |
| **14 + 15 + 16 the CUTOVER** | **done — landed as ONE commit `037798b3`** |
| **17–21** | **not started** |

**Verification at the cutover boundary:** build 0 warnings; SiteRuntime 512, StoreAndForward 130,
Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 —
**all pass, 0 failures.** (SiteRuntime/StoreAndForward fell from 533/153 as 6 obsolete test files
were deleted.)

### The bespoke replicators are gone

`SiteReplicationActor`, `ReplicationMessages.cs`, `ReplicationService` and
`StoreAndForwardStorage.ReplaceAllAsync` are deleted. Ten tables now replicate via LocalDb CDC:
the Phase 1 pair plus `sf_messages` and the 7 site config tables.

`notification_lists` and `smtp_configurations` are created but **never registered** — permanently
empty by design, and registering them would open a standing replication channel whose only
historical payload was plaintext SMTP passwords. That decision is pinned by its own
security-named test and stated in `OnReady`.

---

## Next: Task 17 (config-key cleanup)

Partly **required**, not optional: `SiteRuntimeOptions.cs:66` still documents
`ConfigFetchRetryCount` as "Consumed by `SiteReplicationActor`", which no longer exists.

Per the plan: relax `StartupValidator.cs:113` (`SiteDbPath`) and
`StoreAndForwardOptionsValidator.cs:19-21` (`SqliteDbPath`) to migration-only; delete
`ReplicationEnabled` and `ConfigFetchRetryCount` outright with their validator rules and config
entries; **leave the two path keys present in configs** with a comment — the legacy migrator reads
them, and removing them would strand un-migrated data on a node that has not yet started once.
**10** `appsettings.Site.json` files, including `deploy/wonder-app-vd03/`.

Then 18 (two-node convergence suite), 19 (rig config — `MaxBatchSize = 16`), 20 (live gate),
21 (docs truth pass).

---

## Findings that change the plan (carry these forward)

### From Task 1 (unchanged)

- **D6's premise is wrong, its risk is real.** Largest known production `config_json` is ~60–70 KB,
  not >128 KB → no stop condition. But `MaxBatchSize=500` × 70 KB ≈ 35 MB vs a 4 MB gRPC cap →
  **Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** The one firmly evidence-backed number.
- **D4 is wrong.** `native_alarm_state` is bounded by per-`SourceReference` coalescing on a 100 ms
  flush (`NativeAlarmActor.cs:473-502`).
- **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷ `RetryTimerInterval` (10 s).
- **Task 1 step 7's stop condition is weaker than written.** Exceeding the oplog caps prunes and
  flags `needs_snapshot` → graceful resync, not data loss.

Measured (six 60 s intervals, copy-based `snap()`): `sf_messages` **0.80 rows/sec** insert, **0/sec**
retry-UPDATE, oplog 0, max `payload_json` 76 B, max `config_json` 721 B (rig — *not* representative),
zero SQLite errors in 30 min.

### From waves 1–2

- **LATENT PHASE-1 DEFECT, fixed in-app only.** The LocalDb library never creates the parent
  directory of `LocalDb:Path`, and `SqliteLocalDb` opens the file **eagerly in its constructor**, so
  a missing directory is a **hard boot failure** (`SQLite Error 14`). The default site config uses
  the **relative** `./data/site-localdb.db`; the docker rig escapes only because its volume mount
  creates `/app/data`. Fixed via `SiteLocalDbDirectory.Ensure(config)` before `AddZbLocalDb`, pinned
  by `Host.Tests/SiteLocalDbDirectoryTests`.
  **OPEN DECISION FOR THE USER:** the real fix arguably belongs in the `ZB.MOM.WW.LocalDb` library
  — **every other consumer still has this gap.** Not done here: a library change means a version
  bump and is outside this plan's scope.
- **`SiteStorageService.CreateConnection()` contract flipped** — returns an **already-open**
  connection; calling `Open`/`OpenAsync` on one **throws**.
- **LocalDb has NO in-memory mode.** Every `Mode=Memory;Cache=Shared` fixture became a real temp file.
- **A store constructor change's test blast radius is ~7× the plan's estimate** (Task 5: 40 files,
  7 projects).
- **`tests/ZB.MOM.WW.ScadaBridge.TestSupport`** holds `TestLocalDb` (dispose **before** deleting —
  the master connection anchors the WAL). Use it; don't copy fixtures.
- **Where behaviour moved owner, retarget tests — don't delete them with the code.** WAL is genuinely
  LocalDb's; directory creation is **not**. Both directions occurred.

### From waves 3–5 (new)

- **PLAN DEFECT — Tasks 14/15/16 cannot compile separately.** `SiteReplicationActor` takes a
  `ReplicationService` and calls `ReplaceAllAsync`; `DeploymentManagerActor` Tells types from
  `ReplicationMessages.cs`; `AkkaHostedService` constructs the actor. Landed as one commit, which
  also strengthens Task 14's own invariant (never both mechanisms, never neither).
- **The legacy-copy column intersection.** The plan said "migrate all 16 columns"; that would be a
  data-loss bug. An older legacy file lacks `execution_id`/`parent_execution_id`/
  `last_attempt_at_ms`, naming a missing column throws `no such column`, and the existing reader
  treats that as an unrecognised shape and **silently discards every row**. The copy intersects the
  legacy and current column sets, with a required-column (PK) guard so the tolerance cannot degrade
  into copying NULL-keyed rows.
- **`MigratorColumnLists_MatchTheLiveSchema`** (added beyond the plan). A column-list mismatch is
  invisible at runtime in BOTH directions — a typo is silently dropped by the intersection, a
  missing column silently leaves data behind. `SiteStorageTables`/`LegacyTable` are `internal` for it.
- **Absence assertions need a positive control.** `MessageAddedThenRemoved_NeverConvergesToPresent`
  PASSED with `sf_messages` unregistered entirely — an absent row is also what a pair replicating
  nothing looks like. Fixed with a control row that must converge in the same window.
- **The positional-argument hazard was REAL.** Removing `DeploymentManagerActor`'s optional
  `IActorRef? replicationActor` shifted 6 trailing optionals and 4 test call sites bound the wrong
  arguments, some with no compile error. **`Props.Create` builds an expression tree, which rejects
  OUT-OF-POSITION named arguments** — so named args work only if in position, and the rest must be
  padded positionally.
- **`ReplaceAllAsync` was unsafe to keep, not merely unused.** A mass DELETE on a now-replicated
  table would be captured and shipped to the peer.
- **`LocalDbSitePairHarness`** is the shared two-node fixture (Phase 1 + both Phase 2 suites; Task 18
  makes four). Its `Phase2ReplicatedTables` list is **literal, not derived from production code**, so
  a registration that drifts fails the tests instead of agreeing with itself.
- **Ordinal sort gotcha:** `_` (0x5F) sorts before `b` (0x62), so `data_connection_definitions`
  precedes `database_connections`.
- **Never `git checkout <file>` to undo a non-vacuity experiment** — it reverts to HEAD and takes
  in-progress work with it (cost one redo of Task 8). Use `cp` backups.

---

## Rig operational gotchas (hard-won; not obvious from the repo)

- **`ExternalSystem.Call` does NOT buffer to store-and-forward — only `ExternalSystem.CachedCall`
  does.** The single most important detail for generating S&F load.
- **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use the
  plan's `snap` helper, or `docker run --rm -v <dir>:/d alpine/sqlite3 …`.
- **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a sidecar:
  `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics`
- **Seeded template 4 (`Motor Controller`) is not deployable** — 34 validation errors. The soak used
  a purpose-built `SoakGenerator` template.
- CLI: `template script update` requires `--name` and `--trigger-type` even to change only `--code`.
  In zsh, don't put auth flags in an unquoted variable.
- Rig auth: `--url http://localhost:9000 --username multi-role --password password`.
  `LdapGroupMappings` roles must be canonical `Designer`/`Deployer`; central caches them at startup.

## Rig state as left

Reseeded and healthy; both site-a nodes restarted after the poisoning incident.

**Needs cleanup before the Task 20 live gate:**
- `ExternalSystemDefinitions` id 1 is still repointed to `http://127.0.0.1:9` — restore to
  `http://scadabridge-restapi:5200`.
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed on
  site-a and **still generating load**.

---

## Commits (this branch, newest last)

| Commit | What |
|---|---|
| `25463d52` … `d512572d` | plan review, soak, root-cause + hardening, first resume doc |
| `12eb97e0` | **Task 2** — gate CLOSED |
| `9e3239c5` | **Task 3** — `StoreAndForwardSchema.Apply` |
| `ac5eb12c` | **Task 4** — `SiteStorageSchema.Apply` |
| `f2efeb37` | **Tasks 5+6** — both stores take `ILocalDb`; latent directory defect fixed; `TestSupport` added |
| `fefbbb31` | resume state through wave 2 |
| `f8aa02e2` | **Task 7** — Phase 2 DDL in the consolidated DB (not yet registered) |
| `bdc0dffe` | **Task 8** — migrate legacy `store-and-forward.db` |
| `5ddc7eed` | **Task 9** — migrate legacy `scadabridge.db` config tables |
| `2bbe6631` | **Task 10** — S&F CDC convergence specs (+ `LocalDbSitePairHarness` extraction) |
| `c56bf4ae` | **Task 11** — resync + config convergence specs |
| `79ce5161` | **Tasks 12+13** — purge pin; guarded-write scope |
| `037798b3` | **Tasks 14+15+16 — THE CUTOVER** |
| `0ad11d6b` | task state for the cutover |

Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under
full-suite load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL
fixture, ~1 s warm). Neither reproduced during waves 1–5.

## Execution notes

- Waves ran **sequentially in the main worktree**, not as the plan's parallel worktree agents — the
  tasks were small enough that worktree setup + merge cost more than it saved, and sequential
  execution removes the git race the isolation rule exists to prevent.
- Subagents were used only in waves 1–2, for bulk mechanical fixture rewiring. **Their reported
  results were re-verified independently** — one agent's partial run had missed 2 real failures that
  a full-suite run caught. Re-run the suites yourself.
- Every new test was checked for **non-vacuity** by breaking the thing it claims to pin (removing
  DDL, removing the `Migrate` call, unregistering a table, commenting out the purge). This caught one
  genuinely vacuous test. Keep doing it.