lmxopcua/docs/plans/2026-07-20-localdb-adoption-phase2.md

# OtOpcUa LocalDb Adoption — Phase 2 Implementation Plan (alarm-historian store-and-forward)

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
>
> **Execution model:** Optimized for **Claude Opus** agents (`claude --model opus`); dispatch
> `high-risk` tasks on Opus. Branch **`feat/localdb-phase2`** in `~/Desktop/OtOpcUa` (remote
> `lmxopcua`). **Prerequisite: Phase 1 (`docs/plans/2026-07-20-localdb-adoption-phase1.md`) is
> merged (or this branch is stacked on it) and its live gate passed.** Do not merge as part of this
> plan; stop at the DoD task.
>
> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md` §3.4,
> D8, D9. Reference implementation for "replace a bespoke store" phasing: ScadaBridge Phase 2
> (`~/Desktop/ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md` + its live gate doc).

**Goal:** Move the alarm-historian store-and-forward buffer (today the standalone
`alarm-historian.db` owned by `SqliteStoreAndForwardSink`) into the consolidated LocalDb file as a
replicated table with a primary-gated drain, so a redundant pair no longer loses buffered alarm
history when a node dies — and delete the sink's bespoke file/connection management outright.

**Architecture:** `alarm_sf_events` (TEXT GUID PK) registered in `LocalDbSetup.OnReady`; the sink
rewired onto `ILocalDb` behind its unchanged public seam; drain gated on the delivered-snapshot
Primary role via `PrimaryGatePolicy` (at-least-once across failover, accepted and documented); a
one-time idempotent migrator from the legacy file, running **after** registration.

**Risk framing (from ScadaBridge):** Phase 2 replaces a *working* mechanism — a harder risk class
than Phase 1's "add where none existed." Cutover (delete + rewire) lands in **one commit**; there is
no dual-mechanism period, and the cutover is the test.

**Hard rules:** identical to Phase 1's list (OnReady ordering, no autoincrement/BLOB, fail-closed
auth already in place, no in-memory SQLite in tests, cp-triplet-only rig inspection), plus:
- **Legacy-copy column lists must INTERSECT** with what the legacy file actually has
  (`pragma_table_info` probe) — a missing column throws and readers silently discard every row.
- Absence assertions need a positive control.
- DoD greps phrased as "no references from code" (explanatory comments may survive).

---

### Task 0: Recon (produces `docs/plans/2026-07-20-localdb-phase2-recon.md`)

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Create: `docs/plans/2026-07-20-localdb-phase2-recon.md`
- Read-only: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (and the
  whole `Core.AlarmHistorian` project), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs`,
  the drain worker (whatever forwards batches to `GatewayHistorian`/`SendEvent`),
  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs`,
  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (how `_localRole` +
  driver-member-count reach the gate today)

**Record, with `file:line` citations:**
1. The sink's exact table schema: name, columns, PK type (**autoincrement?** — decides whether the
   migrator needs deterministic `mig-{node}-{legacyId}` ids), indices, and any BLOB columns
   (**STOP condition:** a BLOB payload column cannot be registered — it must become base64 TEXT in
   the new schema, and the recon must size the largest realistic payload against the 171 KB-ish
   chunk guidance; alarm events are small JSON, so expect this to be fine, but verify).

   > **PRE-ANSWERED 2026-07-21 (verified against `master` `d218282c`) — the STOP condition does NOT
   > fire.** Re-verify cheaply during recon, but do not expect a surprise:
   >
   > - **Executed DDL:** `SqliteStoreAndForwardSink.cs:657-667`. It matches the class doc-comment at
   >   `:17-26` exactly — no drift between the documented and executed schema.
   > - **No BLOB.** All 8 columns are TEXT/INTEGER; the payload column is **`PayloadJson TEXT NOT
   >   NULL`**. So no base64 conversion is needed and the chunk-size sizing is moot.
   > - **Payload is small and bounded by shape.** `PayloadJson` is a serialized
   >   `AlarmHistorianEvent` — 10 scalar fields (`AlarmId`, `EquipmentPath`, `AlarmName`,
   >   `AlarmTypeName`, `Severity`, `EventKind`, `Message`, `User`, `Comment?`, `TimestampUtc`).
   >   No collections, no nesting. Realistic worst case is low single-digit KB (operator `Comment`
   >   is the only unbounded-ish field), far under the 171 KB chunk guidance.
   > - **PK IS autoincrement** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. This confirms the
   >   plan's own prediction: LocalDb cannot replicate an autoincrement key, so the migrator
   >   **must** mint deterministic `mig-{node}-{legacyId}` ids, and the new table needs a TEXT
   >   GUID PK as already specified in the Architecture note.
   > - **One index to carry across:** `IX_Queue_Drain ON Queue (DeadLettered, RowId)` (`:667`) —
   >   the drain's covering index. The `alarm_sf_events` equivalent wants the same shape over
   >   (dead-lettered flag, insertion order) so the drain query stays index-covered.
   > - **Legacy column list for the migrator's `pragma_table_info` intersection check:**
   >   `RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError,
   >   DeadLettered`.
2. The public seam: the interface the drain worker and producers use (e.g. `IAlarmHistorianSink` /
   enqueue+dequeue+markDelivered+deadLetter methods), so the rewire can keep it byte-compatible.
3. Semantics to preserve: `Capacity` (1,000,000) enforcement, `MaxAttempts` (10), dead-letter
   retention (30 d), `BatchSize` (100), `DrainIntervalSeconds` (5) — where each lives.
4. The drain worker's lifecycle: hosted service or actor? Where a Primary-role check can be
   injected, and how the delivered-snapshot role (`RedundancyStateChanged` cache) is accessible
   from it (via `DriverHostActor`, a shared status service, or a message). If the role is only
   available inside `DriverHostActor`, note the cleanest bridge (e.g. an `IRedundancyRoleView`
   singleton the actor updates) — that becomes Task 4's shape.
5. Whether the sink is constructed per-node config path (`AlarmHistorian:DatabasePath`) anywhere
   else (tests, tooling).
6. How `AlarmHistorian:Enabled=false` short-circuits (NullAlarmHistorianSink) — the rewire must
   keep the disabled path allocating no LocalDb tables? No: tables are created unconditionally in
   `OnReady` (cheap, empty); only the sink/drain stay Null. Note this in the doc.

Commit: `docs(localdb): phase-2 recon findings`.

---

### Task 1: `alarm_sf_events` schema + registration (+ tests)

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** none (Task 2 depends on it)

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` (depends only on
  `Microsoft.Data.Sqlite`)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
- Test: extend `LocalDbSetupTests`

Schema shape (adjust column names to the recon's findings — preserve today's semantics):
```sql
CREATE TABLE IF NOT EXISTS alarm_sf_events (
    id               TEXT    NOT NULL PRIMARY KEY,   -- app-minted GUID (never autoincrement)
    payload_json     TEXT    NOT NULL,
    enqueued_at_utc  TEXT    NOT NULL,
    attempts         INTEGER NOT NULL DEFAULT 0,
    status           TEXT    NOT NULL DEFAULT 'pending',  -- pending | delivered | dead
    last_attempt_utc TEXT    NULL,
    dead_at_utc      TEXT    NULL
);
CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_status ON alarm_sf_events(status, enqueued_at_utc);
```
`OnReady` order becomes: Phase-1 DDL → `AlarmSfSchema.Apply` → the two Phase-1
`RegisterReplicated` calls → `RegisterReplicated("alarm_sf_events")` → **migrator (Task 5) last**.
(All DDL may run before all registrations; the invariant is registration-before-writes.)

TDD: failing test first — exact replicated set becomes
`["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]` (ordinal-sorted; update the
Phase-1 exact-set pins in the same commit — they are *supposed* to go red here, that's them
working). Oplog-capture test for an `alarm_sf_events` insert. Commit
`feat(localdb): alarm_sf_events replicated table`.

---

### Task 2: Rewire the sink onto `ILocalDb` + delete bespoke file management (the cutover commit, part 1 of 2 — see Task 3)

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (or replace
  with `LocalDbStoreAndForwardSink.cs` — keep the public seam identical either way)
- Modify: its registration (`AddAlarmHistorian`) to inject `ILocalDb`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs` — remove
  `DatabasePath` (a breaking config key removal: note it in the runbook/CHANGELOG task)
- Test: the sink's existing unit tests, rewired to a temp-file `ILocalDb` via `TestLocalDb`-style
  helper (create `tests/.../TestSupport` helper if none exists — real DB, never a stub: a stubbed
  bare `SqliteConnection` lacks `zb_hlc_next()` and fails closed on registered tables)

**Steps:**
1. Write/port failing tests for the seam's semantics: enqueue, drain batch of `BatchSize`,
   `MaxAttempts` → dead-letter, capacity enforcement, dead-letter retention purge.
2. Implement over `ILocalDb.ExecuteAsync/QueryAsync` (anonymous-object params). Delete the private
   connection/pragma/file-open code and any `PRAGMA journal_mode` calls (LocalDb owns pragmas).
   GUIDs minted at enqueue (`Guid.NewGuid().ToString("N")`).
3. `Capacity` enforcement: count-based insert guard (preserve today's overflow behavior per recon).
4. Core.AlarmHistorian gains a package ref on core `ZB.MOM.WW.LocalDb` (interface only).
5. Build + project tests green. **Commit together with Task 3** if the drain gate can't compile
   separately (the ScadaBridge tasks-14/15/16 circular-dependency landmine — check before assuming
   they're independent commits).

---

### Task 3: Primary-gated drain

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:** (exact shape from recon item 4)
- Modify: the drain worker
- Create (if recon says so): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs`
  — a singleton snapshot (`RedundancyRole? LocalRole`, `int DriverMemberCount`) updated by
  `DriverHostActor` where it already caches `_localRole`
- Test: drain-worker tests + an actor test pinning that `DriverHostActor` publishes role changes to
  the view

**Semantics:**
- Drain runs only when `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` is
  true — same policy, same boot-window posture (unknown role drains only when the node is alone).
- Delivered/dead-letter marks are row UPDATEs → they replicate, so the standby's copy tracks drain
  progress and does not re-deliver already-marked rows after failover.
- **At-least-once across failover is accepted:** rows delivered on the old primary whose
  `delivered` mark hadn't replicated yet will be re-sent by the new primary. Document in the
  runbook (Task 7); do NOT build dedup.
- When replication is OFF (default), the gate still applies but `driverMemberCount` for a solo
  node keeps today's behavior — verify with a test: single-node, role unknown → drains (no
  regression for unpaired deployments).

TDD: failing tests — secondary role does not drain; primary drains; unknown+alone drains;
unknown+paired does not. Commit (with Task 2 if coupled):
`feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)`.

---

### Task 4: One-time legacy migrator (`alarm-historian.db` → consolidated)

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs`
- Modify: `LocalDbSetup.OnReady` — call it **last**, after all `RegisterReplicated` calls
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AlarmSfLegacyMigratorTests.cs`

Mirror `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs`:
- Source path from the pre-removal `AlarmHistorian:DatabasePath` default (`alarm-historian.db`) —
  read the raw config key even though the option property is gone.
- `ShouldMigrate`: skip if file missing or `<file>.migrated` exists. `:memory:`/`file:` sources → no-op.
- **If the legacy PK is autoincrement (recon):** deterministic ids `mig-{NodeName}-{legacyId}`
  (node name from the recon-identified config key) — rerunnable without duplicates under
  `INSERT OR IGNORE`, and no cross-node collision. If already GUIDs, copy as-is.
- One transaction for the whole copy; `File.Move(path, path + ".migrated")` only after commit;
  failure throws out of `OnReady` → boot fails, legacy untouched.
- **Column intersection** via `pragma_table_info` on the legacy table; required-PK guard.
- Both pair nodes migrate independently; their rows have distinct ids (node-prefixed), so the
  merged buffer is the union — expected; note that the new primary will drain the standby's
  migrated rows too.

TDD: failing tests — happy path row counts; idempotent re-run; failure leaves legacy file
untouched; migrated rows **enter the oplog** (assert `__localdb_oplog` — the registration-order
pin); older legacy file missing a column still migrates (intersection). Commit
`feat(localdb): one-time alarm-historian.db migrator`.

---

### Task 5: Convergence + failover scenarios in the pair harness

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4

**Files:**
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/AlarmSfConvergenceTests.cs`
  (reuses the Phase-1 `LocalDbPairHarness`)

**Scenarios:**
1. `AlarmBurstOnA_ConvergesToB_AndOplogDrains` — enqueue N events on A; identical rowset on B;
   oplog → 0.
2. `DeliveredMarksReplicate` — mark rows delivered on A; B's copies show delivered (the
   no-redeliver-after-failover property, asserted at the data layer).
3. `WritesWhileTransportDown_SurviveRejoin`.
4. Positive control: with `RegisterReplicated("alarm_sf_events")` commented out, scenarios 1–3 go
   red (run locally, record, restore — do not commit red).

Commit `test(localdb): alarm S&F convergence scenarios`.

---

### Task 6: Rig config + docs

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 4

**Files:**
- Modify: `docker-dev/docker-compose.yml` — enable `AlarmHistorian__Enabled=true` on the site-a
  pair if not already (the gate needs a live buffer); remove any `AlarmHistorian__DatabasePath`
  env vars (key deleted).
- Modify: `docs/operations/2026-07-20-localdb-pair-replication.md` — add: alarm S&F replication
  semantics, at-least-once-across-failover statement, migrator behavior (`.migrated` sidecar),
  `DatabasePath` key removal.
- Modify: `docs/AlarmHistorian.md` + `CLAUDE.md` — sink now lives in the consolidated LocalDb;
  drain is primary-gated.

Commit `docs+chore(localdb): phase-2 rig config + docs`.

---

### Task 7: DoD sweep (offline)

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none

1. Full solution build → 0 warnings; full test suite green (deltas vs pre-branch baseline only).
2. Greps, phrased as "no references from **code**": the old bespoke connection management
   (`AlarmHistorian:DatabasePath`, direct `new SqliteConnection` inside Core.AlarmHistorian except
   via schema helpers/tests) — explanatory comments may remain.
3. Positive-control evidence from Task 5 recorded.
4. Exact-set replicated-tables pin = 3 tables, both directions.
5. Update `…phase2.md.tasks.json`; commit `chore(localdb): phase-2 DoD sweep`; STOP and report.

---

### Task 8: Live gate on the docker-dev rig (run only with explicit user go-ahead)

**Classification:** high-risk (rig)
**Estimated implement time:** ~30 min wall-clock
**Parallelizable with:** none

Same inspection rules as Phase 1's gate (cp-triplet, curl sidecar, observer-suspicion rule).
Record in `docs/plans/2026-07-20-localdb-phase2-live-gate.md`:
1. Migration ran: `.migrated` sidecar present on both site-a nodes; row counts match legacy.
2. Alarm burst (drive a real driver alarm or the historian-gateway-unreachable path) converges:
   identical rowsets, oplog drains to 0, dead letters 0.
3. Only the primary drains: stop the historian gateway egress, buffer builds on BOTH nodes'
   tables (replicated), but only the primary's drain worker logs attempts.
4. Failover: stop the primary; the standby (new primary) resumes draining the shared buffer;
   count of double-delivered events observed and recorded (at-least-once evidence, not a failure).
5. Both-nodes-together restart clean; zero `disk I/O error`/`SQLITE_IOERR` in logs.
6. site-b (default-OFF pin): sink works locally, no sync traffic.

Commit the gate doc; report; do not merge.

---

## Task persistence

Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json`. Record deviations per
task — especially if Tasks 2/3 had to land as one commit (the expected outcome).