# ScadaBridge — ZB.MOM.WW.LocalDb adoption design

**Date:** 2026-07-19
**Status:** Approved (brainstorming session, all sections user-validated)
**Library:** `ZB.MOM.WW.LocalDb` 0.1.0 (feed-published 2026-07-18; no prior app adoption)

ScadaBridge becomes the first consumer of `ZB.MOM.WW.LocalDb` (embedded SQLite cache +
optional 2-node bidirectional gRPC sync, trigger-CDC + HLC last-writer-wins). Target:
the 2-node **site cluster pair** (node-a/node-b), which today mixes a bespoke Akka
replicator (config + store-and-forward) with entirely un-replicated per-node stores
(operation tracking, site events) that lose state on failover.

## Decisions (user-validated)

| Question | Decision |
|---|---|
| Scope | **Both, phased** — Phase 1 adds replication where none exists (tracking + events); Phase 2 migrates the bespoke-replicated DBs (config + store-and-forward) onto LocalDb |
| Audit DB | **Out of scope** — `auditlog.db` diverges per-node by design; central pulls the union. Replicating it would double-forward events |
| Secrets store | Out of scope — has its own replication answer (`ZB.MOM.WW.Secrets.Replicator.SqlServer` hub mode, wired default-off) |
| Central nodes | Out of scope — SQL-Server-first; no LocalDb use case there today |
| Rollout | **Config-gated, default OFF** in the product; enabled + live-proven on the local docker cluster as part of the work (Secrets-replication precedent); production enablement is its own later decision |
| Phase 2 fate of bespoke replicator | **Replace + delete** — `SiteReplicationActor` + StoreAndForward `ReplicationService` removed outright once live-proven (recoverable in git); no dual-mechanism period |
| DB layout | **One consolidated LocalDb file** — the lib is one-DB-per-process (`AddZbLocalDb` first-registration-wins); merge replicated site state into a single file rather than extend the lib to named instances |

## Current state (inventoried 2026-07-19)

Site nodes keep six SQLite files under `/app/data`. Replication today:

| DB | Replicated? | Mechanism |
|---|---|---|
| `scadabridge.db` (site config: deployed configs, overrides, scripts, external systems, …) | Yes | Bespoke `SiteReplicationActor` (Akka, fire-and-forget LWW notify-and-fetch + chunked anti-entropy resync) |
| `store-and-forward.db` (`sf_messages` outbound buffer) | Yes | Bespoke `ReplicationService` op-shipping (Add/Remove/Park/Requeue → newest-wins upserts) via the same actor |
| `site-tracking.db` (`OperationTracking`) | **No — diverges** | — |
| `site_events.db` (`site_events`) | **No — diverges** | — |
| `auditlog.db` | No, by design | Central pulls union over gRPC |
| `scadabridge-secrets.db` | No | Secrets G-7 hub mode (separate) |

Site pair facts that matter: two nodes, active = oldest-Up Akka member, both warm;
site nodes already host Kestrel gRPC on **8083**; central and site clusters are separate
Akka clusters (irrelevant here — LocalDb sync is intra-site-pair only).

## Design

### 1. Shape

One consolidated LocalDb-managed database per site node — `LocalDb:Path` →
`/app/data/site-localdb.db` — replicated between the pair by the library. Adoption
follows the Secrets pattern:

- **Storage move ships unconditionally.** LocalDb with replication off is just a fast
  local SQLite DB; behavior-equivalent, no flag needed.
- **Replication is config-gated, default OFF** (`LocalDb:Replication:PeerAddress` unset
  ⇒ initiator idles; passive endpoint mapped but unused). Enabled in the docker rig as
  part of the definition of done.

**Phase 1 — new replication (low risk, pure win):**
`OperationTracking` + `site_events` move into the consolidated DB as `RegisterReplicated`
tables. `site-tracking.db` and `site_events.db` retire. Failover stops losing the
standby's view of op status and site events.

**Phase 2 — consolidation (starts only after Phase 1 passes the live gate):**
`scadabridge.db` config tables + `sf_messages` join the same DB. `SiteReplicationActor`
and StoreAndForward `ReplicationService` are **deleted**. CDC triggers replace
hand-shipped ops — park/requeue/remove become ordinary captured row updates/deletes —
and the lib's snapshot resync replaces the bespoke chunked anti-entropy.

### 2. Schema + migration

- **`site_events` PK change (required):** `INTEGER PRIMARY KEY AUTOINCREMENT` → TEXT
  GUID minted in `SiteEventLogger`. Two nodes minting the same auto-increment id would
  silently overwrite each other under LWW; GUID rows make events a pure union. Existing
  indexes (timestamp/type/instance/severity) carry over.
- **`OperationTracking`** (TEXT PK `TrackedOperationId`) and **`sf_messages`** (TEXT PK
  `id`) replicate as-is — no BLOB columns anywhere, so all pass `RegisterReplicated`
  validation.
- **One-time idempotent migrator** at site-node startup: after table creation +
  `RegisterReplicated` (inside the `AddZbLocalDb` `onReady`), copy rows from legacy DB
  files via `ILocalDb` inserts — **after** registration, because the lib does not
  capture or snapshot pre-registration rows; migrated rows must enter the oplog to
  replicate. Legacy `site_events` rows get fresh GUIDs. Legacy files rename to
  `*.migrated` on success.
- **Store rewiring:** `OperationTrackingStore`, `SiteEventLogger`,
  `EventLogQueryService` (Phase 2: `SiteStorageService`, `StoreAndForwardStorage`)
  consume `ILocalDb` so every write goes through the capture path. Public interfaces of
  these stores are unchanged — callers don't move.
- **LWW fit per table:** tracking = status transitions updated by whichever node runs
  the op, last update wins (correct); events = append-only GUID rows (no conflicts);
  config tables already use LWW semantics under the bespoke actor; `sf_messages` keeps
  the same bounded-duplicate race the bespoke replicator documents and accepts (N5).
- **Retention/trim jobs** run on both nodes idempotently; concurrent deletes of the
  same rows converge as tombstones.

### 3. Wiring, transport, auth

- `AddZbLocalDb` + `AddZbLocalDbReplication` registered in `SiteServiceRegistration`;
  `MapZbLocalDbSync()` on the site node's existing Kestrel gRPC host (**port 8083**,
  alongside site streaming) — no new port.
- **Roles:** node-a is the initiator — its per-node appsettings set
  `LocalDb:Replication:PeerAddress` = node-b:8083. node-b stays passive (no
  PeerAddress). Both nodes map the passive endpoint, so role assignment doesn't affect
  correctness, only who dials.
- **Auth:** initiator presents `LocalDb:Replication:ApiKey` — a `${secret:}` ref
  resolved by ScadaBridge's existing Secrets integration. The library's passive side
  deliberately does not verify keys, so ScadaBridge adds a small fixed-time-compare
  bearer interceptor on the sync endpoint. h2c in the dev rig; TLS posture matches the
  existing 8083 endpoint.
- **Observability:** the `ZB.MOM.WW.LocalDb.Replication` meter (oplog depth, sync lag,
  applied/conflict/dead-letter counters) flows through the existing `AddZbTelemetry`
  Prometheus export. `ISyncStatus` is surfaced as a site health check / dashboard
  signal.

### 4. Error handling

Inherited from the library (all verified in its 144-test suite): fail-closed handshake
on schema digest mismatch, capped-backoff reconnect, oplog row/age caps with
snapshot-resync fallback (disk safety over delta continuity), per-entry dead-letter for
poison rows, HLC drift guard. ScadaBridge-side additions: migrator failure fails
startup (no half-migrated state — legacy files only renamed after a committed copy),
and the sync-endpoint interceptor rejects unauthenticated streams before the handshake.

### 5. Testing & definition of done

- **DI-graph tests over the real Program.cs** (WebApplicationFactory, container-built)
  — the family's repeated lesson (inert Secrets Akka replicator 0.2.0, DI deadlock
  0.2.2) is that wiring bugs only show with the real host graph built.
- Migration tests: legacy files with data → consolidated DB; idempotent re-run; GUID
  re-keying; failure leaves legacy files untouched.
- 2-node convergence tests in-process (two site hosts, real sync stream): concurrent
  tracking updates, event union, offline-accumulate → reconnect. Phase 2 adds
  config-deploy and S&F park/requeue convergence, porting the bespoke replicator's
  existing test intents before deletion.
- **Live gate (docker rig):** replication enabled for one site pair; run
  `failover-drill.sh`; verify replicated tables converge across the pair and the
  standby serves complete tracking/event (Phase 2: config + S&F) data after takeover.

## Out of scope

- Central-node LocalDb use; audit DB replication; secrets store (all covered above).
- LocalDb library changes (named instances etc.) — the consolidated-file layout uses
  0.1.0 as published. If Phase 2 experience shows S&F churn fighting the shared oplog
  caps, keyed instances are the escape hatch, as its own lib effort.
- Production enablement (real key distribution, TLS) — separate decision after the
  dev-rig proof, per family convention.