diff --git a/docs/plans/2026-07-19-scadabridge-localdb-design.md b/docs/plans/2026-07-19-scadabridge-localdb-design.md new file mode 100644 index 0000000..4c0fd15 --- /dev/null +++ b/docs/plans/2026-07-19-scadabridge-localdb-design.md @@ -0,0 +1,143 @@ +# ScadaBridge — ZB.MOM.WW.LocalDb adoption design + +**Date:** 2026-07-19 +**Status:** Approved (brainstorming session, all sections user-validated) +**Library:** `ZB.MOM.WW.LocalDb` 0.1.0 (feed-published 2026-07-18; no prior app adoption) + +ScadaBridge becomes the first consumer of `ZB.MOM.WW.LocalDb` (embedded SQLite cache + +optional 2-node bidirectional gRPC sync, trigger-CDC + HLC last-writer-wins). Target: +the 2-node **site cluster pair** (node-a/node-b), which today mixes a bespoke Akka +replicator (config + store-and-forward) with entirely un-replicated per-node stores +(operation tracking, site events) that lose state on failover. + +## Decisions (user-validated) + +| Question | Decision | +|---|---| +| Scope | **Both, phased** — Phase 1 adds replication where none exists (tracking + events); Phase 2 migrates the bespoke-replicated DBs (config + store-and-forward) onto LocalDb | +| Audit DB | **Out of scope** — `auditlog.db` diverges per-node by design; central pulls the union. Replicating it would double-forward events | +| Secrets store | Out of scope — has its own replication answer (`ZB.MOM.WW.Secrets.Replicator.SqlServer` hub mode, wired default-off) | +| Central nodes | Out of scope — SQL-Server-first; no LocalDb use case there today | +| Rollout | **Config-gated, default OFF** in the product; enabled + live-proven on the local docker cluster as part of the work (Secrets-replication precedent); production enablement is its own later decision | +| Phase 2 fate of bespoke replicator | **Replace + delete** — `SiteReplicationActor` + StoreAndForward `ReplicationService` removed outright once live-proven (recoverable in git); no dual-mechanism period | +| DB layout | **One consolidated LocalDb file** — the lib is one-DB-per-process (`AddZbLocalDb` first-registration-wins); merge replicated site state into a single file rather than extend the lib to named instances | + +## Current state (inventoried 2026-07-19) + +Site nodes keep six SQLite files under `/app/data`. Replication today: + +| DB | Replicated? | Mechanism | +|---|---|---| +| `scadabridge.db` (site config: deployed configs, overrides, scripts, external systems, …) | Yes | Bespoke `SiteReplicationActor` (Akka, fire-and-forget LWW notify-and-fetch + chunked anti-entropy resync) | +| `store-and-forward.db` (`sf_messages` outbound buffer) | Yes | Bespoke `ReplicationService` op-shipping (Add/Remove/Park/Requeue → newest-wins upserts) via the same actor | +| `site-tracking.db` (`OperationTracking`) | **No — diverges** | — | +| `site_events.db` (`site_events`) | **No — diverges** | — | +| `auditlog.db` | No, by design | Central pulls union over gRPC | +| `scadabridge-secrets.db` | No | Secrets G-7 hub mode (separate) | + +Site pair facts that matter: two nodes, active = oldest-Up Akka member, both warm; +site nodes already host Kestrel gRPC on **8083**; central and site clusters are separate +Akka clusters (irrelevant here — LocalDb sync is intra-site-pair only). + +## Design + +### 1. Shape + +One consolidated LocalDb-managed database per site node — `LocalDb:Path` → +`/app/data/site-localdb.db` — replicated between the pair by the library. Adoption +follows the Secrets pattern: + +- **Storage move ships unconditionally.** LocalDb with replication off is just a fast + local SQLite DB; behavior-equivalent, no flag needed. +- **Replication is config-gated, default OFF** (`LocalDb:Replication:PeerAddress` unset + ⇒ initiator idles; passive endpoint mapped but unused). Enabled in the docker rig as + part of the definition of done. + +**Phase 1 — new replication (low risk, pure win):** +`OperationTracking` + `site_events` move into the consolidated DB as `RegisterReplicated` +tables. `site-tracking.db` and `site_events.db` retire. Failover stops losing the +standby's view of op status and site events. + +**Phase 2 — consolidation (starts only after Phase 1 passes the live gate):** +`scadabridge.db` config tables + `sf_messages` join the same DB. `SiteReplicationActor` +and StoreAndForward `ReplicationService` are **deleted**. CDC triggers replace +hand-shipped ops — park/requeue/remove become ordinary captured row updates/deletes — +and the lib's snapshot resync replaces the bespoke chunked anti-entropy. + +### 2. Schema + migration + +- **`site_events` PK change (required):** `INTEGER PRIMARY KEY AUTOINCREMENT` → TEXT + GUID minted in `SiteEventLogger`. Two nodes minting the same auto-increment id would + silently overwrite each other under LWW; GUID rows make events a pure union. Existing + indexes (timestamp/type/instance/severity) carry over. +- **`OperationTracking`** (TEXT PK `TrackedOperationId`) and **`sf_messages`** (TEXT PK + `id`) replicate as-is — no BLOB columns anywhere, so all pass `RegisterReplicated` + validation. +- **One-time idempotent migrator** at site-node startup: after table creation + + `RegisterReplicated` (inside the `AddZbLocalDb` `onReady`), copy rows from legacy DB + files via `ILocalDb` inserts — **after** registration, because the lib does not + capture or snapshot pre-registration rows; migrated rows must enter the oplog to + replicate. Legacy `site_events` rows get fresh GUIDs. Legacy files rename to + `*.migrated` on success. +- **Store rewiring:** `OperationTrackingStore`, `SiteEventLogger`, + `EventLogQueryService` (Phase 2: `SiteStorageService`, `StoreAndForwardStorage`) + consume `ILocalDb` so every write goes through the capture path. Public interfaces of + these stores are unchanged — callers don't move. +- **LWW fit per table:** tracking = status transitions updated by whichever node runs + the op, last update wins (correct); events = append-only GUID rows (no conflicts); + config tables already use LWW semantics under the bespoke actor; `sf_messages` keeps + the same bounded-duplicate race the bespoke replicator documents and accepts (N5). +- **Retention/trim jobs** run on both nodes idempotently; concurrent deletes of the + same rows converge as tombstones. + +### 3. Wiring, transport, auth + +- `AddZbLocalDb` + `AddZbLocalDbReplication` registered in `SiteServiceRegistration`; + `MapZbLocalDbSync()` on the site node's existing Kestrel gRPC host (**port 8083**, + alongside site streaming) — no new port. +- **Roles:** node-a is the initiator — its per-node appsettings set + `LocalDb:Replication:PeerAddress` = node-b:8083. node-b stays passive (no + PeerAddress). Both nodes map the passive endpoint, so role assignment doesn't affect + correctness, only who dials. +- **Auth:** initiator presents `LocalDb:Replication:ApiKey` — a `${secret:}` ref + resolved by ScadaBridge's existing Secrets integration. The library's passive side + deliberately does not verify keys, so ScadaBridge adds a small fixed-time-compare + bearer interceptor on the sync endpoint. h2c in the dev rig; TLS posture matches the + existing 8083 endpoint. +- **Observability:** the `ZB.MOM.WW.LocalDb.Replication` meter (oplog depth, sync lag, + applied/conflict/dead-letter counters) flows through the existing `AddZbTelemetry` + Prometheus export. `ISyncStatus` is surfaced as a site health check / dashboard + signal. + +### 4. Error handling + +Inherited from the library (all verified in its 144-test suite): fail-closed handshake +on schema digest mismatch, capped-backoff reconnect, oplog row/age caps with +snapshot-resync fallback (disk safety over delta continuity), per-entry dead-letter for +poison rows, HLC drift guard. ScadaBridge-side additions: migrator failure fails +startup (no half-migrated state — legacy files only renamed after a committed copy), +and the sync-endpoint interceptor rejects unauthenticated streams before the handshake. + +### 5. Testing & definition of done + +- **DI-graph tests over the real Program.cs** (WebApplicationFactory, container-built) + — the family's repeated lesson (inert Secrets Akka replicator 0.2.0, DI deadlock + 0.2.2) is that wiring bugs only show with the real host graph built. +- Migration tests: legacy files with data → consolidated DB; idempotent re-run; GUID + re-keying; failure leaves legacy files untouched. +- 2-node convergence tests in-process (two site hosts, real sync stream): concurrent + tracking updates, event union, offline-accumulate → reconnect. Phase 2 adds + config-deploy and S&F park/requeue convergence, porting the bespoke replicator's + existing test intents before deletion. +- **Live gate (docker rig):** replication enabled for one site pair; run + `failover-drill.sh`; verify replicated tables converge across the pair and the + standby serves complete tracking/event (Phase 2: config + S&F) data after takeover. + +## Out of scope + +- Central-node LocalDb use; audit DB replication; secrets store (all covered above). +- LocalDb library changes (named instances etc.) — the consolidated-file layout uses + 0.1.0 as published. If Phase 2 experience shows S&F churn fighting the shared oplog + caps, keyed instances are the escape hatch, as its own lib effort. +- Production enablement (real key distribution, TLS) — separate decision after the + dev-rig proof, per family convention.