scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md

ScadaBridge — ZB.MOM.WW.LocalDb adoption design

Date: 2026-07-19 Status: Approved (brainstorming session, all sections user-validated) Library: ZB.MOM.WW.LocalDb 0.1.0 (feed-published 2026-07-18; no prior app adoption)

ScadaBridge becomes the first consumer of ZB.MOM.WW.LocalDb (embedded SQLite cache + optional 2-node bidirectional gRPC sync, trigger-CDC + HLC last-writer-wins). Target: the 2-node site cluster pair (node-a/node-b), which today mixes a bespoke Akka replicator (config + store-and-forward) with entirely un-replicated per-node stores (operation tracking, site events) that lose state on failover.

Decisions (user-validated)

Question	Decision
Scope	Both, phased — Phase 1 adds replication where none exists (tracking + events); Phase 2 migrates the bespoke-replicated DBs (config + store-and-forward) onto LocalDb
Audit DB	Out of scope — auditlog.db diverges per-node by design; central pulls the union. Replicating it would double-forward events
Secrets store	Out of scope — has its own replication answer (ZB.MOM.WW.Secrets.Replicator.SqlServer hub mode, wired default-off)
Central nodes	Out of scope — SQL-Server-first; no LocalDb use case there today
Rollout	Config-gated, default OFF in the product; enabled + live-proven on the local docker cluster as part of the work (Secrets-replication precedent); production enablement is its own later decision
Phase 2 fate of bespoke replicator	Replace + delete — SiteReplicationActor + StoreAndForward ReplicationService removed outright once live-proven (recoverable in git); no dual-mechanism period
DB layout	One consolidated LocalDb file — the lib is one-DB-per-process (AddZbLocalDb first-registration-wins); merge replicated site state into a single file rather than extend the lib to named instances

Current state (inventoried 2026-07-19)

Site nodes keep six SQLite files under /app/data. Replication today:

DB	Replicated?	Mechanism
scadabridge.db (site config: deployed configs, overrides, scripts, external systems, …)	Yes	Bespoke SiteReplicationActor (Akka, fire-and-forget LWW notify-and-fetch + chunked anti-entropy resync)
store-and-forward.db (sf_messages outbound buffer)	Yes	Bespoke ReplicationService op-shipping (Add/Remove/Park/Requeue → newest-wins upserts) via the same actor
site-tracking.db (OperationTracking)	No — diverges	—
site_events.db (site_events)	No — diverges	—
auditlog.db	No, by design	Central pulls union over gRPC
scadabridge-secrets.db	No	Secrets G-7 hub mode (separate)

Site pair facts that matter: two nodes, active = oldest-Up Akka member, both warm; site nodes already host Kestrel gRPC on 8083; central and site clusters are separate Akka clusters (irrelevant here — LocalDb sync is intra-site-pair only).

Design

1. Shape

One consolidated LocalDb-managed database per site node — LocalDb:Path → /app/data/site-localdb.db — replicated between the pair by the library. Adoption follows the Secrets pattern:

• Storage move ships unconditionally. LocalDb with replication off is just a fast local SQLite DB; behavior-equivalent, no flag needed.
• Replication is config-gated, default OFF (LocalDb:Replication:PeerAddress unset ⇒ initiator idles; passive endpoint mapped but unused). Enabled in the docker rig as part of the definition of done.

Phase 1 — new replication (low risk, pure win): OperationTracking + site_events move into the consolidated DB as RegisterReplicated tables. site-tracking.db and site_events.db retire. Failover stops losing the standby's view of op status and site events.

Phase 2 — consolidation (starts only after Phase 1 passes the live gate): scadabridge.db config tables + sf_messages join the same DB. SiteReplicationActor and StoreAndForward ReplicationService are deleted. CDC triggers replace hand-shipped ops — park/requeue/remove become ordinary captured row updates/deletes — and the lib's snapshot resync replaces the bespoke chunked anti-entropy.

2. Schema + migration

• site_events PK change (required): INTEGER PRIMARY KEY AUTOINCREMENT → TEXT GUID minted in SiteEventLogger. Two nodes minting the same auto-increment id would silently overwrite each other under LWW; GUID rows make events a pure union. Existing indexes (timestamp/type/instance/severity) carry over.
• OperationTracking (TEXT PK TrackedOperationId) and sf_messages (TEXT PK id) replicate as-is — no BLOB columns anywhere, so all pass RegisterReplicated validation.
• One-time idempotent migrator at site-node startup: after table creation + RegisterReplicated (inside the AddZbLocalDb onReady), copy rows from legacy DB files via ILocalDb inserts — after registration, because the lib does not capture or snapshot pre-registration rows; migrated rows must enter the oplog to replicate. Legacy site_events rows get fresh GUIDs. Legacy files rename to *.migrated on success.
• Store rewiring: OperationTrackingStore, SiteEventLogger, EventLogQueryService (Phase 2: SiteStorageService, StoreAndForwardStorage) consume ILocalDb so every write goes through the capture path. Public interfaces of these stores are unchanged — callers don't move.
• LWW fit per table: tracking = status transitions updated by whichever node runs the op, last update wins (correct); events = append-only GUID rows (no conflicts); config tables already use LWW semantics under the bespoke actor; sf_messages keeps the same bounded-duplicate race the bespoke replicator documents and accepts (N5).
• Retention/trim jobs run on both nodes idempotently; concurrent deletes of the same rows converge as tombstones.

3. Wiring, transport, auth

• AddZbLocalDb + AddZbLocalDbReplication registered in SiteServiceRegistration; MapZbLocalDbSync() on the site node's existing Kestrel gRPC host (port 8083, alongside site streaming) — no new port.
• Roles: node-a is the initiator — its per-node appsettings set LocalDb:Replication:PeerAddress = node-b:8083. node-b stays passive (no PeerAddress). Both nodes map the passive endpoint, so role assignment doesn't affect correctness, only who dials.
• Auth: initiator presents LocalDb:Replication:ApiKey — a ${secret:} ref resolved by ScadaBridge's existing Secrets integration. The library's passive side deliberately does not verify keys, so ScadaBridge adds a small fixed-time-compare bearer interceptor on the sync endpoint. h2c in the dev rig; TLS posture matches the existing 8083 endpoint.
• Observability: the ZB.MOM.WW.LocalDb.Replication meter (oplog depth, sync lag, applied/conflict/dead-letter counters) flows through the existing AddZbTelemetry Prometheus export. ISyncStatus is surfaced as a site health check / dashboard signal.

4. Error handling

Inherited from the library (all verified in its 144-test suite): fail-closed handshake on schema digest mismatch, capped-backoff reconnect, oplog row/age caps with snapshot-resync fallback (disk safety over delta continuity), per-entry dead-letter for poison rows, HLC drift guard. ScadaBridge-side additions: migrator failure fails startup (no half-migrated state — legacy files only renamed after a committed copy), and the sync-endpoint interceptor rejects unauthenticated streams before the handshake.

5. Testing & definition of done

• DI-graph tests over the real Program.cs (WebApplicationFactory, container-built) — the family's repeated lesson (inert Secrets Akka replicator 0.2.0, DI deadlock 0.2.2) is that wiring bugs only show with the real host graph built.
• Migration tests: legacy files with data → consolidated DB; idempotent re-run; GUID re-keying; failure leaves legacy files untouched.
• 2-node convergence tests in-process (two site hosts, real sync stream): concurrent tracking updates, event union, offline-accumulate → reconnect. Phase 2 adds config-deploy and S&F park/requeue convergence, porting the bespoke replicator's existing test intents before deletion.
• Live gate (docker rig): replication enabled for one site pair; run failover-drill.sh; verify replicated tables converge across the pair and the standby serves complete tracking/event (Phase 2: config + S&F) data after takeover.

Out of scope

• Central-node LocalDb use; audit DB replication; secrets store (all covered above).
• LocalDb library changes (named instances etc.) — the consolidated-file layout uses 0.1.0 as published. If Phase 2 experience shows S&F churn fighting the shared oplog caps, keyed instances are the escape hatch, as its own lib effort.
• Production enablement (real key distribution, TLS) — separate decision after the dev-rig proof, per family convention.