scadaproj/docs/plans/2026-07-17-localdb-design.md

ZB.MOM.WW.LocalDb — design

Date: 2026-07-17 Status: Approved (brainstorming session, all sections user-validated)

A new shared library: a localized embedded SQLite database cache with optional bidirectional async replication with a second node via gRPC.

Decisions (user-validated)

Question	Decision
Use case	Generic shared ZB.MOM.WW.* library any of the four apps can adopt; consumers define their own tables
Engine	Embedded local DB, Microsoft.Data.Sqlite as the default (and only v1) provider, behind an ILocalDbEngine seam so a libSQL provider can be added later without API change. (libSQL was evaluated: its only relevant wins here are encryption-at-rest and a future Turso-native sync path; its experimental .NET binding is not worth putting on the critical path.)
Replication	Our own protocol over gRPC between two app nodes; fully async; local writes never block on the peer. Optional — a node that never enables replication is just a fast local DB.
Conflict resolution	Last-writer-wins via hybrid logical clocks (HLC); deterministic convergence on both sides
API shape	Arbitrary consumer SQL + CDC — consumers own their tables and SQL; the lib change-captures registered tables
CDC mechanism	Triggers + single shared oplog (evaluated vs update hook / preupdate hook / session extension — triggers win on crash-consistency (same transaction, same fsync), non-bypassability, and engine portability; hook-based capture saves trigger VDBE execution but pays per-row P/Invoke marshaling, roughly a wash; the session extension is faster only because it skips the durable capture write, which a store-and-forward replicator cannot give up)
Replication scope	Opt-in per table (RegisterReplicated), PK required, validated at registration
Timestamps	All UTC — HLC physical component is UTC unix-milliseconds; every stored and wire timestamp is UTC; no local time anywhere in the lib
Name / conventions	ZB.MOM.WW.LocalDb; plain directory in scadaproj (not a nested repo); family conventions (Auth API-key peer auth, Telemetry meters, Configuration validators, TreatWarningsAsErrors)

Packages

Package	Contents
ZB.MOM.WW.LocalDb	Embedded DB host: open/create local DB file, consumer SQL surface (ILocalDb: ExecuteAsync / QueryAsync / transactions), RegisterReplicated("table") (PK validation + trigger install + oplog schema), HybridLogicalClock, AddZbLocalDb() DI
ZB.MOM.WW.LocalDb.Replication	gRPC sync engine: SyncService (passive side), SyncClient + BackgroundService (initiator side), watermarks, LWW apply, tombstone pruning, snapshot full-resync, AddZbLocalDbReplication() / MapZbLocalDbSync()
ZB.MOM.WW.LocalDb.Contracts	localdb_sync.v1 proto + generated types (packable; peer could be non-.NET later)

Repo shape mirrors Secrets/Health: src/, tests/, ZB.MOM.WW.LocalDb.slnx, Directory.Build.props, Directory.Packages.props; packed to the Gitea NuGet feed.

Data model (lib-owned schema)

One oplog per database (not per table) — preserves cross-table transaction ordering and keeps watermark logic trivial:

• __localdb_oplog — (seq INTEGER PK AUTOINCREMENT, table_name, pk_json, row_json NULL, hlc, node_id, is_tombstone). Updates store the full new row; deletes store a tombstone (row_json = NULL). Full-row capture keeps LWW apply idempotent and makes snapshot resync trivial.
• __localdb_peer_state — the peer's acked watermark (last_acked_seq), last-seen HLC, sync status timestamps (one row; one peer).
• __localdb_meta — node_id (GUID minted on first open), persisted HLC high-water (restarts never go backward), lib schema version.
• Per registered table: 3 AFTER triggers (INSERT / UPDATE / DELETE). Re-registering after a schema change regenerates them. Trigger bodies check a session-scoped __localdb_applying guard so replication applies do not re-capture.

Sync protocol (localdb_sync.v1, one bidirectional stream)

The initiator node (configured with the peer address) opens a long-lived Sync stream; the passive node maps the gRPC service. Both directions flow on the one stream:

1. Handshake — exchange node_id, lib schema version, registered-table digests (name + PK-columns hash; mismatch → fail-closed typed error), and each side's last-acked watermark.
2. Delta streaming — each side pushes oplog batches above the peer's watermark (flush-interval + max-batch-size knobs); acks carry the applied watermark back, driving pruning.
3. LWW apply — per incoming row, compare incoming HLC to the local row's latest oplog HLC; newer wins (INSERT OR REPLACE / delete), older is discarded but still acked. Applies are batched in one transaction; watermark advances in the same transaction (idempotent replay after crash).
4. Resync — a reconnecting peer whose watermark predates the pruned horizon gets a snapshot: full contents of registered tables streamed chunked, then deltas resume. Same stream, distinct message types.

Auth: ZB.MOM.WW.Auth.ApiKeys keyId/Bearer on the channel; TLS in prod, h2c in dev.

Error handling

• Fail-closed on schema mismatch at handshake; no partial sync.
• PK-less registration → LocalDbRegistrationException at startup, not sync time.
• Peer down → capped exponential backoff; oplog growth bounded by size/age caps; when exceeded, mark peer as needing snapshot resync and prune anyway (disk safety over delta continuity).
• Apply failures → batch rolls back atomically, watermark does not advance; poison entries go to a dead-letter table + counter metric rather than wedging the stream (same honesty pattern as HistorianGateway store-forward).
• Clock skew → HLC absorbs it; configurable max-drift guard (warn, optionally fail-closed).

Observability

ZB.MOM.WW.Telemetry meters: localdb.oplog.depth, localdb.sync.lag, applied/conflict/ dead-letter counters. ZB.MOM.WW.Configuration validators on all options.

Performance envelope (estimates; benchmark spike in the plan)

• Local batched writes with capture: ~25k–100k rows/sec (roughly half of no-trigger SQLite; the dominant cost is the durable oplog insert, which any crash-safe design pays).
• Single-row autocommit: ~5k–20k writes/sec (commit-cost dominated; trigger overhead marginal).
• Replicated apply: ~10k–50k rows/sec batched; sustained ingest above the peer's apply rate grows lag, never blocks local writes.
• Steady-state replication lag: flush-interval-bound, typically 50–500 ms.

Testing

• Unit: HLC ordering/monotonicity (incl. restart persistence + skew), trigger SQL generation, LWW decision table, watermark/pruning, digest computation.
• Integration (offline): two real SQLite DBs in one process over an in-memory gRPC pipe — bidirectional convergence (concurrent writes → identical final state), delete/update races, offline accumulation → reconnect → convergence, prune-then-reconnect → snapshot resync.
• Property-style: random interleaved op sequences on both nodes → both DBs identical after quiesce.
• Family conventions: TreatWarningsAsErrors, dotnet test green offline, no live dependencies.

Explicitly out of scope (v1)

• libSQL provider (seam exists; add later without API change)
• More than two nodes / mesh topologies
• Per-table pluggable conflict resolvers (LWW only)
• TTL/eviction (consumer concern via their own SQL)

Implemented — deltas from this design

Shipped deviations from the design above (all built + tested; see ZB.MOM.WW.LocalDb/README.md):

• Engine: Microsoft.Data.Sqlite is the default behind an engine seam; libSQL deferred (already noted above).
• PK-column updates: fully supported as OLD-pk tombstone + new-row insert (delete+insert), an upgrade from the original per-design limitation.
• Clock drift: an over-drift entry is per-entry dead-lettered (when FailClosedOnDrift) rather than faulting the session — the session never dies from drift.
• Snapshot trigger: each node computes its own needs_snapshot (pruned-horizon / flag); HandshakeAck.snapshot_required is now informational, not the decision authority.
• onReady API shape: AddZbLocalDb(..., Action<ILocalDb> onReady) for table creation + RegisterReplicated, instead of a registration-builder object.
• Outbound path: a single-writer, two-lane design (unbounded control lane preempts a bounded data lane) rather than an undifferentiated write path.
• Maintenance: a dedicated MaintenanceBackgroundService (both roles) enforces oplog caps + tombstone retention on an interval.
• Snapshot sender returns the as-of oplog seq it covers, so the pump skips already-covered tail deltas.