scadaproj/docs/plans/2026-07-17-localdb-design.md

# ZB.MOM.WW.LocalDb — design

**Date:** 2026-07-17
**Status:** Approved (brainstorming session, all sections user-validated)

A new shared library: a **localized embedded SQLite database cache** with **optional
bidirectional async replication with a second node via gRPC**.

## Decisions (user-validated)

| Question | Decision |
|---|---|
| Use case | Generic shared `ZB.MOM.WW.*` library any of the four apps can adopt; consumers define their own tables |
| Engine | Embedded local DB, **Microsoft.Data.Sqlite as the default (and only v1) provider**, behind an `ILocalDbEngine` seam so a libSQL provider can be added later without API change. (libSQL was evaluated: its only relevant wins here are encryption-at-rest and a future Turso-native sync path; its experimental .NET binding is not worth putting on the critical path.) |
| Replication | Our own protocol over gRPC between two app nodes; fully async; local writes never block on the peer. Optional — a node that never enables replication is just a fast local DB. |
| Conflict resolution | **Last-writer-wins** via hybrid logical clocks (HLC); deterministic convergence on both sides |
| API shape | **Arbitrary consumer SQL + CDC** — consumers own their tables and SQL; the lib change-captures registered tables |
| CDC mechanism | **Triggers + single shared oplog** (evaluated vs update hook / preupdate hook / session extension — triggers win on crash-consistency (same transaction, same fsync), non-bypassability, and engine portability; hook-based capture saves trigger VDBE execution but pays per-row P/Invoke marshaling, roughly a wash; the session extension is faster only because it skips the durable capture write, which a store-and-forward replicator cannot give up) |
| Replication scope | **Opt-in per table** (`RegisterReplicated`), PK required, validated at registration |
| Timestamps | **All UTC** — HLC physical component is UTC unix-milliseconds; every stored and wire timestamp is UTC; no local time anywhere in the lib |
| Name / conventions | `ZB.MOM.WW.LocalDb`; plain directory in scadaproj (not a nested repo); family conventions (Auth API-key peer auth, Telemetry meters, Configuration validators, TreatWarningsAsErrors) |

## Packages

| Package | Contents |
|---|---|
| `ZB.MOM.WW.LocalDb` | Embedded DB host: open/create local DB file, consumer SQL surface (`ILocalDb`: ExecuteAsync / QueryAsync / transactions), `RegisterReplicated("table")` (PK validation + trigger install + oplog schema), `HybridLogicalClock`, `AddZbLocalDb()` DI |
| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: `SyncService` (passive side), `SyncClient` + `BackgroundService` (initiator side), watermarks, LWW apply, tombstone pruning, snapshot full-resync, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()` |
| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated types (packable; peer could be non-.NET later) |

Repo shape mirrors Secrets/Health: `src/`, `tests/`, `ZB.MOM.WW.LocalDb.slnx`,
`Directory.Build.props`, `Directory.Packages.props`; packed to the Gitea NuGet feed.

## Data model (lib-owned schema)

One oplog per database (not per table) — preserves cross-table transaction ordering and
keeps watermark logic trivial:

- `__localdb_oplog` — `(seq INTEGER PK AUTOINCREMENT, table_name, pk_json, row_json NULL, hlc, node_id, is_tombstone)`. Updates store the full new row; deletes store a tombstone (`row_json = NULL`). Full-row capture keeps LWW apply idempotent and makes snapshot resync trivial.
- `__localdb_peer_state` — the peer's acked watermark (`last_acked_seq`), last-seen HLC, sync status timestamps (one row; one peer).
- `__localdb_meta` — `node_id` (GUID minted on first open), persisted HLC high-water (restarts never go backward), lib schema version.
- Per registered table: 3 AFTER triggers (INSERT / UPDATE / DELETE). Re-registering after a schema change regenerates them. Trigger bodies check a session-scoped `__localdb_applying` guard so replication applies do not re-capture.

## Sync protocol (`localdb_sync.v1`, one bidirectional stream)

The *initiator* node (configured with the peer address) opens a long-lived `Sync` stream;
the passive node maps the gRPC service. Both directions flow on the one stream:

1. **Handshake** — exchange `node_id`, lib schema version, registered-table digests (name + PK-columns hash; mismatch → fail-closed typed error), and each side's last-acked watermark.
2. **Delta streaming** — each side pushes oplog batches above the peer's watermark (flush-interval + max-batch-size knobs); acks carry the applied watermark back, driving pruning.
3. **LWW apply** — per incoming row, compare incoming HLC to the local row's latest oplog HLC; newer wins (`INSERT OR REPLACE` / delete), older is discarded but still acked. Applies are batched in one transaction; watermark advances in the same transaction (idempotent replay after crash).
4. **Resync** — a reconnecting peer whose watermark predates the pruned horizon gets a snapshot: full contents of registered tables streamed chunked, then deltas resume. Same stream, distinct message types.

Auth: `ZB.MOM.WW.Auth.ApiKeys` keyId/Bearer on the channel; TLS in prod, h2c in dev.

## Error handling

- **Fail-closed on schema mismatch** at handshake; no partial sync.
- **PK-less registration** → `LocalDbRegistrationException` at startup, not sync time.
- **Peer down** → capped exponential backoff; oplog growth bounded by size/age caps; when exceeded, mark peer as needing snapshot resync and prune anyway (disk safety over delta continuity).
- **Apply failures** → batch rolls back atomically, watermark does not advance; poison entries go to a dead-letter table + counter metric rather than wedging the stream (same honesty pattern as HistorianGateway store-forward).
- **Clock skew** → HLC absorbs it; configurable max-drift guard (warn, optionally fail-closed).

## Observability

`ZB.MOM.WW.Telemetry` meters: `localdb.oplog.depth`, `localdb.sync.lag`, applied/conflict/
dead-letter counters. `ZB.MOM.WW.Configuration` validators on all options.

## Performance envelope (estimates; benchmark spike in the plan)

- Local batched writes with capture: ~25k–100k rows/sec (roughly half of no-trigger SQLite; the dominant cost is the durable oplog insert, which any crash-safe design pays).
- Single-row autocommit: ~5k–20k writes/sec (commit-cost dominated; trigger overhead marginal).
- Replicated apply: ~10k–50k rows/sec batched; sustained ingest above the peer's apply rate grows lag, never blocks local writes.
- Steady-state replication lag: flush-interval-bound, typically 50–500 ms.

## Testing

- Unit: HLC ordering/monotonicity (incl. restart persistence + skew), trigger SQL generation, LWW decision table, watermark/pruning, digest computation.
- Integration (offline): two real SQLite DBs in one process over an in-memory gRPC pipe — bidirectional convergence (concurrent writes → identical final state), delete/update races, offline accumulation → reconnect → convergence, prune-then-reconnect → snapshot resync.
- Property-style: random interleaved op sequences on both nodes → both DBs identical after quiesce.
- Family conventions: `TreatWarningsAsErrors`, `dotnet test` green offline, no live dependencies.

## Explicitly out of scope (v1)

- libSQL provider (seam exists; add later without API change)
- More than two nodes / mesh topologies
- Per-table pluggable conflict resolvers (LWW only)
- TTL/eviction (consumer concern via their own SQL)