diff --git a/docs/plans/2026-07-17-localdb-design.md b/docs/plans/2026-07-17-localdb-design.md new file mode 100644 index 0000000..cebea38 --- /dev/null +++ b/docs/plans/2026-07-17-localdb-design.md @@ -0,0 +1,88 @@ +# ZB.MOM.WW.LocalDb — design + +**Date:** 2026-07-17 +**Status:** Approved (brainstorming session, all sections user-validated) + +A new shared library: a **localized embedded SQLite database cache** with **optional +bidirectional async replication with a second node via gRPC**. + +## Decisions (user-validated) + +| Question | Decision | +|---|---| +| Use case | Generic shared `ZB.MOM.WW.*` library any of the four apps can adopt; consumers define their own tables | +| Engine | Embedded local DB, **Microsoft.Data.Sqlite as the default (and only v1) provider**, behind an `ILocalDbEngine` seam so a libSQL provider can be added later without API change. (libSQL was evaluated: its only relevant wins here are encryption-at-rest and a future Turso-native sync path; its experimental .NET binding is not worth putting on the critical path.) | +| Replication | Our own protocol over gRPC between two app nodes; fully async; local writes never block on the peer. Optional — a node that never enables replication is just a fast local DB. | +| Conflict resolution | **Last-writer-wins** via hybrid logical clocks (HLC); deterministic convergence on both sides | +| API shape | **Arbitrary consumer SQL + CDC** — consumers own their tables and SQL; the lib change-captures registered tables | +| CDC mechanism | **Triggers + single shared oplog** (evaluated vs update hook / preupdate hook / session extension — triggers win on crash-consistency (same transaction, same fsync), non-bypassability, and engine portability; hook-based capture saves trigger VDBE execution but pays per-row P/Invoke marshaling, roughly a wash; the session extension is faster only because it skips the durable capture write, which a store-and-forward replicator cannot give up) | +| Replication scope | **Opt-in per table** (`RegisterReplicated`), PK required, validated at registration | +| Timestamps | **All UTC** — HLC physical component is UTC unix-milliseconds; every stored and wire timestamp is UTC; no local time anywhere in the lib | +| Name / conventions | `ZB.MOM.WW.LocalDb`; plain directory in scadaproj (not a nested repo); family conventions (Auth API-key peer auth, Telemetry meters, Configuration validators, TreatWarningsAsErrors) | + +## Packages + +| Package | Contents | +|---|---| +| `ZB.MOM.WW.LocalDb` | Embedded DB host: open/create local DB file, consumer SQL surface (`ILocalDb`: ExecuteAsync / QueryAsync / transactions), `RegisterReplicated("table")` (PK validation + trigger install + oplog schema), `HybridLogicalClock`, `AddZbLocalDb()` DI | +| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: `SyncService` (passive side), `SyncClient` + `BackgroundService` (initiator side), watermarks, LWW apply, tombstone pruning, snapshot full-resync, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()` | +| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated types (packable; peer could be non-.NET later) | + +Repo shape mirrors Secrets/Health: `src/`, `tests/`, `ZB.MOM.WW.LocalDb.slnx`, +`Directory.Build.props`, `Directory.Packages.props`; packed to the Gitea NuGet feed. + +## Data model (lib-owned schema) + +One oplog per database (not per table) — preserves cross-table transaction ordering and +keeps watermark logic trivial: + +- `__localdb_oplog` — `(seq INTEGER PK AUTOINCREMENT, table_name, pk_json, row_json NULL, hlc, node_id, is_tombstone)`. Updates store the full new row; deletes store a tombstone (`row_json = NULL`). Full-row capture keeps LWW apply idempotent and makes snapshot resync trivial. +- `__localdb_peer_state` — the peer's acked watermark (`last_acked_seq`), last-seen HLC, sync status timestamps (one row; one peer). +- `__localdb_meta` — `node_id` (GUID minted on first open), persisted HLC high-water (restarts never go backward), lib schema version. +- Per registered table: 3 AFTER triggers (INSERT / UPDATE / DELETE). Re-registering after a schema change regenerates them. Trigger bodies check a session-scoped `__localdb_applying` guard so replication applies do not re-capture. + +## Sync protocol (`localdb_sync.v1`, one bidirectional stream) + +The *initiator* node (configured with the peer address) opens a long-lived `Sync` stream; +the passive node maps the gRPC service. Both directions flow on the one stream: + +1. **Handshake** — exchange `node_id`, lib schema version, registered-table digests (name + PK-columns hash; mismatch → fail-closed typed error), and each side's last-acked watermark. +2. **Delta streaming** — each side pushes oplog batches above the peer's watermark (flush-interval + max-batch-size knobs); acks carry the applied watermark back, driving pruning. +3. **LWW apply** — per incoming row, compare incoming HLC to the local row's latest oplog HLC; newer wins (`INSERT OR REPLACE` / delete), older is discarded but still acked. Applies are batched in one transaction; watermark advances in the same transaction (idempotent replay after crash). +4. **Resync** — a reconnecting peer whose watermark predates the pruned horizon gets a snapshot: full contents of registered tables streamed chunked, then deltas resume. Same stream, distinct message types. + +Auth: `ZB.MOM.WW.Auth.ApiKeys` keyId/Bearer on the channel; TLS in prod, h2c in dev. + +## Error handling + +- **Fail-closed on schema mismatch** at handshake; no partial sync. +- **PK-less registration** → `LocalDbRegistrationException` at startup, not sync time. +- **Peer down** → capped exponential backoff; oplog growth bounded by size/age caps; when exceeded, mark peer as needing snapshot resync and prune anyway (disk safety over delta continuity). +- **Apply failures** → batch rolls back atomically, watermark does not advance; poison entries go to a dead-letter table + counter metric rather than wedging the stream (same honesty pattern as HistorianGateway store-forward). +- **Clock skew** → HLC absorbs it; configurable max-drift guard (warn, optionally fail-closed). + +## Observability + +`ZB.MOM.WW.Telemetry` meters: `localdb.oplog.depth`, `localdb.sync.lag`, applied/conflict/ +dead-letter counters. `ZB.MOM.WW.Configuration` validators on all options. + +## Performance envelope (estimates; benchmark spike in the plan) + +- Local batched writes with capture: ~25k–100k rows/sec (roughly half of no-trigger SQLite; the dominant cost is the durable oplog insert, which any crash-safe design pays). +- Single-row autocommit: ~5k–20k writes/sec (commit-cost dominated; trigger overhead marginal). +- Replicated apply: ~10k–50k rows/sec batched; sustained ingest above the peer's apply rate grows lag, never blocks local writes. +- Steady-state replication lag: flush-interval-bound, typically 50–500 ms. + +## Testing + +- Unit: HLC ordering/monotonicity (incl. restart persistence + skew), trigger SQL generation, LWW decision table, watermark/pruning, digest computation. +- Integration (offline): two real SQLite DBs in one process over an in-memory gRPC pipe — bidirectional convergence (concurrent writes → identical final state), delete/update races, offline accumulation → reconnect → convergence, prune-then-reconnect → snapshot resync. +- Property-style: random interleaved op sequences on both nodes → both DBs identical after quiesce. +- Family conventions: `TreatWarningsAsErrors`, `dotnet test` green offline, no live dependencies. + +## Explicitly out of scope (v1) + +- libSQL provider (seam exists; add later without API change) +- More than two nodes / mesh topologies +- Per-table pluggable conflict resolvers (LWW only) +- TTL/eviction (consumer concern via their own SQL)