docs(localdb): approved design for ZB.MOM.WW.LocalDb (embedded SQLite cache + optional 2-node bidirectional async gRPC replication)

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts

docs/plans/2026-07-17-localdb-design.md

@@ -0,0 +1,88 @@
+# ZB.MOM.WW.LocalDb — design
+
+**Date:** 2026-07-17
+**Status:** Approved (brainstorming session, all sections user-validated)
+
+A new shared library: a **localized embedded SQLite database cache** with **optional
+bidirectional async replication with a second node via gRPC**.
+
+## Decisions (user-validated)
+
+| Question | Decision |
+|---|---|
+| Use case | Generic shared `ZB.MOM.WW.*` library any of the four apps can adopt; consumers define their own tables |
+| Engine | Embedded local DB, **Microsoft.Data.Sqlite as the default (and only v1) provider**, behind an `ILocalDbEngine` seam so a libSQL provider can be added later without API change. (libSQL was evaluated: its only relevant wins here are encryption-at-rest and a future Turso-native sync path; its experimental .NET binding is not worth putting on the critical path.) |
+| Replication | Our own protocol over gRPC between two app nodes; fully async; local writes never block on the peer. Optional — a node that never enables replication is just a fast local DB. |
+| Conflict resolution | **Last-writer-wins** via hybrid logical clocks (HLC); deterministic convergence on both sides |
+| API shape | **Arbitrary consumer SQL + CDC** — consumers own their tables and SQL; the lib change-captures registered tables |
+| CDC mechanism | **Triggers + single shared oplog** (evaluated vs update hook / preupdate hook / session extension — triggers win on crash-consistency (same transaction, same fsync), non-bypassability, and engine portability; hook-based capture saves trigger VDBE execution but pays per-row P/Invoke marshaling, roughly a wash; the session extension is faster only because it skips the durable capture write, which a store-and-forward replicator cannot give up) |
+| Replication scope | **Opt-in per table** (`RegisterReplicated`), PK required, validated at registration |
+| Timestamps | **All UTC** — HLC physical component is UTC unix-milliseconds; every stored and wire timestamp is UTC; no local time anywhere in the lib |
+| Name / conventions | `ZB.MOM.WW.LocalDb`; plain directory in scadaproj (not a nested repo); family conventions (Auth API-key peer auth, Telemetry meters, Configuration validators, TreatWarningsAsErrors) |
+
+## Packages
+
+| Package | Contents |
+|---|---|
+| `ZB.MOM.WW.LocalDb` | Embedded DB host: open/create local DB file, consumer SQL surface (`ILocalDb`: ExecuteAsync / QueryAsync / transactions), `RegisterReplicated("table")` (PK validation + trigger install + oplog schema), `HybridLogicalClock`, `AddZbLocalDb()` DI |
+| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: `SyncService` (passive side), `SyncClient` + `BackgroundService` (initiator side), watermarks, LWW apply, tombstone pruning, snapshot full-resync, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()` |
+| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated types (packable; peer could be non-.NET later) |
+
+Repo shape mirrors Secrets/Health: `src/`, `tests/`, `ZB.MOM.WW.LocalDb.slnx`,
+`Directory.Build.props`, `Directory.Packages.props`; packed to the Gitea NuGet feed.
+
+## Data model (lib-owned schema)
+
+One oplog per database (not per table) — preserves cross-table transaction ordering and
+keeps watermark logic trivial:
+
+- `__localdb_oplog` — `(seq INTEGER PK AUTOINCREMENT, table_name, pk_json, row_json NULL, hlc, node_id, is_tombstone)`. Updates store the full new row; deletes store a tombstone (`row_json = NULL`). Full-row capture keeps LWW apply idempotent and makes snapshot resync trivial.
+- `__localdb_peer_state` — the peer's acked watermark (`last_acked_seq`), last-seen HLC, sync status timestamps (one row; one peer).
+- `__localdb_meta` — `node_id` (GUID minted on first open), persisted HLC high-water (restarts never go backward), lib schema version.
+- Per registered table: 3 AFTER triggers (INSERT / UPDATE / DELETE). Re-registering after a schema change regenerates them. Trigger bodies check a session-scoped `__localdb_applying` guard so replication applies do not re-capture.
+
+## Sync protocol (`localdb_sync.v1`, one bidirectional stream)
+
+The *initiator* node (configured with the peer address) opens a long-lived `Sync` stream;
+the passive node maps the gRPC service. Both directions flow on the one stream:
+
+1. **Handshake** — exchange `node_id`, lib schema version, registered-table digests (name + PK-columns hash; mismatch → fail-closed typed error), and each side's last-acked watermark.
+2. **Delta streaming** — each side pushes oplog batches above the peer's watermark (flush-interval + max-batch-size knobs); acks carry the applied watermark back, driving pruning.
+3. **LWW apply** — per incoming row, compare incoming HLC to the local row's latest oplog HLC; newer wins (`INSERT OR REPLACE` / delete), older is discarded but still acked. Applies are batched in one transaction; watermark advances in the same transaction (idempotent replay after crash).
+4. **Resync** — a reconnecting peer whose watermark predates the pruned horizon gets a snapshot: full contents of registered tables streamed chunked, then deltas resume. Same stream, distinct message types.
+
+Auth: `ZB.MOM.WW.Auth.ApiKeys` keyId/Bearer on the channel; TLS in prod, h2c in dev.
+
+## Error handling
+
+- **Fail-closed on schema mismatch** at handshake; no partial sync.
+- **PK-less registration** → `LocalDbRegistrationException` at startup, not sync time.
+- **Peer down** → capped exponential backoff; oplog growth bounded by size/age caps; when exceeded, mark peer as needing snapshot resync and prune anyway (disk safety over delta continuity).
+- **Apply failures** → batch rolls back atomically, watermark does not advance; poison entries go to a dead-letter table + counter metric rather than wedging the stream (same honesty pattern as HistorianGateway store-forward).
+- **Clock skew** → HLC absorbs it; configurable max-drift guard (warn, optionally fail-closed).
+
+## Observability
+
+`ZB.MOM.WW.Telemetry` meters: `localdb.oplog.depth`, `localdb.sync.lag`, applied/conflict/
+dead-letter counters. `ZB.MOM.WW.Configuration` validators on all options.
+
+## Performance envelope (estimates; benchmark spike in the plan)
+
+- Local batched writes with capture: ~25k–100k rows/sec (roughly half of no-trigger SQLite; the dominant cost is the durable oplog insert, which any crash-safe design pays).
+- Single-row autocommit: ~5k–20k writes/sec (commit-cost dominated; trigger overhead marginal).
+- Replicated apply: ~10k–50k rows/sec batched; sustained ingest above the peer's apply rate grows lag, never blocks local writes.
+- Steady-state replication lag: flush-interval-bound, typically 50–500 ms.
+
+## Testing
+
+- Unit: HLC ordering/monotonicity (incl. restart persistence + skew), trigger SQL generation, LWW decision table, watermark/pruning, digest computation.
+- Integration (offline): two real SQLite DBs in one process over an in-memory gRPC pipe — bidirectional convergence (concurrent writes → identical final state), delete/update races, offline accumulation → reconnect → convergence, prune-then-reconnect → snapshot resync.
+- Property-style: random interleaved op sequences on both nodes → both DBs identical after quiesce.
+- Family conventions: `TreatWarningsAsErrors`, `dotnet test` green offline, no live dependencies.
+
+## Explicitly out of scope (v1)
+
+- libSQL provider (seam exists; add later without API change)
+- More than two nodes / mesh topologies
+- Per-table pluggable conflict resolvers (LWW only)
+- TTL/eviction (consumer concern via their own SQL)