scadaproj/ZB.MOM.WW.LocalDb/README.md

# ZB.MOM.WW.LocalDb

An embedded **SQLite local cache** with **optional bidirectional async 2-node
replication over gRPC**, for the ZB.MOM.WW SCADA / OT family. It gives an application a
fast, crash-safe local store that can optionally keep a second node in sync
asynchronously — local writes **never** block on the peer, and a node that never enables
replication is simply a fast local DB.

Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
last-writer-wins over a hybrid logical clock. All timestamps are UTC.

## Packages (all `0.2.1`)

| Package | Contents |
|---|---|
| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK validation + trigger install + oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
| `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
| `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |

> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
>
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
>   `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
>   - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
>     open is `MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc)`, and `hlc_high_water` was
>     written only by graceful disposal — while deregistration deletes the table's oplog AND ledger
>     rows. A crash after a deregister boot could therefore recover the clock below stamps the peer
>     already holds, and LWW discards those as older. The current clock is now flushed inside the
>     deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never
>     regress the anchor).
>   - **The sync inbox is bounded (64 messages) instead of unbounded**, so inbound memory is capped
>     at roughly `64 x MaxBatchBytes` rather than "whatever the peer sends" — the bad case being a
>     mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now
>     stops draining the gRPC stream, which closes the peer's flow-control window.
>   - **A snapshot this node owes is streamed alongside a running receive loop**, not from inside the
>     handshake. That is what makes bounding safe: previously neither side drained its inbox while
>     sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was
>     already safe (snapshot rows and deltas both merge through LWW, the watermark write is
>     MAX-guarded) — only the ordering changed.
>   - **Documented the cost of late baselining**: the snapshot flag is per-database, so baselining
>     one table re-streams every registered table, in both directions on a late-enabled pair. Trim
>     high-volume tables (event logs) before turning replication on late.
> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
>   All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
>   the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
>   - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and
>     row-version rows. A node that does not replicate should not be paying for capture on every
>     write, and until now there was no way to stop once a file had been registered. Idempotent, and
>     usable on triggers left behind by an earlier build.
>   - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows
>     that predate registration were invisible to the peer forever, because the snapshot streamer
>     pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at
>     an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually
>     travel. Default is still off — the documented CDC contract is unchanged unless you ask.
>   - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a
>     `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape
>     (measured 58 VM steps vs 100,020 on a 50k backlog).
>   - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by
>     summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer
>     have to tune the row count against their widest row to stay under the gRPC message limit.
>     Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
>   - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row.
>   - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside
>     every captured consumer write. Existing files lose it on the next open.
> - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN
>   writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq
>   space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's
>   watermark, the rebuilt node started its pump above it, and its first N writes were never sent.
>   Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied
>   more of our stream than we ever produced is treated as evidence that *we* were rebuilt (pump
>   restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig
>   — it only became reachable once `0.1.2` made back-fill work.
> - `0.1.2` (2026-07-21) — fixes snapshot resync for a **wiped peer**: a converged pair prunes its
>   oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer
>   that came back with an empty database and a zero watermark was never snapshotted. The gap is
>   now measured against `last_acked_seq` when the oplog is empty.
> - `0.1.1` (2026-07-20) — creates the parent directory of `LocalDb:Path` (a missing directory was
>   a hard boot failure, `SQLite Error 14`).

## Quickstart

### Local-only (no replication)

```csharp
builder.Services.AddZbLocalDb(builder.Configuration, db =>
{
    // onReady: create your own tables, then opt them into (future) replication.
    db.ExecuteAsync(
        "CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)")
      .GetAwaiter().GetResult();
    db.RegisterReplicated("widget");   // PK required; installs capture triggers
});

// Consume ILocalDb anywhere:
await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)",
    new { id = 1, n = "bolt", q = 42 });
var rows = await db.QueryAsync("SELECT name, qty FROM widget",
    r => (Name: r.GetString(0), Qty: r.GetInt32(1)));
```

`AddZbLocalDb` is first-registration-wins: a second call is a no-op. `RegisterReplicated`
installs capture triggers even without the replication package present — you can register
tables now and turn on sync later.

### Replicated (2 nodes)

Every node registers the passive side (so either can accept a stream) and the initiator
(idle unless a `PeerAddress` is set). Configure exactly **one** side with the peer's
address; the other stays passive.

```csharp
// Both nodes:
builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ });
builder.Services.AddGrpc();
builder.Services.AddZbLocalDbReplication(builder.Configuration);   // passive host + initiator + maintenance

var app = builder.Build();
app.MapZbLocalDbSync();   // passive gRPC endpoint — host must have AddGrpc()
app.Run();
```

```jsonc
// Node A — the INITIATOR: dials node B
"LocalDb": {
  "Path": "nodeA.db",
  "Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" }
}
// Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint
"LocalDb": { "Path": "nodeB.db", "Replication": { } }
```

Both directions flow on the single bidirectional stream the initiator opens; the passive
node still pushes its own deltas back. Only one side needs `PeerAddress`.

## Configuration

### `LocalDb` section (`LocalDbOptions`)

| Key | Default | Meaning |
|---|---|---|
| `Path` | `""` (required) | Filesystem path to the SQLite database file. |
| `BusyTimeoutMs` | `5000` | SQLite `busy_timeout` pragma, in milliseconds. |
| `Synchronous` | `"NORMAL"` | SQLite `synchronous` pragma (`OFF` / `NORMAL` / `FULL`). |

### `LocalDb:Replication` section (`ReplicationOptions`)

| Key | Default | Meaning |
|---|---|---|
| `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
| `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. |
| `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
| `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
| `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
| `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
| `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
| `MaxHlcDriftAhead` | `5 min` | Max tolerated HLC drift where the peer's clock runs ahead of ours. |
| `FailClosedOnDrift` | `false` | When true, an over-drift entry is dead-lettered instead of applied. |

All options are validated at startup (`ZB.MOM.WW.Configuration` validators).

## How it works

- **Capture** — three AFTER triggers per registered table write each change into one
  shared, HLC-stamped `__localdb_oplog` plus a `__localdb_row_version` ledger, in the
  consumer's own transaction. A DB-scoped `__localdb_applying` guard makes replicated
  applies skip re-capture.
- **Convergence** — last-writer-wins: compare incoming HLC, then `node_id` ordinal as the
  tie-break; the strict-greater winner is applied, the loser discarded (but still acked).
- **Streaming** — one long-lived bidirectional gRPC stream. The initiator dials; both
  sides push oplog batches above the peer's watermark and ack the applied watermark back,
  which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded
  control lane for handshake/acks that always preempts a bounded data lane for
  delta/snapshot batches), so receiving never blocks on the ability to ack.
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
  full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
  the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
  peer a snapshot. The snapshot is streamed *after* the handshake, alongside a running
  receive loop, so a node sending one still applies what the peer sends it — a mutual
  snapshot exchanges under flow control instead of being buffered whole in memory.
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
  and prunes expired tombstones every `MaintenanceInterval`.

## Semantics worth knowing

- **PK-column updates** are captured as delete-of-old-PK + insert-of-new — a tombstone on
  the old key plus a fresh insert (full delete+insert support).
- **Upserts** (`INSERT … ON CONFLICT DO UPDATE`) are fully supported (the row-version
  trigger uses an explicit `ON CONFLICT` clause so it is not rewritten by the outer
  statement's conflict algorithm).
- **Clock-drift guard** — warn-once per session when a peer entry's HLC physical time is
  more than `MaxHlcDriftAhead` ahead. With `FailClosedOnDrift`, each over-drift entry is
  dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The **session
  never dies from drift** — no livelock on reconnect.
- **Writes never block on the peer**; local commit is independent of replication.
- **All timestamps are UTC** (HLC physical component is UTC Unix milliseconds).

## Limitations (honest list)

- **Exactly two nodes.** No mesh / N-node topologies.
- **LWW only.** No per-table pluggable conflict resolvers.
- **BLOB columns are rejected at registration** (a column declared `BLOB` fails
  `RegisterReplicated`, because `json_object` cannot capture it). A *typeless* column can
  still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
  registration error.
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
  and are not snapshotted; only changes after registration are replicated. **Opt out with
  `RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
  ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync so
  they reach the peer. The floor is below every genuine stamp, so a baselined row loses
  every comparison against a real remote write and wins only where the peer has no version
  of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
  the node-id tie-break — convergent, but arbitrary as to which content survives.
- **The snapshot flag is per-database, not per-table.** Baselining one table flags a resync
  that streams **every** registered table in full, and on a pair enabling replication late
  both nodes do that to each other simultaneously — the largest transfer this library
  performs. Bounded (batch budget out, bounded inbox in), but **purge or trim high-volume
  tables such as event logs before enabling replication late**, and expect one slow first
  sync. A per-table flag needs an on-disk schema change and is not in `0.2.x`.
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
  SQLite↔SQLite peers this library targets).
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
- **Inbound buffering is bounded, not zero.** The duplex inbox holds up to 64 messages, so
  peak inbound memory is roughly `64 x MaxBatchBytes` (~128 MB at the 2 MB default, far
  less in practice — messages are only that large when the sender has that much backlog).
  Lower `MaxBatchBytes` to lower the ceiling. Beyond the bound the stream stops being
  drained and the peer is slowed by flow control.

## Observability

Metrics publish under the single meter **`ZB.MOM.WW.LocalDb.Replication`** (all instruments
null-guarded — replication runs identically whether or not metrics are collected):

| Instrument | Kind | Notes |
|---|---|---|
| `localdb.oplog.depth` | ObservableGauge | Unacked oplog backlog (polled cheaply at scrape time). |
| `localdb.sync.lag.seconds` | ObservableGauge | Seconds since the last successful sync (no measurement when never synced). |
| `localdb.sync.applied` | Counter | Rows applied from inbound **delta** batches. |
| `localdb.sync.conflicts_discarded` | Counter | Inbound rows that lost LWW. |
| `localdb.sync.dead_lettered` | Counter | Poison / over-drift rows routed to the dead-letter table. |
| `localdb.sync.reconnects` | Counter | Reconnect attempts (initiator). |
| `localdb.sync.snapshots` | Counter | Snapshots, tagged `direction=sent`/`received` (one increment per snapshot). |

**Counter split to note:** rows merged from a **snapshot** are *not* counted under
`localdb.sync.applied`; a completed snapshot instead increments `localdb.sync.snapshots`
once (with its `direction` tag). Only delta-batch rows increment `localdb.sync.applied`.

`ISyncStatus` (registered singleton) surfaces live state for dashboards / health:
`Connected` (true while **any** session is active — a node can run both roles at once),
`PeerNodeId`, `LastSyncUtc`, `OplogBacklog` (**nullable** — null means unknown because no
provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and
`ConnectionAttempts`.

## Ops notes

- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags
  `needs_snapshot` (disk safety over delta continuity); the peer then snapshot-resyncs.
- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
  its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
  registered-table digests fail-closed, so a node that drops a table its peer still
  replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
  neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
  accumulated while capture was off stay invisible to the peer — but trim high-volume
  tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's
  durable anchor is flushed in the same transaction, so a crash after a deregister boot
  cannot bring the clock back below stamps peers already hold.
- **On-disk schema upgrades** are applied in place when the file is opened, in one
  transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
  local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
  keeps syncing.
- The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
  entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
  rejects.
- **Peer auth**: `ApiKey` is sent by the initiator as a static `Authorization: Bearer …`
  header on its channel. The library's passive service does **not** itself verify the key
  — the passive host enforces whatever gRPC auth the host application configures (e.g.
  `ZB.MOM.WW.Auth`). Configure host-side authorization to actually gate inbound streams.

## Testing

188 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
snapshot over a capacity-1 transport), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
Family conventions apply (`TreatWarningsAsErrors`).

Build / test from this directory:

```bash
dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test  ZB.MOM.WW.LocalDb.slnx
```

See the design doc: [`docs/plans/2026-07-17-localdb-design.md`](../docs/plans/2026-07-17-localdb-design.md).