scadaproj/ZB.MOM.WW.LocalDb

ZB.MOM.WW.LocalDb

An embedded SQLite local cache with optional bidirectional async 2-node replication over gRPC, for the ZB.MOM.WW SCADA / OT family. It gives an application a fast, crash-safe local store that can optionally keep a second node in sync asynchronously — local writes never block on the peer, and a node that never enables replication is simply a fast local DB.

Change capture uses SQLite triggers writing a single HLC-stamped oplog inside the consumer's own write transaction (crash-consistent, non-bypassable). Convergence is last-writer-wins over a hybrid logical clock. All timestamps are UTC.

Packages (all 0.2.1)

Package	Contents
ZB.MOM.WW.LocalDb	Embedded DB host: ILocalDb SQL surface (ExecuteAsync / QueryAsync / transactions), RegisterReplicated(table[, baselineExistingRows]) (PK validation + trigger install + oplog schema), DeregisterReplicated(table) (trigger removal + oplog/ledger prune), HybridLogicalClock, AddZbLocalDb() DI.
ZB.MOM.WW.LocalDb.Contracts	localdb_sync.v1 proto + generated gRPC types (packable; a future peer could be non-.NET).
ZB.MOM.WW.LocalDb.Replication	gRPC sync engine: passive LocalDbSyncService, initiating SyncBackgroundService, MaintenanceBackgroundService, LWW apply, watermark acks + pruning, snapshot resync, ISyncStatus, LocalDbMetrics, AddZbLocalDbReplication() / MapZbLocalDbSync().

Status: published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).

• 0.2.1 (2026-08-14) — two adversarial-review findings against 0.2.0. Wire-compatible with 0.1.x/0.2.0 peers; no proto change, no lib_schema_version bump, no on-disk schema change.
  • DeregisterReplicated no longer leaves the HLC without a crash anchor. Clock recovery on open is MAX(hlc_high_water, oplog HLCs, ledger HLCs, last_seen_hlc), and hlc_high_water was written only by graceful disposal — while deregistration deletes the table's oplog AND ledger rows. A crash after a deregister boot could therefore recover the clock below stamps the peer already holds, and LWW discards those as older. The current clock is now flushed inside the deregistration transaction, before the prune (and the flush is MAX-guarded, so it can never regress the anchor).
  • The sync inbox is bounded (64 messages) instead of unbounded, so inbound memory is capped at roughly 64 x MaxBatchBytes rather than "whatever the peer sends" — the bad case being a mutual full snapshot of a large dataset, buffered whole on both nodes at once. A full inbox now stops draining the gRPC stream, which closes the peer's flow-control window.
  • A snapshot this node owes is streamed alongside a running receive loop, not from inside the handshake. That is what makes bounding safe: previously neither side drained its inbox while sending, so a mutual snapshot converged only because the inbox was unbounded. Interleaving was already safe (snapshot rows and deltas both merge through LWW, the watermark write is MAX-guarded) — only the ordering changed.
  • Documented the cost of late baselining: the snapshot flag is per-database, so baselining one table re-streams every registered table, in both directions on a late-enabled pair. Trim high-volume tables (event logs) before turning replication on late.
• 0.2.0 (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review. All of it is wire-compatible with 0.1.x peers (no proto change, no lib_schema_version bump); the on-disk bookkeeping schema goes to v2 and is upgraded in place on open.
  • DeregisterReplicated(table) — drops a table's capture triggers and prunes its oplog and row-version rows. A node that does not replicate should not be paying for capture on every write, and until now there was no way to stop once a file had been registered. Idempotent, and usable on triggers left behind by an earlier build.
  • RegisterReplicated(table, baselineExistingRows: true) — closes the late-opt-in hole: rows that predate registration were invisible to the peer forever, because the snapshot streamer pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at an LWW-safe floor (HLC 0, this node's id) and flags a snapshot resync so they actually travel. Default is still off — the documented CDC contract is unchanged unless you ask.
  • O(1) backlog depth — MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1) replaces a COUNT(*) over the whole unacked backlog that ran once a minute AND on every metrics scrape (measured 58 VM steps vs 100,020 on a 50k backlog).
  • Byte-budget batching — MaxBatchBytes (default 2 MB) bounds delta and snapshot messages by summed serialized size, with MaxBatchSize demoted to a secondary row cap. Consumers no longer have to tune the row count against their widest row to stay under the gRPC message limit. Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
  • LwwApplier prepares once per statement instead of three SQL parses per applied row.
  • Dropped __localdb_oplog_hlc — an index no query ever used, costing a b-tree insert inside every captured consumer write. Existing files lose it on the next open.
• 0.1.3 (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN writes were silently dropped. last_applied_remote_seq is a watermark in the peer's seq space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's watermark, the rebuilt node started its pump above it, and its first N writes were never sent. Now the watermark is reset when the peer's node id changes, and a peer claiming to have applied more of our stream than we ever produced is treated as evidence that we were rebuilt (pump restarts from the beginning; LWW makes re-sending harmless). Found on the OtOpcUa docker-dev rig — it only became reachable once 0.1.2 made back-fill work.
• 0.1.2 (2026-07-21) — fixes snapshot resync for a wiped peer: a converged pair prunes its oplog to empty on ack, and the handshake read an empty oplog as "no gap possible", so a peer that came back with an empty database and a zero watermark was never snapshotted. The gap is now measured against last_acked_seq when the oplog is empty.
• 0.1.1 (2026-07-20) — creates the parent directory of LocalDb:Path (a missing directory was a hard boot failure, SQLite Error 14).

Quickstart

Local-only (no replication)

builder.Services.AddZbLocalDb(builder.Configuration, db =>
{
    // onReady: create your own tables, then opt them into (future) replication.
    db.ExecuteAsync(
        "CREATE TABLE IF NOT EXISTS widget (id INTEGER PRIMARY KEY, name TEXT, qty INTEGER)")
      .GetAwaiter().GetResult();
    db.RegisterReplicated("widget");   // PK required; installs capture triggers
});

// Consume ILocalDb anywhere:
await db.ExecuteAsync("INSERT INTO widget (id, name, qty) VALUES (@id, @n, @q)",
    new { id = 1, n = "bolt", q = 42 });
var rows = await db.QueryAsync("SELECT name, qty FROM widget",
    r => (Name: r.GetString(0), Qty: r.GetInt32(1)));

AddZbLocalDb is first-registration-wins: a second call is a no-op. RegisterReplicated installs capture triggers even without the replication package present — you can register tables now and turn on sync later.

Replicated (2 nodes)

Every node registers the passive side (so either can accept a stream) and the initiator (idle unless a PeerAddress is set). Configure exactly one side with the peer's address; the other stays passive.

// Both nodes:
builder.Services.AddZbLocalDb(builder.Configuration, db => { /* create + RegisterReplicated */ });
builder.Services.AddGrpc();
builder.Services.AddZbLocalDbReplication(builder.Configuration);   // passive host + initiator + maintenance

var app = builder.Build();
app.MapZbLocalDbSync();   // passive gRPC endpoint — host must have AddGrpc()
app.Run();

// Node A — the INITIATOR: dials node B
"LocalDb": {
  "Path": "nodeA.db",
  "Replication": { "PeerAddress": "https://nodeB:5001", "ApiKey": "…" }
}
// Node B — PASSIVE: no PeerAddress, its SyncBackgroundService idles and only serves the endpoint
"LocalDb": { "Path": "nodeB.db", "Replication": { } }

Both directions flow on the single bidirectional stream the initiator opens; the passive node still pushes its own deltas back. Only one side needs PeerAddress.

Configuration

LocalDb section (LocalDbOptions)

Key	Default	Meaning
Path	"" (required)	Filesystem path to the SQLite database file.
BusyTimeoutMs	5000	SQLite busy_timeout pragma, in milliseconds.
Synchronous	"NORMAL"	SQLite synchronous pragma (OFF / NORMAL / FULL).

LocalDb:Replication section (ReplicationOptions)

Key	Default	Meaning
PeerAddress	""	gRPC address to dial. Empty ⇒ passive (initiator idles).
ApiKey	null	Bearer token the initiator presents on its channel; null ⇒ no key header.
FlushInterval	250 ms	How often the change pump flushes pending oplog deltas.
MaxBatchBytes	2 MB	Primary batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone.
MaxBatchSize	500	Secondary row-count cap per outbound batch (also the snapshot DB page size).
MaxOplogRows	1_000_000	Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows.
MaxOplogAge	7 days	Age ceiling; an unpruned entry older than this flags a snapshot resync.
TombstoneRetention	7 days	How long deleted-row tombstones are kept before pruning.
ReconnectBackoffMax	60 s	Upper bound on exponential reconnect backoff (validator caps ≤ 1 day).
MaintenanceInterval	60 s	How often maintenance enforces caps + prunes expired tombstones.
MaxHlcDriftAhead	5 min	Max tolerated HLC drift where the peer's clock runs ahead of ours.
FailClosedOnDrift	false	When true, an over-drift entry is dead-lettered instead of applied.

All options are validated at startup (ZB.MOM.WW.Configuration validators).

How it works

• Capture — three AFTER triggers per registered table write each change into one shared, HLC-stamped __localdb_oplog plus a __localdb_row_version ledger, in the consumer's own transaction. A DB-scoped __localdb_applying guard makes replicated applies skip re-capture.
• Convergence — last-writer-wins: compare incoming HLC, then node_id ordinal as the tie-break; the strict-greater winner is applied, the loser discarded (but still acked).
• Streaming — one long-lived bidirectional gRPC stream. The initiator dials; both sides push oplog batches above the peer's watermark and ack the applied watermark back, which drives pruning. Outbound uses a single writer fed by two lanes (an unbounded control lane for handshake/acks that always preempts a bounded data lane for delta/snapshot batches), so receiving never blocks on the ability to ack.
• Resync — when a peer's watermark falls below the pruned oplog horizon, it is sent a full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above the snapshot's as-of seq. Each node decides from its OWN state whether it owes the peer a snapshot. The snapshot is streamed after the handshake, alongside a running receive loop, so a node sending one still applies what the peer sends it — a mutual snapshot exchanges under flow control instead of being buffered whole in memory.
• Maintenance — a background service on both roles enforces the oplog row/age caps and prunes expired tombstones every MaintenanceInterval.

Semantics worth knowing

• PK-column updates are captured as delete-of-old-PK + insert-of-new — a tombstone on the old key plus a fresh insert (full delete+insert support).
• Upserts (INSERT … ON CONFLICT DO UPDATE) are fully supported (the row-version trigger uses an explicit ON CONFLICT clause so it is not rewritten by the outer statement's conflict algorithm).
• Clock-drift guard — warn-once per session when a peer entry's HLC physical time is more than MaxHlcDriftAhead ahead. With FailClosedOnDrift, each over-drift entry is dead-lettered; otherwise it is applied (the HLC merge absorbs the skew). The session never dies from drift — no livelock on reconnect.
• Writes never block on the peer; local commit is independent of replication.
• All timestamps are UTC (HLC physical component is UTC Unix milliseconds).

Limitations (honest list)

• Exactly two nodes. No mesh / N-node topologies.
• LWW only. No per-table pluggable conflict resolvers.
• BLOB columns are rejected at registration (a column declared BLOB fails RegisterReplicated, because json_object cannot capture it). A typeless column can still hold a BLOB value at runtime, which then surfaces as a write-time error, not a registration error.
• CDC semantics — rows that existed before RegisterReplicated are not captured and are not snapshotted; only changes after registration are replicated. Opt out with RegisterReplicated(table, baselineExistingRows: true), which seeds the row-version ledger for existing rows at the baseline floor (HLC 0) and flags a snapshot resync so they reach the peer. The floor is below every genuine stamp, so a baselined row loses every comparison against a real remote write and wins only where the peer has no version of that key at all. Two nodes that baseline the same key both hold HLC 0 and resolve by the node-id tie-break — convergent, but arbitrary as to which content survives.
• The snapshot flag is per-database, not per-table. Baselining one table flags a resync that streams every registered table in full, and on a pair enabling replication late both nodes do that to each other simultaneously — the largest transfer this library performs. Bounded (batch budget out, bounded inbox in), but purge or trim high-volume tables such as event logs before enabling replication late, and expect one slow first sync. A per-table flag needs an on-disk schema change and is not in 0.2.x.
• REAL ±Infinity is carried in SQLite's JSON dialect on the wire (fine for the SQLite↔SQLite peers this library targets).
• AddZbLocalDb / AddZbLocalDbReplication are first-registration-wins (idempotent).
• Inbound buffering is bounded, not zero. The duplex inbox holds up to 64 messages, so peak inbound memory is roughly 64 x MaxBatchBytes (~128 MB at the 2 MB default, far less in practice — messages are only that large when the sender has that much backlog). Lower MaxBatchBytes to lower the ceiling. Beyond the bound the stream stops being drained and the peer is slowed by flow control.

Observability

Metrics publish under the single meter ZB.MOM.WW.LocalDb.Replication (all instruments null-guarded — replication runs identically whether or not metrics are collected):

Instrument	Kind	Notes
localdb.oplog.depth	ObservableGauge	Unacked oplog backlog (polled cheaply at scrape time).
localdb.sync.lag.seconds	ObservableGauge	Seconds since the last successful sync (no measurement when never synced).
localdb.sync.applied	Counter	Rows applied from inbound delta batches.
localdb.sync.conflicts_discarded	Counter	Inbound rows that lost LWW.
localdb.sync.dead_lettered	Counter	Poison / over-drift rows routed to the dead-letter table.
localdb.sync.reconnects	Counter	Reconnect attempts (initiator).
localdb.sync.snapshots	Counter	Snapshots, tagged direction=sent/received (one increment per snapshot).

Counter split to note: rows merged from a snapshot are not counted under localdb.sync.applied; a completed snapshot instead increments localdb.sync.snapshots once (with its direction tag). Only delta-batch rows increment localdb.sync.applied.

ISyncStatus (registered singleton) surfaces live state for dashboards / health: Connected (true while any session is active — a node can run both roles at once), PeerNodeId, LastSyncUtc, OplogBacklog (nullable — null means unknown because no provider is wired or the poll threw, so a DB failure never reads as "0 backlog"), and ConnectionAttempts.

Ops notes

• When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags needs_snapshot (disk safety over delta continuity); the peer then snapshot-resyncs.
• Turning a table off — DeregisterReplicated(table) drops its capture triggers and prunes its oplog + ledger rows. It must be applied symmetrically: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on neither. Re-enabling later should pass baselineExistingRows: true, or the rows that accumulated while capture was off stay invisible to the peer — but trim high-volume tables first (see Limitations: baselining flags a full, all-tables snapshot). The HLC's durable anchor is flushed in the same transaction, so a crash after a deregister boot cannot bring the clock back below stamps peers already hold.
• On-disk schema upgrades are applied in place when the file is opened, in one transaction, and every step is safe to re-run. schema_version in __localdb_meta is local-file state only — it is NOT the wire lib_schema_version, so a pair mid-upgrade keeps syncing.
• The __localdb_dead_letter table holds poison rows (bad JSON, null-row non-tombstone entries) and over-drift rejects (when FailClosedOnDrift). Inspect it to diagnose rejects.
• Peer auth: ApiKey is sent by the initiator as a static Authorization: Bearer … header on its channel. The library's passive service does not itself verify the key — the passive host enforces whatever gRPC auth the host application configures (e.g. ZB.MOM.WW.Auth). Configure host-side authorization to actually gate inbound streams.

Testing

188 tests, all offline / no live dependencies — unit (HLC ordering + restart persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL generation, LWW decision table, watermark/pruning, digest computation, schema-version upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence, delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual snapshot over a capacity-1 transport), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce). Family conventions apply (TreatWarningsAsErrors).

Build / test from this directory:

dotnet build ZB.MOM.WW.LocalDb.slnx
dotnet test  ZB.MOM.WW.LocalDb.slnx

See the design doc: docs/plans/2026-07-17-localdb-design.md.