lmxopcua/docs/operations/2026-07-20-localdb-pair-replication.md

LocalDb pair replication — operations runbook

Scope. Every driver-role OtOpcUa node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database. It holds two things: the deployed-configuration artifact (chunked, Phase 1) so a node can boot from cache when central SQL Server is unreachable, and the alarm store-and-forward buffer (Phase 2) so a node that dies holding undelivered alarm history does not take it to the grave. Both can optionally replicate to the node's redundant pair peer over the library's gRPC sync. Replication is default-OFF and fail-closed. This runbook covers enabling it, the operational rules that keep a pair converging, and the DB-inspection safety rules.

What replicates, and what it buys you

• Replicated tables: deployment_artifacts (the artifact, split into base64 chunks), deployment_pointer (one current-deployment pointer per cluster), and alarm_sf_events (the alarm store-and-forward buffer). Registered — in this order, after the DDL — by LocalDbSetup.OnReady.
• The payoff: in a redundant driver pair, either node can be the one that applied the latest deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node that restarts into a central-SQL outage can boot the current config even if it never applied that deploy itself — its peer did, and the row replicated.
• Convergence model: last-writer-wins per primary key, HLC-stamped. Both nodes also store locally after every apply, so the pair converges to identical content regardless of which node deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes replicate as tombstones.

Alarm store-and-forward — what replication changes

Phase 2 moved the alarm buffer out of its own alarm-historian.db and into this database. Three consequences an operator needs to know:

• Only the Primary drains. The buffer replicates, so the Secondary holds a full copy of the Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and native-ack gates use, and reports DrainState = NotPrimary. A rising queue depth on a Secondary is correct. A rising queue depth on both nodes is not — see below.
• Delivery is at-least-once across a failover, by design. Rows the old Primary delivered moments before it died may not have had their deletes replicated yet, so the new Primary will re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a missing one is not. There is deliberately no dedup layer.
• The buffer is bounded. AlarmHistorian:Capacity (1,000,000 by default) still evicts the oldest undelivered rows when the historian has been unreachable long enough. Watch EvictedCount — non-zero means accepted alarm events were dropped before reaching the historian.

If BOTH nodes report NotPrimary

Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate default-DENIES while the redundancy role is unknown and a driver peer exists, so this is what a redundancy snapshot that never names either node looks like — the identity-mismatch shape DriverHostActor also warns about once ("redundancy snapshot omitted this node"). Check node identity (Cluster:PublicHostname / Cluster:Port skew) before suspecting the sink.

One-time migration from alarm-historian.db

On first boot after the upgrade, AlarmSfLegacyMigrator copies any pre-existing alarm-historian.db into alarm_sf_events and renames the file to alarm-historian.db.migrated.

• Both nodes of a pair migrate independently, and their files overlap — row ids are derived from the event payload, so the same event on both nodes converges to one row rather than duplicating. The new Primary drains the union.
• The rename happens only after the copy commits. A failure fails host startup with the legacy file untouched; the .migrated sidecar is what makes a later boot a no-op.
• A file whose shape is unrecognised is left alone — not renamed, not copied — so it stays available for inspection.
• The AlarmHistorian:DatabasePath key is removed. It is still read by the migrator, to find the legacy file; it no longer configures anything. Delete it from appsettings once the .migrated sidecar exists on every node.

Enabling replication on a pair

Replication has two knobs, both under LocalDb:

Key	Role	Notes
LocalDb:SyncListenPort	binds the dedicated h2c sync listener	0 (default) = no listener, replication off. Set the same non-zero port on both nodes of the pair.
LocalDb:Replication:PeerAddress	the address this node dials	Set on one node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions.
LocalDb:Replication:ApiKey	bearer token for the sync stream	Must be byte-identical on both nodes. See fail-closed rule below. Supply via ${secret:...} / env — never a cleartext literal in production config.
LocalDb:Replication:MaxBatchSize	secondary row cap and the DB read-page size	16 for the artifact cache. Since LocalDb 0.2.0 this no longer bounds the wire message — MaxBatchBytes does — so it is not the byte proxy the note here used to describe. Keep it at 16 anyway: OplogStore/SnapshotStreamer materialise a whole read page in memory before the byte-budget split runs, and 16 × ≈171 KB artifact chunk rows ≈ 2.7 MB bounds that transient allocation. Raising it no longer risks tripping the gRPC cap, only the allocation.
LocalDb:Replication:MaxBatchBytes	primary cap on one outbound message (bytes)	Library default 2 MB, sized under gRPC's 4 MB receive limit. Left unset on this rig. A single row over budget is sent alone rather than stalling the stream.

Library version. This repo pins ZB.MOM.WW.LocalDb / .Replication 0.3.4 (Directory.Packages.props; bumped from 0.2.1 on 2026-09-03, itself bumped from 0.1.3 on 2026-08-15). Every move in that chain needed no config or code change here: the wire protocol is unchanged end to end (0.1.x/0.2.x/0.3.x peers all still sync), and the on-disk bookkeeping schema upgrades in place on open (now v3) with no operator action on an existing node's database file. Behaviours worth knowing at the current pin: batching is byte-budgeted (the two rows above); both of 0.2.x's lifecycle APIs are now used (2026-08-18, see the box below); the snapshot-resync flag introduced by baselining is per-table, not per-database, since 0.3.0; and 0.3.2 added validated Replication:KeepAlivePingDelay/KeepAlivePingTimeout config (defaults 60s/20s, left unset on this rig).

⚠️ Registration follows this config, and flipping it is a BOTH-NODES change. Since 2026-08-18 LocalDbSetup.OnReady installs the change-capture triggers only when the node is configured to replicate — the predicate is PeerAddress OR ApiKey non-blank (the OR because the passive node has no PeerAddress; SyncListenPort is deliberately not part of it, since a port with no key is fail-closed and syncs with nobody). Consequences at enablement time:

• A configured node registers with baselineExistingRows: true, so the rows already in the file are seeded into the row-version ledger at the LWW floor (HLC 0) and actually travel. Before this, enabling replication on a node that had been running converged on subsequent writes only. ⚠️ On 0.2.1 the snapshot flag is per-database: baselining streams every registered table in full, in both directions at once. Trim a long-standing alarm_sf_events backlog before enabling replication late, and expect one slow first sync.
• An unconfigured node deregisters at boot — the capture triggers a pre-2026-08-18 build left behind are dropped and that table's oplog/row-version rows pruned. It logs one Information line naming the table count, and stays silent when there was nothing to clean.
• The sync handshake compares registered-table digests, fail-closed, so a node that deregisters a table its peer still replicates stops syncing with a schema mismatch. Set the keys on both nodes and restart both — never one. (0.2.1's HLC-anchor fix now matters here: the clock is flushed inside the deregistration transaction, so a crash right after a deregister boot cannot bring the clock below stamps the peer already holds.)

Example (site-a-1 initiator, site-a-2 passive):

site-a-1:  LocalDb__SyncListenPort=9001
           LocalDb__Replication__PeerAddress=http://site-a-2:9001
           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
           LocalDb__Replication__MaxBatchSize=16
site-a-2:  LocalDb__SyncListenPort=9001
           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
           LocalDb__Replication__MaxBatchSize=16

Kestrel note. Setting SyncListenPort makes the Host add an explicit Listen* for the h2c sync listener. An explicit Listen* makes Kestrel ignore ASPNETCORE_URLS entirely, so the Host also re-binds the primary HTTP port in the same block. If you change the primary port, verify both :<httpPort> and :<syncPort> appear in the startup "Now listening on" lines.

The ApiKey rule (fail-closed)

The replication library's passive endpoint verifies no authentication — the host LocalDbSyncAuthInterceptor is the only gate, and it is fail-closed:

• No key configured ⇒ every sync call is refused (PermissionDenied). "No key" is never "no auth required".
• A key mismatch ⇒ the pair silently stops converging. The initiator's stream is rejected at the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is broken". If a pair is not converging, check the keys match first.

Operational rules

• Stop / start the pair together where you can. Each node keeps working (and caching locally) while its peer is down; the outage is not a data-loss event — the surviving node accumulates writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its peer, so avoid leaving a pair split for extended periods.

• Tombstone-retention resurrection window. Retention prunes to the newest 2 deployments and replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window. If a node is offline longer than the tombstone-retention window, a delete that happened during its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.

• A rebuilt node is back-filled by its peer. If a node loses its LocalDb file — a wiped volume, a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the cached configuration back to it, with no new deploy and no central SQL. Two things are worth knowing: this needs ZB.MOM.WW.LocalDb ≥ 0.1.3, and it heals the cache, so the node regains boot-from-cache for future outages. (On 0.1.1 a converged pair has pruned every oplog row on ack and the wiped node was never snapshotted — it stayed empty until the next deploy. On 0.1.2 it is back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node applies a deploy.) A node wiped during a central outage is still repopulated by its peer under 0.1.2; a node with no peer replication configured self-heals only from central on its next successful apply.

• What boot-from-cache does NOT cover. The cache is a fallback for central-SQL outages at boot, not a replacement for central. A new deployment still requires central SQL — the cache is only read when the central fetch fails at startup. When central returns, the node resumes fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays on it as a central-connectivity incident, not steady state.

Inspecting the LocalDb file — safety rules

The database runs in WAL mode. These rules exist because violating them corrupted a live DB in the 2026-07-20 ScadaBridge incident:

• Never run sqlite3 on the live file (host-side, against a bind-mounted or container path). Opening a live WAL DB from a second process across virtiofs poisons the WAL. Always copy the triplet first and query the copy:

  docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
  docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
  docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
  sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
  

• Metrics come from the container, not a host curl (aspnet:10.0 has no curl):

  docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
  

• If an anomaly appears within seconds of your own measurement, suspect the measurement. Restart both nodes and re-observe untouched before blaming the library.

Useful queries (on a copied triplet)

-- What each node thinks the current deployment is
SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;

-- Convergence check: the pointer's origin stamp should match on both nodes
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;

-- Replication backlog (should drain to 0 when a pair is caught up)
SELECT COUNT(*) FROM __localdb_oplog;

-- Undelivered alarm history, and how much of it has given up
SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;

-- Why the dead-lettered rows died
SELECT alarm_id, attempt_count, last_attempt_utc, last_error
FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;

-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;

See also

• docs/Redundancy.md — the pair-local config cache section.
• docs/Configuration.md — the LocalDb appsettings section.
• docs/AlarmHistorian.md — the alarm sink, its knobs, and the drain.
• The two-node convergence harness: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/.