lmxopcua/docs/operations/2026-07-20-localdb-pair-replication.md

# LocalDb pair replication — operations runbook

> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
> node that dies holding undelivered alarm history does not take it to the grave. Both can
> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
> rules that keep a pair converging, and the DB-inspection safety rules.

## What replicates, and what it buys you

- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
  `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
  alarm store-and-forward buffer). Registered — in this order, after the DDL — by
  `LocalDbSetup.OnReady`.
- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
  deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
  that restarts into a central-SQL outage can boot the current config **even if it never applied
  that deploy itself** — its peer did, and the row replicated.
- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store
  locally after every apply, so the pair converges to identical content regardless of which node
  deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
  replicate as tombstones.

## Alarm store-and-forward — what replication changes

Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
consequences an operator needs to know:

- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
  Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
  native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
  Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
  moments before it died may not have had their deletes replicated yet, so the new Primary will
  re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
  missing one is not. There is deliberately no dedup layer.
- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
  oldest undelivered rows when the historian has been unreachable long enough. Watch
  `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.

### If BOTH nodes report `NotPrimary`

Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
redundancy snapshot that never names either node looks like — the identity-mismatch shape
`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.

### One-time migration from `alarm-historian.db`

On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.

- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
  from the event payload, so the same event on both nodes converges to one row rather than
  duplicating. The new Primary drains the union.
- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
  file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
  available for inspection.
- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
  find the legacy file; it no longer configures anything. Delete it from appsettings once the
  `.migrated` sidecar exists on every node.

## Enabling replication on a pair

Replication has two knobs, both under `LocalDb`:

| Key | Role | Notes |
|---|---|---|
| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. |
| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. |
| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. |
| `LocalDb:Replication:MaxBatchSize` | secondary row cap **and** the DB read-page size | `16` for the artifact cache. **Since LocalDb 0.2.0 this no longer bounds the wire message** — `MaxBatchBytes` does — so it is not the byte proxy the note here used to describe. Keep it at `16` anyway: `OplogStore`/`SnapshotStreamer` materialise a whole read page in memory *before* the byte-budget split runs, and 16 × ≈171 KB artifact chunk rows ≈ 2.7 MB bounds that transient allocation. Raising it no longer risks tripping the gRPC cap, only the allocation. |
| `LocalDb:Replication:MaxBatchBytes` | primary cap on one outbound message (bytes) | Library default **2 MB**, sized under gRPC's 4 MB receive limit. Left unset on this rig. A single row over budget is sent alone rather than stalling the stream. |

> **Library version.** This repo pins **`ZB.MOM.WW.LocalDb` / `.Replication` 0.2.1**
> (`Directory.Packages.props`). The 0.1.3 → 0.2.1 move needed no config or code change here: the
> wire protocol is unchanged (0.1.x peers still sync), and the on-disk bookkeeping schema v1 → v2
> is upgraded **in place on open**, so an existing node's database file needs no operator action.
> Two behaviours changed that an operator should know: batching is now byte-budgeted (the two rows
> above), and **both of 0.2.x's lifecycle APIs are now used** (2026-08-18) — see the box below.

> **⚠️ Registration follows this config, and flipping it is a BOTH-NODES change.** Since
> 2026-08-18 `LocalDbSetup.OnReady` installs the change-capture triggers only when the node is
> configured to replicate — the predicate is **`PeerAddress` OR `ApiKey` non-blank** (the OR because
> the passive node has no `PeerAddress`; `SyncListenPort` is deliberately not part of it, since a
> port with no key is fail-closed and syncs with nobody). Consequences at enablement time:
>
> - A configured node registers with **`baselineExistingRows: true`**, so the rows **already in the
>   file** are seeded into the row-version ledger at the LWW floor (HLC `0`) and actually travel.
>   Before this, enabling replication on a node that had been running converged on *subsequent
>   writes only*. ⚠️ On 0.2.1 the snapshot flag is **per-database**: baselining streams *every*
>   registered table in full, in both directions at once. **Trim a long-standing `alarm_sf_events`
>   backlog before enabling replication late**, and expect one slow first sync.
> - An unconfigured node **deregisters** at boot — the capture triggers a pre-2026-08-18 build left
>   behind are dropped and that table's oplog/row-version rows pruned. It logs one Information line
>   naming the table count, and stays silent when there was nothing to clean.
> - The sync handshake compares **registered-table digests, fail-closed**, so a node that
>   deregisters a table its peer still replicates stops syncing with a schema mismatch. **Set the
>   keys on both nodes and restart both** — never one. (0.2.1's HLC-anchor fix now matters here:
>   the clock is flushed inside the deregistration transaction, so a crash right after a
>   deregister boot cannot bring the clock below stamps the peer already holds.)

Example (site-a-1 initiator, site-a-2 passive):

```
site-a-1:  LocalDb__SyncListenPort=9001
           LocalDb__Replication__PeerAddress=http://site-a-2:9001
           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
           LocalDb__Replication__MaxBatchSize=16
site-a-2:  LocalDb__SyncListenPort=9001
           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
           LocalDb__Replication__MaxBatchSize=16
```

> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c
> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the
> Host also re-binds the primary HTTP port in the same block. If you change the primary port,
> verify both `:<httpPort>` and `:<syncPort>` appear in the startup "Now listening on" lines.

### The ApiKey rule (fail-closed)

The replication library's passive endpoint verifies **no** authentication — the host
`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**:

- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never
  "no auth required".
- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at
  the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is
  broken". If a pair is not converging, **check the keys match first.**

## Operational rules

- **Stop / start the pair together where you can.** Each node keeps working (and caching locally)
  while its peer is down; the outage is not a data-loss event — the surviving node accumulates
  writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its
  peer, so avoid leaving a pair split for extended periods.
- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and
  replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window.
  If a node is offline **longer than the tombstone-retention window**, a delete that happened during
  its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
  briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
  a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
  cached configuration back to it, with no new deploy and no central SQL. Two things are worth
  knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
  boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
  and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
  back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
  peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
  applies a deploy.) A node
  wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
  peer replication configured self-heals only from central on its next successful apply.

- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
  boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
  is only read when the central fetch fails at startup. When central returns, the node resumes
  fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays
  on it as a central-connectivity incident, not steady state.

## Inspecting the LocalDb file — safety rules

The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in
the 2026-07-20 ScadaBridge incident:

- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path).
  Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the
  triplet first** and query the copy:

  ```bash
  docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
  docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
  docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
  sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
  ```

- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`):

  ```bash
  docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
  ```

- **If an anomaly appears within seconds of your own measurement, suspect the measurement.**
  Restart both nodes and re-observe untouched before blaming the library.

## Useful queries (on a copied triplet)

```sql
-- What each node thinks the current deployment is
SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;

-- Convergence check: the pointer's origin stamp should match on both nodes
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;

-- Replication backlog (should drain to 0 when a pair is caught up)
SELECT COUNT(*) FROM __localdb_oplog;

-- Undelivered alarm history, and how much of it has given up
SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;

-- Why the dead-lettered rows died
SELECT alarm_id, attempt_count, last_attempt_utc, last_error
FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;

-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
```

## See also

- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.