Files

T

Joseph Doherty f587e7972e docs(localdb): live-gate the two Phase 1 follow-ups; pin LocalDb 0.1.3

The gate closed both follow-ups on the docker-dev rig and found a third defect
that offline tests could not have found first: it lived behind the one being
fixed. Once 0.1.2 made back-fill work, a rebuilt node could be observed writing
for the first time — and its writes went nowhere, because the healthy node kept
the old peer's seq watermark. Library 0.1.1 -> 0.1.2 -> 0.1.3 over the course
of the gate; both consumers now pinned to 0.1.3.

Check 4 (was PARTIAL, now PASS): with SQL stopped and site-a-2's LocalDb volume
destroyed, a-1 logged "Snapshot sent (as_of_seq 0, 4 rows)" — a line that could
not appear on 0.1.1 — and a-2 came back with a row_version dump byte-identical
to a-1's, carrying a-1's origin node ids and its ORIGINAL applied_at_utc. It
then booted from cache and served 17 ns=2 nodes, diff-identical to its peer,
from a configuration it never applied and could not have fetched.

Check 8 (PASS): two restarts with an unchanged artifact left the oplog at 7 and
the pointer timestamp frozen — the re-cache was skipped outright. Positive
control: a real config change still wrote, 7 -> 10 -> 13.

Also records one unrelated pre-existing finding the gate's SQL flapping
surfaced: a transient ConfigDb error during artifact load empties the served
address space while logging "rebuild becomes no-op". The cache layer correctly
refused to store the bad artifact and the node recovered on restart, but the
two logs disagree and it is the same class as the Phase 1 gate's check-3
defect. Untouched by this work; wants its own issue.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

2026-07-21 02:14:00 -04:00

8.1 KiB

Raw Blame History

LocalDb pair replication — operations runbook

Phase 1 scope. Every driver-role OtOpcUa node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database. Today it caches exactly one thing: the deployed-configuration artifact, chunked, so a node can boot from cache when central SQL Server is unreachable. That cache can optionally replicate to the node's redundant pair peer over the library's gRPC sync. Replication is default-OFF and fail-closed. This runbook covers enabling it, the operational rules that keep a pair converging, and the DB-inspection safety rules.

What replicates, and what it buys you

Replicated tables: deployment_artifacts (the artifact, split into base64 chunks) and deployment_pointer (one current-deployment pointer per cluster). Registered — in this order, after the DDL — by LocalDbSetup.OnReady.
The payoff: in a redundant driver pair, either node can be the one that applied the latest deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node that restarts into a central-SQL outage can boot the current config even if it never applied that deploy itself — its peer did, and the row replicated.
Convergence model: last-writer-wins per primary key, HLC-stamped. Both nodes also store locally after every apply, so the pair converges to identical content regardless of which node deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes replicate as tombstones.

Enabling replication on a pair

Replication has two knobs, both under LocalDb:

Key	Role	Notes
`LocalDb:SyncListenPort`	binds the dedicated h2c sync listener	`0` (default) = no listener, replication off. Set the same non-zero port on both nodes of the pair.
`LocalDb:Replication:PeerAddress`	the address this node dials	Set on one node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions.
`LocalDb:Replication:ApiKey`	bearer token for the sync stream	Must be byte-identical on both nodes. See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config.
`LocalDb:Replication:MaxBatchSize`	rows per replication batch	`16` for the artifact cache. Batching is row-count-only against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap.

Example (site-a-1 initiator, site-a-2 passive):

site-a-1:  LocalDb__SyncListenPort=9001
           LocalDb__Replication__PeerAddress=http://site-a-2:9001
           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
           LocalDb__Replication__MaxBatchSize=16
site-a-2:  LocalDb__SyncListenPort=9001
           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
           LocalDb__Replication__MaxBatchSize=16

Kestrel note. Setting SyncListenPort makes the Host add an explicit Listen* for the h2c sync listener. An explicit Listen* makes Kestrel ignore ASPNETCORE_URLS entirely, so the Host also re-binds the primary HTTP port in the same block. If you change the primary port, verify both :<httpPort> and :<syncPort> appear in the startup "Now listening on" lines.

The ApiKey rule (fail-closed)

The replication library's passive endpoint verifies no authentication — the host LocalDbSyncAuthInterceptor is the only gate, and it is fail-closed:

No key configured ⇒ every sync call is refused (PermissionDenied). "No key" is never "no auth required".
A key mismatch ⇒ the pair silently stops converging. The initiator's stream is rejected at the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is broken". If a pair is not converging, check the keys match first.

Operational rules

Stop / start the pair together where you can. Each node keeps working (and caching locally) while its peer is down; the outage is not a data-loss event — the surviving node accumulates writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its peer, so avoid leaving a pair split for extended periods.
Tombstone-retention resurrection window. Retention prunes to the newest 2 deployments and replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window. If a node is offline longer than the tombstone-retention window, a delete that happened during its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
A rebuilt node is back-filled by its peer. If a node loses its LocalDb file — a wiped volume, a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the cached configuration back to it, with no new deploy and no central SQL. Two things are worth knowing: this needs ZB.MOM.WW.LocalDb ≥ 0.1.3, and it heals the cache, so the node regains boot-from-cache for future outages. (On 0.1.1 a converged pair has pruned every oplog row on ack and the wiped node was never snapshotted — it stayed empty until the next deploy. On 0.1.2 it is back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node applies a deploy.) A node wiped during a central outage is still repopulated by its peer under 0.1.2; a node with no peer replication configured self-heals only from central on its next successful apply.
What boot-from-cache does NOT cover. The cache is a fallback for central-SQL outages at boot, not a replacement for central. A new deployment still requires central SQL — the cache is only read when the central fetch fails at startup. When central returns, the node resumes fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays on it as a central-connectivity incident, not steady state.

Inspecting the LocalDb file — safety rules

The database runs in WAL mode. These rules exist because violating them corrupted a live DB in the 2026-07-20 ScadaBridge incident:

Never run sqlite3 on the live file (host-side, against a bind-mounted or container path). Opening a live WAL DB from a second process across virtiofs poisons the WAL. Always copy the triplet first and query the copy:

docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'

Metrics come from the container, not a host curl (aspnet:10.0 has no curl):

docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_

If an anomaly appears within seconds of your own measurement, suspect the measurement. Restart both nodes and re-observe untouched before blaming the library.

Useful queries (on a copied triplet)

-- What each node thinks the current deployment is
SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;

-- Convergence check: the pointer's origin stamp should match on both nodes
SELECT pk_json, hlc, node_id, is_tombstone
FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;

-- Replication backlog (should drain to 0 when a pair is caught up)
SELECT COUNT(*) FROM __localdb_oplog;

8.1 KiB Raw Blame History Unescape Escape