docs+chore(localdb): phase-2 rig config + docs

Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.

Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.

Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -1,18 +1,20 @@
 # LocalDb pair replication — operations runbook
 
-> **Phase 1 scope.** Every driver-role OtOpcUa node keeps a consolidated
-> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. Today it caches exactly one thing: the
-> **deployed-configuration artifact**, chunked, so a node can **boot from cache when central SQL
-> Server is unreachable**. That cache can *optionally* replicate to the node's redundant pair peer
-> over the library's gRPC sync. **Replication is default-OFF and fail-closed.** This runbook covers
-> enabling it, the operational rules that keep a pair converging, and the DB-inspection safety
-> rules.
+> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
+> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
+> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
+> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
+> node that dies holding undelivered alarm history does not take it to the grave. Both can
+> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
+> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
+> rules that keep a pair converging, and the DB-inspection safety rules.
 
 ## What replicates, and what it buys you
 
-- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks) and
-  `deployment_pointer` (one current-deployment pointer per cluster). Registered — in this order,
-  after the DDL — by `LocalDbSetup.OnReady`.
+- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
+  `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
+  alarm store-and-forward buffer). Registered — in this order, after the DDL — by
+  `LocalDbSetup.OnReady`.
 - **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
   deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
   that restarts into a central-SQL outage can boot the current config **even if it never applied
@@ -22,6 +24,47 @@
   deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
   replicate as tombstones.
 
+## Alarm store-and-forward — what replication changes
+
+Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
+consequences an operator needs to know:
+
+- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
+  Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
+  native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
+  Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
+- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
+  moments before it died may not have had their deletes replicated yet, so the new Primary will
+  re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
+  missing one is not. There is deliberately no dedup layer.
+- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
+  oldest undelivered rows when the historian has been unreachable long enough. Watch
+  `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
+
+### If BOTH nodes report `NotPrimary`
+
+Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
+default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
+redundancy snapshot that never names either node looks like — the identity-mismatch shape
+`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
+identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
+
+### One-time migration from `alarm-historian.db`
+
+On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
+`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
+
+- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
+  from the event payload, so the same event on both nodes converges to one row rather than
+  duplicating. The new Primary drains the union.
+- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
+  file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
+- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
+  available for inspection.
+- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
+  find the legacy file; it no longer configures anything. Delete it from appsettings once the
+  `.migrated` sidecar exists on every node.
+
 ## Enabling replication on a pair
 
 Replication has two knobs, both under `LocalDb`:
@@ -127,10 +170,22 @@ FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_j
 
 -- Replication backlog (should drain to 0 when a pair is caught up)
 SELECT COUNT(*) FROM __localdb_oplog;
+
+-- Undelivered alarm history, and how much of it has given up
+SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
+
+-- Why the dead-lettered rows died
+SELECT alarm_id, attempt_count, last_attempt_utc, last_error
+FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
+
+-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
 ```
 
 ## See also
 
 - [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
 - [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
+- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
 - The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.