docs+chore(localdb): phase-2 rig config + docs

Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.

Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.

Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

CLAUDE.md

@@ -219,10 +219,13 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
 
 The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
 
-## LocalDb pair-local config cache (Phase 1)
+## LocalDb pair-local store (Phases 1 + 2)
 
 Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
-LiteDB `LocalCache` subsystem was deleted as superseded). Phase-1 scope: it caches the
+LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
+`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
+
+**Phase 1** caches the
 **deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
 driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
 the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
@@ -234,8 +237,27 @@ library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAu
 on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
 docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
 the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
-exception). See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
+exception).
-cache), and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`.
+
+**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
+and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
+history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
+`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
+`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
+to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
+commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
+sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
+`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
+`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
+design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
+same event accepted on both nodes — which happens in every boot window, since
+`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
+docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
+deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
+
+See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
+cache), `docs/AlarmHistorian.md`, and the runbook
+`docs/operations/2026-07-20-localdb-pair-replication.md`.
 
 ## LDAP Authentication
 

docker-dev/docker-compose.yml

@@ -328,6 +328,22 @@ services:
       # site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
       # Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
       # re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
       LocalDb__Path: "/app/data/otopcua-localdb.db"
       LocalDb__SyncListenPort: "9001"
       LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
@@ -365,6 +381,22 @@ services:
       # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
       # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
       # is mandatory — the interceptor fail-closes on a mismatch).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
       LocalDb__Path: "/app/data/otopcua-localdb.db"
       LocalDb__SyncListenPort: "9001"
       LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
@@ -397,6 +429,11 @@ services:
       # site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
       # replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
       # This is what the live gate's check 8 verifies — the cache works locally with replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
       LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4844:4840"
@@ -422,6 +459,11 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
       # site-b default-OFF pin (see site-b-1): cache on, replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
       LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4845:4840"

docs/AlarmHistorian.md

@@ -42,7 +42,7 @@ unless noted.
 - **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
   that don't historize alarms. It is the default DI binding (registered in the
   Runtime's `AddOtOpcUaRuntime`); production overrides it with
-  `SqliteStoreAndForwardSink`.
+  `LocalDbStoreAndForwardSink`.
 - **`AlarmHistorianEvent`**
   ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
   — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
@@ -61,42 +61,84 @@ unless noted.
 - **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
   `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
   `LastError`, `DrainState`, and `EvictedCount`.
-- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
+- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
+  **`NotPrimary`** (ticking, but leaving the replicated queue for the node that
+  holds the Primary role).
 
 ---
 
-## SqliteStoreAndForwardSink
+## LocalDbStoreAndForwardSink
 
-[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
+[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
-is the production `IAlarmHistorianSink`. Construction takes a SQLite database
+is the production `IAlarmHistorianSink`. Construction takes the node's
-path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
+`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
-100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
+(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
-and a test clock.
+days), a test clock, and a **`drainGate`**.
+
+### The drain gate
+
+`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
+the tick entirely, leaving the queue untouched and reporting
+`DrainState = NotPrimary`.
+
+It exists because the queue **replicates**. The Secondary holds a full copy of
+the Primary's rows, and an ungated drain there would re-deliver every event,
+continuously — not just at a failover. Runtime supplies the gate from
+`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
+`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
+with the inbound-write and native-ack gates by construction.
+
+Two postures are deliberate:
+
+- **Unset (the default) means drain.** So does an unpublished view. A deployment
+  that runs no redundancy never publishes a role, and defaulting closed would
+  silently stop its alarm history forever.
+- **A gate that throws means "not now", never permission.** Draining on a gate
+  fault would put a pair into dual delivery exactly when its redundancy signal is
+  sick.
 
 ### Queue table
 
-The sink owns one SQLite table (created on construction, WAL journal mode):
+The sink owns one table in the node's consolidated LocalDb — created and
+registered for replication by `LocalDbSetup.OnReady`, before any consumer can
+resolve the sink:
 
 ```sql
-CREATE TABLE Queue (
+CREATE TABLE alarm_sf_events (
-    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
+    id               TEXT    NOT NULL PRIMARY KEY,   -- SHA-256 of payload_json
-    AlarmId        TEXT    NOT NULL,
+    alarm_id         TEXT    NOT NULL,
-    EnqueuedUtc    TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
-    PayloadJson    TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
+    payload_json     TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
-    AttemptCount   INTEGER NOT NULL DEFAULT 0,
+    attempt_count    INTEGER NOT NULL DEFAULT 0,
-    LastAttemptUtc TEXT    NULL,
+    last_attempt_utc TEXT    NULL,
-    LastError      TEXT    NULL,
+    last_error       TEXT    NULL,
-    DeadLettered   INTEGER NOT NULL DEFAULT 0
+    dead_lettered    INTEGER NOT NULL DEFAULT 0
 );
-CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
+CREATE INDEX ix_alarm_sf_events_drain
+    ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
 ```
 
-`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
+**The primary key is a hash of the payload, not an autoincrement rowid.** Two
-`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
+reasons. Replication converges last-writer-wins *per primary key*, so two nodes
-row counter (seeded at startup, kept current by every mutation, and re-synced
+independently allocating rowid 7 to different alarms would silently overwrite one
-from storage every 10,000 enqueues to defend against drift). SQLite writer
+another. And an equal-payload key makes the same event arriving twice collapse
-contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
+into one row — which happens for real, because `HistorianAdapterActor`
-collision waits out the file lock instead of failing fast.
+default-writes while its redundancy role is unknown and both nodes of a pair
+therefore accept the same fanned transition in every boot window. Two genuinely
+distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+timestamp alongside the alarm id, kind, message and user.
+
+Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
+as a tiebreak so the ordering is total), because a hashed key carries no
+insertion sequence.
+
+`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
+To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
+non-dead-lettered row counter (seeded at startup, kept current by every mutation,
+and re-synced from storage every 10,000 enqueues to defend against drift — now
+doubly warranted, since replication applies the peer's rows straight into the
+table where no local code path can observe them). Pragmas and connection
+lifetime belong to LocalDb; the sink no longer manages either.
 
 ### Drain worker
 
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
 `System.Threading.Timer`** (not started automatically — tests drive
 `DrainOnceAsync` deterministically). Each tick:
 
+0. Consults the drain gate. A closed gate returns immediately, leaving the queue
+   untouched and `DrainState = NotPrimary`.
 1. Purges aged dead-lettered rows past the retention window.
-2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
+2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
 3. Rows with un-deserializable payloads are dead-lettered immediately (by their
-   own `RowId`) so they can't stall the queue head.
+   own `id`) so they can't stall the queue head.
 4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
-   each outcome is applied in one transaction: `Ack` deletes the row,
+   each outcome is applied in one transaction: `Ack` deletes the row (the delete
-   `PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
+   replicates as a tombstone, so the peer drops its copy and a promoted standby
+   cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
+   `RetryPlease` bumps its attempt
    count and leaves it queued. A row whose `AttemptCount` has reached the configured
    **`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
    tick rather than retried — this breaks infinite retry loops for poison events whose
@@ -131,7 +177,7 @@ an unobserved task exception.
 
 **The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
 When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
-oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
+oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
 and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
 outage, accepted alarm events can therefore be dropped before delivery. A
 non-zero `EvictedCount` is a data-loss signal that requires operator attention —
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
 ### Dead-letter + operator recovery
 
 `PermanentFail` and corrupt-payload rows are retained in-place with
-`DeadLettered = 1` for the retention window (default 30 days) so operators can
+`dead_lettered = 1` for the retention window (default 30 days) so operators can
 inspect them before the sweeper purges them. `RetryDeadLettered()` is the
 operator action (from the AdminUI) that clears the dead-letter flag and attempt
 count on every dead-lettered row, returning them to the regular queue with a
@@ -173,17 +219,28 @@ the `alerts` topic (future).
 The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
 When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
 `NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
-`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
+`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
 `GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
-**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
+**only** the `Enabled` gate + the store-and-forward knobs — the downstream
 gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
-section (see [Historian.md](Historian.md)).
+section (see [Historian.md](Historian.md)), and the queue's storage location is the
+node's consolidated `LocalDb:Path` database (see
+[Configuration.md](Configuration.md)).
+
+> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
+> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
+> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
+> Two consequences follow, both covered in the
+> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
+> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
+> everything), and **delivery is at-least-once across a failover** by design. On first boot after
+> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
+> it `.migrated`.
 
 ```json
 {
   "AlarmHistorian": {
     "Enabled": true,
-    "DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
     "BatchSize": 100,
     "DrainIntervalSeconds": 5,
     "Capacity": 1000000,
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
+| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
-| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
 | `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
 | `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
 | `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
 > `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
 > `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
 
+> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
+> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
+> so leave it in place through the upgrade, then delete it once every node has an
+> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
+
+> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
+> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
+> nobody is draining — see the runbook.
+
 > Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
 
 ---
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
 - [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
   most events into this sink.
 - [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
+- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
+  — where the queue lives, the Primary-only drain, and the one-time legacy migration.

docs/AlarmTracking.md

@@ -365,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
 - `IAlarmHistorianSink` is the DI-registered intake contract. The
   default binding is `NullAlarmHistorianSink` (registered in
   `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
-  deployments override it with `SqliteStoreAndForwardSink` wrapping
+  deployments override it with `LocalDbStoreAndForwardSink` wrapping
   `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
   — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
-- `SqliteStoreAndForwardSink` queues each transition to a local
+- `LocalDbStoreAndForwardSink` queues each transition into the node's
-  SQLite database and drains in the background via an
+  consolidated LocalDb — where it **replicates to the redundant pair peer**,
-  `IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
+  so undelivered alarm history survives losing the node holding it — and
+  drains in the background via an `IAlarmHistorianWriter`. **Only the node
+  holding the Primary role drains**, since the peer holds a full replica;
+  delivery is consequently at-least-once across a failover, by design. See
+  [AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
   queue capacity defaults to 1,000,000 rows; under a sustained
   historian outage, older non-dead-lettered rows are evicted (oldest
   first) to make room for new events. The `HistorianSinkStatus.EvictedCount`

docs/Configuration.md

@@ -136,8 +136,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar,
 `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
 [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
 appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
-(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward
+(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
-alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in
+alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
+file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
 `ServerHistorian`; the other two sections source it from there.
 
 The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed

docs/README.md

@@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess
 | [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) |
 | [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) |
 | [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) |
-| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
+| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
 | [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) |
 | [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` |
 | [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) |

docs/Redundancy.md

@@ -221,15 +221,19 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
 
 - **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
 - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
+- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
 
 Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
 
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
 
-## Pair-local config cache (LocalDb — Phase 1)
+## Pair-local store (LocalDb — Phases 1 + 2)
 
 Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
-[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database that caches the **deployed-configuration
+[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
+buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
+undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
+and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
 artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
 node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
 the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -1,18 +1,20 @@
 # LocalDb pair replication — operations runbook
 
-> **Phase 1 scope.** Every driver-role OtOpcUa node keeps a consolidated
+> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
-> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. Today it caches exactly one thing: the
+> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
-> **deployed-configuration artifact**, chunked, so a node can **boot from cache when central SQL
+> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
-> Server is unreachable**. That cache can *optionally* replicate to the node's redundant pair peer
+> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
-> over the library's gRPC sync. **Replication is default-OFF and fail-closed.** This runbook covers
+> node that dies holding undelivered alarm history does not take it to the grave. Both can
-> enabling it, the operational rules that keep a pair converging, and the DB-inspection safety
+> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
-> rules.
+> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
+> rules that keep a pair converging, and the DB-inspection safety rules.
 
 ## What replicates, and what it buys you
 
-- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks) and
+- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
-  `deployment_pointer` (one current-deployment pointer per cluster). Registered — in this order,
+  `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
-  after the DDL — by `LocalDbSetup.OnReady`.
+  alarm store-and-forward buffer). Registered — in this order, after the DDL — by
+  `LocalDbSetup.OnReady`.
 - **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
   deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
   that restarts into a central-SQL outage can boot the current config **even if it never applied
@@ -22,6 +24,47 @@
   deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
   replicate as tombstones.
 
+## Alarm store-and-forward — what replication changes
+
+Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
+consequences an operator needs to know:
+
+- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
+  Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
+  native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
+  Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
+- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
+  moments before it died may not have had their deletes replicated yet, so the new Primary will
+  re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
+  missing one is not. There is deliberately no dedup layer.
+- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
+  oldest undelivered rows when the historian has been unreachable long enough. Watch
+  `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
+
+### If BOTH nodes report `NotPrimary`
+
+Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
+default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
+redundancy snapshot that never names either node looks like — the identity-mismatch shape
+`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
+identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
+
+### One-time migration from `alarm-historian.db`
+
+On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
+`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
+
+- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
+  from the event payload, so the same event on both nodes converges to one row rather than
+  duplicating. The new Primary drains the union.
+- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
+  file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
+- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
+  available for inspection.
+- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
+  find the legacy file; it no longer configures anything. Delete it from appsettings once the
+  `.migrated` sidecar exists on every node.
+
 ## Enabling replication on a pair
 
 Replication has two knobs, both under `LocalDb`:
@@ -127,10 +170,22 @@ FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_j
 
 -- Replication backlog (should drain to 0 when a pair is caught up)
 SELECT COUNT(*) FROM __localdb_oplog;
+
+-- Undelivered alarm history, and how much of it has given up
+SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
+
+-- Why the dead-lettered rows died
+SELECT alarm_id, attempt_count, last_attempt_utc, last_error
+FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
+
+-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
 ```
 
 ## See also
 
 - [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
 - [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
+- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
 - The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.

docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json

@@ -55,10 +55,11 @@
     {
       "id": 6,
       "subject": "Task 6: Rig config + docs",
-      "status": "pending",
+      "status": "completed",
       "blockedBy": [
         3
-      ]
+      ],
+      "note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md."
     },
     {
       "id": 7,