# Alarm Historian — store-and-forward SQLite sink

Reference for `ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian`
([`src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/)),
the durable local queue that historizes alarm transitions to AVEVA Historian
without ever blocking the alarm engine or operator actions.

This is the *sink mechanics* doc. For how the three alarm sources converge on
the OPC UA Part 9 surface and which alarms route here, see
[AlarmTracking.md](AlarmTracking.md). For the historian client that drains this
queue, see [DriverLifecycle.md](DriverLifecycle.md#ihistoriandatasource--server-side-historian-read-surface)
and [ServiceHosting.md](ServiceHosting.md).

---

## Why store-and-forward

Scripted alarms (and any future non-Galaxy `IAlarmSource`, e.g. AB CIP ALMD)
must reach AVEVA Historian, but the historian gateway can be slow, busy, or
disconnected. The sink decouples the alarm engine from historian reachability:
every qualifying transition is committed to a **local SQLite queue first**, and
a background drain worker forwards rows to the historian on a backoff-aware
cadence. Operator acks and alarm-state transitions are never blocked waiting on
the historian.

> Galaxy-native alarms with `$Alarm*` extensions reach AVEVA Historian directly
> via System Platform's `HistorizeToAveva` toggle — they do **not** flow through
> this sink. This path is exclusively for non-Galaxy alarm producers.

---

## Contracts

All in
[`IAlarmHistorianSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs)
unless noted.

- **`IAlarmHistorianSink`** — the intake contract. `EnqueueAsync(evt, ct)`
  durably enqueues an event and returns as soon as the queue row is committed
  (fire-and-forget from the engine's perspective; the sink must not block the
  emitting thread). `GetStatus()` returns a `HistorianSinkStatus` snapshot.
- **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
  that don't historize alarms. It is the default DI binding (registered in the
  Runtime's `AddOtOpcUaRuntime`); production overrides it with
  `LocalDbStoreAndForwardSink`.
- **`AlarmHistorianEvent`**
  ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
  — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
  doubles as Historian's SourceNode), `AlarmName`, `AlarmTypeName` (Part 9
  subtype), `Severity`, `EventKind` (free-form transition string —
  "Activated"/"Cleared"/"Acknowledged"/etc.), `Message`, `User`, `Comment`,
  `TimestampUtc`.
- **`IAlarmHistorianWriter`** — what the drain worker delegates writes to.
  `WriteBatchAsync(batch, ct)` returns one `HistorianWriteOutcome` per event,
  in order. Production binds this to `GatewayAlarmHistorianWriter` (the
  HistorianGateway `SendEvent` path).
- **`HistorianWriteOutcome`** — per-event drain result: `Ack` (persisted,
  remove from queue), `RetryPlease` (transient failure — leave queued, retry
  after backoff), `PermanentFail` (malformed/unrecoverable — move to
  dead-letter).
- **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
  `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
  `LastError`, `DrainState`, and `EvictedCount`.
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
  **`NotPrimary`** (ticking, but leaving the replicated queue for the node that
  holds the Primary role).

---

## LocalDbStoreAndForwardSink

[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
is the production `IAlarmHistorianSink`. Construction takes the node's
`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
days), a test clock, and a **`drainGate`**.

### The drain gate

`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
the tick entirely, leaving the queue untouched and reporting
`DrainState = NotPrimary`.

It exists because the queue **replicates**. The Secondary holds a full copy of
the Primary's rows, and an ungated drain there would re-deliver every event,
continuously — not just at a failover. Runtime supplies the gate from
`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
with the inbound-write and native-ack gates by construction.

Two postures are deliberate:

- **Unset (the default) means drain.** So does an unpublished view. A deployment
  that runs no redundancy never publishes a role, and defaulting closed would
  silently stop its alarm history forever.
- **Only a shared queue is gated.** Standing down is safe only because a peer
  holds the same rows, so the gate applies only when
  `LocalDbReplicationPolicy.SharesRowsWithPeer` holds:
  `LocalDb:Replication:ApiKey` **AND** (`LocalDb:Replication:PeerAddress` **OR**
  `LocalDb:SyncListenPort > 0`) — an authenticated sync path in either
  direction. Any other node always drains its own queue. This is deliberately
  stricter than the table-registration predicate (`PeerAddress` OR `ApiKey`):
  a key with no dial and no listener (e.g. a fleet-wide secret on an unpaired
  node) captures changes but shares them with nobody, and gating it would park
  its events behind a Primary that does not hold them (review R1 m3).
- **A gated node stands down only for the peer that holds its rows — and BOTH
  halves of a pair know it.** The dialling half identifies its peer by the host
  of `LocalDb:Replication:PeerAddress`. The passive half (`ApiKey` +
  `SyncListenPort`, no `PeerAddress`) identifies it as its **per-cluster mesh
  partner** (the other member of its `cluster-{ClusterId}` role). Before review
  R6 MINOR-4 the passive half knew no peer at all, so as Secondary it kept
  draining beside its Primary partner and the historian received every alarm
  event twice. ⚠️ A passive half with **no** cluster-scoped role (a legacy,
  pre-Phase-6 node) still cannot identify its partner and keeps draining — it
  cannot prove which Primary holds its rows, and deferring to the wrong one
  would strand them.
- **A gate that throws means "not now", never permission.** Draining on a gate
  fault would put a pair into dual delivery exactly when its redundancy signal is
  sick.

### Queue table

The sink owns one table in the node's consolidated LocalDb — created and
registered for replication by `LocalDbSetup.OnReady`, before any consumer can
resolve the sink:

```sql
CREATE TABLE alarm_sf_events (
    id               TEXT    NOT NULL PRIMARY KEY,   -- SHA-256 of payload_json
    alarm_id         TEXT    NOT NULL,
    enqueued_at_utc  TEXT    NOT NULL,
    payload_json     TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
    attempt_count    INTEGER NOT NULL DEFAULT 0,
    last_attempt_utc TEXT    NULL,
    last_error       TEXT    NULL,
    dead_lettered    INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX ix_alarm_sf_events_drain
    ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
```

**The primary key is a hash of the payload, not an autoincrement rowid.** Two
reasons. Replication converges last-writer-wins *per primary key*, so two nodes
independently allocating rowid 7 to different alarms would silently overwrite one
another. And an equal-payload key makes the same event arriving twice collapse
into one row — which happens for real, because `HistorianAdapterActor`
default-writes while its redundancy role is unknown and both nodes of a pair
therefore accept the same fanned transition in every boot window. Two genuinely
distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
timestamp alongside the alarm id, kind, message and user.

Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
as a tiebreak so the ordering is total), because a hashed key carries no
insertion sequence.

Gateway delivery carries a separate nonzero `DeliveryId` GUID. Direct event instances
create it once; retries and record copies preserve it. It is excluded from queue JSON
and business equality, so the existing payload fingerprint and pair deduplication stay
unchanged. The drain derives its delivery GUID solely from the persisted row `id`,
including migrated legacy rows. It never hashes a reconstructed payload or mints an ID
per send. The versioned encoding in `AlarmDeliveryIds` therefore survives retry,
restart, payload edits and replicated failover without a schema migration.

Different business payloads at the same timestamp remain distinct; completely identical
payloads deliberately converge. An explicitly empty direct delivery ID is rejected.
The Gateway mapper sends this GUID as the wire event ID; the live alarm readback test
requires that exact ID. SQL Server and QuestDB retain their existing event-ID schemes.

`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
non-dead-lettered row counter (seeded at startup, kept current by every mutation,
and re-synced from storage every 10,000 enqueues to defend against drift — now
doubly warranted, since replication applies the peer's rows straight into the
table where no local code path can observe them). Pragmas and connection
lifetime belong to LocalDb; the sink no longer manages either.

### Drain worker

`StartDrainLoop(tickInterval)` starts a **self-rescheduling one-shot
`System.Threading.Timer`** (not started automatically — tests drive
`DrainOnceAsync` deterministically). Each tick:

0. Consults the drain gate. A closed gate returns immediately, leaving the queue
   untouched and `DrainState = NotPrimary`.
1. Purges aged dead-lettered rows past the retention window.
2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
3. Rows with un-deserializable payloads are dead-lettered immediately (by their
   own `id`) so they can't stall the queue head.
4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
   each outcome is applied in one transaction: `Ack` deletes the row (the delete
   replicates as a tombstone, so the peer drops its copy and a promoted standby
   cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
   `RetryPlease` bumps its attempt
   count and leaves it queued. A row whose `AttemptCount` has reached the configured
   **`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
   tick rather than retried — this breaks infinite retry loops for poison events whose
   payload the historian will always reject (e.g. a malformed alarm record that triggers
   a permanent SDK error on every attempt). The dead-lettered row remains inspectable
   via `RetryDeadLettered()` for the configured retention window.
5. The timer re-arms its next due-time to `max(tickInterval, currentBackoff)`.

**Backoff ladder** (applied to the timer's next due-time, so a historian outage
genuinely slows the drain cadence): 1s → 2s → 5s → 15s → 60s cap. Any
`RetryPlease` outcome — or a writer exception, or a writer cardinality violation
(outcome count ≠ event count) — bumps the backoff and sets `DrainState =
BackingOff`; a clean batch resets it. The async-void timer callback is fully
guarded: a fault is logged and recorded into `GetStatus()` rather than lost as
an unobserved task exception.

### Durability bound (important)

**The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
outage, accepted alarm events can therefore be dropped before delivery. A
non-zero `EvictedCount` is a data-loss signal that requires operator attention —
it surfaces silent loss without log scraping.

### Dead-letter + operator recovery

`PermanentFail` and corrupt-payload rows are retained in-place with
`dead_lettered = 1` for the retention window (default 30 days) so operators can
inspect them before the sweeper purges them. `RetryDeadLettered()` is the
operator action (from the AdminUI) that clears the dead-letter flag and attempt
count on every dead-lettered row, returning them to the regular queue with a
fresh backoff.

---

## Runtime wiring

`HistorianAdapterActor`
([`Runtime/Historian/HistorianAdapterActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs))
subscribes to the cluster **`alerts` DPS topic** and translates each
`AlarmTransitionEvent` into an `AlarmHistorianEvent`, then calls
`IAlarmHistorianSink.EnqueueAsync` fire-and-forget so the actor loop is never
blocked on historian reachability. The actor is **Primary-gated**: only the
node whose `RedundancyRole` is `Primary` historizes, giving exactly-once
writes across a redundant pair. `AlarmTransitionEvent` carries `AlarmTypeName`
(the Part 9 subtype string) and `Comment` (the operator comment from the
originating ack/shelve command) that populate the corresponding fields of
`AlarmHistorianEvent`. `GatewayAlarmHistorianWriter` is the `IAlarmHistorianWriter`
the drain worker delegates to (the gateway `SendEvent` path). See
[ServiceHosting.md](ServiceHosting.md) for the (external) HistorianGateway setup.

**Scope:** scripted alarms only. Galaxy-native alarms historize via System
Platform's `HistorizeToAveva` toggle (not this actor); AB CIP ALMD is not on
the `alerts` topic (future).

## Configuration

The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
`NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
`GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
**only** the `Enabled` gate + the store-and-forward knobs — the downstream
gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
section (see [Historian.md](Historian.md)), and the queue's storage location is the
node's consolidated `LocalDb:Path` database (see
[Configuration.md](Configuration.md)).

> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
> Two consequences follow, both covered in the
> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
> everything), and **delivery is at-least-once across a failover** by design. On first boot after
> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
> it `.migrated`.

```json
{
  "AlarmHistorian": {
    "Enabled": true,
    "BatchSize": 100,
    "DrainIntervalSeconds": 5,
    "Capacity": 1000000,
    "DeadLetterRetentionDays": 30
  }
}
```

| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
| `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
| `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
| `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
| `DeadLetterRetentionDays` | int | `30` | Days to retain dead-lettered rows before purge. |
| `MaxAttempts` | int | `10` | Maximum delivery attempts before a poison (perpetually-retrying) row is dead-lettered automatically. Must be > 0. |

> The downstream gateway connection lives in `ServerHistorian` (`Endpoint` + env `ServerHistorian__ApiKey`,
> `UseTls`, `CaCertificatePath`); alarm-history `ReadEvents` additionally requires the gateway running
> `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
> `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.

> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
> so leave it in place through the upgrade, then delete it once every node has an
> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.

> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
> nobody is draining — see the runbook.

> Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.

---

## See also

- [AlarmTracking.md](AlarmTracking.md) — the three alarm sources and the OPC UA
  Part 9 surface; which alarms route to this sink.
- [DriverLifecycle.md](DriverLifecycle.md) — `IHistorianDataSource` (the
  historian *read* surface; this page covers the *write* path) and the
  `GatewayHistorianDataSource`.
- [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
  most events into this sink.
- [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
  — where the queue lives, the Primary-only drain, and the one-time legacy migration.