dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

docs+chore(localdb): phase-2 rig config + docs

Browse Source 
Enables the alarm store-and-forward sink on all four driver nodes of the
docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin
-- so the live gate has a real buffer to watch converge, and can see that
site-b's sink works as a plain node-local queue with no peer traffic.

Two rig details worth stating rather than rediscovering. The endpoint is
deliberately unresolvable: there is no HistorianGateway here, so every drain
attempt fails, which is exactly the historian-outage state the buffer exists
for. But it still has to be a syntactically valid absolute http(s) URI or the
host refuses to start, because ServerHistorianOptionsValidator is
consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required
even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above
the production default of 10, which against a permanently unreachable gateway
would dead-letter the entire queue about five minutes in, turning a buffering
test into a dead-letter test.

Docs record what an operator now has to know: that a rising queue depth on a
Secondary is correct and on BOTH nodes is not (that shape is a redundancy
snapshot naming neither node -- check node identity before suspecting the
sink), that delivery is at-least-once across a failover by design with no
dedup layer, and that AlarmHistorian:DatabasePath is removed but should be
left in place through the upgrade because the migrator still reads it to find
the file to copy.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 04:43:37 -04:00
parent 271fcc4e15
commit f76d1f91e5
9 changed files with 258 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/AlarmHistorian.md
+104 -37
View File
@@ -42,7 +42,7 @@ unless noted.
- **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
that don't historize alarms. It is the default DI binding (registered in the
Runtime's `AddOtOpcUaRuntime`); production overrides it with
`SqliteStoreAndForwardSink`.
`LocalDbStoreAndForwardSink`.
- **`AlarmHistorianEvent`**
([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
— the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
@@ -61,42 +61,84 @@ unless noted.
- **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
`/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
`LastError`, `DrainState`, and `EvictedCount`.
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
**`NotPrimary`** (ticking, but leaving the replicated queue for the node that
holds the Primary role).
---
## SqliteStoreAndForwardSink
## LocalDbStoreAndForwardSink
[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
is the production `IAlarmHistorianSink`. Construction takes a SQLite database
path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
and a test clock.
[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
is the production `IAlarmHistorianSink`. Construction takes the node's
`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
days), a test clock, and a **`drainGate`**.
### The drain gate
`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
the tick entirely, leaving the queue untouched and reporting
`DrainState = NotPrimary`.
It exists because the queue **replicates**. The Secondary holds a full copy of
the Primary's rows, and an ungated drain there would re-deliver every event,
continuously — not just at a failover. Runtime supplies the gate from
`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
with the inbound-write and native-ack gates by construction.
Two postures are deliberate:
- **Unset (the default) means drain.** So does an unpublished view. A deployment
that runs no redundancy never publishes a role, and defaulting closed would
silently stop its alarm history forever.
- **A gate that throws means "not now", never permission.** Draining on a gate
fault would put a pair into dual delivery exactly when its redundancy signal is
sick.
### Queue table
The sink owns one SQLite table (created on construction, WAL journal mode):
The sink owns one table in the node's consolidated LocalDb — created and
registered for replication by `LocalDbSetup.OnReady`, before any consumer can
resolve the sink:
```sql
CREATE TABLE Queue (
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
AlarmId TEXT NOT NULL,
EnqueuedUtc TEXT NOT NULL,
PayloadJson TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
AttemptCount INTEGER NOT NULL DEFAULT 0,
LastAttemptUtc TEXT NULL,
LastError TEXT NULL,
DeadLettered INTEGER NOT NULL DEFAULT 0
CREATE TABLE alarm_sf_events (
id TEXT NOT NULL PRIMARY KEY, -- SHA-256 of payload_json
alarm_id TEXT NOT NULL,
enqueued_at_utc TEXT NOT NULL,
payload_json TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
attempt_count INTEGER NOT NULL DEFAULT 0,
last_attempt_utc TEXT NULL,
last_error TEXT NULL,
dead_lettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
CREATE INDEX ix_alarm_sf_events_drain
ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
```
`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
row counter (seeded at startup, kept current by every mutation, and re-synced
from storage every 10,000 enqueues to defend against drift). SQLite writer
contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
collision waits out the file lock instead of failing fast.
**The primary key is a hash of the payload, not an autoincrement rowid.** Two
reasons. Replication converges last-writer-wins *per primary key*, so two nodes
independently allocating rowid 7 to different alarms would silently overwrite one
another. And an equal-payload key makes the same event arriving twice collapse
into one row — which happens for real, because `HistorianAdapterActor`
default-writes while its redundancy role is unknown and both nodes of a pair
therefore accept the same fanned transition in every boot window. Two genuinely
distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
timestamp alongside the alarm id, kind, message and user.
Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
as a tiebreak so the ordering is total), because a hashed key carries no
insertion sequence.
`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
non-dead-lettered row counter (seeded at startup, kept current by every mutation,
and re-synced from storage every 10,000 enqueues to defend against drift — now
doubly warranted, since replication applies the peer's rows straight into the
table where no local code path can observe them). Pragmas and connection
lifetime belong to LocalDb; the sink no longer manages either.
### Drain worker
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
`System.Threading.Timer`** (not started automatically — tests drive
`DrainOnceAsync` deterministically). Each tick:
0. Consults the drain gate. A closed gate returns immediately, leaving the queue
untouched and `DrainState = NotPrimary`.
1. Purges aged dead-lettered rows past the retention window.
2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
3. Rows with un-deserializable payloads are dead-lettered immediately (by their
own `RowId`) so they can't stall the queue head.
own `id`) so they can't stall the queue head.
4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
each outcome is applied in one transaction: `Ack` deletes the row,
`PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
each outcome is applied in one transaction: `Ack` deletes the row (the delete
replicates as a tombstone, so the peer drops its copy and a promoted standby
cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
`RetryPlease` bumps its attempt
count and leaves it queued. A row whose `AttemptCount` has reached the configured
**`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
tick rather than retried — this breaks infinite retry loops for poison events whose
@@ -131,7 +177,7 @@ an unobserved task exception.
**The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
outage, accepted alarm events can therefore be dropped before delivery. A
non-zero `EvictedCount` is a data-loss signal that requires operator attention —
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
### Dead-letter + operator recovery
`PermanentFail` and corrupt-payload rows are retained in-place with
`DeadLettered = 1` for the retention window (default 30 days) so operators can
`dead_lettered = 1` for the retention window (default 30 days) so operators can
inspect them before the sweeper purges them. `RetryDeadLettered()` is the
operator action (from the AdminUI) that clears the dead-letter flag and attempt
count on every dead-lettered row, returning them to the regular queue with a
@@ -173,17 +219,28 @@ the `alerts` topic (future).
The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
`NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
`GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
**only** the `Enabled` gate + the store-and-forward knobs — the downstream
gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
section (see [Historian.md](Historian.md)).
section (see [Historian.md](Historian.md)), and the queue's storage location is the
node's consolidated `LocalDb:Path` database (see
[Configuration.md](Configuration.md)).
> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
> Two consequences follow, both covered in the
> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
> everything), and **delivery is at-least-once across a failover** by design. On first boot after
> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
> it `.migrated`.
```json
{
"AlarmHistorian": {
"Enabled": true,
"DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
"BatchSize": 100,
"DrainIntervalSeconds": 5,
"Capacity": 1000000,
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
| `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
| `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
| `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
> `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
> `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
> so leave it in place through the upgrade, then delete it once every node has an
> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
> nobody is draining — see the runbook.
> Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
---
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
- [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
most events into this sink.
- [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
— where the queue lives, the Primary-only drain, and the one-time legacy migration.