docs+chore(localdb): phase-2 rig config + docs
Enables the alarm store-and-forward sink on all four driver nodes of the docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin -- so the live gate has a real buffer to watch converge, and can see that site-b's sink works as a plain node-local queue with no peer traffic. Two rig details worth stating rather than rediscovering. The endpoint is deliberately unresolvable: there is no HistorianGateway here, so every drain attempt fails, which is exactly the historian-outage state the buffer exists for. But it still has to be a syntactically valid absolute http(s) URI or the host refuses to start, because ServerHistorianOptionsValidator is consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above the production default of 10, which against a permanently unreachable gateway would dead-letter the entire queue about five minutes in, turning a buffering test into a dead-letter test. Docs record what an operator now has to know: that a rising queue depth on a Secondary is correct and on BOTH nodes is not (that shape is a redundancy snapshot naming neither node -- check node identity before suspecting the sink), that delivery is at-least-once across a failover by design with no dedup layer, and that AlarmHistorian:DatabasePath is removed but should be left in place through the upgrade because the migrator still reads it to find the file to copy. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -219,10 +219,13 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
|
||||
|
||||
The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
|
||||
|
||||
## LocalDb pair-local config cache (Phase 1)
|
||||
## LocalDb pair-local store (Phases 1 + 2)
|
||||
|
||||
Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
|
||||
LiteDB `LocalCache` subsystem was deleted as superseded). Phase-1 scope: it caches the
|
||||
LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
|
||||
`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
|
||||
|
||||
**Phase 1** caches the
|
||||
**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
|
||||
driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
|
||||
the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
|
||||
@@ -234,8 +237,27 @@ library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAu
|
||||
on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
|
||||
docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
|
||||
the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
|
||||
exception). See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
|
||||
cache), and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`.
|
||||
exception).
|
||||
|
||||
**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
|
||||
and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
|
||||
history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
|
||||
`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
|
||||
`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
|
||||
to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
|
||||
commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
|
||||
sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
|
||||
`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
|
||||
`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
|
||||
design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
|
||||
same event accepted on both nodes — which happens in every boot window, since
|
||||
`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
|
||||
docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
|
||||
deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
|
||||
|
||||
See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
|
||||
cache), `docs/AlarmHistorian.md`, and the runbook
|
||||
`docs/operations/2026-07-20-localdb-pair-replication.md`.
|
||||
|
||||
## LDAP Authentication
|
||||
|
||||
|
||||
@@ -328,6 +328,22 @@ services:
|
||||
# site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
|
||||
# Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
|
||||
# re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
|
||||
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
|
||||
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
|
||||
# queue to watch converge, and needs to see that only the Primary drains it.
|
||||
#
|
||||
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
|
||||
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
|
||||
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
|
||||
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
|
||||
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
|
||||
#
|
||||
# MaxAttempts is raised far above the production default of 10. With a permanently
|
||||
# unreachable gateway the default would dead-letter the whole queue about five minutes into
|
||||
# the outage, turning a buffering test into a dead-letter test.
|
||||
AlarmHistorian__Enabled: "true"
|
||||
AlarmHistorian__MaxAttempts: "1000000"
|
||||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||||
LocalDb__SyncListenPort: "9001"
|
||||
LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
|
||||
@@ -365,6 +381,22 @@ services:
|
||||
# but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
|
||||
# single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
|
||||
# is mandatory — the interceptor fail-closes on a mismatch).
|
||||
# Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
|
||||
# buffer is a live, replicated table rather than an empty one — the live gate needs a real
|
||||
# queue to watch converge, and needs to see that only the Primary drains it.
|
||||
#
|
||||
# There is no HistorianGateway on this rig, so the endpoint below is deliberately
|
||||
# unresolvable: every drain attempt fails, which is exactly the historian-outage state the
|
||||
# buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
|
||||
# URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
|
||||
# AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
|
||||
#
|
||||
# MaxAttempts is raised far above the production default of 10. With a permanently
|
||||
# unreachable gateway the default would dead-letter the whole queue about five minutes into
|
||||
# the outage, turning a buffering test into a dead-letter test.
|
||||
AlarmHistorian__Enabled: "true"
|
||||
AlarmHistorian__MaxAttempts: "1000000"
|
||||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||||
LocalDb__SyncListenPort: "9001"
|
||||
LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
|
||||
@@ -397,6 +429,11 @@ services:
|
||||
# site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
|
||||
# replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
|
||||
# This is what the live gate's check 8 verifies — the cache works locally with replication off.
|
||||
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
|
||||
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
|
||||
AlarmHistorian__Enabled: "true"
|
||||
AlarmHistorian__MaxAttempts: "1000000"
|
||||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||||
ports:
|
||||
- "4844:4840"
|
||||
@@ -422,6 +459,11 @@ services:
|
||||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||||
GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
|
||||
# site-b default-OFF pin (see site-b-1): cache on, replication off.
|
||||
# Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
|
||||
# must work as a plain node-local buffer here, with no sync listener and no peer traffic.
|
||||
AlarmHistorian__Enabled: "true"
|
||||
AlarmHistorian__MaxAttempts: "1000000"
|
||||
ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
|
||||
LocalDb__Path: "/app/data/otopcua-localdb.db"
|
||||
ports:
|
||||
- "4845:4840"
|
||||
|
||||
+104
-37
@@ -42,7 +42,7 @@ unless noted.
|
||||
- **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
|
||||
that don't historize alarms. It is the default DI binding (registered in the
|
||||
Runtime's `AddOtOpcUaRuntime`); production overrides it with
|
||||
`SqliteStoreAndForwardSink`.
|
||||
`LocalDbStoreAndForwardSink`.
|
||||
- **`AlarmHistorianEvent`**
|
||||
([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
|
||||
— the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
|
||||
@@ -61,42 +61,84 @@ unless noted.
|
||||
- **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
|
||||
`/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
|
||||
`LastError`, `DrainState`, and `EvictedCount`.
|
||||
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
|
||||
- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
|
||||
**`NotPrimary`** (ticking, but leaving the replicated queue for the node that
|
||||
holds the Primary role).
|
||||
|
||||
---
|
||||
|
||||
## SqliteStoreAndForwardSink
|
||||
## LocalDbStoreAndForwardSink
|
||||
|
||||
[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
|
||||
is the production `IAlarmHistorianSink`. Construction takes a SQLite database
|
||||
path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
|
||||
100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
|
||||
and a test clock.
|
||||
[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
|
||||
is the production `IAlarmHistorianSink`. Construction takes the node's
|
||||
`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
|
||||
(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
|
||||
days), a test clock, and a **`drainGate`**.
|
||||
|
||||
### The drain gate
|
||||
|
||||
`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
|
||||
the tick entirely, leaving the queue untouched and reporting
|
||||
`DrainState = NotPrimary`.
|
||||
|
||||
It exists because the queue **replicates**. The Secondary holds a full copy of
|
||||
the Primary's rows, and an ungated drain there would re-deliver every event,
|
||||
continuously — not just at a failover. Runtime supplies the gate from
|
||||
`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
|
||||
`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
|
||||
with the inbound-write and native-ack gates by construction.
|
||||
|
||||
Two postures are deliberate:
|
||||
|
||||
- **Unset (the default) means drain.** So does an unpublished view. A deployment
|
||||
that runs no redundancy never publishes a role, and defaulting closed would
|
||||
silently stop its alarm history forever.
|
||||
- **A gate that throws means "not now", never permission.** Draining on a gate
|
||||
fault would put a pair into dual delivery exactly when its redundancy signal is
|
||||
sick.
|
||||
|
||||
### Queue table
|
||||
|
||||
The sink owns one SQLite table (created on construction, WAL journal mode):
|
||||
The sink owns one table in the node's consolidated LocalDb — created and
|
||||
registered for replication by `LocalDbSetup.OnReady`, before any consumer can
|
||||
resolve the sink:
|
||||
|
||||
```sql
|
||||
CREATE TABLE Queue (
|
||||
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
AlarmId TEXT NOT NULL,
|
||||
EnqueuedUtc TEXT NOT NULL,
|
||||
PayloadJson TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
|
||||
AttemptCount INTEGER NOT NULL DEFAULT 0,
|
||||
LastAttemptUtc TEXT NULL,
|
||||
LastError TEXT NULL,
|
||||
DeadLettered INTEGER NOT NULL DEFAULT 0
|
||||
CREATE TABLE alarm_sf_events (
|
||||
id TEXT NOT NULL PRIMARY KEY, -- SHA-256 of payload_json
|
||||
alarm_id TEXT NOT NULL,
|
||||
enqueued_at_utc TEXT NOT NULL,
|
||||
payload_json TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_attempt_utc TEXT NULL,
|
||||
last_error TEXT NULL,
|
||||
dead_lettered INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
|
||||
CREATE INDEX ix_alarm_sf_events_drain
|
||||
ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
|
||||
```
|
||||
|
||||
`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
|
||||
`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
|
||||
row counter (seeded at startup, kept current by every mutation, and re-synced
|
||||
from storage every 10,000 enqueues to defend against drift). SQLite writer
|
||||
contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
|
||||
collision waits out the file lock instead of failing fast.
|
||||
**The primary key is a hash of the payload, not an autoincrement rowid.** Two
|
||||
reasons. Replication converges last-writer-wins *per primary key*, so two nodes
|
||||
independently allocating rowid 7 to different alarms would silently overwrite one
|
||||
another. And an equal-payload key makes the same event arriving twice collapse
|
||||
into one row — which happens for real, because `HistorianAdapterActor`
|
||||
default-writes while its redundancy role is unknown and both nodes of a pair
|
||||
therefore accept the same fanned transition in every boot window. Two genuinely
|
||||
distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
|
||||
timestamp alongside the alarm id, kind, message and user.
|
||||
|
||||
Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
|
||||
as a tiebreak so the ordering is total), because a hashed key carries no
|
||||
insertion sequence.
|
||||
|
||||
`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
|
||||
To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
|
||||
non-dead-lettered row counter (seeded at startup, kept current by every mutation,
|
||||
and re-synced from storage every 10,000 enqueues to defend against drift — now
|
||||
doubly warranted, since replication applies the peer's rows straight into the
|
||||
table where no local code path can observe them). Pragmas and connection
|
||||
lifetime belong to LocalDb; the sink no longer manages either.
|
||||
|
||||
### Drain worker
|
||||
|
||||
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
|
||||
`System.Threading.Timer`** (not started automatically — tests drive
|
||||
`DrainOnceAsync` deterministically). Each tick:
|
||||
|
||||
0. Consults the drain gate. A closed gate returns immediately, leaving the queue
|
||||
untouched and `DrainState = NotPrimary`.
|
||||
1. Purges aged dead-lettered rows past the retention window.
|
||||
2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
|
||||
2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
|
||||
3. Rows with un-deserializable payloads are dead-lettered immediately (by their
|
||||
own `RowId`) so they can't stall the queue head.
|
||||
own `id`) so they can't stall the queue head.
|
||||
4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
|
||||
each outcome is applied in one transaction: `Ack` deletes the row,
|
||||
`PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
|
||||
each outcome is applied in one transaction: `Ack` deletes the row (the delete
|
||||
replicates as a tombstone, so the peer drops its copy and a promoted standby
|
||||
cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
|
||||
`RetryPlease` bumps its attempt
|
||||
count and leaves it queued. A row whose `AttemptCount` has reached the configured
|
||||
**`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
|
||||
tick rather than retried — this breaks infinite retry loops for poison events whose
|
||||
@@ -131,7 +177,7 @@ an unobserved task exception.
|
||||
|
||||
**The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
|
||||
When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
|
||||
oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
|
||||
oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
|
||||
and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
|
||||
outage, accepted alarm events can therefore be dropped before delivery. A
|
||||
non-zero `EvictedCount` is a data-loss signal that requires operator attention —
|
||||
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
|
||||
### Dead-letter + operator recovery
|
||||
|
||||
`PermanentFail` and corrupt-payload rows are retained in-place with
|
||||
`DeadLettered = 1` for the retention window (default 30 days) so operators can
|
||||
`dead_lettered = 1` for the retention window (default 30 days) so operators can
|
||||
inspect them before the sweeper purges them. `RetryDeadLettered()` is the
|
||||
operator action (from the AdminUI) that clears the dead-letter flag and attempt
|
||||
count on every dead-lettered row, returning them to the regular queue with a
|
||||
@@ -173,17 +219,28 @@ the `alerts` topic (future).
|
||||
The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
|
||||
When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
|
||||
`NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
|
||||
`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
|
||||
`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
|
||||
`GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
|
||||
**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
|
||||
**only** the `Enabled` gate + the store-and-forward knobs — the downstream
|
||||
gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
|
||||
section (see [Historian.md](Historian.md)).
|
||||
section (see [Historian.md](Historian.md)), and the queue's storage location is the
|
||||
node's consolidated `LocalDb:Path` database (see
|
||||
[Configuration.md](Configuration.md)).
|
||||
|
||||
> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
|
||||
> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
|
||||
> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
|
||||
> Two consequences follow, both covered in the
|
||||
> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
|
||||
> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
|
||||
> everything), and **delivery is at-least-once across a failover** by design. On first boot after
|
||||
> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
|
||||
> it `.migrated`.
|
||||
|
||||
```json
|
||||
{
|
||||
"AlarmHistorian": {
|
||||
"Enabled": true,
|
||||
"DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
|
||||
"BatchSize": 100,
|
||||
"DrainIntervalSeconds": 5,
|
||||
"Capacity": 1000000,
|
||||
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
|
||||
| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
|
||||
| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
|
||||
| `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
|
||||
| `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
|
||||
| `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
|
||||
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
|
||||
> `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
|
||||
> `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
|
||||
|
||||
> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
|
||||
> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
|
||||
> so leave it in place through the upgrade, then delete it once every node has an
|
||||
> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
|
||||
|
||||
> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
|
||||
> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
|
||||
> nobody is draining — see the runbook.
|
||||
|
||||
> Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
|
||||
|
||||
---
|
||||
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
|
||||
- [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
|
||||
most events into this sink.
|
||||
- [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
|
||||
- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
|
||||
— where the queue lives, the Primary-only drain, and the one-time legacy migration.
|
||||
|
||||
@@ -365,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
|
||||
- `IAlarmHistorianSink` is the DI-registered intake contract. The
|
||||
default binding is `NullAlarmHistorianSink` (registered in
|
||||
`ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
|
||||
deployments override it with `SqliteStoreAndForwardSink` wrapping
|
||||
deployments override it with `LocalDbStoreAndForwardSink` wrapping
|
||||
`GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
|
||||
— see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
|
||||
- `SqliteStoreAndForwardSink` queues each transition to a local
|
||||
SQLite database and drains in the background via an
|
||||
`IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
|
||||
- `LocalDbStoreAndForwardSink` queues each transition into the node's
|
||||
consolidated LocalDb — where it **replicates to the redundant pair peer**,
|
||||
so undelivered alarm history survives losing the node holding it — and
|
||||
drains in the background via an `IAlarmHistorianWriter`. **Only the node
|
||||
holding the Primary role drains**, since the peer holds a full replica;
|
||||
delivery is consequently at-least-once across a failover, by design. See
|
||||
[AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
|
||||
queue capacity defaults to 1,000,000 rows; under a sustained
|
||||
historian outage, older non-dead-lettered rows are evicted (oldest
|
||||
first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
|
||||
|
||||
@@ -136,8 +136,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar,
|
||||
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
|
||||
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
|
||||
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
|
||||
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward
|
||||
alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in
|
||||
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
|
||||
alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
|
||||
file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
|
||||
`ServerHistorian`; the other two sections source it from there.
|
||||
|
||||
The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess
|
||||
| [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) |
|
||||
| [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) |
|
||||
| [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) |
|
||||
| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
|
||||
| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
|
||||
| [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) |
|
||||
| [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` |
|
||||
| [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) |
|
||||
|
||||
+6
-2
@@ -221,15 +221,19 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
|
||||
|
||||
- **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
|
||||
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
|
||||
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
|
||||
|
||||
Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
|
||||
|
||||
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
|
||||
|
||||
## Pair-local config cache (LocalDb — Phase 1)
|
||||
## Pair-local store (LocalDb — Phases 1 + 2)
|
||||
|
||||
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
|
||||
[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database that caches the **deployed-configuration
|
||||
[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
|
||||
buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
|
||||
undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
|
||||
and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
|
||||
artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
|
||||
node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
|
||||
the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
|
||||
|
||||
@@ -1,18 +1,20 @@
|
||||
# LocalDb pair replication — operations runbook
|
||||
|
||||
> **Phase 1 scope.** Every driver-role OtOpcUa node keeps a consolidated
|
||||
> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. Today it caches exactly one thing: the
|
||||
> **deployed-configuration artifact**, chunked, so a node can **boot from cache when central SQL
|
||||
> Server is unreachable**. That cache can *optionally* replicate to the node's redundant pair peer
|
||||
> over the library's gRPC sync. **Replication is default-OFF and fail-closed.** This runbook covers
|
||||
> enabling it, the operational rules that keep a pair converging, and the DB-inspection safety
|
||||
> rules.
|
||||
> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
|
||||
> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
|
||||
> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
|
||||
> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
|
||||
> node that dies holding undelivered alarm history does not take it to the grave. Both can
|
||||
> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
|
||||
> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
|
||||
> rules that keep a pair converging, and the DB-inspection safety rules.
|
||||
|
||||
## What replicates, and what it buys you
|
||||
|
||||
- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks) and
|
||||
`deployment_pointer` (one current-deployment pointer per cluster). Registered — in this order,
|
||||
after the DDL — by `LocalDbSetup.OnReady`.
|
||||
- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
|
||||
`deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
|
||||
alarm store-and-forward buffer). Registered — in this order, after the DDL — by
|
||||
`LocalDbSetup.OnReady`.
|
||||
- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
|
||||
deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
|
||||
that restarts into a central-SQL outage can boot the current config **even if it never applied
|
||||
@@ -22,6 +24,47 @@
|
||||
deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
|
||||
replicate as tombstones.
|
||||
|
||||
## Alarm store-and-forward — what replication changes
|
||||
|
||||
Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
|
||||
consequences an operator needs to know:
|
||||
|
||||
- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
|
||||
Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
|
||||
native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
|
||||
Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
|
||||
- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
|
||||
moments before it died may not have had their deletes replicated yet, so the new Primary will
|
||||
re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
|
||||
missing one is not. There is deliberately no dedup layer.
|
||||
- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
|
||||
oldest undelivered rows when the historian has been unreachable long enough. Watch
|
||||
`EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
|
||||
|
||||
### If BOTH nodes report `NotPrimary`
|
||||
|
||||
Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
|
||||
default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
|
||||
redundancy snapshot that never names either node looks like — the identity-mismatch shape
|
||||
`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
|
||||
identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
|
||||
|
||||
### One-time migration from `alarm-historian.db`
|
||||
|
||||
On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
|
||||
`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
|
||||
|
||||
- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
|
||||
from the event payload, so the same event on both nodes converges to one row rather than
|
||||
duplicating. The new Primary drains the union.
|
||||
- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
|
||||
file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
|
||||
- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
|
||||
available for inspection.
|
||||
- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
|
||||
find the legacy file; it no longer configures anything. Delete it from appsettings once the
|
||||
`.migrated` sidecar exists on every node.
|
||||
|
||||
## Enabling replication on a pair
|
||||
|
||||
Replication has two knobs, both under `LocalDb`:
|
||||
@@ -127,10 +170,22 @@ FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_j
|
||||
|
||||
-- Replication backlog (should drain to 0 when a pair is caught up)
|
||||
SELECT COUNT(*) FROM __localdb_oplog;
|
||||
|
||||
-- Undelivered alarm history, and how much of it has given up
|
||||
SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
|
||||
|
||||
-- Why the dead-lettered rows died
|
||||
SELECT alarm_id, attempt_count, last_attempt_utc, last_error
|
||||
FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
|
||||
|
||||
-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
|
||||
SELECT pk_json, hlc, node_id, is_tombstone
|
||||
FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
|
||||
- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
|
||||
- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
|
||||
- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.
|
||||
|
||||
@@ -55,10 +55,11 @@
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: Rig config + docs",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
]
|
||||
],
|
||||
"note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md."
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
|
||||
Reference in New Issue
Block a user