From f76d1f91e5d3401dce84d144d1379c58a4dbdefd Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 21 Jul 2026 04:43:37 -0400 Subject: [PATCH] docs+chore(localdb): phase-2 rig config + docs Enables the alarm store-and-forward sink on all four driver nodes of the docker-dev rig -- the replicating site-a pair and the default-OFF site-b pin -- so the live gate has a real buffer to watch converge, and can see that site-b's sink works as a plain node-local queue with no peer traffic. Two rig details worth stating rather than rediscovering. The endpoint is deliberately unresolvable: there is no HistorianGateway here, so every drain attempt fails, which is exactly the historian-outage state the buffer exists for. But it still has to be a syntactically valid absolute http(s) URI or the host refuses to start, because ServerHistorianOptionsValidator is consumer-gated -- AlarmHistorian:Enabled=true makes the endpoint required even while ServerHistorian:Enabled=false. And MaxAttempts is raised far above the production default of 10, which against a permanently unreachable gateway would dead-letter the entire queue about five minutes in, turning a buffering test into a dead-letter test. Docs record what an operator now has to know: that a rising queue depth on a Secondary is correct and on BOTH nodes is not (that shape is a redundancy snapshot naming neither node -- check node identity before suspecting the sink), that delivery is at-least-once across a failover by design with no dedup layer, and that AlarmHistorian:DatabasePath is removed but should be left in place through the upgrade because the migrator still reads it to find the file to copy. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 30 +++- docker-dev/docker-compose.yml | 42 ++++++ docs/AlarmHistorian.md | 141 +++++++++++++----- docs/AlarmTracking.md | 12 +- docs/Configuration.md | 5 +- docs/README.md | 2 +- docs/Redundancy.md | 8 +- .../2026-07-20-localdb-pair-replication.md | 75 ++++++++-- ...7-20-localdb-adoption-phase2.md.tasks.json | 5 +- 9 files changed, 258 insertions(+), 62 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 06c1f7e5..0bed5c12 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -219,10 +219,13 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide. -## LocalDb pair-local config cache (Phase 1) +## LocalDb pair-local store (Phases 1 + 2) Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired -LiteDB `LocalCache` subsystem was deleted as superseded). Phase-1 scope: it caches the +LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**: +`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2). + +**Phase 1** caches the **deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`, @@ -234,8 +237,27 @@ library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAu on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret -exception). See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config -cache), and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`. +exception). + +**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db` +and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm +history no longer takes it to the grave. `SqliteStoreAndForwardSink` became +`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key +`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely +to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy +commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the +sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that +`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new +`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by +design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the +same event accepted on both nodes — which happens in every boot window, since +`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the +docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a +deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue. + +See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config +cache), `docs/AlarmHistorian.md`, and the runbook +`docs/operations/2026-07-20-localdb-pair-replication.md`. ## LDAP Authentication diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 2d2d712d..be3f3c59 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -328,6 +328,22 @@ services: # site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer. # Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and # re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS). + # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the + # buffer is a live, replicated table rather than an empty one — the live gate needs a real + # queue to watch converge, and needs to see that only the Primary drains it. + # + # There is no HistorianGateway on this rig, so the endpoint below is deliberately + # unresolvable: every drain attempt fails, which is exactly the historian-outage state the + # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s) + # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so + # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false. + # + # MaxAttempts is raised far above the production default of 10. With a permanently + # unreachable gateway the default would dead-letter the whole queue about five minutes into + # the outage, turning a buffering test into a dead-letter test. + AlarmHistorian__Enabled: "true" + AlarmHistorian__MaxAttempts: "1000000" + ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}" LocalDb__Path: "/app/data/otopcua-localdb.db" LocalDb__SyncListenPort: "9001" LocalDb__Replication__PeerAddress: "http://site-a-2:9001" @@ -365,6 +381,22 @@ services: # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key # is mandatory — the interceptor fail-closes on a mismatch). + # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the + # buffer is a live, replicated table rather than an empty one — the live gate needs a real + # queue to watch converge, and needs to see that only the Primary drains it. + # + # There is no HistorianGateway on this rig, so the endpoint below is deliberately + # unresolvable: every drain attempt fails, which is exactly the historian-outage state the + # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s) + # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so + # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false. + # + # MaxAttempts is raised far above the production default of 10. With a permanently + # unreachable gateway the default would dead-letter the whole queue about five minutes into + # the outage, turning a buffering test into a dead-letter test. + AlarmHistorian__Enabled: "true" + AlarmHistorian__MaxAttempts: "1000000" + ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}" LocalDb__Path: "/app/data/otopcua-localdb.db" LocalDb__SyncListenPort: "9001" LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key" @@ -397,6 +429,11 @@ services: # site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO # replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy. # This is what the live gate's check 8 verifies — the cache works locally with replication off. + # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink + # must work as a plain node-local buffer here, with no sync listener and no peer traffic. + AlarmHistorian__Enabled: "true" + AlarmHistorian__MaxAttempts: "1000000" + ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}" LocalDb__Path: "/app/data/otopcua-localdb.db" ports: - "4844:4840" @@ -422,6 +459,11 @@ services: Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning" GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}" # site-b default-OFF pin (see site-b-1): cache on, replication off. + # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink + # must work as a plain node-local buffer here, with no sync listener and no peer traffic. + AlarmHistorian__Enabled: "true" + AlarmHistorian__MaxAttempts: "1000000" + ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}" LocalDb__Path: "/app/data/otopcua-localdb.db" ports: - "4845:4840" diff --git a/docs/AlarmHistorian.md b/docs/AlarmHistorian.md index 212afe4c..26dc8efd 100644 --- a/docs/AlarmHistorian.md +++ b/docs/AlarmHistorian.md @@ -42,7 +42,7 @@ unless noted. - **`NullAlarmHistorianSink`** — the no-op default for tests and deployments that don't historize alarms. It is the default DI binding (registered in the Runtime's `AddOtOpcUaRuntime`); production overrides it with - `SqliteStoreAndForwardSink`. + `LocalDbStoreAndForwardSink`. - **`AlarmHistorianEvent`** ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs)) — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path, @@ -61,42 +61,84 @@ unless noted. - **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`, `LastError`, `DrainState`, and `EvictedCount`. -- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`. +- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` / + **`NotPrimary`** (ticking, but leaving the replicated queue for the node that + holds the Primary role). --- -## SqliteStoreAndForwardSink +## LocalDbStoreAndForwardSink -[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs) -is the production `IAlarmHistorianSink`. Construction takes a SQLite database -path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default -100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days), -and a test clock. +[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs) +is the production `IAlarmHistorianSink`. Construction takes the node's +`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` +(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 +days), a test clock, and a **`drainGate`**. + +### The drain gate + +`drainGate` is a `Func` consulted at the top of every tick; `false` skips +the tick entirely, leaving the queue untouched and reporting +`DrainState = NotPrimary`. + +It exists because the queue **replicates**. The Secondary holds a full copy of +the Primary's rows, and an ungated drain there would re-deliver every event, +continuously — not just at a failover. Runtime supplies the gate from +`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its +`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees +with the inbound-write and native-ack gates by construction. + +Two postures are deliberate: + +- **Unset (the default) means drain.** So does an unpublished view. A deployment + that runs no redundancy never publishes a role, and defaulting closed would + silently stop its alarm history forever. +- **A gate that throws means "not now", never permission.** Draining on a gate + fault would put a pair into dual delivery exactly when its redundancy signal is + sick. ### Queue table -The sink owns one SQLite table (created on construction, WAL journal mode): +The sink owns one table in the node's consolidated LocalDb — created and +registered for replication by `LocalDbSetup.OnReady`, before any consumer can +resolve the sink: ```sql -CREATE TABLE Queue ( - RowId INTEGER PRIMARY KEY AUTOINCREMENT, - AlarmId TEXT NOT NULL, - EnqueuedUtc TEXT NOT NULL, - PayloadJson TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent - AttemptCount INTEGER NOT NULL DEFAULT 0, - LastAttemptUtc TEXT NULL, - LastError TEXT NULL, - DeadLettered INTEGER NOT NULL DEFAULT 0 +CREATE TABLE alarm_sf_events ( + id TEXT NOT NULL PRIMARY KEY, -- SHA-256 of payload_json + alarm_id TEXT NOT NULL, + enqueued_at_utc TEXT NOT NULL, + payload_json TEXT NOT NULL, -- JSON-serialized AlarmHistorianEvent + attempt_count INTEGER NOT NULL DEFAULT 0, + last_attempt_utc TEXT NULL, + last_error TEXT NULL, + dead_lettered INTEGER NOT NULL DEFAULT 0 ); -CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId); +CREATE INDEX ix_alarm_sf_events_drain + ON alarm_sf_events (dead_lettered, enqueued_at_utc, id); ``` -`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a -`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered -row counter (seeded at startup, kept current by every mutation, and re-synced -from storage every 10,000 enqueues to defend against drift). SQLite writer -contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain -collision waits out the file lock instead of failing fast. +**The primary key is a hash of the payload, not an autoincrement rowid.** Two +reasons. Replication converges last-writer-wins *per primary key*, so two nodes +independently allocating rowid 7 to different alarms would silently overwrite one +another. And an equal-payload key makes the same event arriving twice collapse +into one row — which happens for real, because `HistorianAdapterActor` +default-writes while its redundancy role is unknown and both nodes of a pair +therefore accept the same fanned transition in every boot window. Two genuinely +distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision +timestamp alongside the alarm id, kind, message and user. + +Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id` +as a tiebreak so the ordering is total), because a hashed key carries no +insertion sequence. + +`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path. +To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory +non-dead-lettered row counter (seeded at startup, kept current by every mutation, +and re-synced from storage every 10,000 enqueues to defend against drift — now +doubly warranted, since replication applies the peer's rows straight into the +table where no local code path can observe them). Pragmas and connection +lifetime belong to LocalDb; the sink no longer manages either. ### Drain worker @@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast. `System.Threading.Timer`** (not started automatically — tests drive `DrainOnceAsync` deterministically). Each tick: +0. Consults the drain gate. A closed gate returns immediately, leaving the queue + untouched and `DrainState = NotPrimary`. 1. Purges aged dead-lettered rows past the retention window. -2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order. +2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order. 3. Rows with un-deserializable payloads are dead-lettered immediately (by their - own `RowId`) so they can't stall the queue head. + own `id`) so they can't stall the queue head. 4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and - each outcome is applied in one transaction: `Ack` deletes the row, - `PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt + each outcome is applied in one transaction: `Ack` deletes the row (the delete + replicates as a tombstone, so the peer drops its copy and a promoted standby + cannot re-send it), `PermanentFail` flips its `dead_lettered` flag, + `RetryPlease` bumps its attempt count and leaves it queued. A row whose `AttemptCount` has reached the configured **`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain tick rather than retried — this breaks infinite retry loops for poison events whose @@ -131,7 +177,7 @@ an unobserved task exception. **The durability guarantee is bounded by `capacity` (default 1,000,000 rows).** When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the -oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN, +oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN, and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian outage, accepted alarm events can therefore be dropped before delivery. A non-zero `EvictedCount` is a data-loss signal that requires operator attention — @@ -140,7 +186,7 @@ it surfaces silent loss without log scraping. ### Dead-letter + operator recovery `PermanentFail` and corrupt-payload rows are retained in-place with -`DeadLettered = 1` for the retention window (default 30 days) so operators can +`dead_lettered = 1` for the retention window (default 30 days) so operators can inspect them before the sweeper purges them. `RetryDeadLettered()` is the operator action (from the AdminUI) that clears the dead-letter flag and attempt count on every dead-lettered row, returning them to the regular queue with a @@ -173,17 +219,28 @@ the `alerts` topic (future). The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`. When `Enabled` is `false` (the default), `AddAlarmHistorian` registers `NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`, -`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers +`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers `GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries -**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream +**only** the `Enabled` gate + the store-and-forward knobs — the downstream gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian` -section (see [Historian.md](Historian.md)). +section (see [Historian.md](Historian.md)), and the queue's storage location is the +node's consolidated `LocalDb:Path` database (see +[Configuration.md](Configuration.md)). + +> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's +> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to +> the redundant pair peer**, so undelivered alarm history survives losing the node holding it. +> Two consequences follow, both covered in the +> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md): +> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send +> everything), and **delivery is at-least-once across a failover** by design. On first boot after +> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames +> it `.migrated`. ```json { "AlarmHistorian": { "Enabled": true, - "DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db", "BatchSize": 100, "DrainIntervalSeconds": 5, "Capacity": 1000000, @@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)). | Key | Type | Default | Description | |---|---|---|---| -| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. | -| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. | +| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. | | `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. | | `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. | | `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). | @@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)). > `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` / > `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned. +> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by +> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate — +> so leave it in place through the upgrade, then delete it once every node has an +> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing. + +> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a +> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means +> nobody is draining — see the runbook. + > Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured. --- @@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)). - [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits most events into this sink. - [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend. +- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md) + — where the queue lives, the Primary-only drain, and the one-time legacy migration. diff --git a/docs/AlarmTracking.md b/docs/AlarmTracking.md index 4f00551d..399359b7 100644 --- a/docs/AlarmTracking.md +++ b/docs/AlarmTracking.md @@ -365,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway: - `IAlarmHistorianSink` is the DI-registered intake contract. The default binding is `NullAlarmHistorianSink` (registered in `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production - deployments override it with `SqliteStoreAndForwardSink` wrapping + deployments override it with `LocalDbStoreAndForwardSink` wrapping `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path) — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup. -- `SqliteStoreAndForwardSink` queues each transition to a local - SQLite database and drains in the background via an - `IAlarmHistorianWriter`. **The durability guarantee is bounded**: the +- `LocalDbStoreAndForwardSink` queues each transition into the node's + consolidated LocalDb — where it **replicates to the redundant pair peer**, + so undelivered alarm history survives losing the node holding it — and + drains in the background via an `IAlarmHistorianWriter`. **Only the node + holding the Primary role drains**, since the peer holds a full replica; + delivery is consequently at-least-once across a failover, by design. See + [AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the queue capacity defaults to 1,000,000 rows; under a sustained historian outage, older non-dead-lettered rows are evicted (oldest first) to make room for new events. The `HistorianSinkStatus.EvictedCount` diff --git a/docs/Configuration.md b/docs/Configuration.md index 5f01bfdc..c4b1ecb3 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -136,8 +136,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar, `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization` -(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward -alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in +(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward +alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a +file of its own). The gateway connection (endpoint / key / TLS) lives **only** in `ServerHistorian`; the other two sections source it from there. The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed diff --git a/docs/README.md b/docs/README.md index c5df5b74..6c320064 100644 --- a/docs/README.md +++ b/docs/README.md @@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess | [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) | | [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) | | [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) | -| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction | +| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction | | [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) | | [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` | | [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) | diff --git a/docs/Redundancy.md b/docs/Redundancy.md index 6692bed9..0a0dff1d 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -221,15 +221,19 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev - **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover. - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size. +- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md). Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node. See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals. -## Pair-local config cache (LocalDb — Phase 1) +## Pair-local store (LocalDb — Phases 1 + 2) Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated -[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database that caches the **deployed-configuration +[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward +buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding +undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above +and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache diff --git a/docs/operations/2026-07-20-localdb-pair-replication.md b/docs/operations/2026-07-20-localdb-pair-replication.md index 78422482..8b7fc525 100644 --- a/docs/operations/2026-07-20-localdb-pair-replication.md +++ b/docs/operations/2026-07-20-localdb-pair-replication.md @@ -1,18 +1,20 @@ # LocalDb pair replication — operations runbook -> **Phase 1 scope.** Every driver-role OtOpcUa node keeps a consolidated -> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. Today it caches exactly one thing: the -> **deployed-configuration artifact**, chunked, so a node can **boot from cache when central SQL -> Server is unreachable**. That cache can *optionally* replicate to the node's redundant pair peer -> over the library's gRPC sync. **Replication is default-OFF and fail-closed.** This runbook covers -> enabling it, the operational rules that keep a pair converging, and the DB-inspection safety -> rules. +> **Scope.** Every driver-role OtOpcUa node keeps a consolidated +> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the +> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when +> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a +> node that dies holding undelivered alarm history does not take it to the grave. Both can +> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync. +> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational +> rules that keep a pair converging, and the DB-inspection safety rules. ## What replicates, and what it buys you -- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks) and - `deployment_pointer` (one current-deployment pointer per cluster). Registered — in this order, - after the DDL — by `LocalDbSetup.OnReady`. +- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks), + `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the + alarm store-and-forward buffer). Registered — in this order, after the DDL — by + `LocalDbSetup.OnReady`. - **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node that restarts into a central-SQL outage can boot the current config **even if it never applied @@ -22,6 +24,47 @@ deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes replicate as tombstones. +## Alarm store-and-forward — what replication changes + +Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three +consequences an operator needs to know: + +- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the + Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and + native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a + Secondary is correct.** A rising queue depth on *both* nodes is not — see below. +- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered + moments before it died may not have had their deletes replicated yet, so the new Primary will + re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a + missing one is not. There is deliberately no dedup layer. +- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the + oldest undelivered rows when the historian has been unreachable long enough. Watch + `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian. + +### If BOTH nodes report `NotPrimary` + +Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate +default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a +redundancy snapshot that never names either node looks like — the identity-mismatch shape +`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node +identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink. + +### One-time migration from `alarm-historian.db` + +On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing +`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`. + +- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived + from the event payload, so the same event on both nodes converges to one row rather than + duplicating. The new Primary drains the union. +- **The rename happens only after the copy commits.** A failure fails host startup with the legacy + file untouched; the `.migrated` sidecar is what makes a later boot a no-op. +- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays + available for inspection. +- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to + find the legacy file; it no longer configures anything. Delete it from appsettings once the + `.migrated` sidecar exists on every node. + ## Enabling replication on a pair Replication has two knobs, both under `LocalDb`: @@ -127,10 +170,22 @@ FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_j -- Replication backlog (should drain to 0 when a pair is caught up) SELECT COUNT(*) FROM __localdb_oplog; + +-- Undelivered alarm history, and how much of it has given up +SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered; + +-- Why the dead-lettered rows died +SELECT alarm_id, attempt_count, last_attempt_utc, last_error +FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20; + +-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows +SELECT pk_json, hlc, node_id, is_tombstone +FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json; ``` ## See also - [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section. - [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section. +- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain. - The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`. diff --git a/docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json index 7e93a5bd..104b2e38 100644 --- a/docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json @@ -55,10 +55,11 @@ { "id": 6, "subject": "Task 6: Rig config + docs", - "status": "pending", + "status": "completed", "blockedBy": [ 3 - ] + ], + "note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md." }, { "id": 7,