dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)

Browse Source 
Moves the alarm store-and-forward buffer out of its own alarm-historian.db
and into the node's consolidated LocalDb, where it replicates to the
redundant pair peer. A node that dies holding undelivered alarm history no
longer takes it to the grave.

Tasks 2 and 3 land together, as the plan anticipated. They are not separable:
the gate is a constructor argument of the rewritten sink, and a commit that
replicated the queue without gating the drain would be a commit in which both
nodes of a pair deliver every alarm event, continuously.

That drain gate is the load-bearing part of the change, and the recon
explains why it is new work rather than a refinement. Exactly-once delivery
across a pair is enforced today on the ENQUEUE side, by
HistorianAdapterActor: only the Primary enqueues, so the Secondary's queue is
empty and its ungated drain has nothing to send. Replicating the table
destroys that invariant.

The gate is a Func<bool> the caller supplies, because the drain runs on a
timer the sink owns rather than on a mailbox, and Core.AlarmHistorian cannot
reference PrimaryGatePolicy in Runtime. Runtime supplies it via a new
IRedundancyRoleView singleton that DriverHostActor publishes its Primary-gate
verdict to -- the same verdict the inbound-write and native-ack gates use, so
there is no second notion of am-I-the-Primary to drift.

Two failure modes are deliberately closed:

- The view is seeded OPEN, matching the policy's own answer for an unknown
  role with no driver peer. A deployment that runs no redundancy never
  publishes to it, and defaulting closed would silently stop its alarm
  history forever.

- A gate that throws is read as not-now, never as permission, and a closed
  gate reports the new HistorianDrainState.NotPrimary rather than Idle. A
  Secondary's rising queue is supposed to look different from a stalled
  drain, and if BOTH nodes report NotPrimary the pair is misconfigured and
  says so instead of quietly filling toward the capacity ceiling.

Row ids are a hash of the payload rather than fresh GUIDs. Both adapters
accept the same fanned transition in the window before the first redundancy
snapshot arrives, and under last-writer-wins an equal key converges those two
accepts into one row instead of duplicating them.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 04:13:00 -04:00
parent 8a9cb40a72
commit 71379816e7
14 changed files with 804 additions and 365 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Directory.Packages.props
+1
View File
@@ -57,6 +57,7 @@
<PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
<PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.7" />
<PackageVersion Include="Microsoft.FASTER.Core" Version="2.6.5" />
<PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
<PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.7" />
docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json
+6 -4
View File
@@ -19,18 +19,20 @@
{
"id": 2,
"subject": "Task 2: Rewire sink onto ILocalDb + delete bespoke file management (cutover 1/2)",
"status": "pending",
"status": "completed",
"blockedBy": [
1
]
],
"note": "Sink rewritten as LocalDbStoreAndForwardSink over ILocalDb; bespoke file/pragma/schema management deleted with the old class. AlarmHistorian:DatabasePath removed (breaking config key). Ids are a deterministic payload hash (D-1), not GUIDs."
},
{
"id": 3,
"subject": "Task 3: Primary-gated drain via PrimaryGatePolicy (cutover 2/2 \u2014 may co-commit with Task 2)",
"status": "pending",
"status": "completed",
"blockedBy": [
2
]
],
"note": "Drain gated on IRedundancyRoleView, a singleton DriverHostActor publishes PrimaryGatePolicy's verdict to on every snapshot. Fails closed on a throwing gate; seeded OPEN so a non-redundant deployment is never silently stopped. New HistorianDrainState.NotPrimary + transition-logged (D-2). Landed with Task 2 as one commit (D-3)."
},
{
"id": 4,
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs
+14 -2
View File
@@ -4,13 +4,13 @@ namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
/// The historian sink contract — where qualifying alarm events land. Ingestion routes
/// through the HistorianGateway alarm writer (the gateway's <c>SendEvent</c> gRPC path)
/// behind the durable store-and-forward queue. Tests use an in-memory fake; production uses
/// <see cref="SqliteStoreAndForwardSink"/>.
/// <see cref="LocalDbStoreAndForwardSink"/>.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="EnqueueAsync"/> is fire-and-forget from the engine's perspective —
/// the sink MUST NOT block the emitting thread. Production implementations
/// (<see cref="SqliteStoreAndForwardSink"/>) persist to a local SQLite queue
/// (<see cref="LocalDbStoreAndForwardSink"/>) persist to the node's local SQLite queue
/// first, then drain asynchronously to the actual historian. Per the Phase 7 plan,
/// failed downstream writes replay with exponential backoff;
/// operator actions are never blocked waiting on the historian.
@@ -79,6 +79,18 @@ public enum HistorianDrainState
Idle,
Draining,
BackingOff,
/// <summary>
/// Ticking, but not draining: this node does not hold the Primary role, so it leaves the
/// (replicated) queue for the node that does.
/// </summary>
/// <remarks>
/// Distinct from <see cref="Idle"/> so a rising queue depth on a Secondary reads as the
/// designed behaviour rather than a stalled drain. It is also how a misconfigured pair
/// becomes diagnosable: if BOTH nodes report this, no one is draining and alarm history is
/// silently accumulating toward the capacity ceiling.
/// </remarks>
NotPrimary,
}
/// <summary>Returned by the historian alarm writer per event — drain worker uses this to decide retry cadence.</summary>
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs → src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs
+279 -243
View File
@@ -1,53 +1,56 @@
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using Microsoft.Data.Sqlite;
using Serilog;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
/// <summary>
/// Durable SQLite queue on the node absorbs every qualifying alarm event, a drain
/// worker batches rows to the Wonderware historian sidecar via
/// <see cref="IAlarmHistorianWriter"/> on an exponential-backoff cadence, and
/// operator acks never block on the historian being reachable.
/// Durable queue in the node's consolidated LocalDb absorbs every qualifying alarm event, a
/// drain worker batches rows to the historian gateway via <see cref="IAlarmHistorianWriter"/>
/// on an exponential-backoff cadence, and operator acks never block on the historian being
/// reachable.
/// </summary>
/// <remarks>
/// <para>
/// Queue schema:
/// <code>
/// CREATE TABLE Queue (
/// RowId INTEGER PRIMARY KEY AUTOINCREMENT,
/// AlarmId TEXT NOT NULL,
/// EnqueuedUtc TEXT NOT NULL,
/// PayloadJson TEXT NOT NULL,
/// AttemptCount INTEGER NOT NULL DEFAULT 0,
/// LastAttemptUtc TEXT NULL,
/// LastError TEXT NULL,
/// DeadLettered INTEGER NOT NULL DEFAULT 0
/// );
/// </code>
/// Dead-lettered rows stay in place for the configured retention window (default
/// 30 days) so operators can inspect + manually
/// retry before the sweeper purges them. Regular queue capacity is bounded —
/// overflow evicts the oldest non-dead-lettered rows with a WARN log. The
/// durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>:
/// under a sustained historian outage, accepted events may be evicted before
/// delivery. The <see cref="HistorianSinkStatus.EvictedCount"/> counter makes
/// overflow visible to operators without requiring the WARN log to be scraped.
/// Rows live in <c>alarm_sf_events</c> (see <see cref="AlarmSfSchema"/>), which the host
/// registers for replication. That is the point of this type living on
/// <see cref="ILocalDb"/> rather than owning its own file: the buffer mirrors to the
/// redundant pair peer, so a node that dies holding undelivered alarm history no longer
/// takes it to the grave.
/// </para>
/// <para>
/// <b>Only one node of a pair may drain.</b> Replication puts the Primary's queued rows in
/// the Secondary's table too; an ungated drain there would re-deliver every event,
/// continuously. The <c>drainGate</c> constructor argument is how the caller scopes
/// draining to the Primary. Enqueue is gated separately and upstream, by
/// <c>HistorianAdapterActor</c>.
/// </para>
/// <para>
/// Dead-lettered rows stay in place for the configured retention window (default 30 days)
/// so operators can inspect + manually retry before the sweeper purges them. Regular queue
/// capacity is bounded — overflow evicts the oldest non-dead-lettered rows with a WARN log.
/// The durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>: under a
/// sustained historian outage, accepted events may be evicted before delivery. The
/// <see cref="HistorianSinkStatus.EvictedCount"/> counter makes overflow visible to
/// operators without requiring the WARN log to be scraped.
/// </para>
/// <para>
/// Drain runs on a self-rescheduling one-shot <see cref="System.Threading.Timer"/>.
/// Exponential backoff on <see cref="HistorianWriteOutcome.RetryPlease"/>:
/// 1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next
/// due-time, so a historian outage genuinely slows the drain cadence.
/// <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip
/// the <c>DeadLettered</c> flag on the individual row; neighbors in the batch
/// still retry on their own cadence.
/// 1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next due-time, so a
/// historian outage genuinely slows the drain cadence.
/// <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip the <c>dead_lettered</c> flag
/// on the individual row; neighbors in the batch still retry on their own cadence.
/// </para>
/// </remarks>
public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public sealed class LocalDbStoreAndForwardSink : IAlarmHistorianSink, IDisposable
{
/// <summary>Default queue capacity — oldest non-dead-lettered rows evicted past this.</summary>
public const long DefaultCapacity = 1_000_000;
/// <summary>Default window dead-lettered rows are retained for before the sweeper purges them.</summary>
public static readonly TimeSpan DefaultDeadLetterRetention = TimeSpan.FromDays(30);
/// <summary>Default max delivery attempts before a perpetually-retrying (poison) row is dead-lettered.</summary>
@@ -62,7 +65,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
TimeSpan.FromSeconds(60),
];
private readonly string _connectionString;
private readonly ILocalDb _db;
private readonly IAlarmHistorianWriter _writer;
private readonly ILogger _logger;
private readonly int _batchSize;
@@ -70,8 +73,9 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
private readonly TimeSpan _deadLetterRetention;
private readonly int _maxAttempts;
private readonly Func<DateTime> _clock;
private readonly Func<bool> _drainGate;
private readonly SemaphoreSlim _drainGate = new(1, 1);
private readonly SemaphoreSlim _drainGateLock = new(1, 1);
private Timer? _drainTimer;
private TimeSpan _tickInterval;
private volatile int _backoffIndex;
@@ -85,13 +89,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
private string? _lastError;
private HistorianDrainState _drainState = HistorianDrainState.Idle;
private long _evictedCount;
// Tracks the last gate decision so a denial is logged on the transition rather than on every
// tick. Null until the first drain runs.
private bool? _lastGateDecision;
private long _queuedRowCount;
// Probe counter — incremented every time we actually issue a real COUNT(*) for
// capacity enforcement. Public for test instrumentation only.
private long _capacityProbeCount;
// After every Nth enqueue we resync the in-memory counter from storage to defend
// against silent drift (e.g. an external process editing the DB).
// against silent drift (e.g. the peer replicating rows in underneath us).
private const long ResyncEnqueueInterval = 10_000;
private long _enqueuesSinceResync;
@@ -99,9 +106,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public long DebugCapacityProbeCount => Interlocked.Read(ref _capacityProbeCount);
/// <summary>
/// Initializes a new instance of the <see cref="SqliteStoreAndForwardSink"/> class with the specified configuration.
/// Initializes a new instance of the <see cref="LocalDbStoreAndForwardSink"/> class.
/// </summary>
/// <param name="databasePath">The filesystem path to the SQLite database file.</param>
/// <param name="db">
/// The node's local database. Its <c>alarm_sf_events</c> table must already exist and be
/// registered for replication — the host does both in <c>LocalDbSetup.OnReady</c>, before
/// any consumer can resolve this sink.
/// </param>
/// <param name="writer">The alarm historian writer to handle batch forwarding.</param>
/// <param name="logger">The logger for diagnostic output.</param>
/// <param name="batchSize">The maximum number of rows to forward in a single batch. Defaults to 100.</param>
@@ -109,18 +120,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <param name="deadLetterRetention">The timespan to retain dead-lettered rows before purging. Defaults to 30 days.</param>
/// <param name="maxAttempts">The maximum number of delivery attempts before a perpetually-retrying (poison) row is dead-lettered. Defaults to 10.</param>
/// <param name="clock">Optional clock function for testing; defaults to <see cref="DateTime.UtcNow"/>.</param>
public SqliteStoreAndForwardSink(
string databasePath,
/// <param name="drainGate">
/// Consulted at the top of every drain tick; <c>false</c> skips the tick entirely, leaving
/// the queue intact. Callers running a redundant pair MUST supply a gate that is true on at
/// most one node, because the queue replicates — see the class remarks. The default
/// (<c>null</c> ⇒ always drain) is the correct single-node and test posture.
/// </param>
public LocalDbStoreAndForwardSink(
ILocalDb db,
IAlarmHistorianWriter writer,
ILogger logger,
int batchSize = 100,
long capacity = DefaultCapacity,
TimeSpan? deadLetterRetention = null,
int maxAttempts = DefaultMaxAttempts,
Func<DateTime>? clock = null)
Func<DateTime>? clock = null,
Func<bool>? drainGate = null)
{
if (string.IsNullOrWhiteSpace(databasePath))
throw new ArgumentException("Database path required.", nameof(databasePath));
_db = db ?? throw new ArgumentNullException(nameof(db));
_writer = writer ?? throw new ArgumentNullException(nameof(writer));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_batchSize = batchSize > 0 ? batchSize : throw new ArgumentOutOfRangeException(nameof(batchSize));
@@ -128,50 +145,11 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
_deadLetterRetention = deadLetterRetention ?? DefaultDeadLetterRetention;
_maxAttempts = maxAttempts > 0 ? maxAttempts : throw new ArgumentOutOfRangeException(nameof(maxAttempts));
_clock = clock ?? (() => DateTime.UtcNow);
// DefaultTimeout gives ADO.NET command-level retry; the PRAGMA busy_timeout
// applied in OpenConnection backs it with SQLite's own busy-handler so an
// enqueue/drain collision waits out the file lock instead of throwing
// SQLITE_BUSY immediately.
_connectionString = new SqliteConnectionStringBuilder
{
DataSource = databasePath,
DefaultTimeout = 5,
}.ToString();
_drainGate = drainGate ?? (static () => true);
InitializeSchema();
_queuedRowCount = ProbeQueuedRowCount();
}
/// <summary>
/// Open a connection with the busy timeout + WAL journal applied. SQLite
/// serializes writers with a file lock; the busy_timeout lets a writer wait
/// out a competing lock (default is 0 — fail fast), and WAL lets readers and
/// the single writer proceed without blocking each other.
/// </summary>
private SqliteConnection OpenConnection()
{
var conn = new SqliteConnection(_connectionString);
conn.Open();
ApplyPragmas(conn);
return conn;
}
/// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (sync).</summary>
private static void ApplyPragmas(SqliteConnection conn)
{
using var pragma = conn.CreateCommand();
pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
pragma.ExecuteNonQuery();
}
/// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (async).</summary>
private static async Task ApplyPragmasAsync(SqliteConnection conn, CancellationToken ct)
{
using var pragma = conn.CreateCommand();
pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
await pragma.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
/// <summary>
/// Start the background drain worker. Not started automatically so tests can
/// drive <see cref="DrainOnceAsync"/> deterministically.
@@ -188,7 +166,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <param name="tickInterval">The base interval between drain attempts.</param>
public void StartDrainLoop(TimeSpan tickInterval)
{
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
_tickInterval = tickInterval;
_drainTimer?.Dispose();
// One-shot: dueTime = tickInterval, period = Infinite. RescheduleDrain re-arms
@@ -235,29 +213,50 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
catch (ObjectDisposedException) { /* raced with Dispose — nothing to re-arm */ }
}
/// <summary>
/// Derives the row's primary key from its payload.
/// </summary>
/// <remarks>
/// Deterministic rather than a fresh GUID so that the same event enqueued independently on
/// both nodes of a pair converges to one row under last-writer-wins instead of duplicating.
/// That happens for real: <c>HistorianAdapterActor</c> default-writes while its redundancy
/// role is unknown, so in the window before the first snapshot arrives both adapters accept
/// the same fanned transition. Two distinct events cannot collide —
/// <see cref="AlarmHistorianEvent"/> carries a full-precision timestamp alongside the alarm
/// id, kind, message and user, so an equal hash means an equal event.
/// </remarks>
private static string DeriveId(string payloadJson) =>
Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(payloadJson)));
/// <inheritdoc />
public async Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
{
if (evt is null) throw new ArgumentNullException(nameof(evt));
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
using var conn = new SqliteConnection(_connectionString);
await conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await ApplyPragmasAsync(conn, cancellationToken).ConfigureAwait(false);
await EnforceCapacityFastPathAsync(cancellationToken).ConfigureAwait(false);
await EnforceCapacityFastPathAsync(conn, cancellationToken).ConfigureAwait(false);
var payload = JsonSerializer.Serialize(evt);
using var cmd = conn.CreateCommand();
cmd.CommandText = """
INSERT INTO Queue (AlarmId, EnqueuedUtc, PayloadJson, AttemptCount)
VALUES ($alarmId, $enqueued, $payload, 0);
""";
cmd.Parameters.AddWithValue("$alarmId", evt.AlarmId);
cmd.Parameters.AddWithValue("$enqueued", _clock().ToString("O"));
cmd.Parameters.AddWithValue("$payload", JsonSerializer.Serialize(evt));
await cmd.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
// ON CONFLICT DO NOTHING: re-enqueuing an identical event (the pair's boot-window
// double-accept) must not resurrect a row the drain already bumped or dead-lettered.
var inserted = await _db.ExecuteAsync(
"""
INSERT INTO alarm_sf_events
(id, alarm_id, enqueued_at_utc, payload_json, attempt_count)
VALUES (@Id, @AlarmId, @EnqueuedAtUtc, @PayloadJson, 0)
ON CONFLICT(id) DO NOTHING
""",
new
{
Id = DeriveId(payload),
AlarmId = evt.AlarmId,
EnqueuedAtUtc = _clock().ToString("O"),
PayloadJson = payload,
},
cancellationToken).ConfigureAwait(false);
Interlocked.Increment(ref _queuedRowCount);
if (inserted > 0) Interlocked.Increment(ref _queuedRowCount);
}
/// <summary>
@@ -268,15 +267,17 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// through <see cref="EnforceCapacityAsync"/> which still runs a precise
/// COUNT to compute the exact number of rows to evict.
/// </summary>
private async Task EnforceCapacityFastPathAsync(SqliteConnection conn, CancellationToken ct)
private async Task EnforceCapacityFastPathAsync(CancellationToken ct)
{
var enqueuesSinceResync = Interlocked.Increment(ref _enqueuesSinceResync);
var cached = Interlocked.Read(ref _queuedRowCount);
// Periodic resync — bounded amount of drift even under exotic conditions.
// Periodic resync — bounded amount of drift even under exotic conditions. Now doubly
// warranted: replication applies the peer's rows straight into the table, so the counter
// has a second way to fall behind that no local code path can observe.
if (enqueuesSinceResync >= ResyncEnqueueInterval)
{
await ResyncQueuedRowCountAsync(conn, ct).ConfigureAwait(false);
await ResyncQueuedRowCountAsync(ct).ConfigureAwait(false);
cached = Interlocked.Read(ref _queuedRowCount);
Interlocked.Exchange(ref _enqueuesSinceResync, 0);
}
@@ -286,55 +287,70 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
// Cached counter says we're at or above the capacity wall — fall back to the
// precise path which probes COUNT(*) and evicts whatever's needed.
await EnforceCapacityAsync(conn, ct).ConfigureAwait(false);
await EnforceCapacityAsync(ct).ConfigureAwait(false);
}
/// <summary>Synchronously query <c>COUNT(*)</c> of non-dead-lettered rows. Used at startup.</summary>
private long ProbeQueuedRowCount()
{
Interlocked.Increment(ref _capacityProbeCount);
using var conn = OpenConnection();
using var conn = _db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 0";
return (long)(cmd.ExecuteScalar() ?? 0L);
}
/// <summary>Re-sync the in-memory counter from storage (async path).</summary>
private async Task ResyncQueuedRowCountAsync(SqliteConnection conn, CancellationToken ct)
private async Task ResyncQueuedRowCountAsync(CancellationToken ct)
{
Interlocked.Increment(ref _capacityProbeCount);
using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
var live = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
var live = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
Interlocked.Exchange(ref _queuedRowCount, live);
}
private async Task<long> CountAsync(string predicate, CancellationToken ct)
{
var rows = await _db.QueryAsync(
$"SELECT COUNT(*) FROM alarm_sf_events WHERE {predicate}",
r => r.GetInt64(0),
parameters: null,
ct).ConfigureAwait(false);
return rows.Count > 0 ? rows[0] : 0L;
}
/// <summary>
/// Read up to <see cref="_batchSize"/> queued rows, forward through the writer,
/// remove Ack'd rows, dead-letter PermanentFail rows, and extend the backoff
/// on RetryPlease. Safe to call from multiple threads; the semaphore enforces
/// serial execution.
/// </summary>
/// <remarks>
/// Returns without touching the queue when the drain gate is closed. On a redundant pair
/// that is the Secondary's steady state: its table fills by replication and stays put until
/// it is promoted.
/// </remarks>
/// <param name="ct">Cancellation token for the operation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task DrainOnceAsync(CancellationToken ct)
{
if (_disposed) return;
if (!await _drainGate.WaitAsync(0, ct).ConfigureAwait(false)) return;
if (!await _drainGateLock.WaitAsync(0, ct).ConfigureAwait(false)) return;
try
{
if (!EvaluateDrainGate())
{
lock (_statusLock) { _drainState = HistorianDrainState.NotPrimary; }
return;
}
lock (_statusLock)
{
_drainState = HistorianDrainState.Draining;
_lastDrainUtc = _clock();
}
// One connection per drain tick — used by purge, read, corrupt-dead-letter,
// and the outcome-applying transaction.
using var conn = OpenConnection();
PurgeAgedDeadLetters(conn);
var batch = ReadBatch(conn);
await PurgeAgedDeadLettersAsync(ct).ConfigureAwait(false);
var batch = await ReadBatchAsync(ct).ConfigureAwait(false);
if (batch.Count == 0)
{
lock (_statusLock) { _drainState = HistorianDrainState.Idle; }
@@ -342,22 +358,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
}
// A null/un-deserializable payload can never succeed — dead-letter it
// immediately for its own RowId so it cannot stall the queue head, and
// immediately for its own id so it cannot stall the queue head, and
// exclude it from the batch handed to the writer.
var corruptRowIds = batch.Where(r => r.Event is null).Select(r => r.RowId).ToList();
var corruptIds = batch.Where(r => r.Event is null).Select(r => r.Id).ToList();
var liveRows = batch.Where(r => r.Event is not null).ToList();
var events = liveRows.Select(r => r.Event!).ToList();
if (corruptRowIds.Count > 0)
if (corruptIds.Count > 0)
{
using var corruptTx = conn.BeginTransaction();
foreach (var rowId in corruptRowIds)
DeadLetterRow(conn, corruptTx, rowId, $"corrupt payload at {_clock():O}");
corruptTx.Commit();
Interlocked.Add(ref _queuedRowCount, -corruptRowIds.Count);
await using (var corruptTx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
{
foreach (var id in corruptIds)
await DeadLetterRowAsync(corruptTx, id, $"corrupt payload at {_clock():O}", ct).ConfigureAwait(false);
await corruptTx.CommitAsync(ct).ConfigureAwait(false);
}
Interlocked.Add(ref _queuedRowCount, -corruptIds.Count);
_logger.Warning(
"Dead-lettered {Count} historian queue row(s) with un-deserializable payload",
corruptRowIds.Count);
corruptIds.Count);
}
if (events.Count == 0)
@@ -404,41 +422,44 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
}
int rowsLeavingQueue = 0;
using (var tx = conn.BeginTransaction())
await using (var tx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
{
for (var i = 0; i < outcomes.Count; i++)
{
var outcome = outcomes[i];
var rowId = liveRows[i].RowId;
var id = liveRows[i].Id;
switch (outcome)
{
case HistorianWriteOutcome.Ack:
DeleteRow(conn, tx, rowId);
// Deleted, not marked delivered. The replication engine carries the
// delete as a tombstone, so the peer drops its copy and cannot
// re-deliver the row after a failover.
await DeleteRowAsync(tx, id, ct).ConfigureAwait(false);
rowsLeavingQueue++;
break;
case HistorianWriteOutcome.PermanentFail:
DeadLetterRow(conn, tx, rowId, $"permanent fail at {_clock():O}");
await DeadLetterRowAsync(tx, id, $"permanent fail at {_clock():O}", ct).ConfigureAwait(false);
rowsLeavingQueue++;
break;
case HistorianWriteOutcome.RetryPlease:
// finding 002: cap retries so a perpetually-RetryPlease (poison)
// row cannot retry forever at the 60s backoff floor. The incoming
// AttemptCount is the count BEFORE this attempt; +1 accounts for the
// attempt count is the count BEFORE this attempt; +1 accounts for the
// bump this drain represents. At the cap, dead-letter instead of
// bumping — and count it as leaving the live queue like PermanentFail.
if (liveRows[i].AttemptCount + 1 >= _maxAttempts)
{
DeadLetterRow(conn, tx, rowId, $"max attempts ({_maxAttempts}) exceeded");
await DeadLetterRowAsync(tx, id, $"max attempts ({_maxAttempts}) exceeded", ct).ConfigureAwait(false);
rowsLeavingQueue++;
}
else
{
BumpAttempt(conn, tx, rowId, "retry-please");
await BumpAttemptAsync(tx, id, "retry-please", ct).ConfigureAwait(false);
}
break;
}
}
tx.Commit();
await tx.CommitAsync(ct).ConfigureAwait(false);
}
// Ack-deleted + PermanentFail-dead-lettered rows both leave the
// non-dead-lettered queue, as do RetryPlease rows that hit the max-attempts
@@ -464,22 +485,66 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
}
finally
{
_drainGate.Release();
_drainGateLock.Release();
}
}
/// <summary>
/// Consults the injected gate, logging only when the decision changes.
/// </summary>
/// <remarks>
/// Logged at all because the closed state is indistinguishable from a healthy idle queue
/// from the outside, and one way to reach it is a redundancy snapshot that never names this
/// node — the documented identity-mismatch shape. Without this line, that misconfiguration
/// would present as alarm history quietly ceasing on both nodes of a pair. Logged on the
/// transition rather than per tick because the drain runs every few seconds.
/// </remarks>
/// <returns><c>true</c> when this tick may proceed.</returns>
private bool EvaluateDrainGate()
{
bool open;
try
{
open = _drainGate();
}
catch (Exception ex)
{
// A throwing gate must not wedge the queue silently, and must not be treated as
// permission either: skip this tick, say why, and re-ask on the next one.
_logger.Error(ex, "Historian drain gate threw; skipping this tick");
return false;
}
bool? previous;
lock (_statusLock)
{
previous = _lastGateDecision;
_lastGateDecision = open;
}
if (previous == open) return open;
if (open)
_logger.Information("Historian drain resumed — this node now services the alarm queue");
else
_logger.Information(
"Historian drain suspended — this node is not the Primary. Queued alarm events are retained and will drain from whichever node holds the Primary role");
return open;
}
/// <inheritdoc />
public HistorianSinkStatus GetStatus()
{
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
var queued = Interlocked.Read(ref _queuedRowCount);
if (queued < 0) queued = 0;
long deadlettered;
using (var conn = OpenConnection())
using (var conn = _db.CreateConnection())
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 1";
cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 1";
deadlettered = (long)(cmd.ExecuteScalar() ?? 0L);
}
@@ -510,10 +575,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <returns>The number of rows revived from the dead-letter state.</returns>
public int RetryDeadLettered()
{
if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
using var conn = OpenConnection();
if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
// Through a LocalDb connection, not a private one: the capture triggers are what make this
// revival replicate to the peer, and they only exist on the registered table.
using var conn = _db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = "UPDATE Queue SET DeadLettered = 0, AttemptCount = 0, LastError = NULL WHERE DeadLettered = 1";
cmd.CommandText =
"UPDATE alarm_sf_events SET dead_lettered = 0, attempt_count = 0, last_error = NULL WHERE dead_lettered = 1";
var revived = cmd.ExecuteNonQuery();
// Dead-lettered rows rejoin the non-dead-lettered queue — keep the in-memory
// counter aligned.
@@ -523,29 +591,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
/// <summary>
/// One queued row paired with its deserialized event. <see cref="Event"/> is
/// <c>null</c> when the row's <c>PayloadJson</c> is corrupt or un-deserializable —
/// the <see cref="RowId"/> always stays bound to its own row so outcomes can
/// <c>null</c> when the row's <c>payload_json</c> is corrupt or un-deserializable —
/// the <see cref="Id"/> always stays bound to its own row so outcomes can
/// never be mapped to the wrong row.
/// </summary>
private readonly record struct QueueRow(long RowId, AlarmHistorianEvent? Event, long AttemptCount);
private readonly record struct QueueRow(string Id, AlarmHistorianEvent? Event, long AttemptCount);
private List<QueueRow> ReadBatch(SqliteConnection conn)
private async Task<List<QueueRow>> ReadBatchAsync(CancellationToken ct)
{
var rows = new List<QueueRow>();
using var cmd = conn.CreateCommand();
cmd.CommandText = """
SELECT RowId, PayloadJson, AttemptCount FROM Queue
WHERE DeadLettered = 0
ORDER BY RowId ASC
LIMIT $limit
""";
cmd.Parameters.AddWithValue("$limit", _batchSize);
using var reader = cmd.ExecuteReader();
while (reader.Read())
// Ordered by enqueue time rather than insertion order: a hashed TEXT primary key carries
// no sequence the way the legacy AUTOINCREMENT rowid did. Round-trip ("O") timestamps sort
// lexicographically in chronological order; id makes the ordering total.
var raw = await _db.QueryAsync(
"""
SELECT id, payload_json, attempt_count FROM alarm_sf_events
WHERE dead_lettered = 0
ORDER BY enqueued_at_utc ASC, id ASC
LIMIT @Limit
""",
r => (Id: r.GetString(0), Payload: r.GetString(1), AttemptCount: r.GetInt64(2)),
new { Limit = _batchSize },
ct).ConfigureAwait(false);
var rows = new List<QueueRow>(raw.Count);
foreach (var (id, payload, attemptCount) in raw)
{
var rowId = reader.GetInt64(0);
var payload = reader.GetString(1);
var attemptCount = reader.GetInt64(2);
AlarmHistorianEvent? evt;
try
{
@@ -556,77 +626,58 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
// Malformed JSON — carry a null event so the caller dead-letters this row.
evt = null;
}
rows.Add(new QueueRow(rowId, evt, attemptCount));
rows.Add(new QueueRow(id, evt, attemptCount));
}
return rows;
}
private static void DeleteRow(SqliteConnection conn, SqliteTransaction tx, long rowId)
{
using var cmd = conn.CreateCommand();
cmd.Transaction = tx;
cmd.CommandText = "DELETE FROM Queue WHERE RowId = $id";
cmd.Parameters.AddWithValue("$id", rowId);
cmd.ExecuteNonQuery();
}
private static Task DeleteRowAsync(ILocalDbTransaction tx, string id, CancellationToken ct) =>
tx.ExecuteAsync("DELETE FROM alarm_sf_events WHERE id = @Id", new { Id = id }, ct);
private void DeadLetterRow(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
{
using var cmd = conn.CreateCommand();
cmd.Transaction = tx;
cmd.CommandText = """
UPDATE Queue SET DeadLettered = 1, LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
WHERE RowId = $id
""";
cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
cmd.Parameters.AddWithValue("$err", reason);
cmd.Parameters.AddWithValue("$id", rowId);
cmd.ExecuteNonQuery();
}
private Task DeadLetterRowAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
tx.ExecuteAsync(
"""
UPDATE alarm_sf_events
SET dead_lettered = 1, last_attempt_utc = @Now, last_error = @Err,
attempt_count = attempt_count + 1
WHERE id = @Id
""",
new { Now = _clock().ToString("O"), Err = reason, Id = id },
ct);
private void BumpAttempt(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
{
using var cmd = conn.CreateCommand();
cmd.Transaction = tx;
cmd.CommandText = """
UPDATE Queue SET LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
WHERE RowId = $id
""";
cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
cmd.Parameters.AddWithValue("$err", reason);
cmd.Parameters.AddWithValue("$id", rowId);
cmd.ExecuteNonQuery();
}
private Task BumpAttemptAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
tx.ExecuteAsync(
"""
UPDATE alarm_sf_events
SET last_attempt_utc = @Now, last_error = @Err, attempt_count = attempt_count + 1
WHERE id = @Id
""",
new { Now = _clock().ToString("O"), Err = reason, Id = id },
ct);
// Async variant used by EnqueueAsync.
private async Task EnforceCapacityAsync(SqliteConnection conn, CancellationToken ct)
private async Task EnforceCapacityAsync(CancellationToken ct)
{
Interlocked.Increment(ref _capacityProbeCount);
long count;
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
count = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
}
var count = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
// Resync the in-memory counter while we have a fresh number.
Interlocked.Exchange(ref _queuedRowCount, count);
if (count < _capacity) return;
var toEvict = count - _capacity + 1;
using (var cmd = conn.CreateCommand())
{
cmd.CommandText = """
DELETE FROM Queue
WHERE RowId IN (
SELECT RowId FROM Queue
WHERE DeadLettered = 0
ORDER BY RowId ASC
LIMIT $n
)
""";
cmd.Parameters.AddWithValue("$n", toEvict);
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
await _db.ExecuteAsync(
"""
DELETE FROM alarm_sf_events
WHERE id IN (
SELECT id FROM alarm_sf_events
WHERE dead_lettered = 0
ORDER BY enqueued_at_utc ASC, id ASC
LIMIT @N
)
""",
new { N = toEvict },
ct).ConfigureAwait(false);
Interlocked.Add(ref _queuedRowCount, -toEvict);
long lifetimeEvicted;
lock (_statusLock) { _evictedCount += toEvict; lifetimeEvicted = _evictedCount; }
@@ -635,40 +686,21 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
_capacity, toEvict, lifetimeEvicted);
}
private void PurgeAgedDeadLetters(SqliteConnection conn)
private async Task PurgeAgedDeadLettersAsync(CancellationToken ct)
{
var cutoff = (_clock() - _deadLetterRetention).ToString("O");
using var cmd = conn.CreateCommand();
cmd.CommandText = """
DELETE FROM Queue
WHERE DeadLettered = 1 AND LastAttemptUtc IS NOT NULL AND LastAttemptUtc < $cutoff
""";
cmd.Parameters.AddWithValue("$cutoff", cutoff);
var purged = cmd.ExecuteNonQuery();
var purged = await _db.ExecuteAsync(
"""
DELETE FROM alarm_sf_events
WHERE dead_lettered = 1 AND last_attempt_utc IS NOT NULL AND last_attempt_utc < @Cutoff
""",
new { Cutoff = cutoff },
ct).ConfigureAwait(false);
if (purged > 0)
_logger.Information("Purged {Count} dead-lettered row(s) past retention window", purged);
}
private void InitializeSchema()
{
using var conn = OpenConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS Queue (
RowId INTEGER PRIMARY KEY AUTOINCREMENT,
AlarmId TEXT NOT NULL,
EnqueuedUtc TEXT NOT NULL,
PayloadJson TEXT NOT NULL,
AttemptCount INTEGER NOT NULL DEFAULT 0,
LastAttemptUtc TEXT NULL,
LastError TEXT NULL,
DeadLettered INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
""";
cmd.ExecuteNonQuery();
}
private void BumpBackoff() => _backoffIndex = Math.Min(_backoffIndex + 1, BackoffLadder.Length - 1);
private void ResetBackoff() => _backoffIndex = 0;
@@ -676,12 +708,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
public TimeSpan CurrentBackoff => BackoffLadder[_backoffIndex];
/// <summary>Disposes the sink and releases all held resources including the drain timer and the writer.</summary>
/// <remarks>
/// The <see cref="ILocalDb"/> is NOT disposed here — it is a shared singleton owned by the
/// host and used by the deployment cache too.
/// </remarks>
public void Dispose()
{
if (_disposed) return;
_disposed = true;
_drainTimer?.Dispose();
_drainGate.Dispose();
_drainGateLock.Dispose();
if (_writer is IDisposable writerDisposable) writerDisposable.Dispose();
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj
+6
View File
@@ -20,6 +20,12 @@
-->
<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
<PackageReference Include="Serilog"/>
<!--
The store-and-forward buffer lives in the node's consolidated LocalDb file so it
replicates to the redundant pair peer. Only the ILocalDb SQL surface is used here; the
Host owns the DI registration and the replication package.
-->
<PackageReference Include="ZB.MOM.WW.LocalDb"/>
</ItemGroup>
<ItemGroup>
src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json
+1 -2
View File
@@ -52,9 +52,8 @@
"MaxBackoffSeconds": 30
},
"AlarmHistorian": {
"_comment": "Durable SQLite store-and-forward alarm sink. Drains alarm events to the ServerHistorian gateway's SendEvent path; the downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
"_comment": "Durable store-and-forward alarm sink. Buffers into the node's consolidated LocalDb (see the LocalDb section) so the queue replicates to the redundant pair peer, and drains alarm events to the ServerHistorian gateway's SendEvent path from whichever node holds the Primary role. The downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
"Enabled": false,
"DatabasePath": "alarm-historian.db",
"DrainIntervalSeconds": 5,
"Capacity": 1000000,
"DeadLetterRetentionDays": 30
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
+21 -9
View File
@@ -24,6 +24,7 @@ using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
using CommonsNodeId = ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId;
@@ -252,6 +253,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// non-cluster ActorRefProvider). See <see cref="DriverMemberCount"/>.</summary>
private readonly Func<int>? _driverMemberCountProvider;
/// <summary>Where the Primary-gate decision is published for consumers that live outside a mailbox —
/// today, the alarm store-and-forward drain. Null on nodes that do not wire one.</summary>
private readonly IRedundancyRoleView? _redundancyRoleView;
/// <summary>Debounces the S5 "snapshot omitted this node" Warning to once per process — a persistent
/// identity mismatch (03/S5) would otherwise log on every snapshot.</summary>
private bool _warnedSnapshotMissingLocalNode;
@@ -366,7 +371,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
IActorRef? scriptedAlarmHostOverride = null,
IDriverCapabilityInvokerFactory? invokerFactory = null,
Func<int>? driverMemberCountProvider = null,
IDeploymentArtifactCache? deploymentArtifactCache = null) =>
IDeploymentArtifactCache? deploymentArtifactCache = null,
IRedundancyRoleView? redundancyRoleView = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -376,7 +382,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
dbFactory, localNode, coordinator, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
deploymentArtifactCache));
deploymentArtifactCache, redundancyRoleView));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -426,9 +432,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
IActorRef? scriptedAlarmHostOverride = null,
IDriverCapabilityInvokerFactory? invokerFactory = null,
Func<int>? driverMemberCountProvider = null,
IDeploymentArtifactCache? deploymentArtifactCache = null)
IDeploymentArtifactCache? deploymentArtifactCache = null,
IRedundancyRoleView? redundancyRoleView = null)
{
_deploymentArtifactCache = deploymentArtifactCache;
_redundancyRoleView = redundancyRoleView;
_dbFactory = dbFactory;
_localNode = localNode;
_coordinatorOverride = coordinator;
@@ -1476,19 +1484,23 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
if (local is not null)
{
_localRole = local.Role;
return;
}
// S5 diagnostic: the snapshot never mentioned this node. On a multi-driver cluster that means the
// Primary gate stays default-DENY indefinitely for an unknown role — surface it once so a silent
// identity mismatch (e.g. an ApplicationUri/NodeId skew) is diagnosable.
if (!_warnedSnapshotMissingLocalNode && DriverMemberCount() > 1)
else if (!_warnedSnapshotMissingLocalNode && DriverMemberCount() > 1)
{
// S5 diagnostic: the snapshot never mentioned this node. On a multi-driver cluster that means the
// Primary gate stays default-DENY indefinitely for an unknown role — surface it once so a silent
// identity mismatch (e.g. an ApplicationUri/NodeId skew) is diagnosable.
_warnedSnapshotMissingLocalNode = true;
_log.Warning(
"DriverHost {Node}: redundancy snapshot omitted this node (snapshotNodes={SnapshotNodes}); Primary data-plane gate stays default-DENY while role is unknown on a multi-driver cluster",
_localNode, string.Join(",", msg.Nodes.Select(n => n.NodeId)));
}
// Publish on EVERY snapshot, including ones that leave the cached role untouched: the other
// half of the decision is the driver member count, which moves with cluster membership and
// not with this message. Snapshots are re-published on a heartbeat, so a membership change
// is picked up within one heartbeat rather than waiting for a role change that may never come.
_redundancyRoleView?.Publish(ShouldServiceAsPrimary());
}
private void Stale()
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs
+8 -13
View File
@@ -1,5 +1,4 @@
using System.Collections.Generic;
using System.IO;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -7,11 +6,12 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
/// <summary>
/// Binds the <c>AlarmHistorian</c> configuration section that gates the durable
/// store-and-forward alarm sink. When <see cref="Enabled"/> is <c>true</c>,
/// <c>AddAlarmHistorian</c> registers a <c>SqliteStoreAndForwardSink</c> (draining to the
/// <c>AddAlarmHistorian</c> registers a <c>LocalDbStoreAndForwardSink</c> (draining to the
/// gateway alarm writer supplied by the Host) in place of the
/// <c>NullAlarmHistorianSink</c> default; otherwise the Null default survives. This section
/// supplies only the <see cref="Enabled"/> gate and the SQLite store-and-forward knobs — the
/// downstream connection (endpoint/key/TLS) is sourced from the <c>ServerHistorian</c> section.
/// supplies only the <see cref="Enabled"/> gate and the store-and-forward knobs — the
/// downstream connection (endpoint/key/TLS) is sourced from the <c>ServerHistorian</c> section,
/// and the queue's storage location is the node's consolidated <c>LocalDb:Path</c> database.
/// </summary>
public sealed class AlarmHistorianOptions
{
@@ -24,9 +24,6 @@ public sealed class AlarmHistorianOptions
/// </summary>
public bool Enabled { get; init; }
/// <summary>Filesystem path to the local SQLite store-and-forward queue database.</summary>
public string DatabasePath { get; init; } = "alarm-historian.db";
/// <summary>Maximum number of queued rows the drain worker forwards in a single batch.</summary>
public int BatchSize { get; init; } = 100;
@@ -34,15 +31,15 @@ public sealed class AlarmHistorianOptions
public int DrainIntervalSeconds { get; init; } = 5;
/// <summary>Maximum queued rows before the sink evicts the oldest. Defaults to 1,000,000
/// (matches <c>SqliteStoreAndForwardSink</c>'s <c>DefaultCapacity</c>).</summary>
public long Capacity { get; init; } = SqliteStoreAndForwardSink.DefaultCapacity;
/// (matches <c>LocalDbStoreAndForwardSink</c>'s <c>DefaultCapacity</c>).</summary>
public long Capacity { get; init; } = LocalDbStoreAndForwardSink.DefaultCapacity;
/// <summary>Days to retain dead-lettered rows before purge. Defaults to 30.</summary>
public int DeadLetterRetentionDays { get; init; } = 30;
/// <summary>Maximum delivery attempts before a perpetually-retrying (poison) row is dead-lettered.
/// Defaults to 10 (matches <c>SqliteStoreAndForwardSink</c>'s <c>DefaultMaxAttempts</c>).</summary>
public int MaxAttempts { get; init; } = SqliteStoreAndForwardSink.DefaultMaxAttempts;
/// Defaults to 10 (matches <c>LocalDbStoreAndForwardSink</c>'s <c>DefaultMaxAttempts</c>).</summary>
public int MaxAttempts { get; init; } = LocalDbStoreAndForwardSink.DefaultMaxAttempts;
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian
/// (empty when disabled or correctly configured). Pure — the registration logs each entry.</summary>
@@ -51,8 +48,6 @@ public sealed class AlarmHistorianOptions
{
var warnings = new List<string>();
if (!Enabled) return warnings;
if (!Path.IsPathRooted(DatabasePath))
warnings.Add($"AlarmHistorian:DatabasePath '{DatabasePath}' is relative — it resolves against the process working directory (e.g. System32 for a Windows service). Set an absolute path.");
if (DrainIntervalSeconds <= 0)
warnings.Add($"AlarmHistorian:DrainIntervalSeconds is {DrainIntervalSeconds} — must be > 0; the drain timer will throw or spin at startup.");
if (Capacity <= 0)
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs
+46
View File
@@ -0,0 +1,46 @@
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
/// <summary>
/// A singleton snapshot of the Primary-gate decision, so code that lives outside an actor can
/// ask "should this node be servicing Primary-only work right now?".
/// </summary>
/// <remarks>
/// <para>
/// Exists for the alarm store-and-forward drain. That drain runs on a timer owned by the
/// sink, not on an actor mailbox, so it cannot receive <c>RedundancyStateChanged</c>
/// directly — but it must be Primary-scoped, because the queue it drains replicates to the
/// pair peer and two draining nodes would deliver every event twice.
/// </para>
/// <para>
/// Deliberately publishes the <i>decision</i> rather than the role and member count that
/// produced it, so <see cref="PrimaryGatePolicy"/> stays the single place that decides.
/// </para>
/// </remarks>
public interface IRedundancyRoleView
{
/// <summary>Whether this node should service Primary-only work right now.</summary>
bool ShouldServiceAsPrimary { get; }
/// <summary>Records a fresh decision. Called by <c>DriverHostActor</c> on every redundancy snapshot.</summary>
/// <param name="shouldServiceAsPrimary">The decision <see cref="PrimaryGatePolicy"/> produced.</param>
void Publish(bool shouldServiceAsPrimary);
}
/// <summary>Thread-safe <see cref="IRedundancyRoleView"/> backed by a volatile field.</summary>
public sealed class RedundancyRoleView : IRedundancyRoleView
{
// Seeded with the decision for "role unknown, no driver peer" rather than a bare false. An
// unpublished view and a node with no peer are genuinely the same situation, and they must
// behave the same: a deployment that runs no redundancy at all never publishes here, and
// defaulting closed would silently stop its alarm history forever.
private volatile bool _shouldServiceAsPrimary =
PrimaryGatePolicy.ShouldServiceAsPrimary(localRole: null, driverMemberCount: 0);
/// <inheritdoc/>
public bool ShouldServiceAsPrimary => _shouldServiceAsPrimary;
/// <inheritdoc/>
public void Publish(bool shouldServiceAsPrimary) => _shouldServiceAsPrimary = shouldServiceAsPrimary;
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs
+34 -8
View File
@@ -20,7 +20,9 @@ using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Health;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.OtOpcUa.Runtime;
@@ -65,12 +67,23 @@ public static class ServiceCollectionExtensions
/// <summary>
/// Config-gated durable alarm-historian sink. When the <c>AlarmHistorian</c> section has
/// <c>Enabled=true</c>, registers a <see cref="SqliteStoreAndForwardSink"/> (draining via the
/// <c>Enabled=true</c>, registers a <see cref="LocalDbStoreAndForwardSink"/> (draining via the
/// <paramref name="writerFactory"/>-supplied writer) as the <see cref="IAlarmHistorianSink"/>,
/// overriding the <see cref="NullAlarmHistorianSink"/> default. Otherwise a no-op (Null stays).
/// The writer is injected so the durable downstream (the HistorianGateway alarm writer) can be
/// supplied by the Host, which is the only project that references it.
/// </summary>
/// <remarks>
/// <para>
/// The queue lives in the node's consolidated <see cref="ILocalDb"/>, which replicates it
/// to the redundant pair peer — so undelivered alarm history survives losing the node
/// holding it. That is also why the drain is gated on <see cref="IRedundancyRoleView"/>:
/// the Secondary holds a full replica of the Primary's queue, and an ungated drain there
/// would re-deliver every event. Both services are resolved from the provider, so a
/// deployment that enables this section without registering LocalDb fails loudly at
/// resolution rather than silently buffering to nowhere.
/// </para>
/// </remarks>
/// <param name="services">The service collection to register with.</param>
/// <param name="configuration">The configuration carrying the <c>AlarmHistorian</c> section.</param>
/// <param name="writerFactory">
@@ -87,21 +100,28 @@ public static class ServiceCollectionExtensions
if (opts is not { Enabled: true }) return services; // leave the Null default from AddOtOpcUaRuntime
foreach (var warning in opts.Validate())
Serilog.Log.Logger.ForContext<SqliteStoreAndForwardSink>().Warning("Historian config: {HistorianConfigWarning}", warning);
Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>().Warning("Historian config: {HistorianConfigWarning}", warning);
// The view is a plain singleton so it exists even on nodes whose DriverHostActor never
// publishes to it — an unpublished view reads as "no peer, drain", which is the correct
// posture for a deployment that runs no redundancy at all.
services.TryAddSingleton<IRedundancyRoleView, RedundancyRoleView>();
services.AddSingleton<IAlarmHistorianSink>(sp =>
{
// SqliteStoreAndForwardSink takes a Serilog ILogger (not Microsoft.Extensions.Logging).
// LocalDbStoreAndForwardSink takes a Serilog ILogger (not Microsoft.Extensions.Logging).
// Resolve it off the host's configured static logger so the drain worker's WARN/INFO
// lines land in the same sinks as the rest of the process.
var sink = new SqliteStoreAndForwardSink(
opts.DatabasePath,
var roleView = sp.GetRequiredService<IRedundancyRoleView>();
var sink = new LocalDbStoreAndForwardSink(
sp.GetRequiredService<ILocalDb>(),
writerFactory(opts, sp),
Serilog.Log.Logger.ForContext<SqliteStoreAndForwardSink>(),
Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>(),
batchSize: opts.BatchSize,
capacity: opts.Capacity,
deadLetterRetention: TimeSpan.FromDays(opts.DeadLetterRetentionDays),
maxAttempts: opts.MaxAttempts);
maxAttempts: opts.MaxAttempts,
drainGate: () => roleView.ShouldServiceAsPrimary);
sink.StartDrainLoop(TimeSpan.FromSeconds(opts.DrainIntervalSeconds));
return sink;
});
@@ -227,6 +247,11 @@ public static class ServiceCollectionExtensions
// harnesses) rather than given a null-object, so DriverHostActor skips caching outright
// instead of pretending to cache into a sink that drops everything.
var deploymentArtifactCache = resolver.GetService<IDeploymentArtifactCache>();
// Where this actor publishes its Primary-gate verdict for the alarm store-and-forward
// drain, which runs on a timer and so cannot read RedundancyStateChanged itself.
// Registered by AddAlarmHistorian; absent when no durable sink is configured, in which
// case there is nothing downstream to inform.
var redundancyRoleView = resolver.GetService<IRedundancyRoleView>();
// Root script logger backs the ScriptedAlarm host's engine + script logging. Registered in
// Host DI inside the hasDriver block; may be absent in some role configs / test harnesses,
// in which case the DriverHostActor gracefully skips spawning the ScriptedAlarm host.
@@ -345,7 +370,8 @@ public static class ServiceCollectionExtensions
loggerFactory: loggerFactory,
scriptRootLogger: scriptRootLogger,
invokerFactory: invokerFactory,
deploymentArtifactCache: deploymentArtifactCache),
deploymentArtifactCache: deploymentArtifactCache,
redundancyRoleView: redundancyRoleView),
DriverHostActorName);
registry.Register<DriverHostActorKey>(driverHost);
tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/SqliteStoreAndForwardSinkTests.cs → tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/LocalDbStoreAndForwardSinkTests.cs
+204 -62
View File
@@ -1,9 +1,12 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Serilog;
using Serilog.Core;
using Serilog.Events;
using Shouldly;
using Xunit;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests;
@@ -14,22 +17,66 @@ namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests;
/// capacity eviction, and retention-based dead-letter purge.
/// </summary>
[Trait("Category", "Unit")]
public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public sealed class LocalDbStoreAndForwardSinkTests : IDisposable
{
private readonly string _dbPath;
private readonly ILogger _log;
private ServiceProvider? _provider;
private ILocalDb? _db;
/// <summary>Initializes a new test instance.</summary>
public SqliteStoreAndForwardSinkTests()
public LocalDbStoreAndForwardSinkTests()
{
_dbPath = Path.Combine(Path.GetTempPath(), $"otopcua-historian-{Guid.NewGuid():N}.sqlite");
_log = new LoggerConfiguration().MinimumLevel.Verbose().CreateLogger();
}
/// <summary>
/// A real temp-file local database with the buffer table created and registered, built on
/// first use.
/// </summary>
/// <remarks>
/// A real database rather than a stub. The library has no in-memory mode, and a bare
/// <see cref="SqliteConnection"/> would lack the <c>zb_hlc_next()</c> UDF the capture
/// triggers call — so every write to a registered table would fail closed. Registering here
/// also means these tests exercise the same capture path production replicates through.
/// </remarks>
private ILocalDb Db
{
get
{
if (_db is not null) return _db;
var configuration = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["LocalDb:Path"] = _dbPath })
.Build();
_provider = new ServiceCollection()
.AddZbLocalDb(configuration, db =>
{
using var connection = db.CreateConnection();
AlarmSfSchema.Apply(connection);
db.RegisterReplicated(AlarmSfSchema.EventsTable);
})
.BuildServiceProvider();
return _db = _provider.GetRequiredService<ILocalDb>();
}
}
/// <summary>Cleans up test resources.</summary>
public void Dispose()
{
try { if (File.Exists(_dbPath)) File.Delete(_dbPath); } catch { }
_provider?.Dispose();
// Pooled connections keep a handle open past dispose; clearing the pools first is what
// makes the delete actually succeed.
SqliteConnection.ClearAllPools();
foreach (var path in new[] { _dbPath, $"{_dbPath}-wal", $"{_dbPath}-shm" })
{
try { if (File.Exists(path)) File.Delete(path); } catch { /* best effort */ }
}
}
private sealed class FakeWriter : IAlarmHistorianWriter
@@ -81,7 +128,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task EnqueueThenDrain_Ack_removes_row()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
sink.GetStatus().QueueDepth.ShouldBe(1);
@@ -102,7 +149,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Drain_with_empty_queue_is_noop()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.DrainOnceAsync(CancellationToken.None);
@@ -116,7 +163,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
{
var writer = new FakeWriter();
writer.NextOutcomePerEvent.Enqueue(HistorianWriteOutcome.RetryPlease);
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
var before = sink.CurrentBackoff;
@@ -133,7 +180,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
{
var writer = new FakeWriter();
writer.NextOutcomePerEvent.Enqueue(HistorianWriteOutcome.RetryPlease);
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
@@ -153,7 +200,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
var writer = new FakeWriter();
writer.NextOutcomePerEvent.Enqueue(HistorianWriteOutcome.PermanentFail);
writer.NextOutcomePerEvent.Enqueue(HistorianWriteOutcome.Ack);
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("bad"), CancellationToken.None);
await sink.EnqueueAsync(Event("good"), CancellationToken.None);
@@ -169,7 +216,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Writer_exception_treated_as_retry_for_whole_batch()
{
var writer = new FakeWriter { ThrowOnce = new InvalidOperationException("pipe broken") };
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
@@ -189,8 +236,8 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Capacity_eviction_drops_oldest_nondeadlettered_row()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(
_dbPath, writer, _log, batchSize: 100, capacity: 3);
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, batchSize: 100, capacity: 3);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.EnqueueAsync(Event("A2"), CancellationToken.None);
@@ -217,8 +264,8 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
var writer = new FakeWriter();
writer.NextOutcomePerEvent.Enqueue(HistorianWriteOutcome.PermanentFail);
using var sink = new SqliteStoreAndForwardSink(
_dbPath, writer, _log, deadLetterRetention: TimeSpan.FromDays(30),
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, deadLetterRetention: TimeSpan.FromDays(30),
clock: () => clock);
await sink.EnqueueAsync(Event("bad"), CancellationToken.None);
@@ -238,7 +285,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
{
var writer = new FakeWriter();
writer.NextOutcomePerEvent.Enqueue(HistorianWriteOutcome.PermanentFail);
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("bad"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
@@ -257,7 +304,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Backoff_ladder_caps_at_60s()
{
var writer = new FakeWriter { DefaultOutcome = HistorianWriteOutcome.RetryPlease };
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
@@ -278,8 +325,8 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task RetryPlease_dead_letters_row_after_MaxAttempts()
{
var writer = new FakeWriter { DefaultOutcome = HistorianWriteOutcome.RetryPlease };
using var sink = new SqliteStoreAndForwardSink(
_dbPath, writer, _log, maxAttempts: 3);
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, maxAttempts: 3);
await sink.EnqueueAsync(Event("poison"), CancellationToken.None);
@@ -316,17 +363,89 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
await NullAlarmHistorianSink.Instance.EnqueueAsync(Event("A1"), CancellationToken.None);
}
/// <summary>
/// A closed drain gate must leave the queue completely alone — no writer call, no attempt
/// bump, no purge.
/// </summary>
/// <remarks>
/// This is the property that keeps alarm delivery exactly-once across a redundant pair.
/// The buffer replicates, so the Secondary holds a full copy of the Primary's queue; a
/// Secondary that drained would re-send every event, continuously.
/// </remarks>
[Fact]
public async Task Drain_does_nothing_while_the_gate_is_closed()
{
var writer = new FakeWriter();
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log, drainGate: () => false);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.EnqueueAsync(Event("A2"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
writer.Batches.Count.ShouldBe(0, "a gated-off node must not deliver");
var status = sink.GetStatus();
status.QueueDepth.ShouldBe(2, "the rows stay queued for whichever node holds the Primary role");
status.DrainState.ShouldBe(
HistorianDrainState.NotPrimary,
"a Secondary's rising queue must be distinguishable from a stalled drain");
}
/// <summary>
/// The positive control for the case above: the same queue, the same sink, drains as soon
/// as the gate opens. Without this, a sink that was simply broken would pass the
/// gated-off assertion.
/// </summary>
[Fact]
public async Task Drain_delivers_the_retained_queue_once_the_gate_opens()
{
var writer = new FakeWriter();
var isPrimary = false;
// ReSharper disable once AccessToModifiedClosure — flipping it mid-test is the point.
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log, drainGate: () => isPrimary);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
writer.Batches.Count.ShouldBe(0);
isPrimary = true; // this node is promoted
await sink.DrainOnceAsync(CancellationToken.None);
writer.Batches.Count.ShouldBe(1);
writer.Batches[0].Select(e => e.AlarmId).ShouldBe(["A1"]);
sink.GetStatus().QueueDepth.ShouldBe(0);
}
/// <summary>
/// A gate that throws must be read as "not now", never as permission. Draining on a gate
/// fault would put a pair into dual-delivery precisely when its redundancy signal is sick.
/// </summary>
[Fact]
public async Task Drain_treats_a_throwing_gate_as_closed()
{
var writer = new FakeWriter();
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, drainGate: () => throw new InvalidOperationException("role unavailable"));
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
writer.Batches.Count.ShouldBe(0, "a faulted gate must fail closed");
sink.GetStatus().QueueDepth.ShouldBe(1);
}
/// <summary>Verifies that the constructor rejects invalid arguments.</summary>
[Fact]
public void Ctor_rejects_bad_args()
{
var w = new FakeWriter();
Should.Throw<ArgumentException>(() => new SqliteStoreAndForwardSink("", w, _log));
Should.Throw<ArgumentNullException>(() => new SqliteStoreAndForwardSink(_dbPath, null!, _log));
Should.Throw<ArgumentNullException>(() => new SqliteStoreAndForwardSink(_dbPath, w, null!));
Should.Throw<ArgumentOutOfRangeException>(() => new SqliteStoreAndForwardSink(_dbPath, w, _log, batchSize: 0));
Should.Throw<ArgumentOutOfRangeException>(() => new SqliteStoreAndForwardSink(_dbPath, w, _log, capacity: 0));
Should.Throw<ArgumentOutOfRangeException>(() => new SqliteStoreAndForwardSink(_dbPath, w, _log, maxAttempts: 0));
Should.Throw<ArgumentNullException>(() => new LocalDbStoreAndForwardSink(null!, w, _log));
Should.Throw<ArgumentNullException>(() => new LocalDbStoreAndForwardSink(Db, null!, _log));
Should.Throw<ArgumentNullException>(() => new LocalDbStoreAndForwardSink(Db, w, null!));
Should.Throw<ArgumentOutOfRangeException>(() => new LocalDbStoreAndForwardSink(Db, w, _log, batchSize: 0));
Should.Throw<ArgumentOutOfRangeException>(() => new LocalDbStoreAndForwardSink(Db, w, _log, capacity: 0));
Should.Throw<ArgumentOutOfRangeException>(() => new LocalDbStoreAndForwardSink(Db, w, _log, maxAttempts: 0));
}
/// <summary>Verifies that a disposed sink rejects enqueue operations.</summary>
@@ -334,7 +453,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Disposed_sink_rejects_enqueue()
{
var writer = new FakeWriter();
var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
sink.Dispose();
await Should.ThrowAsync<ObjectDisposedException>(
@@ -352,11 +471,14 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Drain_with_corrupt_payload_row_deadletters_it_and_keeps_good_rows_aligned()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
var clock = new TickingClock();
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log, clock: clock.Next);
// Row 1: good. Row 2: corrupt JSON (inserted directly). Row 3: good.
// Row 1: good. Row 2: corrupt JSON (inserted directly). Row 3: good. The shared ticking
// clock is what pins the corrupt row BETWEEN the good ones — which is the whole point of
// this case, as distinct from the corrupt-at-the-head case below.
await sink.EnqueueAsync(Event("good-1"), CancellationToken.None);
InsertCorruptRow("corrupt");
InsertCorruptRow("corrupt", clock.Next());
await sink.EnqueueAsync(Event("good-2"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
@@ -383,9 +505,10 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Drain_with_corrupt_head_row_does_not_stall_queue()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
var clock = new TickingClock();
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log, clock: clock.Next);
InsertCorruptRow("corrupt-head");
InsertCorruptRow("corrupt-head", clock.Next());
await sink.EnqueueAsync(Event("good-1"), CancellationToken.None);
await sink.DrainOnceAsync(CancellationToken.None);
@@ -409,7 +532,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task StartDrainLoop_honors_backoff_and_slows_cadence_under_retry()
{
var writer = new FakeWriter { DefaultOutcome = HistorianWriteOutcome.RetryPlease };
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
@@ -436,7 +559,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task StartDrainLoop_keeps_steady_cadence_when_writer_is_healthy()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
sink.StartDrainLoop(TimeSpan.FromMilliseconds(30));
@@ -465,7 +588,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
// exception... so to exercise the *callback* catch we instead make the
// fault originate from the writer itself but assert the loop self-heals.
var writer = new ThrowingThenHealingWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
sink.StartDrainLoop(TimeSpan.FromMilliseconds(30));
@@ -490,7 +613,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Concurrent_enqueue_and_drain_do_not_throw_sqlite_busy()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
var faults = new List<Exception>();
var enqueuers = Enumerable.Range(0, 4).Select(t => Task.Run(async () =>
@@ -561,7 +684,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Writer_returning_wrong_cardinality_outcomes_sets_backing_off_and_keeps_rows()
{
var writer = new WrongCardinalityWriter(returnExtraOutcome: true);
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.EnqueueAsync(Event("A2"), CancellationToken.None);
@@ -591,8 +714,8 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Capacity_eviction_increments_evicted_count_on_status()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(
_dbPath, writer, _log, batchSize: 100, capacity: 3);
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, batchSize: 100, capacity: 3);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
await sink.EnqueueAsync(Event("A2"), CancellationToken.None);
@@ -616,7 +739,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task GetStatus_snapshot_is_consistent_under_concurrent_drain()
{
var writer = new FakeWriter { DefaultOutcome = HistorianWriteOutcome.RetryPlease };
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
// Fill the queue.
for (var i = 0; i < 10; i++)
@@ -690,8 +813,8 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task EnqueueAsync_does_not_count_all_rows_on_every_call_below_capacity()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(
_dbPath, writer, _log, batchSize: 100, capacity: 10_000);
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, batchSize: 100, capacity: 10_000);
for (var i = 0; i < 200; i++)
await sink.EnqueueAsync(Event($"E{i}"), CancellationToken.None);
@@ -705,7 +828,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
/// <summary>
/// Regression for Core.AlarmHistorian-008: across every queue mutation (enqueue,
/// Ack drain, PermanentFail drain, capacity eviction, RetryDeadLettered) the
/// queue depth reported by <see cref="SqliteStoreAndForwardSink.GetStatus"/> must
/// queue depth reported by <see cref="LocalDbStoreAndForwardSink.GetStatus"/> must
/// stay aligned with a fresh <c>COUNT(*)</c> against the database. Catches drift
/// bugs in the in-memory counter introduced by the perf optimisation.
/// </summary>
@@ -713,8 +836,8 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Enqueue_and_drain_keep_queue_depth_consistent_with_storage()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(
_dbPath, writer, _log, batchSize: 5, capacity: 8);
using var sink = new LocalDbStoreAndForwardSink(
Db, writer, _log, batchSize: 5, capacity: 8);
// Burst-enqueue below capacity — the in-memory counter must stay aligned with the
// SELECT COUNT(*) that GetStatus runs.
@@ -760,7 +883,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Counter_remains_consistent_under_concurrent_enqueue_and_drain()
{
var writer = new FakeWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
var enqueuers = Enumerable.Range(0, 3).Select(t => Task.Run(async () =>
{
@@ -790,7 +913,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
/// Regression for Core.AlarmHistorian-012: when WriteBatchAsync throws
/// <see cref="OperationCanceledException"/> the drain exits via re-throw
/// without resetting <c>_drainState</c>, leaving the status surface permanently
/// showing <c>Draining</c>. The next call to <see cref="SqliteStoreAndForwardSink.GetStatus"/>
/// showing <c>Draining</c>. The next call to <see cref="LocalDbStoreAndForwardSink.GetStatus"/>
/// must reflect <c>Idle</c> or <c>BackingOff</c>, not the stale <c>Draining</c>
/// state that was set at the top of the drain tick.
/// </summary>
@@ -798,7 +921,7 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
public async Task Drain_cancelled_mid_write_resets_drain_state_to_not_draining()
{
var writer = new CancellableWriter();
using var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
using var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
await sink.EnqueueAsync(Event("A1"), CancellationToken.None);
using var cts = new CancellationTokenSource();
@@ -840,49 +963,68 @@ public sealed class SqliteStoreAndForwardSinkTests : IDisposable
/// COUNT(*) read directly from storage — proves the in-memory counter has not
/// drifted from the persisted truth.
/// </summary>
private void AssertQueueDepthMatchesStorage(SqliteStoreAndForwardSink sink)
private void AssertQueueDepthMatchesStorage(LocalDbStoreAndForwardSink sink)
{
using var conn = new SqliteConnection($"Data Source={_dbPath}");
conn.Open();
using var conn = Db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 0";
var live = (long)(cmd.ExecuteScalar() ?? 0L);
sink.GetStatus().QueueDepth.ShouldBe(live, "GetStatus must agree with a fresh COUNT(*)");
}
/// <summary>
/// Regression for Core.AlarmHistorian-014: <see cref="SqliteStoreAndForwardSink.GetStatus"/>
/// and <see cref="SqliteStoreAndForwardSink.RetryDeadLettered"/> must throw
/// <see cref="ObjectDisposedException"/> after <see cref="SqliteStoreAndForwardSink.Dispose"/>
/// Regression for Core.AlarmHistorian-014: <see cref="LocalDbStoreAndForwardSink.GetStatus"/>
/// and <see cref="LocalDbStoreAndForwardSink.RetryDeadLettered"/> must throw
/// <see cref="ObjectDisposedException"/> after <see cref="LocalDbStoreAndForwardSink.Dispose"/>
/// is called, consistent with the guards already present on
/// <see cref="SqliteStoreAndForwardSink.EnqueueAsync"/> and
/// <see cref="SqliteStoreAndForwardSink.StartDrainLoop"/>.
/// <see cref="LocalDbStoreAndForwardSink.EnqueueAsync"/> and
/// <see cref="LocalDbStoreAndForwardSink.StartDrainLoop"/>.
/// </summary>
[Fact]
public void Disposed_sink_throws_ObjectDisposedException_on_GetStatus_and_RetryDeadLettered()
{
var writer = new FakeWriter();
var sink = new SqliteStoreAndForwardSink(_dbPath, writer, _log);
var sink = new LocalDbStoreAndForwardSink(Db, writer, _log);
sink.Dispose();
Should.Throw<ObjectDisposedException>(() => sink.GetStatus());
Should.Throw<ObjectDisposedException>(() => sink.RetryDeadLettered());
}
/// <summary>Insert a queue row whose PayloadJson cannot deserialize into an AlarmHistorianEvent.</summary>
private void InsertCorruptRow(string alarmId)
/// <summary>
/// Insert a queue row whose payload cannot deserialize into an AlarmHistorianEvent, stamped
/// at an explicit enqueue time.
/// </summary>
/// <remarks>
/// The timestamp is a parameter rather than <c>DateTime.UtcNow</c> because the drain reads
/// in <c>enqueued_at_utc</c> order, and the tests that use this helper care precisely about
/// where the corrupt row sits in that order. Three unsynchronised clock reads can land on
/// one tick, which would make the interleaving — and therefore the test — a coin flip.
/// </remarks>
private void InsertCorruptRow(string alarmId, DateTime enqueuedAt)
{
using var conn = new SqliteConnection($"Data Source={_dbPath}");
conn.Open();
using var conn = Db.CreateConnection();
using var cmd = conn.CreateCommand();
cmd.CommandText = """
INSERT INTO Queue (AlarmId, EnqueuedUtc, PayloadJson, AttemptCount)
VALUES ($alarmId, $enqueued, $payload, 0);
INSERT INTO alarm_sf_events (id, alarm_id, enqueued_at_utc, payload_json, attempt_count)
VALUES ($id, $alarmId, $enqueued, $payload, 0);
""";
cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N"));
cmd.Parameters.AddWithValue("$alarmId", alarmId);
cmd.Parameters.AddWithValue("$enqueued", DateTime.UtcNow.ToString("O"));
cmd.Parameters.AddWithValue("$enqueued", enqueuedAt.ToString("O"));
// JSON literal "null" round-trips through Deserialize<T> as a null reference.
cmd.Parameters.AddWithValue("$payload", "null");
cmd.ExecuteNonQuery();
}
/// <summary>
/// Monotonic test clock — every read advances one second, so enqueue order is total and
/// the drain's <c>ORDER BY enqueued_at_utc</c> is deterministic.
/// </summary>
private sealed class TickingClock
{
private DateTime _now = new(2026, 7, 21, 0, 0, 0, DateTimeKind.Utc);
public DateTime Next() => _now = _now.AddSeconds(1);
}
}
tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests.csproj
+8
View File
@@ -17,6 +17,14 @@
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<!--
The sink is built over ILocalDb, whose only construction path is the DI extension — so
these tests need the DI + configuration hosts to stand up a real temp-file database.
The library has no in-memory mode, and a bare SqliteConnection would lack the
zb_hlc_next() UDF the capture triggers call, failing closed on registered tables.
-->
<PackageReference Include="Microsoft.Extensions.DependencyInjection"/>
<PackageReference Include="Microsoft.Extensions.Configuration"/>
</ItemGroup>
<ItemGroup>
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorRoleViewTests.cs
+132
View File
@@ -0,0 +1,132 @@
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
/// <summary>
/// The bridge that carries the Primary-gate decision out of the actor and into
/// <see cref="IRedundancyRoleView"/>, where the alarm store-and-forward drain can read it.
/// </summary>
/// <remarks>
/// <para>
/// The drain runs on a timer owned by the sink, not on a mailbox, so it cannot receive
/// <see cref="RedundancyStateChanged"/> itself — but it must be Primary-scoped, because
/// the queue it drains replicates to the pair peer and two draining nodes would deliver
/// every alarm event twice.
/// </para>
/// <para>
/// These cases mirror <see cref="DriverHostActorPrimaryGateTests"/>'s matrix on purpose:
/// the view must reach exactly the same verdict as the inbound-write and native-ack gates,
/// because a second, subtly different notion of "am I the Primary" is how a pair ends up
/// with one node writing and neither draining.
/// </para>
/// </remarks>
public sealed class DriverHostActorRoleViewTests : RuntimeActorTestBase
{
private static readonly NodeId TestNode = NodeId.Parse("driver-roleview-test");
private static readonly TimeSpan Timeout = TimeSpan.FromSeconds(5);
/// <summary>
/// Before any snapshot arrives the view must read open, not closed.
/// </summary>
/// <remarks>
/// A deployment that runs no redundancy never publishes here at all. Defaulting closed
/// would silently stop its alarm history forever — the exact failure the drain gate exists
/// to avoid causing.
/// </remarks>
[Fact]
public void An_unpublished_view_reads_as_primary()
{
new RedundancyRoleView().ShouldServiceAsPrimary.ShouldBeTrue();
}
[Fact]
public void Secondary_role_closes_the_view()
{
var view = SpawnAndTellRole(RedundancyRole.Secondary, driverMemberCount: 2);
AwaitAssert(() => view.ShouldServiceAsPrimary.ShouldBeFalse(), duration: Timeout);
}
[Fact]
public void Primary_role_opens_the_view()
{
var view = SpawnAndTellRole(RedundancyRole.Primary, driverMemberCount: 2);
AwaitAssert(() => view.ShouldServiceAsPrimary.ShouldBeTrue(), duration: Timeout);
}
/// <summary>Unknown role + a real driver peer ⇒ closed, matching the write gate's default-DENY.</summary>
[Fact]
public void Unknown_role_with_a_driver_peer_closes_the_view()
{
// A snapshot that never names this node is how the role stays unknown in practice — the
// documented identity-mismatch shape, not a contrived case.
var view = SpawnAndTellForeignSnapshot(driverMemberCount: 2);
AwaitAssert(() => view.ShouldServiceAsPrimary.ShouldBeFalse(), duration: Timeout);
}
/// <summary>Unknown role + no driver peer ⇒ open, preserving the single-node posture.</summary>
[Fact]
public void Unknown_role_without_a_driver_peer_leaves_the_view_open()
{
var view = SpawnAndTellForeignSnapshot(driverMemberCount: 1);
AwaitAssert(() => view.ShouldServiceAsPrimary.ShouldBeTrue(), duration: Timeout);
}
/// <summary>A demotion must move the view, not just an initial promotion.</summary>
[Fact]
public void A_demotion_closes_a_previously_open_view()
{
var view = new RedundancyRoleView();
var host = SpawnHost(view, driverMemberCount: 2);
host.Tell(Snapshot(TestNode, RedundancyRole.Primary));
AwaitAssert(() => view.ShouldServiceAsPrimary.ShouldBeTrue(), duration: Timeout);
host.Tell(Snapshot(TestNode, RedundancyRole.Secondary));
AwaitAssert(() => view.ShouldServiceAsPrimary.ShouldBeFalse(), duration: Timeout);
}
// ---------------- helpers ----------------
private RedundancyRoleView SpawnAndTellRole(RedundancyRole role, int driverMemberCount)
{
var view = new RedundancyRoleView();
SpawnHost(view, driverMemberCount).Tell(Snapshot(TestNode, role));
return view;
}
private RedundancyRoleView SpawnAndTellForeignSnapshot(int driverMemberCount)
{
var view = new RedundancyRoleView();
SpawnHost(view, driverMemberCount).Tell(Snapshot(NodeId.Parse("some-other-node"), RedundancyRole.Primary));
return view;
}
private IActorRef SpawnHost(IRedundancyRoleView view, int driverMemberCount) =>
Sys.ActorOf(DriverHostActor.Props(
NewInMemoryDbFactory(),
TestNode,
coordinator: null,
driverMemberCountProvider: () => driverMemberCount,
redundancyRoleView: view));
private static RedundancyStateChanged Snapshot(NodeId node, RedundancyRole role) =>
new(
[
new NodeRedundancyState(node, role,
IsClusterLeader: role == RedundancyRole.Primary,
IsRoleLeaderForDriver: role == RedundancyRole.Primary,
AsOfUtc: DateTime.UtcNow),
],
CorrelationId.NewId());
}
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/AlarmHistorianRegistrationTests.cs
+44 -22
View File
@@ -1,8 +1,10 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -11,12 +13,12 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Historian;
/// <summary>
/// Verifies the config-gated <c>AddAlarmHistorian</c> registration: when the
/// <c>AlarmHistorian</c> section is absent or disabled the <see cref="NullAlarmHistorianSink"/>
/// default survives; when it is enabled a real <see cref="SqliteStoreAndForwardSink"/> wins
/// default survives; when it is enabled a real <see cref="LocalDbStoreAndForwardSink"/> wins
/// (last-registration-wins over the <c>TryAddSingleton</c> Null default).
/// </summary>
public sealed class AlarmHistorianRegistrationTests
{
/// <summary>A no-op writer the factory hands the Sqlite sink; never actually invoked in these tests.</summary>
/// <summary>A no-op writer the factory hands the sink; never actually invoked in these tests.</summary>
private sealed class FakeWriter : IAlarmHistorianWriter
{
public Task<IReadOnlyList<HistorianWriteOutcome>> WriteBatchAsync(
@@ -65,34 +67,61 @@ public sealed class AlarmHistorianRegistrationTests
}
[Fact]
public void Section_enabled_registers_sqlite_sink()
public void Section_enabled_registers_the_localdb_sink()
{
var tempDir = Path.Combine(Path.GetTempPath(), "otopcua-alarmhist-test-" + Guid.NewGuid().ToString("N"));
Directory.CreateDirectory(tempDir);
var dbPath = Path.Combine(tempDir, "alarm-historian-test.db");
var dbPath = Path.Combine(tempDir, "node-local.db");
try
{
var services = BaseServices();
var config = ConfigFrom(new Dictionary<string, string?>
{
["AlarmHistorian:Enabled"] = "true",
["AlarmHistorian:DatabasePath"] = dbPath,
["LocalDb:Path"] = dbPath,
});
var services = BaseServices();
services.AddZbLocalDb(config, db =>
{
using var connection = db.CreateConnection();
AlarmSfSchema.Apply(connection);
db.RegisterReplicated(AlarmSfSchema.EventsTable);
});
services.AddAlarmHistorian(config, (_, _) => new FakeWriter());
using (var provider = services.BuildServiceProvider())
{
provider.GetRequiredService<IAlarmHistorianSink>().ShouldBeOfType<SqliteStoreAndForwardSink>();
provider.GetRequiredService<IAlarmHistorianSink>().ShouldBeOfType<LocalDbStoreAndForwardSink>();
} // dispose stops the drain loop + releases the SQLite file handle
}
finally
{
SqliteConnection.ClearAllPools();
try { Directory.Delete(tempDir, recursive: true); } catch (IOException) { /* best effort */ }
}
}
/// <summary>
/// An enabled historian with no LocalDb registered must fail at resolution, loudly.
/// </summary>
/// <remarks>
/// The queue has nowhere to live without it. Falling back to the Null sink here would look
/// like a working historian while silently discarding every alarm event — the failure mode
/// this whole registration is meant to make impossible.
/// </remarks>
[Fact]
public void Section_enabled_without_localdb_throws_rather_than_silently_discarding()
{
var services = BaseServices();
var config = ConfigFrom(new Dictionary<string, string?> { ["AlarmHistorian:Enabled"] = "true" });
services.AddAlarmHistorian(config, (_, _) => new FakeWriter());
using var provider = services.BuildServiceProvider();
Should.Throw<InvalidOperationException>(() => provider.GetRequiredService<IAlarmHistorianSink>());
}
[Fact]
public void Section_binds_drain_capacity_and_retention_knobs()
{
@@ -112,17 +141,10 @@ public sealed class AlarmHistorianRegistrationTests
opts.DeadLetterRetentionDays.ShouldBe(7);
}
[Fact]
public void Validate_warns_on_relative_database_path_when_enabled()
{
var opts = new AlarmHistorianOptions { Enabled = true, DatabasePath = "alarm-historian.db" };
opts.Validate().ShouldContain(w => w.Contains("DatabasePath"));
}
[Fact]
public void Validate_is_silent_when_correctly_configured()
{
new AlarmHistorianOptions { Enabled = true, DatabasePath = "/abs/h.db" }.Validate().ShouldBeEmpty();
new AlarmHistorianOptions { Enabled = true }.Validate().ShouldBeEmpty();
}
[Fact]
@@ -134,39 +156,39 @@ public sealed class AlarmHistorianRegistrationTests
[Fact]
public void Validate_warns_on_non_positive_drain_interval()
{
var opts = new AlarmHistorianOptions { Enabled = true, DatabasePath = "/abs/h.db", DrainIntervalSeconds = 0 };
var opts = new AlarmHistorianOptions { Enabled = true, DrainIntervalSeconds = 0 };
opts.Validate().ShouldContain(w => w.Contains("DrainIntervalSeconds"));
}
[Fact]
public void Validate_warns_on_non_positive_capacity()
{
var opts = new AlarmHistorianOptions { Enabled = true, DatabasePath = "/abs/h.db", Capacity = 0 };
var opts = new AlarmHistorianOptions { Enabled = true, Capacity = 0 };
opts.Validate().ShouldContain(w => w.Contains("Capacity"));
}
[Fact]
public void Validate_warns_on_non_positive_retention()
{
var opts = new AlarmHistorianOptions { Enabled = true, DatabasePath = "/abs/h.db", DeadLetterRetentionDays = 0 };
var opts = new AlarmHistorianOptions { Enabled = true, DeadLetterRetentionDays = 0 };
opts.Validate().ShouldContain(w => w.Contains("DeadLetterRetentionDays"));
}
[Fact]
public void Validate_warns_on_non_positive_max_attempts()
{
var opts = new AlarmHistorianOptions { Enabled = true, DatabasePath = "/abs/h.db", MaxAttempts = 0 };
var opts = new AlarmHistorianOptions { Enabled = true, MaxAttempts = 0 };
opts.Validate().ShouldContain(w => w.Contains("MaxAttempts"));
}
[Fact]
public void Validate_accumulates_multiple_warnings()
{
// relative path + non-positive drain interval ⇒ both warnings, not short-circuited on the first.
var opts = new AlarmHistorianOptions { Enabled = true, DatabasePath = "alarm-historian.db", DrainIntervalSeconds = 0 };
// Two bad knobs ⇒ both warnings, not short-circuited on the first.
var opts = new AlarmHistorianOptions { Enabled = true, DrainIntervalSeconds = 0, Capacity = 0 };
var warnings = opts.Validate();
warnings.ShouldContain(w => w.Contains("DatabasePath"));
warnings.ShouldContain(w => w.Contains("DrainIntervalSeconds"));
warnings.ShouldContain(w => w.Contains("Capacity"));
warnings.Count.ShouldBeGreaterThanOrEqualTo(2);
}
}