dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

test(historian): pin the failed-batch-append branch; correct its loss-accounting docs

Browse Source 
The batched ingest's append-failure catch had zero coverage, and two doc
statements overstated the loss: the Warning said the burst's values were
"dropped", and RecorderStatus.OutboxAppendFailures said "the value was
dropped, not recorded". The outbox contract is prefix-never-orphan, so a
partial-prefix failure leaves entries durably queued that drain normally.

- New test pins the branch: one 3-item burst whose group commit commits a
  1-entry prefix then throws => OutboxAppendFailures == 1 (per burst, not
  per value), TotalRecorded == 0, actor alive, and the prefix still reaches
  the writer via the post-failure DrainTick (drainInterval 30s, so the
  steady cadence cannot account for it). Falsified twice: counting per item
  and dropping the drain nudge each turn it red.
- Second test pins the OperationCanceledException filter from its
  observable side - a cancel that is NOT shutdown is metered, not swallowed.
  Removing the 'when (_lifetime.IsCancellationRequested)' turns it red. The
  shutdown side has no cheap seam (the counter dies with the actor).
- Docs corrected: OutboxAppendFailures counts FAILED BATCH APPENDS, one per
  burst whatever its size (a granularity change from the per-value era), and
  is not a count of values lost; TotalRecorded under-counts when a commit
  fails after a partial prefix. Log message says "failed to append" and
  names the prefix. No functional change beyond the message string.

Also folds in the DrainBatchSize default alignment behind 0298a0b9: the
recorder's two ctor defaults and its non-positive fallback still said 64
while ContinuousHistorizationOptions.DrainBatchSize now defaults to 1024.
No test pinned the old 64.
This commit is contained in:
Joseph Doherty
2026-08-16 04:50:30 -04:00
parent 0298a0b944
commit 451273c860
2 changed files with 222 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs
+36 -13
View File
@@ -92,11 +92,17 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
/// <summary>A point-in-time snapshot of the recorder's counters.</summary>
/// <param name="QueuedDepth">Un-acked entries currently held in the durable outbox.</param>
/// <param name="TotalRecorded">Lifetime count of values appended to the outbox.</param>
/// <param name="TotalRecorded">Lifetime count of values durably appended to the outbox. Credited per
/// BURST, on a successful group commit — so it UNDER-counts when a batch append fails after the outbox
/// durably committed a partial prefix: the outbox does not report how far it got, so the recorder cannot
/// credit that prefix. Those entries still drain; they are simply never counted here.</param>
/// <param name="DroppedNonNumeric">Lifetime count of values dropped for not being numeric-coercible.</param>
/// <param name="OutboxDropped">Lifetime count of entries the outbox dropped on capacity overflow.</param>
/// <param name="OutboxAppendFailures">Lifetime count of durable-boundary append failures (the value
/// was dropped, not recorded, and the actor stayed alive rather than restart-looping).</param>
/// <param name="OutboxAppendFailures">Lifetime count of FAILED BATCH APPENDS at the durable boundary —
/// ONE per failed burst whatever its size, never one per value (the granularity changed with the batched
/// ingest; the per-value era counted one per value). The actor stays alive rather than restart-looping.
/// It is NOT a count of values lost: the outbox contract is "prefix, never orphan", so a failed burst
/// may still have left entries durably queued that drain normally — see <see cref="IngestAsync"/>.</param>
/// <param name="LastDrainSucceeded">Whether the most recent drain pass acked cleanly.</param>
/// <param name="SuppressedNotPrimary">Lifetime count of values not captured because this node is
/// the Secondary of a pair whose Primary records the same refs (Gitea #537).</param>
@@ -200,7 +206,7 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
IHistorianValueWriter writer,
IHistorizationOutbox outbox,
IReadOnlyList<string> historizedRefs,
int drainBatchSize = 64,
int drainBatchSize = 1024,
TimeSpan? drainInterval = null,
TimeSpan? minBackoff = null,
TimeSpan? maxBackoff = null,
@@ -232,7 +238,7 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
IHistorianValueWriter writer,
IHistorizationOutbox outbox,
IReadOnlyList<string> historizedRefs,
int drainBatchSize = 64,
int drainBatchSize = 1024,
TimeSpan? drainInterval = null,
TimeSpan? minBackoff = null,
TimeSpan? maxBackoff = null,
@@ -252,7 +258,10 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
AddRef(r, r);
}
_drainBatchSize = drainBatchSize > 0 ? drainBatchSize : 64;
// Both this fallback and the ctor/Props defaults above must track ContinuousHistorizationOptions
// .DrainBatchSize (1024): production always passes the option through, so a divergent literal here
// only ever shows up as a quietly different batch size in tests and in degenerate-input callers.
_drainBatchSize = drainBatchSize > 0 ? drainBatchSize : 1024;
_drainInterval = drainInterval is { } di && di > TimeSpan.Zero ? di : TimeSpan.FromSeconds(2);
_minBackoff = minBackoff is { } mb && mb > TimeSpan.Zero ? mb : TimeSpan.FromSeconds(1);
_maxBackoff = maxBackoff is { } xb && xb > TimeSpan.Zero ? xb : TimeSpan.FromSeconds(30);
@@ -339,11 +348,14 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
/// #491 note below).
/// </para>
/// <para>
/// <b>Append failure is batch-wide.</b> A failed group commit is metered once
/// (<c>OutboxAppendFailures</c>), logged by category, and the burst's values are dropped — the
/// same never-restart-loop policy the per-value path had, applied to the batch. Note the outbox
/// contract makes any partially-appended prefix fully tracked and ackable, so the trailing drain
/// nudge still flushes it; nothing is left buffered-but-unackable.
/// <b>Append failure is batch-wide, and it is NOT a whole-burst loss.</b> A failed group commit
/// is metered ONCE (<c>OutboxAppendFailures</c> — one per failed burst whatever its size, never
/// one per value) and logged by category, keeping the same never-restart-loop policy the
/// per-value path had. What is actually lost is only what the outbox never committed: its
/// contract is "prefix, never orphan", so anything it durably appended before the failure stays
/// fully tracked and ackable and the trailing drain nudge flushes it normally — nothing is left
/// buffered-but-unackable. That prefix does NOT credit <c>TotalRecorded</c>, because the outbox
/// does not report how far it got; the counter under-counts rather than over-claims.
/// </para>
/// </remarks>
/// <param name="items">The burst's matched dependency values, in fan-out order.</param>
@@ -433,9 +445,20 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
// out of the handler — that would trip Akka supervision into a restart, and under a persistent
// disk fault the actor would restart-loop (re-register → next burst → append fails → restart →
// …). Mirror the drain path's catch-all: meter the failure (category only, no value content)
// and drop this burst's values.
// and carry on.
//
// The meter is ONE per failed burst, whatever the burst's size — a deliberate granularity change
// from the per-value era, where it was one per value. And it is NOT a count of values lost: the
// outbox commits a PREFIX or nothing (never an orphan), so an unknown number of this burst's
// entries may already be durable. Hence the message says "failed to append", not "dropped" —
// claiming total loss here would be wrong about the prefix the drain nudge below is about to
// flush. TotalRecorded is likewise not credited for that prefix: the outbox does not report how
// far it got, so the counter under-counts rather than over-claims.
_outboxAppendFailures++;
_log.Warning("ContinuousHistorization: outbox batch append failed ({Exception}); {Count} value(s) dropped.",
_log.Warning(
"ContinuousHistorization: outbox batch append failed ({Exception}); {Count} value(s) in the "
+ "burst failed to append. Any prefix the outbox committed before the failure stays queued "
+ "and drains normally; the remainder are lost.",
ex.GetType().Name, entries.Count);
}