perf(historian): recorder appends off its mailbox; Periodic commit default (lmxopcua#746)

The continuous-historization recorder awaited every outbox append (an fsync
under PerEntry) on its mailbox, so ingestion was capped at 1/fsync_latency
bursts per second while the mux kept telling into an unbounded, unmetered
mailbox. GetStatus also suspended the mailbox on an unbounded outbox count.

- Gated ingest bursts go to a bounded channel (64 bursts) drained by one
  ordered append loop, which commits every burst waiting behind a slow
  append in a single AppendBatchAsync (FIFO kept). Only a full queue holds
  the mailbox (backpressure, bounded memory). Counters are Interlocked; the
  loop nudges the drain. PostStop flushes queued bursts within the shared
  5 s stop budget.
- GetStatus reads counters on the actor thread and pipes a count bounded by
  a 10 s timeout (-1 when it does not answer); it no longer suspends the
  mailbox. RecorderStatus gains PendingAppends, MailboxDepth and
  AppendBackpressureWaits; two gauges publish pending appends and mailbox
  depth.
- ContinuousHistorization:CommitMode defaults to Periodic (100 ms). The
  Host's appsettings.json, its unparseable-value fallback and docker-dev
  still say PerEntry: filed as a Host follow-up (outside this lane).

Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-10-22 17:30:58 -04:00
parent f4907d098c
commit a5557375ee
7 changed files with 407 additions and 87 deletions
@@ -38,22 +38,20 @@ public sealed class ContinuousHistorizationOptions
public string OutboxPath { get; init; } = "";
/// <summary>
/// Outbox commit cadence: <c>PerEntry</c> (the default) fsyncs the log before each append
/// returns (safest, no loss window); <c>Periodic</c> batches commits onto a background timer
/// every <see cref="CommitIntervalMs"/> ms (higher throughput, a bounded worst-case loss window).
/// Parsed case-insensitively against <c>HistorizationCommitMode</c>; an unrecognized value falls
/// back to <c>PerEntry</c>.
/// Outbox commit cadence: <c>Periodic</c> (the default) commits on a background timer every
/// <see cref="CommitIntervalMs"/> ms (100 ms by default — a crash can lose at most the last interval of captured
/// values); <c>PerEntry</c> fsyncs before each append returns (no loss window, one fsync per append). Parsed
/// case-insensitively against <c>HistorizationCommitMode</c>.
/// <para>
/// <b><c>PerEntry</c> now means one fsync per ingest <em>burst</em>, not per value.</b> The
/// recorder appends a whole value burst through
/// <c>IHistorizationOutbox.AppendBatchAsync</c>, which enqueues every entry and then issues a
/// <em>single</em> group commit — so a burst of N values costs one fsync instead of N. The
/// durability contract is unchanged: <c>AppendBatchAsync</c> does not return until that
/// commit completes, so nothing the caller has been told is buffered can be lost to an
/// unclean restart.
/// <b>Why Periodic.</b> Under <c>PerEntry</c> every ingest append waits for an fsync, so ingestion is capped at
/// <c>1 / fsync_latency</c> appends per second (50 drivers × 1 Hz on a 10 ms-fsync disk kept the recorder half
/// busy; at 100+ bursts/s it fell behind for good). The recorder now appends off its mailbox and coalesces
/// every burst waiting behind a slow append into one group commit, which helps <c>PerEntry</c> too — but it is
/// still one fsync per group on the critical path. <c>PerEntry</c> remains available where no loss window is
/// acceptable.
/// </para>
/// </summary>
public string CommitMode { get; init; } = "PerEntry";
public string CommitMode { get; init; } = "Periodic";
/// <summary>Periodic-mode commit cadence in milliseconds; must be positive when
/// <see cref="CommitMode"/> is <c>Periodic</c>. Ignored under <c>PerEntry</c>.</summary>