perf(historian): recorder appends off its mailbox; Periodic commit default (lmxopcua#746)

The continuous-historization recorder awaited every outbox append (an fsync
under PerEntry) on its mailbox, so ingestion was capped at 1/fsync_latency
bursts per second while the mux kept telling into an unbounded, unmetered
mailbox. GetStatus also suspended the mailbox on an unbounded outbox count.

- Gated ingest bursts go to a bounded channel (64 bursts) drained by one
  ordered append loop, which commits every burst waiting behind a slow
  append in a single AppendBatchAsync (FIFO kept). Only a full queue holds
  the mailbox (backpressure, bounded memory). Counters are Interlocked; the
  loop nudges the drain. PostStop flushes queued bursts within the shared
  5 s stop budget.
- GetStatus reads counters on the actor thread and pipes a count bounded by
  a 10 s timeout (-1 when it does not answer); it no longer suspends the
  mailbox. RecorderStatus gains PendingAppends, MailboxDepth and
  AppendBackpressureWaits; two gauges publish pending appends and mailbox
  depth.
- ContinuousHistorization:CommitMode defaults to Periodic (100 ms). The
  Host's appsettings.json, its unparseable-value fallback and docker-dev
  still say PerEntry: filed as a Host follow-up (outside this lane).

Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-10-22 17:30:58 -04:00
parent f4907d098c
commit a5557375ee
7 changed files with 407 additions and 87 deletions
+11 -2
View File
@@ -266,7 +266,7 @@ and is registered the same way for every backend.
|---|---|---|---|
| `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + the selected backend's value writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. |
| `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. |
| `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** |
| `CommitMode` | string | `Periodic` | ⚠️ **Default changed from `PerEntry`.** `Periodic` commits on a background timer every `CommitIntervalMs` (a crash can lose at most the last interval of captured values); `PerEntry` fsyncs before the append returns (no loss window). Under `PerEntry` ingestion was capped at `1 / fsync_latency` appends/s. Parsed case-insensitively. ⚠️ The shipped `appsettings.json` / docker-dev still restate `PerEntry` (and the Host falls back to `PerEntry` for an unparseable value) until the Host-side follow-up lands — set `Periodic` explicitly to adopt it now. |
| `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. |
| `DrainBatchSize` | int | `1024` | Outbox entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over `DrainConcurrency` rather than sequential, so a wide batch no longer costs `tags × latency`. At 64 a drain drew down at most 64 entries every `DrainIntervalSeconds` — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples instead of catching up. |
| `DrainIntervalSeconds` | double | `2` | Steady drain cadence (and the post-success reschedule). |
@@ -277,7 +277,16 @@ and is registered the same way for every backend.
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
> **`PerEntry` now means one fsync per ingest *burst*, not per value.** The recorder appends a whole
> **The append is off the recorder's mailbox.** An ingest burst is queued to one ordered append loop, which commits
> every burst waiting behind it with a single `AppendBatchAsync` (FIFO kept). Up to **64 bursts** may be queued or
> appending; only a full queue holds the recorder's mailbox until room frees (backpressure on the mux, bounded memory) — the
> append used to be awaited on the mailbox, so a slow fsync throttled ingestion while the mux kept telling into an unbounded
> mailbox. Stopping the recorder flushes the queued bursts within the shared 5 s stop budget. `GetStatus` reports
> `PendingAppends`, `MailboxDepth` and `AppendBackpressureWaits`, no longer suspends the mailbox on the outbox count, and
> answers `QueuedDepth = -1` when the count does not return within 10 s. Gauges: `otopcua.historization.recorder.pending_appends`
> and `otopcua.historization.recorder.mailbox_depth`.
> **`PerEntry` means one fsync per append, not per value.** The recorder appends a whole
> value burst through `IHistorizationOutbox.AppendBatchAsync`, which enqueues every entry and then
> issues a **single** group commit — so a burst of N values costs one fsync rather than N. The
> durability contract is unchanged: the call does not return until that commit completes, so nothing