perf(historian): recorder appends off its mailbox; Periodic commit default (lmxopcua#746)
The continuous-historization recorder awaited every outbox append (an fsync under PerEntry) on its mailbox, so ingestion was capped at 1/fsync_latency bursts per second while the mux kept telling into an unbounded, unmetered mailbox. GetStatus also suspended the mailbox on an unbounded outbox count. - Gated ingest bursts go to a bounded channel (64 bursts) drained by one ordered append loop, which commits every burst waiting behind a slow append in a single AppendBatchAsync (FIFO kept). Only a full queue holds the mailbox (backpressure, bounded memory). Counters are Interlocked; the loop nudges the drain. PostStop flushes queued bursts within the shared 5 s stop budget. - GetStatus reads counters on the actor thread and pipes a count bounded by a 10 s timeout (-1 when it does not answer); it no longer suspends the mailbox. RecorderStatus gains PendingAppends, MailboxDepth and AppendBackpressureWaits; two gauges publish pending appends and mailbox depth. - ContinuousHistorization:CommitMode defaults to Periodic (100 ms). The Host's appsettings.json, its unparseable-value fallback and docker-dev still say PerEntry: filed as a Host follow-up (outside this lane). Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22 in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
+11
-2
@@ -266,7 +266,7 @@ and is registered the same way for every backend.
|
||||
|---|---|---|---|
|
||||
| `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + the selected backend's value writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. |
|
||||
| `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. |
|
||||
| `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** |
|
||||
| `CommitMode` | string | `Periodic` | ⚠️ **Default changed from `PerEntry`.** `Periodic` commits on a background timer every `CommitIntervalMs` (a crash can lose at most the last interval of captured values); `PerEntry` fsyncs before the append returns (no loss window). Under `PerEntry` ingestion was capped at `1 / fsync_latency` appends/s. Parsed case-insensitively. ⚠️ The shipped `appsettings.json` / docker-dev still restate `PerEntry` (and the Host falls back to `PerEntry` for an unparseable value) until the Host-side follow-up lands — set `Periodic` explicitly to adopt it now. |
|
||||
| `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. |
|
||||
| `DrainBatchSize` | int | `1024` | Outbox entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over `DrainConcurrency` rather than sequential, so a wide batch no longer costs `tags × latency`. At 64 a drain drew down at most 64 entries every `DrainIntervalSeconds` — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples instead of catching up. |
|
||||
| `DrainIntervalSeconds` | double | `2` | Steady drain cadence (and the post-success reschedule). |
|
||||
@@ -277,7 +277,16 @@ and is registered the same way for every backend.
|
||||
| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
|
||||
| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
|
||||
|
||||
> **`PerEntry` now means one fsync per ingest *burst*, not per value.** The recorder appends a whole
|
||||
> **The append is off the recorder's mailbox.** An ingest burst is queued to one ordered append loop, which commits
|
||||
> every burst waiting behind it with a single `AppendBatchAsync` (FIFO kept). Up to **64 bursts** may be queued or
|
||||
> appending; only a full queue holds the recorder's mailbox until room frees (backpressure on the mux, bounded memory) — the
|
||||
> append used to be awaited on the mailbox, so a slow fsync throttled ingestion while the mux kept telling into an unbounded
|
||||
> mailbox. Stopping the recorder flushes the queued bursts within the shared 5 s stop budget. `GetStatus` reports
|
||||
> `PendingAppends`, `MailboxDepth` and `AppendBackpressureWaits`, no longer suspends the mailbox on the outbox count, and
|
||||
> answers `QueuedDepth = -1` when the count does not return within 10 s. Gauges: `otopcua.historization.recorder.pending_appends`
|
||||
> and `otopcua.historization.recorder.mailbox_depth`.
|
||||
|
||||
> **`PerEntry` means one fsync per append, not per value.** The recorder appends a whole
|
||||
> value burst through `IHistorizationOutbox.AppendBatchAsync`, which enqueues every entry and then
|
||||
> issues a **single** group commit — so a burst of N values costs one fsync rather than N. The
|
||||
> durability contract is unchanged: the call does not return until that commit completes, so nothing
|
||||
|
||||
Reference in New Issue
Block a user