chore(historian): batch-era defaults + appsettings/doc alignment

CLAUDE.md

@@ -934,10 +934,11 @@ only the recorder + outbox knobs:
 |---|---|---|
 | `Enabled` | `false` | `true` (with `ServerHistorian` configured) wires + spawns the recorder |
 | `OutboxPath` | `""` (required when enabled) | **Directory** holding the FasterLog segment + commit files. In production set an **absolute** path on durable storage |
-| `CommitMode` | `PerEntry` | `PerEntry` = fsync before each append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs` |
+| `CommitMode` | `PerEntry` | `PerEntry` = fsync before the append returns (no loss window); `Periodic` = batched commits every `CommitIntervalMs`. ⚠️ `PerEntry` now means **one fsync per ingest burst, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands |
 | `CommitIntervalMs` | `100` | Periodic-mode commit cadence; required `> 0` only under `Periodic` |
-| `DrainBatchSize` | `64` | Entries peeked + written per drain pass |
+| `DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs `tags × latency` — while 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, letting the outbox fall permanently behind during any gateway hiccup |
 | `DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule) |
+| `DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential `tags × latency` pass becomes `⌈tags / concurrency⌉ × latency` with outstanding round-trips bounded. `1` = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass" |
 | `Capacity` | `500000` | Max un-acked outbox entries before drop-oldest; `0` = unbounded. Non-zero **by default since #538** — the disk backstop for a historian refusing *every* write, the one case the per-tag quarantine deliberately cannot resolve |
 | `MaxWriteAttempts` | `5` | #538: consecutive failed drain passes one tag may accumulate — counted **only across passes where other tags wrote successfully** — before its queued entries are abandoned so the FIFO head can advance |
 | `MinBackoffSeconds` | `1` | Initial retry backoff after a failed drain pass |

docs/Configuration.md

@@ -383,6 +383,24 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
 [`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
 gateway keys), and the deployment prerequisites.
 
+#### `ContinuousHistorization` (recorder + outbox knobs)
+
+Only the knobs; the gateway connection comes from `ServerHistorian`. Full descriptions in
+[`docs/Historian.md` §ContinuousHistorization configuration](Historian.md#continuoushistorization-configuration).
+
+| Key | Default | Meaning |
+|---|---|---|
+| `ContinuousHistorization:Enabled` | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox and spawns the recorder. |
+| `ContinuousHistorization:OutboxPath` | `""` | **Directory** for the FasterLog segment + commit files. Required when enabled; use an **absolute** path on durable storage. |
+| `ContinuousHistorization:CommitMode` | `PerEntry` | `PerEntry` fsyncs before the append returns; `Periodic` commits on a timer every `CommitIntervalMs`. **`PerEntry` is one fsync per ingest *burst*, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry and issues a single group commit. Durability is unchanged: the call does not return until that commit lands. |
+| `ContinuousHistorization:CommitIntervalMs` | `100` | Periodic-mode cadence (ms); must be `> 0` under `Periodic`. |
+| `ContinuousHistorization:DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan (widening it costs allocation, not I/O) and the pass is now bounded-parallel rather than sequential, so a wide batch no longer costs `tags × latency`. 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples. |
+| `ContinuousHistorization:DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule). |
+| `ContinuousHistorization:DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** — a batch is grouped per tag and each group is its own gateway round-trip, so this turns a sequential `tags × latency` pass into `⌈tags / concurrency⌉ × latency` while bounding outstanding round-trips. `1` = fully sequential; `< 1` is clamped to 1 with a `Validate()` warning. The ack is unchanged (all-or-nothing after every group reports), as is the #538 quarantine rule. |
+| `ContinuousHistorization:Capacity` | `500000` | Max un-acked entries before drop-oldest; `0` = unbounded. The #538 disk backstop for a historian refusing **every** write — the one case the per-tag quarantine cannot resolve. |
+| `ContinuousHistorization:MaxWriteAttempts` | `5` | #538 poison-tag budget: consecutive failed passes one tag may accumulate — **counted only across passes where *other* tags wrote successfully** — before its entries are abandoned so the FIFO head can advance. |
+| `ContinuousHistorization:MinBackoffSeconds` / `MaxBackoffSeconds` | `1` / `30` | Initial and capped exponential retry backoff after a failed drain pass. |
+
 ### `LocalDb` (pair-local config cache — driver role only)
 
 Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed

docs/Historian.md

@@ -142,6 +142,47 @@ source it from there.
 
 ---
 
+## ContinuousHistorization configuration
+
+`ContinuousHistorizationOptions` binds the `ContinuousHistorization` section. It carries **only** the
+recorder + outbox knobs — the gateway connection (endpoint / key / TLS) is sourced from
+`ServerHistorian` (above), never restated here.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + gateway value-writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. |
+| `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. |
+| `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** |
+| `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. |
+| `DrainBatchSize` | int | `1024` | Outbox entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan of the outbox's live index (widening it costs allocation, not I/O), and a pass is now bounded-parallel over `DrainConcurrency` rather than sequential, so a wide batch no longer costs `tags × latency`. At 64 a drain drew down at most 64 entries every `DrainIntervalSeconds` — below the arrival rate of even a modest historized tag set — so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples instead of catching up. |
+| `DrainIntervalSeconds` | double | `2` | Steady drain cadence (and the post-success reschedule). |
+| `DrainConcurrency` | int | `8` | Historian tags a single drain pass writes **concurrently**. One peeked batch is grouped per historian tag and each group is an independent gateway round-trip, so a sequential pass cost `tags × latency`; this window makes it `⌈tags / concurrency⌉ × latency` while bounding how many round-trips can be outstanding against the gateway at once. `1` restores the fully sequential drain; a value `< 1` is clamped to 1 (with a `Validate()` warning). **The ack is unchanged** — still all-or-nothing after every group has reported, and the #538 per-tag quarantine still keys on "another group wrote this pass". |
+| `Capacity` | int | `500000` | Max un-acked outbox entries before the drop-oldest policy sheds the front of the queue; `0` = unbounded. Roughly 50 MB of FasterLog. The #538 **disk backstop** for the one case the per-tag quarantine deliberately cannot resolve — a historian unreachable or refusing *every* write, where nothing proves any particular tag is at fault so the recorder correctly retains. |
+| `MaxWriteAttempts` | int | `5` | #538 poison-tag budget: consecutive failed drain passes one historian tag may accumulate before its queued entries are abandoned and the FIFO head is allowed to advance. **Counted only across passes where *other* tags wrote successfully**, so an ordinary gateway outage never advances it. Clamped to at least 1; each abandonment logs a Warning naming the tag. |
+| `MinBackoffSeconds` | double | `1` | Initial retry backoff after a failed drain pass. |
+| `MaxBackoffSeconds` | double | `30` | Cap on the exponential retry backoff. |
+
+> **`PerEntry` now means one fsync per ingest *burst*, not per value.** The recorder appends a whole
+> value burst through `IHistorizationOutbox.AppendBatchAsync`, which enqueues every entry and then
+> issues a **single** group commit — so a burst of N values costs one fsync rather than N. The
+> durability contract is unchanged: the call does not return until that commit completes, so nothing
+> the caller has been told is buffered can be lost to an unclean restart. (The documented residual is
+> unchanged too: a throw *before* the commit leaves the enqueued prefix tracked-but-uncommitted —
+> served correctly for the life of the process, absent after an unclean restart — exactly the
+> enqueue-then-commit window the single-entry path always had, widened from one entry to the batch.)
+
+> **Cross-repo follow-up — a multi-tag batch `WriteLiveValues` on the gateway contract.** The drain's
+> unit of work is one gateway round-trip **per historian tag**, because
+> `ZB.MOM.WW.HistorianGateway.Client` **0.3.0** exposes no RPC that writes values for more than one tag
+> in a call. Bounded parallelism (`DrainConcurrency`) is therefore the ceiling: it overlaps the
+> round-trips but cannot remove them, so a wide tag set still pays `⌈tags / concurrency⌉` network
+> hops per drain pass. Collapsing that to a single hop needs a **batch/multi-tag `WriteLiveValues`** on
+> the `historian_gateway.v1` contract in the `~/Desktop/HistorianGateway` sister repo. Until that
+> lands, raising `DrainBatchSize` widens each hop's payload but not the hop count — which is why the
+> concurrency window, not the batch width, is what governs drain throughput today.
+
+---
+
 ## A poison tag no longer pins the outbox (#538)
 
 `DrainOnceAsync` peeks a batch, writes **every** per-tag group in it, and then acks **all-or-nothing** —

src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json

@@ -43,10 +43,17 @@
     "OutboxPath": "",
     "_OutboxPathComment": "Directory holding the FasterLog segment + commit files. Required when Enabled=true. In production set an ABSOLUTE path on durable storage.",
     "CommitMode": "PerEntry",
+    "_CommitModeComment": "PerEntry means one fsync per ingest BURST, not per value: the recorder appends a whole burst through AppendBatchAsync, which enqueues every entry and issues a single group commit. Periodic batches commits onto a timer every CommitIntervalMs (bounded loss window).",
     "CommitIntervalMs": 100,
-    "DrainBatchSize": 64,
+    "DrainBatchSize": 1024,
+    "_DrainBatchSizeComment": "Entries peeked + written per drain pass. Was 64 when a pass wrote its per-tag groups sequentially; the peek is an in-memory scan and the pass is now bounded-parallel over DrainConcurrency, so a wide batch is cheap — and 64 let the outbox fall permanently behind during any gateway hiccup.",
     "DrainIntervalSeconds": 2,
-    "Capacity": 0,
+    "DrainConcurrency": 8,
+    "_DrainConcurrencyComment": "Historian tags a single drain pass writes concurrently (one gateway round-trip per tag group). 1 restores the fully sequential drain. The ack is unchanged: still all-or-nothing after every group reports.",
+    "Capacity": 500000,
+    "_CapacityComment": "Max un-acked outbox entries before drop-oldest; 0 = unbounded. The Gitea #538 disk backstop for a historian refusing EVERY write — the one case the per-tag quarantine cannot resolve.",
+    "MaxWriteAttempts": 5,
+    "_MaxWriteAttemptsComment": "Gitea #538: consecutive failed drain passes one tag may accumulate — counted only across passes where OTHER tags wrote successfully — before its queued entries are abandoned so the FIFO head can advance.",
     "MinBackoffSeconds": 1,
     "MaxBackoffSeconds": 30
   },

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationOptions.cs

@@ -43,6 +43,15 @@ public sealed class ContinuousHistorizationOptions
     ///     every <see cref="CommitIntervalMs"/> ms (higher throughput, a bounded worst-case loss window).
     ///     Parsed case-insensitively against <c>HistorizationCommitMode</c>; an unrecognized value falls
     ///     back to <c>PerEntry</c>.
+    ///     <para>
+    ///         <b><c>PerEntry</c> now means one fsync per ingest <em>burst</em>, not per value.</b> The
+    ///         recorder appends a whole value burst through
+    ///         <c>IHistorizationOutbox.AppendBatchAsync</c>, which enqueues every entry and then issues a
+    ///         <em>single</em> group commit — so a burst of N values costs one fsync instead of N. The
+    ///         durability contract is unchanged: <c>AppendBatchAsync</c> does not return until that
+    ///         commit completes, so nothing the caller has been told is buffered can be lost to an
+    ///         unclean restart.
+    ///     </para>
     /// </summary>
     public string CommitMode { get; init; } = "PerEntry";
 
@@ -50,9 +59,23 @@ public sealed class ContinuousHistorizationOptions
     /// <see cref="CommitMode"/> is <c>Periodic</c>. Ignored under <c>PerEntry</c>.</summary>
     public int CommitIntervalMs { get; init; } = 100;
 
-    /// <summary>Maximum outbox entries peeked + written per drain pass; clamped to a positive value by
-    /// the recorder (a non-positive value falls back to 64).</summary>
-    public int DrainBatchSize { get; init; } = 64;
+    /// <summary>
+    ///     Maximum outbox entries peeked + written per drain pass; clamped to a positive value by the
+    ///     recorder (a non-positive value falls back to the default).
+    ///     <para>
+    ///         Defaults to 1024. It was 64 when a pass wrote its per-tag groups <em>sequentially</em>, so
+    ///         a wide batch cost <c>tags × latency</c> on the drain thread. Both halves of that reasoning
+    ///         are gone: the peek is an in-memory scan of the outbox's live index (widening it costs
+    ///         allocation, not I/O), and the pass is now bounded-parallel over
+    ///         <see cref="DrainConcurrency"/>, so its cost is <c>⌈tags / concurrency⌉ × latency</c>
+    ///         regardless of batch width. At 64, a drain draws down at most 64 entries every
+    ///         <see cref="DrainIntervalSeconds"/> — below the arrival rate of even a modest historized
+    ///         tag set — so any gateway hiccup left the outbox permanently behind, growing toward
+    ///         <see cref="Capacity"/> and shedding oldest samples, rather than catching up once the
+    ///         gateway recovered.
+    ///     </para>
+    /// </summary>
+    public int DrainBatchSize { get; init; } = 1024;
 
     /// <summary>Steady drain cadence in seconds (also the post-success reschedule). Defaults to 2.</summary>
     public double DrainIntervalSeconds { get; init; } = 2;

tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ContinuousHistorizationOptionsTests.cs

@@ -36,6 +36,26 @@ public sealed class ContinuousHistorizationOptionsTests
             }.Validate(),
             m => m.Contains("CommitIntervalMs"));
 
+    /// <summary>
+    ///     Pins the shipped defaults, which are a tuned set rather than arbitrary numbers: the drain is
+    ///     bounded-parallel over an in-memory peek (so a wide <c>DrainBatchSize</c> is cheap and a narrow
+    ///     one lets the outbox fill during a gateway hiccup), <c>Capacity</c> is the #538 disk backstop,
+    ///     and <c>MaxWriteAttempts</c> is the #538 poison-tag budget. The shipped
+    ///     <c>appsettings.json</c> restates these values, so a silent drift between the two is exactly
+    ///     what this test exists to catch.
+    /// </summary>
+    [Fact]
+    public void Defaults_are_the_documented_batch_era_values()
+    {
+        var options = new ContinuousHistorizationOptions();
+
+        Assert.Equal("PerEntry", options.CommitMode);
+        Assert.Equal(1024, options.DrainBatchSize);
+        Assert.Equal(8, options.DrainConcurrency);
+        Assert.Equal(500_000, options.Capacity);
+        Assert.Equal(5, options.MaxWriteAttempts);
+    }
+
     /// <summary>A fully-configured enabled recorder produces no warnings.</summary>
     [Fact]
     public void Valid_config_is_clean()