perf(site-runtime): operation tracking store writes off the script thread (single-writer queue, batch per transaction)

A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.

OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.

No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29

docs/plans/2026-09-06-tracking-store-async-writes-design.md

@@ -0,0 +1,81 @@
+# Operation tracking store: writes off the script thread
+
+**Date:** 2026-09-06 · **Status:** implemented the same day (branch `feat/tracking-store-async-writes`)
+
+## Problem
+
+A Flow Debugger live run of `FlowOutboundDemo` (execution `1130c01f-…`) spent 194 ms end to end, of
+which the site script run was 176 ms. Reading the audit rows against the code path:
+
+| Segment | Time | What it is |
+|---|---|---|
+| `ExternalSystem.Call` RigRest.Echo | 4 ms | the REST round trip |
+| `Database.CachedWrite` submit → write completed | 61 ms | one tracking-store commit (`RecordEnqueueAsync`) + the MS SQL write |
+| write completed → `Notify.Send` starts | **108 ms** | no script statement: the immediate-completion telemetry — two more tracking-store commits (`RecordAttemptAsync`, `RecordTerminalAsync`) |
+| `Notify.Send` | ~2 ms | S&F enqueue |
+
+`OperationTrackingStore` executed every `Record*Async` as its own autocommit SQLite transaction on the
+consolidated LocalDb file, **awaited inline on the script thread** through
+`CachedCallTelemetryForwarder.ForwardAsync`. Each commit is a WAL append + fsync
+(`synchronous=NORMAL`, Docker volume): ~30–50 ms apiece here. The audit half of the same forwarder was
+already cheap — `SqliteAuditWriter.WriteAsync` is a `Channel.TryWrite` — so the tracking half was the
+whole cost. The API cached-call path (`ExternalSystem.CachedCall`) pays the same three commits.
+
+## Decision
+
+Make `OperationTrackingStore` a **single-writer queue**: `Record*Async` enqueue an ordered work item
+and return once it is queued; one background pump drains the queue and applies each drained batch in
+**one transaction** (one fsync per batch instead of per row). Nothing above the store changes —
+`IOperationTrackingStore`, the forwarder, `ScriptRuntimeContext` and the S&F lifecycle bridge are
+untouched, and their existing `await`s now cost a channel write.
+
+### Invariants kept
+
+- **FIFO per store.** One bounded channel, one reader, so enqueue → attempt → terminal apply in the
+  order they were recorded. The idempotent/first-write-wins SQL is unchanged.
+- **Read-your-writes.** `Tracking.Status(id)` (script), the telemetry actor's status join and central's
+  `PullSiteCalls` reconciliation all go through `GetStatusAsync`/`ReadChangedSinceAsync`, which now
+  **wait for every write queued before the call to be applied** (a barrier through the same queue,
+  skipped when nothing is pending). A script that calls `CachedWrite` and then `Tracking.Status` in
+  the next statement sees the row exactly as before.
+- **Timestamps.** `CreatedAtUtc`/`UpdatedAtUtc`/`TerminalAtUtc` are stamped when the call is
+  *recorded*, not when the batch is applied.
+- **`PurgeTerminalAsync` returns after the purge is applied** (it rides the queue with a completion).
+- **Shutdown drains.** `DisposeAsync` completes the queue and awaits the pump, so queued writes land
+  before the connection closes. The synchronous `Dispose()` keeps its documented "in-flight writes
+  fail" semantics and must not be used where a drain matters (host shutdown uses the async path).
+- **Best-effort failure handling, per row.** A batch that fails is rolled back and re-applied one
+  row at a time so a single poison row cannot take the others with it; each failed row logs a
+  warning, as the forwarder already did.
+
+### Costs accepted
+
+- **Crash window.** A row queued but not yet applied is lost on a hard crash (SIGKILL, power loss);
+  the tracking row then stays at its previous state — e.g. `Submitted` for a call that actually
+  delivered — until central's mirror is reconciled from the audit rows, or forever if it was the only
+  record. The window is the pump's cadence (sub-millisecond to a few milliseconds under load), and
+  the S&F buffer and audit rows, which are the delivery and evidence records, are unaffected. This is
+  the same trade the site audit writer made (`FlushIntervalMs`, `synchronous=NORMAL`).
+- **Bounded backpressure.** The queue holds `MaxQueuedWrites` (8 192) items; beyond that
+  `Record*Async` waits rather than drops — tracking is the status source of truth, so dropping is
+  not acceptable. At the volumes this table sees (a handful of rows per cached call) the bound is
+  never reached; if it is, the SQLite file is the bottleneck and the wait is the honest signal.
+- **A barrier read pays one pump round trip** when writes are pending. It is queued behind at most
+  one batch, so worst case it waits for one transaction (~one fsync) — the same cost a read used to
+  pay contending for the old write gate.
+
+### Rejected
+
+- *Fire-and-forget from the forwarder* (`_ = _trackingStore.Record…`): loses ordering across the
+  three lifecycle rows and read-your-writes, and moves the failure handling to an unobserved task.
+- *Batching in the forwarder*: the forwarder is per-packet; only the store sees consecutive rows from
+  different runs, which is where batching pays.
+- *A bigger LocalDb `synchronous` relaxation*: `OFF` would remove the fsync but risk corruption on
+  power loss for every replicated table, not just this one.
+
+## Verification
+
+- Store tests: a write returns while another connection holds the database write lock and lands
+  once it is released; reads observe every earlier write in order; `DisposeAsync` drains the queue.
+- Rig: rerun `FlowOutboundDemo` through the Flow Debugger and compare the `DbCall`-to-`NotifySend`
+  gap and the run total against the 108 ms / 176 ms baseline above.

docs/requirements/Component-SiteRuntime.md

@@ -504,6 +504,7 @@ Available to all script runs and alarm on-trigger runs:
 
 ### Operation Tracking
 - `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)
+- **Tracking writes are off the script thread (2026-09-06).** `OperationTrackingStore` is a single-writer queue: `RecordEnqueue/Attempt/TerminalAsync` (reached from the script through `CachedCallTelemetryForwarder`) return once the row is *queued*, and one background pump applies each drained batch in one SQLite transaction — one fsync per batch instead of one per lifecycle row. Before this, a `Database.CachedWrite` that delivered immediately paid three awaited autocommit transactions on the consolidated LocalDb file, ~108 ms of a 176 ms script run measured on the rig. Preserved: FIFO order across the three lifecycle rows; **read-your-writes** — `GetStatusAsync` (behind `Tracking.Status`, and the telemetry actor's status join) and `ReadChangedSinceAsync` (central's `PullSiteCalls`) wait for every write queued before them; timestamps are stamped when recorded, not when applied; `PurgeTerminalAsync` returns after the purge lands; `DisposeAsync` drains the queue (the sync `Dispose()` does not — host shutdown uses the async path). Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. `FlushAsync()` is the explicit barrier for callers that go to the file directly. Design: `docs/plans/2026-09-06-tracking-store-async-writes-design.md`.
 
 ### Alarm State (read-only)
 - `Alarms.CurrentAsync()` — Returns a snapshot of the instance's **current** alarm conditions as `IReadOnlyList<ScriptAlarm>` (`Commons.Types.Scripts`), covering computed alarms and mirrored native (OPC UA A&C / MxAccess Gateway) conditions alike. Served by a **local Ask** (`GetAlarmSnapshotRequest`/`GetAlarmSnapshotResponse`) to the script's own Instance Actor — the same mechanism attribute reads use, never a cross-cluster hop.

docs/requirements/Component-StoreAndForward.md

@@ -53,7 +53,7 @@ For notifications, "delivery" means forwarding the message to the central cluste
 
 There are now exactly two ways a notification parks. First, a **corrupt buffered payload** (a row whose stored JSON no longer deserialises to a `NotificationSubmit`, or deserialises to null): the forwarder returns the delivery-handler contract's permanent-failure signal (`false`) so the engine parks the row, preserving the payload for operator forensics. This **supersedes `StoreAndForward-018`**, which discarded such rows by reporting them as delivered — a silent data loss with no parked row, no central `Notifications` row, and no audit trail. Retrying a corrupt payload is pointless (it can never deserialise), so parking, not retrying, is the correct home for it. Second, retry exhaustion under the `DefaultMaxRetries` cap — but because `Notify.Send` enqueues with `maxRetries: 0` (its unbounded-retry contract, above), that second cause is **off for script-produced notifications**, leaving corrupt payload as their only parking cause. (Retry-exhaustion parking remains reachable only for a hypothetical notification enqueued with a positive `maxRetries`.)
 
-For the cached-call categories (`ExternalCall` and `DatabaseWrite`), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal `Delivered` tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to `Pending`/`Retrying`; the buffered message carries its `TrackedOperationId` so the tracking row and the retry record stay linked. When immediate delivery fails **permanently** (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal `Failed` row capturing the error. On every tracking-table status transition the site emits `CachedCallTelemetry` to central.
+For the cached-call categories (`ExternalCall` and `DatabaseWrite`), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal `Delivered` tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to `Pending`/`Retrying`; the buffered message carries its `TrackedOperationId` so the tracking row and the retry record stay linked. When immediate delivery fails **permanently** (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal `Failed` row capturing the error. On every tracking-table status transition the site emits `CachedCallTelemetry` to central. The tracking row itself is written through a single-writer queue (Site Runtime, 2026-09-06): the script's `await` on each lifecycle emission returns once the row is queued, not committed, and reads of the table wait for earlier queued writes so status stays read-your-writes — see `Component-SiteRuntime.md` → *Operation Tracking*.
 
 Every cached-call outcome maps to a tracking-table state: immediate success → `Delivered`; transient failure → `Pending`/`Retrying`, eventually `Delivered` or `Parked`; permanent failure → terminal `Failed`; operator discard of a parked row → terminal `Discarded`.