perf(site-runtime): operation tracking store writes off the script thread (single-writer queue, batch per transaction)

A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run
between the cached DB write completing and Notify.Send starting: the immediate-
completion telemetry awaited two autocommit SQLite transactions on the consolidated
LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a
WAL fsync on a Docker volume; the submit row was a third. The audit half of the same
forwarder was already a channel write, so the tracking half was the whole cost.

OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered
work item and return; one background pump drains the bounded channel (8192, wait
backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction,
falling back to per-row apply when a batch fails so a poison row cannot take its
neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your-
writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before
them, skipped when idle) so Tracking.Status, the telemetry actor's status join and
central's PullSiteCalls never see a stale row; timestamps stamped when recorded;
PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before
closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public
FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a
hard crash (ms window); S&F buffer and audit rows are unaffected.

No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while
another connection holds the DB write lock and lands once released (fails on the old
store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200
queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the
SQLite file right after a store call now FlushAsync first.

Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29

docs/plans/2026-09-06-tracking-store-async-writes-design.md

@@ -0,0 +1,81 @@
+# Operation tracking store: writes off the script thread
+
+**Date:** 2026-09-06 · **Status:** implemented the same day (branch `feat/tracking-store-async-writes`)
+
+## Problem
+
+A Flow Debugger live run of `FlowOutboundDemo` (execution `1130c01f-…`) spent 194 ms end to end, of
+which the site script run was 176 ms. Reading the audit rows against the code path:
+
+| Segment | Time | What it is |
+|---|---|---|
+| `ExternalSystem.Call` RigRest.Echo | 4 ms | the REST round trip |
+| `Database.CachedWrite` submit → write completed | 61 ms | one tracking-store commit (`RecordEnqueueAsync`) + the MS SQL write |
+| write completed → `Notify.Send` starts | **108 ms** | no script statement: the immediate-completion telemetry — two more tracking-store commits (`RecordAttemptAsync`, `RecordTerminalAsync`) |
+| `Notify.Send` | ~2 ms | S&F enqueue |
+
+`OperationTrackingStore` executed every `Record*Async` as its own autocommit SQLite transaction on the
+consolidated LocalDb file, **awaited inline on the script thread** through
+`CachedCallTelemetryForwarder.ForwardAsync`. Each commit is a WAL append + fsync
+(`synchronous=NORMAL`, Docker volume): ~30–50 ms apiece here. The audit half of the same forwarder was
+already cheap — `SqliteAuditWriter.WriteAsync` is a `Channel.TryWrite` — so the tracking half was the
+whole cost. The API cached-call path (`ExternalSystem.CachedCall`) pays the same three commits.
+
+## Decision
+
+Make `OperationTrackingStore` a **single-writer queue**: `Record*Async` enqueue an ordered work item
+and return once it is queued; one background pump drains the queue and applies each drained batch in
+**one transaction** (one fsync per batch instead of per row). Nothing above the store changes —
+`IOperationTrackingStore`, the forwarder, `ScriptRuntimeContext` and the S&F lifecycle bridge are
+untouched, and their existing `await`s now cost a channel write.
+
+### Invariants kept
+
+- **FIFO per store.** One bounded channel, one reader, so enqueue → attempt → terminal apply in the
+  order they were recorded. The idempotent/first-write-wins SQL is unchanged.
+- **Read-your-writes.** `Tracking.Status(id)` (script), the telemetry actor's status join and central's
+  `PullSiteCalls` reconciliation all go through `GetStatusAsync`/`ReadChangedSinceAsync`, which now
+  **wait for every write queued before the call to be applied** (a barrier through the same queue,
+  skipped when nothing is pending). A script that calls `CachedWrite` and then `Tracking.Status` in
+  the next statement sees the row exactly as before.
+- **Timestamps.** `CreatedAtUtc`/`UpdatedAtUtc`/`TerminalAtUtc` are stamped when the call is
+  *recorded*, not when the batch is applied.
+- **`PurgeTerminalAsync` returns after the purge is applied** (it rides the queue with a completion).
+- **Shutdown drains.** `DisposeAsync` completes the queue and awaits the pump, so queued writes land
+  before the connection closes. The synchronous `Dispose()` keeps its documented "in-flight writes
+  fail" semantics and must not be used where a drain matters (host shutdown uses the async path).
+- **Best-effort failure handling, per row.** A batch that fails is rolled back and re-applied one
+  row at a time so a single poison row cannot take the others with it; each failed row logs a
+  warning, as the forwarder already did.
+
+### Costs accepted
+
+- **Crash window.** A row queued but not yet applied is lost on a hard crash (SIGKILL, power loss);
+  the tracking row then stays at its previous state — e.g. `Submitted` for a call that actually
+  delivered — until central's mirror is reconciled from the audit rows, or forever if it was the only
+  record. The window is the pump's cadence (sub-millisecond to a few milliseconds under load), and
+  the S&F buffer and audit rows, which are the delivery and evidence records, are unaffected. This is
+  the same trade the site audit writer made (`FlushIntervalMs`, `synchronous=NORMAL`).
+- **Bounded backpressure.** The queue holds `MaxQueuedWrites` (8 192) items; beyond that
+  `Record*Async` waits rather than drops — tracking is the status source of truth, so dropping is
+  not acceptable. At the volumes this table sees (a handful of rows per cached call) the bound is
+  never reached; if it is, the SQLite file is the bottleneck and the wait is the honest signal.
+- **A barrier read pays one pump round trip** when writes are pending. It is queued behind at most
+  one batch, so worst case it waits for one transaction (~one fsync) — the same cost a read used to
+  pay contending for the old write gate.
+
+### Rejected
+
+- *Fire-and-forget from the forwarder* (`_ = _trackingStore.Record…`): loses ordering across the
+  three lifecycle rows and read-your-writes, and moves the failure handling to an unobserved task.
+- *Batching in the forwarder*: the forwarder is per-packet; only the store sees consecutive rows from
+  different runs, which is where batching pays.
+- *A bigger LocalDb `synchronous` relaxation*: `OFF` would remove the fsync but risk corruption on
+  power loss for every replicated table, not just this one.
+
+## Verification
+
+- Store tests: a write returns while another connection holds the database write lock and lands
+  once it is released; reads observe every earlier write in order; `DisposeAsync` drains the queue.
+- Rig: rerun `FlowOutboundDemo` through the Flow Debugger and compare the `DbCall`-to-`NotifySend`
+  gap and the run total against the 108 ms / 176 ms baseline above.