Joseph Doherty
d91135f960
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run between the cached DB write completing and Notify.Send starting: the immediate- completion telemetry awaited two autocommit SQLite transactions on the consolidated LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a WAL fsync on a Docker volume; the submit row was a third. The audit half of the same forwarder was already a channel write, so the tracking half was the whole cost. OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered work item and return; one background pump drains the bounded channel (8192, wait backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction, falling back to per-row apply when a batch fails so a poison row cannot take its neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your- writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before them, skipped when idle) so Tracking.Status, the telemetry actor's status join and central's PullSiteCalls never see a stale row; timestamps stamped when recorded; PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a hard crash (ms window); S&F buffer and audit rows are unaffected. No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while another connection holds the DB write lock and lands once released (fails on the old store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200 queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the SQLite file right after a store call now FlushAsync first. Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
82 lines
5.3 KiB
Markdown
82 lines
5.3 KiB
Markdown
# Operation tracking store: writes off the script thread
|
||
|
||
**Date:** 2026-09-06 · **Status:** implemented the same day (branch `feat/tracking-store-async-writes`)
|
||
|
||
## Problem
|
||
|
||
A Flow Debugger live run of `FlowOutboundDemo` (execution `1130c01f-…`) spent 194 ms end to end, of
|
||
which the site script run was 176 ms. Reading the audit rows against the code path:
|
||
|
||
| Segment | Time | What it is |
|
||
|---|---|---|
|
||
| `ExternalSystem.Call` RigRest.Echo | 4 ms | the REST round trip |
|
||
| `Database.CachedWrite` submit → write completed | 61 ms | one tracking-store commit (`RecordEnqueueAsync`) + the MS SQL write |
|
||
| write completed → `Notify.Send` starts | **108 ms** | no script statement: the immediate-completion telemetry — two more tracking-store commits (`RecordAttemptAsync`, `RecordTerminalAsync`) |
|
||
| `Notify.Send` | ~2 ms | S&F enqueue |
|
||
|
||
`OperationTrackingStore` executed every `Record*Async` as its own autocommit SQLite transaction on the
|
||
consolidated LocalDb file, **awaited inline on the script thread** through
|
||
`CachedCallTelemetryForwarder.ForwardAsync`. Each commit is a WAL append + fsync
|
||
(`synchronous=NORMAL`, Docker volume): ~30–50 ms apiece here. The audit half of the same forwarder was
|
||
already cheap — `SqliteAuditWriter.WriteAsync` is a `Channel.TryWrite` — so the tracking half was the
|
||
whole cost. The API cached-call path (`ExternalSystem.CachedCall`) pays the same three commits.
|
||
|
||
## Decision
|
||
|
||
Make `OperationTrackingStore` a **single-writer queue**: `Record*Async` enqueue an ordered work item
|
||
and return once it is queued; one background pump drains the queue and applies each drained batch in
|
||
**one transaction** (one fsync per batch instead of per row). Nothing above the store changes —
|
||
`IOperationTrackingStore`, the forwarder, `ScriptRuntimeContext` and the S&F lifecycle bridge are
|
||
untouched, and their existing `await`s now cost a channel write.
|
||
|
||
### Invariants kept
|
||
|
||
- **FIFO per store.** One bounded channel, one reader, so enqueue → attempt → terminal apply in the
|
||
order they were recorded. The idempotent/first-write-wins SQL is unchanged.
|
||
- **Read-your-writes.** `Tracking.Status(id)` (script), the telemetry actor's status join and central's
|
||
`PullSiteCalls` reconciliation all go through `GetStatusAsync`/`ReadChangedSinceAsync`, which now
|
||
**wait for every write queued before the call to be applied** (a barrier through the same queue,
|
||
skipped when nothing is pending). A script that calls `CachedWrite` and then `Tracking.Status` in
|
||
the next statement sees the row exactly as before.
|
||
- **Timestamps.** `CreatedAtUtc`/`UpdatedAtUtc`/`TerminalAtUtc` are stamped when the call is
|
||
*recorded*, not when the batch is applied.
|
||
- **`PurgeTerminalAsync` returns after the purge is applied** (it rides the queue with a completion).
|
||
- **Shutdown drains.** `DisposeAsync` completes the queue and awaits the pump, so queued writes land
|
||
before the connection closes. The synchronous `Dispose()` keeps its documented "in-flight writes
|
||
fail" semantics and must not be used where a drain matters (host shutdown uses the async path).
|
||
- **Best-effort failure handling, per row.** A batch that fails is rolled back and re-applied one
|
||
row at a time so a single poison row cannot take the others with it; each failed row logs a
|
||
warning, as the forwarder already did.
|
||
|
||
### Costs accepted
|
||
|
||
- **Crash window.** A row queued but not yet applied is lost on a hard crash (SIGKILL, power loss);
|
||
the tracking row then stays at its previous state — e.g. `Submitted` for a call that actually
|
||
delivered — until central's mirror is reconciled from the audit rows, or forever if it was the only
|
||
record. The window is the pump's cadence (sub-millisecond to a few milliseconds under load), and
|
||
the S&F buffer and audit rows, which are the delivery and evidence records, are unaffected. This is
|
||
the same trade the site audit writer made (`FlushIntervalMs`, `synchronous=NORMAL`).
|
||
- **Bounded backpressure.** The queue holds `MaxQueuedWrites` (8 192) items; beyond that
|
||
`Record*Async` waits rather than drops — tracking is the status source of truth, so dropping is
|
||
not acceptable. At the volumes this table sees (a handful of rows per cached call) the bound is
|
||
never reached; if it is, the SQLite file is the bottleneck and the wait is the honest signal.
|
||
- **A barrier read pays one pump round trip** when writes are pending. It is queued behind at most
|
||
one batch, so worst case it waits for one transaction (~one fsync) — the same cost a read used to
|
||
pay contending for the old write gate.
|
||
|
||
### Rejected
|
||
|
||
- *Fire-and-forget from the forwarder* (`_ = _trackingStore.Record…`): loses ordering across the
|
||
three lifecycle rows and read-your-writes, and moves the failure handling to an unobserved task.
|
||
- *Batching in the forwarder*: the forwarder is per-packet; only the store sees consecutive rows from
|
||
different runs, which is where batching pays.
|
||
- *A bigger LocalDb `synchronous` relaxation*: `OFF` would remove the fsync but risk corruption on
|
||
power loss for every replicated table, not just this one.
|
||
|
||
## Verification
|
||
|
||
- Store tests: a write returns while another connection holds the database write lock and lands
|
||
once it is released; reads observe every earlier write in order; `DisposeAsync` drains the queue.
|
||
- Rig: rerun `FlowOutboundDemo` through the Flow Debugger and compare the `DbCall`-to-`NotifySend`
|
||
gap and the run total against the 108 ms / 176 ms baseline above.
|