perf(site-runtime): operation tracking store writes off the script thread (single-writer queue, batch per transaction)
A Flow Debugger live run (execution 1130c01f) spent 108 of a 176 ms site script run between the cached DB write completing and Notify.Send starting: the immediate- completion telemetry awaited two autocommit SQLite transactions on the consolidated LocalDb file (RecordAttemptAsync + RecordTerminalAsync), on the script thread, each a WAL fsync on a Docker volume; the submit row was a third. The audit half of the same forwarder was already a channel write, so the tracking half was the whole cost. OperationTrackingStore is now a single-writer queue: Record*Async enqueue an ordered work item and return; one background pump drains the bounded channel (8192, wait backpressure - tracking never drops) and applies each batch (<=256) in ONE transaction, falling back to per-row apply when a batch fails so a poison row cannot take its neighbours. Preserved invariants: FIFO across enqueue/attempt/terminal; read-your- writes (GetStatusAsync and ReadChangedSinceAsync barrier on every write queued before them, skipped when idle) so Tracking.Status, the telemetry actor's status join and central's PullSiteCalls never see a stale row; timestamps stamped when recorded; PurgeTerminalAsync returns after the purge lands; DisposeAsync drains the queue before closing; sync Dispose keeps its documented in-flight-writes-fail semantics. New public FlushAsync() is the explicit barrier. Accepted: a queued-but-unapplied row is lost on a hard crash (ms window); S&F buffer and audit rows are unaffected. No interface, forwarder or ScriptRuntimeContext change. Tests: write returns while another connection holds the DB write lock and lands once released (fails on the old store: 31 s); reads see 100 ordered attempts + terminal; DisposeAsync drains 200 queued rows; write after dispose throws ObjectDisposed. Existing tests that poke the SQLite file right after a store call now FlushAsync first. Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
This commit is contained in:
@@ -53,7 +53,7 @@ For notifications, "delivery" means forwarding the message to the central cluste
|
||||
|
||||
There are now exactly two ways a notification parks. First, a **corrupt buffered payload** (a row whose stored JSON no longer deserialises to a `NotificationSubmit`, or deserialises to null): the forwarder returns the delivery-handler contract's permanent-failure signal (`false`) so the engine parks the row, preserving the payload for operator forensics. This **supersedes `StoreAndForward-018`**, which discarded such rows by reporting them as delivered — a silent data loss with no parked row, no central `Notifications` row, and no audit trail. Retrying a corrupt payload is pointless (it can never deserialise), so parking, not retrying, is the correct home for it. Second, retry exhaustion under the `DefaultMaxRetries` cap — but because `Notify.Send` enqueues with `maxRetries: 0` (its unbounded-retry contract, above), that second cause is **off for script-produced notifications**, leaving corrupt payload as their only parking cause. (Retry-exhaustion parking remains reachable only for a hypothetical notification enqueued with a positive `maxRetries`.)
|
||||
|
||||
For the cached-call categories (`ExternalCall` and `DatabaseWrite`), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal `Delivered` tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to `Pending`/`Retrying`; the buffered message carries its `TrackedOperationId` so the tracking row and the retry record stay linked. When immediate delivery fails **permanently** (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal `Failed` row capturing the error. On every tracking-table status transition the site emits `CachedCallTelemetry` to central.
|
||||
For the cached-call categories (`ExternalCall` and `DatabaseWrite`), the operation tracking table is the status record and the S&F buffer is purely the retry mechanism. A cached call that succeeds on its first immediate attempt is written directly as a terminal `Delivered` tracking row and never enters the S&F buffer. When immediate delivery fails transiently, the message is buffered and its tracking row moves to `Pending`/`Retrying`; the buffered message carries its `TrackedOperationId` so the tracking row and the retry record stay linked. When immediate delivery fails **permanently** (e.g. HTTP 4xx), the message is not buffered — the error is returned synchronously to the calling script as before — but the tracking row is written directly as a terminal `Failed` row capturing the error. On every tracking-table status transition the site emits `CachedCallTelemetry` to central. The tracking row itself is written through a single-writer queue (Site Runtime, 2026-09-06): the script's `await` on each lifecycle emission returns once the row is queued, not committed, and reads of the table wait for earlier queued writes so status stays read-your-writes — see `Component-SiteRuntime.md` → *Operation Tracking*.
|
||||
|
||||
Every cached-call outcome maps to a tracking-table state: immediate success → `Delivered`; transient failure → `Pending`/`Retrying`, eventually `Delivered` or `Parked`; permanent failure → terminal `Failed`; operator discard of a parked row → terminal `Discarded`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user