Four changes that compound:
1) **SQLite busy_timeout 5.0s -> 0.1s** in chat/db/connection.py. Root
cause of the bulk of the slowness. The embedding worker contends
for the WAL write lock while the request handler holds an open
transaction; conn.execute's busy-wait does NOT release the GIL, so
every state_update LLM call after the narrative was silently
freezing the asyncio event loop for ~5s. With 0.1s the worker
fails fast and logs (already handled), the chat keeps moving, and
any missed embedding can be backfilled out of band. Also takes the
test suite from ~290s -> 13s as a bonus.
2) **Parallel state-update pairs** in multi_state_update.py. Each
directed (src, tgt) pair becomes a coroutine in asyncio.gather
instead of a sequential for-loop. Returned order is preserved.
3) **Classifier on OpenRouter, provider-pinned to Cerebras**. New
prefix-based router: model id with mlx-community/ -> local MLX,
model == narrative_model -> narrative remote, else -> classifier
remote. Settings.classifier_provider_order populates extra_body for
the classifier client only (FeatherlessClient now accepts
default_extra_body to merge into every chat.completions.create).
Llama-3.1-8B on Cerebras runs at ~423 tok/s, ~10x the default
provider. narrative still routes to mistral-nemo:nitro (Friendli).
4) **Cap classify max_tokens at 512**. A misbehaving classifier
(response_format=json_object ignored) could otherwise generate
thousands of tokens of prose before classify's JSON validation
trips the retry. 512 is generous; usual completions are 50-150.
CHAT_LLM_TIMING=1 env var enables per-call timing logs on stderr;
zero overhead when unset. Useful for finding the slow link.
Suite: 464 passed in 13s (was 290s).
Joseph Doherty
de7f6624f0