Joseph Doherty
5c039c8e56
Two related issues blocking real-world use of the kickoff parse: 1. Classifier calls take ~12s end-to-end on Featherless for the complex KickoffParse schema (Hermes-3-8B generating ~1.3KB of structured JSON). The 10s timeout was firing on most attempts, causing all 3 retries to time out and the empty-fallback to render with blank form values. Bumping the default classifier_timeout_s 10 → 30s gives generous headroom; measured p99 is ~13s, so 30s is comfortable. 2. Featherless caps concurrent connections per account (2 on free / lower paid tiers). Each turn flow can fire 4–5 calls (parse, scene-close detect, narrative stream, two state-update passes) plus the background significance worker. Without a gate, we'd exceed the cap and fail. Added a class-level ``asyncio.Semaphore`` to FeatherlessClient, shared across all instances, configured once in lifespan from ``Settings.featherless_max_concurrent`` (default 2). Both ``generate`` and ``stream`` acquire the semaphore for the duration of the call; the stream holds it until the async generator completes, so token streaming is correctly accounted for. Verified live: 4/4 sequential kickoff parses for the same bot all succeed with real parsed values (previously ~50% blank-fallback). Full suite: 168 passed.
56 lines
2.1 KiB
Python
56 lines
2.1 KiB
Python
from __future__ import annotations
|
|
import asyncio
|
|
from typing import AsyncIterator, Sequence
|
|
from openai import AsyncOpenAI
|
|
from .client import Message
|
|
|
|
|
|
class FeatherlessClient:
|
|
"""Client for Featherless's OpenAI-compatible API.
|
|
|
|
Featherless caps concurrent connections per account (2 on free / lower
|
|
paid tiers). A class-level semaphore gates every ``generate`` and
|
|
``stream`` call so the orchestrator never exceeds the configured cap,
|
|
regardless of how many ``FeatherlessClient`` instances are alive.
|
|
|
|
Configure once at app startup via :meth:`configure_concurrency`. The
|
|
default is 2.
|
|
"""
|
|
|
|
_semaphore: asyncio.Semaphore | None = None
|
|
|
|
@classmethod
|
|
def configure_concurrency(cls, max_concurrent: int) -> None:
|
|
cls._semaphore = asyncio.Semaphore(max(1, int(max_concurrent)))
|
|
|
|
@classmethod
|
|
def _sem(cls) -> asyncio.Semaphore:
|
|
if cls._semaphore is None:
|
|
cls._semaphore = asyncio.Semaphore(2)
|
|
return cls._semaphore
|
|
|
|
def __init__(self, api_key: str, base_url: str = "https://api.featherless.ai/v1"):
|
|
self._client = AsyncOpenAI(api_key=api_key, base_url=base_url)
|
|
|
|
async def generate(self, messages: Sequence[Message], *, model: str, **params) -> str:
|
|
async with self._sem():
|
|
resp = await self._client.chat.completions.create(
|
|
model=model,
|
|
messages=[{"role": m.role, "content": m.content} for m in messages],
|
|
**params,
|
|
)
|
|
return resp.choices[0].message.content or ""
|
|
|
|
async def stream(self, messages: Sequence[Message], *, model: str, **params) -> AsyncIterator[str]:
|
|
async with self._sem():
|
|
stream = await self._client.chat.completions.create(
|
|
model=model,
|
|
messages=[{"role": m.role, "content": m.content} for m in messages],
|
|
stream=True,
|
|
**params,
|
|
)
|
|
async for chunk in stream:
|
|
delta = chunk.choices[0].delta.content or ""
|
|
if delta:
|
|
yield delta
|