fix: classifier robustness — schema in prompt, retries, kickoff fallback
The kickoff parse-and-confirm route was 500-ing intermittently because
Hermes-3 + Featherless's response_format={"type":"json_object"} only
guarantees JSON output, NOT a particular schema. The model was inventing
its own field names (sceneTime, entities, settingDetails) instead of
the KickoffParse fields, causing Pydantic validation to fail on both
classify() retries.
Three changes:
1. Include the Pydantic JSON schema in the system prompt so the model
knows exactly which keys to produce. Affects every classify() call
(kickoff parse, turn parse, scene-close detect, significance,
state-update, scene summarize). Strip ```json fences if the model
wraps its output. Bump retries 2 → 3 (model is stochastic; one extra
attempt closes most of the remaining gap).
2. parse_kickoff() now passes a default empty KickoffParse so the
route degrades to a fillable form instead of 500 when the classifier
ultimately fails. The confirm form is the human-in-the-loop; an
empty form is strictly better UX than a stack trace.
3. Tests updated: bumped canned-failure arrays from 2 → 3 entries to
match the new attempt count; renamed kickoff test from
"raises_when_classifier_fails_twice" to
"falls_back_to_empty_when_classifier_fails" reflecting the new
degraded-but-usable behavior.
Verified live with all 3 sample bots (maya/eli/sam) — kickoff route
returns 200 across multiple attempts. Full suite: 168 passed.
This commit is contained in:
@@ -85,6 +85,36 @@ def _build_user_prompt(
|
||||
)
|
||||
|
||||
|
||||
def _empty_activity() -> ActivityShape:
|
||||
return ActivityShape(
|
||||
posture="",
|
||||
action_verb="",
|
||||
action_interruptible=True,
|
||||
action_required_attention="low",
|
||||
action_expected_duration="brief",
|
||||
)
|
||||
|
||||
|
||||
def _empty_kickoff_parse() -> KickoffParse:
|
||||
"""Default returned when the classifier can't produce a valid parse.
|
||||
|
||||
The user gets a mostly-empty confirm form they can fill in by hand
|
||||
instead of a 500. ``initial_time_iso`` is left as the current UTC.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
return KickoffParse(
|
||||
container_name="",
|
||||
container_type="",
|
||||
container_properties={},
|
||||
you_activity=_empty_activity(),
|
||||
bot_activity=_empty_activity(),
|
||||
initial_time_iso=datetime.now(timezone.utc).isoformat(timespec="seconds"),
|
||||
edge_seed_summary="",
|
||||
edge_seed_knowledge_facts=[],
|
||||
)
|
||||
|
||||
|
||||
async def parse_kickoff(
|
||||
client: LLMClient,
|
||||
*,
|
||||
@@ -98,11 +128,9 @@ async def parse_kickoff(
|
||||
) -> KickoffParse:
|
||||
"""Parse authored kickoff prose into a structured ``KickoffParse``.
|
||||
|
||||
Internally calls :func:`chat.llm.classify.classify` with a labeled
|
||||
user prompt. Raises ``RuntimeError`` if the classifier fails twice in
|
||||
a row — no default is supplied at this layer, since the caller (T13's
|
||||
confirm form) is responsible for showing an error and letting the
|
||||
user edit.
|
||||
Falls back to a mostly-empty default if the classifier fails — the
|
||||
confirm-and-edit form is the human-in-the-loop, so a degraded form
|
||||
that the user can fill in is preferable to a 500.
|
||||
"""
|
||||
user_prompt = _build_user_prompt(
|
||||
bot_name=bot_name,
|
||||
@@ -117,5 +145,6 @@ async def parse_kickoff(
|
||||
system=_SYSTEM_PROMPT,
|
||||
user=user_prompt,
|
||||
schema=KickoffParse,
|
||||
default=_empty_kickoff_parse(),
|
||||
timeout_s=timeout_s,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user