fix: classifier robustness — schema in prompt, retries, kickoff fallback
The kickoff parse-and-confirm route was 500-ing intermittently because
Hermes-3 + Featherless's response_format={"type":"json_object"} only
guarantees JSON output, NOT a particular schema. The model was inventing
its own field names (sceneTime, entities, settingDetails) instead of
the KickoffParse fields, causing Pydantic validation to fail on both
classify() retries.
Three changes:
1. Include the Pydantic JSON schema in the system prompt so the model
knows exactly which keys to produce. Affects every classify() call
(kickoff parse, turn parse, scene-close detect, significance,
state-update, scene summarize). Strip ```json fences if the model
wraps its output. Bump retries 2 → 3 (model is stochastic; one extra
attempt closes most of the remaining gap).
2. parse_kickoff() now passes a default empty KickoffParse so the
route degrades to a fillable form instead of 500 when the classifier
ultimately fails. The confirm form is the human-in-the-loop; an
empty form is strictly better UX than a stack trace.
3. Tests updated: bumped canned-failure arrays from 2 → 3 entries to
match the new attempt count; renamed kickoff test from
"raises_when_classifier_fails_twice" to
"falls_back_to_empty_when_classifier_fails" reflecting the new
degraded-but-usable behavior.
Verified live with all 3 sample bots (maya/eli/sam) — kickoff route
returns 200 across multiple attempts. Full suite: 168 passed.
This commit is contained in:
+23
-12
@@ -117,15 +117,26 @@ async def test_parse_kickoff_applies_activity_defaults_for_missing_fields():
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parse_kickoff_raises_when_classifier_fails_twice():
|
||||
mock = MockLLMClient(canned=["nope", "still nope"])
|
||||
with pytest.raises(RuntimeError):
|
||||
await parse_kickoff(
|
||||
mock,
|
||||
model="m",
|
||||
bot_name="BotA",
|
||||
bot_persona="x",
|
||||
initial_relationship_to_you="y",
|
||||
kickoff_prose="z",
|
||||
you_name="You",
|
||||
)
|
||||
async def test_parse_kickoff_falls_back_to_empty_when_classifier_fails():
|
||||
"""When the classifier fails three times, return an empty KickoffParse
|
||||
instead of raising — the confirm form lets the user fill in by hand.
|
||||
"""
|
||||
mock = MockLLMClient(canned=["nope", "still nope", "still bad"])
|
||||
result = await parse_kickoff(
|
||||
mock,
|
||||
model="m",
|
||||
bot_name="BotA",
|
||||
bot_persona="x",
|
||||
initial_relationship_to_you="y",
|
||||
kickoff_prose="z",
|
||||
you_name="You",
|
||||
)
|
||||
assert isinstance(result, KickoffParse)
|
||||
assert result.container_name == ""
|
||||
assert result.container_type == ""
|
||||
assert result.edge_seed_summary == ""
|
||||
assert result.edge_seed_knowledge_facts == []
|
||||
# Activity defaults sane (action_interruptible defaults to True so the
|
||||
# confirm form's checkbox is in a reasonable initial state).
|
||||
assert result.you_activity.action_interruptible is True
|
||||
assert result.bot_activity.action_interruptible is True
|
||||
|
||||
Reference in New Issue
Block a user