Compare commits
99 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| fae6edef6b | |||
| 2ab8fcbdf0 | |||
| 5d5c888acf | |||
| a45a33534f | |||
| f3827706df | |||
| 2afbb9fefc | |||
| 1f8b4d2078 | |||
| 3f1a284acb | |||
| 87f93f00b5 | |||
| d1e2902655 | |||
| 54dfa8d611 | |||
| 5d36d3456f | |||
| 0e9421dcf7 | |||
| baffeb3a44 | |||
| 29b7c90b29 | |||
| 64c9ca634a | |||
| 374a76c867 | |||
| b65e1e1098 | |||
| 996a16cfb5 | |||
| a06f90a164 | |||
| df977fc985 | |||
| 51a12afbec | |||
| fc3020a0ee | |||
| 228f9abb19 | |||
| b6119879e5 | |||
| 3b4c7b9cef | |||
| 36d75fa6e7 | |||
| 0a2c5924f9 | |||
| a5f0e69d44 | |||
| 3dbe1a01ff | |||
| 4546bc0d9c | |||
| c4fa11fe78 | |||
| 461d441078 | |||
| b25007eb44 | |||
| d39d31479d | |||
| 7899c50b6c | |||
| 177e39d59c | |||
| d85ed8aaa6 | |||
| 9c63d6b24c | |||
| 64a07aa87f | |||
| 6674f9475c | |||
| 50448b72f8 | |||
| b8b4aed6d9 | |||
| 5ff107574c | |||
| 915d625d7f | |||
| 28e13d416f | |||
| 296e8fdddd | |||
| 013b563f21 | |||
| 62d5cdd826 | |||
| a25c166174 | |||
| 8f66e1123a | |||
| caa17b4174 | |||
| c7cb0eb01e | |||
| 1d6768e980 | |||
| 8b086d4bb8 | |||
| 6c7ac8f69f | |||
| fe34d4f4c0 | |||
| 0d76a6b2d6 | |||
| cc71fb4d01 | |||
| c06a32767b | |||
| 0ba374b790 | |||
| 77f1636086 | |||
| bffd9a2f38 | |||
| 1b66a2821c | |||
| 74bb42397d | |||
| 3be8ed8915 | |||
| 097073ede5 | |||
| 4a2617565b | |||
| 73625c0ac4 | |||
| aea20a2c83 | |||
| 9493d24a53 | |||
| da7aa88b8e | |||
| 82701d3c18 | |||
| 0de4d1252c | |||
| b667a21c99 | |||
| a1e2d9a24d | |||
| d833bbc3e7 | |||
| f2fd30c5a9 | |||
| 9e7c16de40 | |||
| 71245fb85a | |||
| be92691f9a | |||
| 6f50ce5b7a | |||
| f816d44438 | |||
| 6f0716201f | |||
| 0d3bbf4272 | |||
| b91a5e9293 | |||
| 9d06eaf57a | |||
| dae481eb92 | |||
| d123684f9a | |||
| 29e6f346ef | |||
| cb570a5adc | |||
| ce4f56adfa | |||
| bdc93b4b67 | |||
| 9c9d71eb31 | |||
| 4199038b8b | |||
| e3dfe18811 | |||
| d759b90aa1 | |||
| fb7e97260b | |||
| 753cec327f |
@@ -204,15 +204,7 @@ Phase 2.5 cleanup shipped end-to-end across 8 tasks (T68–T75). Two CLAUDE.md b
|
|||||||
|
|
||||||
### Phase 2.6 / 3 backlog
|
### Phase 2.6 / 3 backlog
|
||||||
|
|
||||||
New follow-ups discovered during Phase 2.5 execution. None are blocking; pick up at any time.
|
All items shipped — see Phase 3.5 status below.
|
||||||
|
|
||||||
- **Frontend handler for `turn_html_replace` SSE event (from T73.1 review)**: regenerate's backend broadcast lands, but no live tab swaps the regenerated turn until a JS handler is wired. The existing `turn_html` event uses HTMX `sse-swap` to append; `turn_html_replace` ships JSON with `supersedes_id` for replacement semantics. Phase 2.6 should wire the JS to swap the prior turn's DOM node in place.
|
|
||||||
- **Cancel/stop hook for in-flight regenerate streams (from T73 review)**: `post_turn` registers stream tasks in `_in_flight_tasks` so the user can stop them. Regenerate doesn't. A user clicking "Stop" mid-regenerate has no cancel hook today.
|
|
||||||
- **DRY: regenerate vs post_turn (from T73 review)**: recent-dialogue assembly and prior-edges block are duplicated between `chat/services/regenerate.py` and `chat/web/turns.py`. Extract to shared helpers analogous to `_gather_state_update_inputs`.
|
|
||||||
- **Sibling-discovery query optimization (from T73 review)**: `regenerate.py`'s sibling-assistant-turn lookup scans all non-superseded `assistant_turn` rows globally. Adding a `chat_id` predicate via JSON extraction (or a denormalized column) bounds the cost to per-chat scale.
|
|
||||||
- **`_witness_role_for` defensive coding (from T71 review)**: helper returns `"guest"` when `host_bot_id is None`, which is wrong for Phase-1 chats. Defensive: `return "host" if host_bot_id is None or speaker_bot_id == host_bot_id else "guest"`. Not exercised by current tests; harden as a precaution.
|
|
||||||
- **Confidence type tightening (from T74 review)**: `chat/services/addressee.py::AddresseeDecision.confidence` could be typed as `Literal["high","medium","low"]` for stricter validation. Currently `str` with a comment.
|
|
||||||
- **Scene-close-on-cancel UX revisit**: T74.3 pinned the existing behavior (close fires even on cancel). If real play-testing surfaces a regression, revisit.
|
|
||||||
|
|
||||||
## Phase 3 status
|
## Phase 3 status
|
||||||
|
|
||||||
@@ -249,51 +241,134 @@ Phase 3 shipped end-to-end across 19 tasks (T49–T67). Events with full lifecyc
|
|||||||
|
|
||||||
### Phase 3.5 / 4 backlog
|
### Phase 3.5 / 4 backlog
|
||||||
|
|
||||||
New follow-ups discovered during Phase 3 reviews and execution. None are blocking; pick up at any time.
|
All items shipped — see Phase 3.5 status below.
|
||||||
|
|
||||||
#### From T53 review
|
## Phase 3.5 status
|
||||||
|
|
||||||
- **`narrate_skip` `timeout_s` not piped through to `client.generate`**: parameter accepted but ignored. Fix: pass `timeout_s=timeout_s` to `client.generate(**...)`, or drop the parameter entirely if Featherless's client doesn't honor it.
|
Phase 3.5 cleanup shipped end-to-end across 12 tasks (T76–T87). Two CLAUDE.md backlogs (Phase 2.6/3, Phase 3.5/4) are now empty; deferred follow-ups discovered during execution are tracked in a new "Phase 3.6 / 4 backlog" section below. Test count grew from 315 (Phase 3) to 343 (+28 new tests).
|
||||||
|
|
||||||
#### From T57 review
|
- **Wave 1 — trivial polish (parallel)**:
|
||||||
|
- **T76** `narrate_skip` `timeout_s` plumbed through to `client.generate`.
|
||||||
|
- **T77** `AddresseeDecision.confidence` typed as `Literal["high","medium","low"]`.
|
||||||
|
- **T78** `search_memories` docstring notes SQL-side significance bias (`SIGNIFICANCE_RANK_BIAS`).
|
||||||
|
- **T79** `_witness_role_for` defensive `host_bot_id is None` handling (returns `"host"` for Phase-1 chats).
|
||||||
|
- **Wave 2 — scene_summarize polish (single)**:
|
||||||
|
- **T80** five T58 follow-ups: re-close suffix bloat guard, transcript scoping by scene, swallowed-exception logging in `detect_threads`, chat-clock `closed_at`, and three new tests covering T58 gaps (200-char truncation, `thread_updated`/`thread_closed` candidate paths, try/except fallback).
|
||||||
|
- **Wave 3 — typed exception (single)**:
|
||||||
|
- **T81** `ChatNotFoundError` replaces string-prefix sniff in skip routes; mapped to 404 (vs 400 for other `ValueError` cases).
|
||||||
|
- **Wave 4 — turn-flow wiring (single)**:
|
||||||
|
- **T82** `consume_pending_meanwhile_digests` wired into `post_turn` (closes T66 gap; meanwhile digests no longer pile up); natural-language skip dispatch now runs scene close detection first.
|
||||||
|
- **Wave 5 — regenerate polish (single)**:
|
||||||
|
- **T83** five sub-fixes — cancel/stop hook (regenerate registers stream task in `_in_flight_tasks`); DRY extraction of `read_recent_dialogue` and `gather_prior_edges` into `chat/services/turn_common.py`; chat-scoped sibling-assistant-turn lookup; lifecycle-rollback warning log on regenerate; ordering-symmetry comment between post_turn and regenerate event-detection paths.
|
||||||
|
- **Wave 6 — final polish (parallel)**:
|
||||||
|
- **T84** unified `record_turn_memory` API with `you_present` kwarg; `record_meanwhile_memory` becomes a thin wrapper.
|
||||||
|
- **T85** JSON-build audit (no findings) + meanwhile cancel route-level test.
|
||||||
|
- **T86** frontend `turn_html_replace` SSE handler + turn_id stamping on rendered HTML so the in-place swap actually works.
|
||||||
|
|
||||||
- **`search_memories` docstring should mention SQL-side significance bias**: the function docstring still describes only the Python composite re-rank; add a one-line note about `SIGNIFICANCE_RANK_BIAS`.
|
### Phase 3.6 / 4 backlog
|
||||||
|
|
||||||
#### From T58 review
|
New follow-ups discovered during Phase 3.5 reviews and execution. None are blocking; pick up at any time.
|
||||||
|
|
||||||
- **Scene close re-close suffix bloat risk**: `_build_key_quotes_suffix` reads from `memories.pov_summary`. If a scene close runs twice, the second pass would read the rewritten text plus the previous "Key quotes:" suffix and append a second one. Either guard for double-suffix or source quotes from `event_log` `assistant_turn`/`user_turn` text instead.
|
#### From T80 review
|
||||||
- **Thread detection transcript scoping**: `_read_recent_dialogue` returns chat-wide history with no `scene_id` filter (Phase 1 turns lack one). Feeding chat-wide history to `detect_threads` will misattribute threads to the closing scene when the scene boundary falls inside the last 50 turns. Scope by `scene_id` once turns carry it, or by `started_at` against scene-open timestamp.
|
|
||||||
- **Swallowed exceptions in `detect_threads` try/except**: bare `Exception` swallows programmer errors silently. Log at debug level so silent regressions are recoverable.
|
|
||||||
- **Scene close `closed_at` clock divergence**: T58 uses `datetime.now(timezone.utc).isoformat()` instead of chat-clock time. Diverges from chat-clock semantics elsewhere; revisit if event reconstructions need chat-clock ordering.
|
|
||||||
- **Test coverage gaps in T58**: no test for 200-char quote truncation; no test for `thread_updated`/`thread_closed` candidate paths; no test for the `try/except` fallback.
|
|
||||||
|
|
||||||
#### From T61 review
|
- **`read_recent_dialogue` chat-id pushdown**: helper filters `chat_id` post-fetch in Python. Could push the `json_extract(payload_json, '$.chat_id') = ?` predicate into SQL (matching T83.3's pattern) for tighter LIMIT semantics. Currently a chat-with-many-other-chats can have its 50-row LIMIT consumed by foreign rows.
|
||||||
|
- **Lifecycle warning wording in regenerate**: T83.4's warning log lists ALL lifecycle event ids that exist after the original `assistant_turn` id, not just ones produced by the superseded turn. For the typical "regenerate the most recent" flow these are identical, but if a user regenerates an OLDER turn, the warning will list intervening-turn lifecycle events that legitimately stand. Tighten warning wording to "lifecycle transitions at-or-after turn X" (operator-friendly); a code-level fix would require a schema change to add explicit back-reference from lifecycle events to their producing turn.
|
||||||
|
|
||||||
- **Regenerate doesn't roll back lifecycle transitions from superseded turn**: `event_started`/`event_completed` rows from a superseded turn remain. Phase 3.5 should add a lifecycle-undo step. Caveat: regenerate-after-completion may double-emit promotion artifacts if the new text re-completes the same event.
|
#### From T84 review
|
||||||
- **Asymmetry in event-detection ordering**: post_turn runs lifecycle BETWEEN interjection and scene-close; regenerate runs lifecycle at the END. Benign because regenerate has no scene-close path, but worth tidying.
|
|
||||||
|
|
||||||
#### From T62 review
|
- **`record_turn_memory` legacy single-bot function** still exists alongside the unified `record_turn_memory_for_present`. Could be consolidated in a follow-up.
|
||||||
|
|
||||||
- **Error-message prefix sniff for 404 vs 400 routing**: drawer skip routes use `str(exc).startswith("chat not found")` to distinguish 404 from 400. Fragile if error wording changes. Use a typed exception subclass.
|
#### From T86 fix-up
|
||||||
- **Skip command bypasses scene close detection**: a user typing "fade out, skip an hour" would skip without closing the scene. Acceptable for Phase 3 but worth noting.
|
|
||||||
|
|
||||||
#### From T63 review
|
- **Test fixtures + `tests/test_phase3_integration.py`** that seed turns directly via `append_event`+`project` may need updating once any new test asserts the rendered HTML carries the new turn ids end-to-end. Existing tests pass because they don't read the stamped attribute, but they're brittle if the contract evolves.
|
||||||
|
|
||||||
- **`participants_json` JSON injection** (FIXED in T63 but worth noting in backlog as a "double-check other JSON-string-build sites" task): T63 originally used f-string interpolation; fixed to use `json.dumps`. Audit other state modules for similar patterns.
|
#### Deferred items (carry-over)
|
||||||
|
|
||||||
#### From T64 review
|
- **Scene-close-on-cancel UX revisit** (Phase 2.5 carry-over): T74.3 pinned the existing behavior; revisit if real play-testing surfaces a regression.
|
||||||
|
- **Cross-feature canned-queue brittleness**: meanwhile-scene close test required a canned response for T65's digest call after T64+T65 merge. Future close-path additions will keep extending the queue. Consider a structured fixture builder rather than positional canned arrays. NOT addressed in Phase 3.5.
|
||||||
|
- **Lifecycle-transition rollback in regenerate**: T83.4 added a warning log; actual rollback (with proper schema linkage from lifecycle event back to producing turn) is Phase 4 work.
|
||||||
|
|
||||||
- **`record_meanwhile_memory` and `record_turn_memory_for_present` share private `_write_one_memory` helper**: minor DRY note; both helpers are similar enough that a unified API with a `you_present: bool` kwarg might be cleaner long-term.
|
## Phase 4 status
|
||||||
- **Stop button cancellation for meanwhile turns**: T64 fix-up registered tasks in `_in_flight_tasks`; verify the `/turns/cancel` endpoint actually cancels meanwhile streams (the test pins registration but not the cancel-from-route path).
|
|
||||||
|
|
||||||
#### From cross-feature interactions discovered in Wave 6b merge
|
Phase 4 polish shipped end-to-end across 15 tasks (T88–T102). Vector retrieval is functional via pure-Python cosine over a JSON-blob embeddings table (sqlite-vec deferred — host Python lacks loadable extensions). Branching is data-model + drawer UI. Surgical delete with cascade preview, hide-from-view soft delete, significance review panel, snapshot UX, and cross-chat search all surface from the drawer or top-bar. Test count grew from 343 (Phase 3.5) to ~413 (+70 new tests).
|
||||||
|
|
||||||
- **Cross-feature canned-queue brittleness**: meanwhile-scene close test required a canned response for T65's digest call after T64+T65 merge. Future close-path additions will keep extending the queue; consider a structured fixture builder rather than positional canned arrays.
|
- **Wave 1 — schema + Phase 3.6 carry-overs (parallel)**:
|
||||||
|
- **T88** `embeddings` table + projector handlers (pure-Python cosine, JSON-blob storage; sqlite-vec deferred).
|
||||||
|
- **T89** `branches` table + handlers (main bootstrapped; `is_active` flag; partial unique index).
|
||||||
|
- **T90** Phase 3.6 carry-overs trio — `read_recent_dialogue` chat-id SQL pushdown, lifecycle warning wording tightening, legacy `record_turn_memory` removed.
|
||||||
|
- **Wave 2 — services (parallel)**:
|
||||||
|
- **T91** embedding generation service (Phase 4 ships a deterministic SHA-256-derived pseudo-embedding; real model swap is Phase 4.5+).
|
||||||
|
- **T92** vector search service via pure-Python cosine.
|
||||||
|
- **T93** cross-chat search service (FTS5 across all owners, no witness filter — admin-style).
|
||||||
|
- **Wave 3 — services (parallel)**:
|
||||||
|
- **T94** branching service (`branch_from_event`, `switch_active_branch`, `list_branches_with_metadata`).
|
||||||
|
- **T95** delete-impact computation service (cascade preview, no DB mutation).
|
||||||
|
- **Wave 4 — combined retrieval (single)**:
|
||||||
|
- **T96** combined FTS + vector retrieval ranking via reciprocal-rank fusion (RRF, `RRF_CONST=60`); existing significance/recency boost applied as final pass.
|
||||||
|
- **Wave 5 — memory write hook + backfill (single)**:
|
||||||
|
- **T97** `EmbeddingWorker` drains queue and emits `embedding_indexed` events; `memory_write` enqueues per `memory_written`; `backfill_embeddings` script for existing memories; ALL 4 production call sites wired (turns, regenerate, meanwhile, drawer).
|
||||||
|
- **Wave 6 — drawer Phase 4 bundle (single, 5 sub-features)**:
|
||||||
|
- **T98.1** branching UI (Branches panel + 3 routes).
|
||||||
|
- **T98.2** significance review panel (distribution bar chart + per-memory edit).
|
||||||
|
- **T98.3** hide-from-view toggle + `turn_hidden` `manual_edit` branch.
|
||||||
|
- **T98.4** surgical delete with cascade preview (reuses existing rewind path; pre-rewind snapshot preserved).
|
||||||
|
- **T98.5** remaining v1 edits — `narrative_anchor` + weather drawer affordances + 2 new `manual_edit` branches.
|
||||||
|
- **Wave 7 — UX surfaces (parallel)**:
|
||||||
|
- **T99** snapshot UX (manual trigger, list, restore with hard-confirm, preview).
|
||||||
|
- **T100** cross-chat search UX (top-bar form + results page).
|
||||||
|
- **Wave 8 — polish (parallel)**:
|
||||||
|
- **T101** cross-feature integration tests (5 multi-feature scenarios).
|
||||||
|
- **T102** documentation (this section).
|
||||||
|
|
||||||
#### From T66 integration tests
|
### Phase 4.5 / 5 backlog
|
||||||
|
|
||||||
- **`consume_pending_meanwhile_digests` is defined but NOT wired into `post_turn`**: the helper lives in `chat/services/prompt.py` (T65) but `chat/web/turns.py` never calls it. Meanwhile digests stay pending forever in production. Phase 3.5 should call the helper after the first you-turn following a meanwhile close — probably right after the assistant_turn lands but before the next prompt assembly. Pinned by `tests/test_phase3_integration.py::test_meanwhile_close_digest_surfaces_then_consumed` which currently calls the helper directly.
|
New follow-ups discovered during Phase 4 reviews and execution. None are blocking; pick up at any time.
|
||||||
|
|
||||||
#### Discovered during Phase 3 execution
|
#### From T88 review
|
||||||
|
|
||||||
- **`_witness_role_for` defensive `host_bot_id is None`** (carry-over from Phase 2.5 T71 backlog) — still pending.
|
- **`embeddings` FK lacks `ON DELETE CASCADE`**: deindex events are the only deletion path; if memories ever get deleted directly (raw SQL), embedding rows orphan. Defensible since projector model uses explicit deindex events, but worth a comment or `ON DELETE CASCADE` addition.
|
||||||
|
|
||||||
|
#### From T89 review
|
||||||
|
|
||||||
|
- **`list_branches(chat_id=...)` filter leaks global branches** (`chat_id IS NULL`) into every chat scope. Intentional? Document.
|
||||||
|
- **Branch-switch to nonexistent silently leaves zero active branches** — log a warning when this would happen.
|
||||||
|
|
||||||
|
#### From T91 review
|
||||||
|
|
||||||
|
- **Real embedding model swap**: Phase 4 ships pseudo-embedding (deterministic SHA-256 hash). Phase 4.5+ should swap to a real model (Featherless `bge-small-en-v1.5` if available; or local `sentence-transformers/all-MiniLM-L6-v2`). The 384-dim is hardcoded in `0012_embeddings.sql`; if dim changes, migrate first.
|
||||||
|
- **`timeout_s` unused on pseudo path** — fine, but log when non-default model falls through to fallback so misconfigured callers don't silently degrade.
|
||||||
|
|
||||||
|
#### From T96 review
|
||||||
|
|
||||||
|
- **Duplicate `MAX(id)` lookup** between `_composite_rerank` and the fused-path tail — DRY follow-up.
|
||||||
|
- **`fts_rank=None` for vector-only rows** — document downstream contract.
|
||||||
|
|
||||||
|
#### From T98 review
|
||||||
|
|
||||||
|
- **`event_id <= 0` guard in `delete_turn`** — currently silently rewinds everything if `event_id` is 0. Add `if event_id <= 0: 400`.
|
||||||
|
- **`html.escape()` on `compute_delete_impact` output rendered into the modal** — defense in depth (currently model-controlled strings, but if event payload fields ever appear in descriptions, autoescape needed).
|
||||||
|
- **Extract delete-impact modal HTML to a Jinja partial** — testability + autoescape inheritance.
|
||||||
|
|
||||||
|
#### From T99 review
|
||||||
|
|
||||||
|
- **Hoist `datetime`/`timezone` imports to module level** in `chat/web/snapshots.py`.
|
||||||
|
- **`kind` defaulting in restore/preview** — reject missing `kind` rather than silent 404.
|
||||||
|
- **`created_at` from file mtime** vs filename-encoded timestamp — small drift if files copied; document.
|
||||||
|
|
||||||
|
#### From T100 review
|
||||||
|
|
||||||
|
- **Hardcoded `k=50`** — extract to module constant.
|
||||||
|
- **N+1 lookups (`get_bot`/`get_chat`/`get_scene` per row)** — fine at `k=50`, revisit if `k` grows.
|
||||||
|
- **FTS highlighting via `snippet()`** — Phase 4 skipped this; UX nice-to-have.
|
||||||
|
- **Result links chat-level only** — `memories` table has no `event_id` column; deep-linking to specific turn requires schema addition.
|
||||||
|
|
||||||
|
#### Deferred items
|
||||||
|
|
||||||
|
- **sqlite-vec swap** when host Python supports `enable_load_extension`.
|
||||||
|
- **Real embedding model** with proper semantic similarity.
|
||||||
|
- **Branching read-side filter**: T89 ships data-model + UI but event readers don't yet consult `is_active`. Each branch is metadata-only labeled ranges. Consult-on-read is Phase 4.5+ work.
|
||||||
|
- **Bulk significance re-rate** in drawer (T98.2 deferred — only per-memory edit shipped).
|
||||||
|
- **Vector index optimization** (HNSW) — only relevant if memory counts grow past pure-Python feasibility.
|
||||||
|
- **`scene-close-on-cancel` UX revisit** (Phase 2.5 carry-over).
|
||||||
|
- **Cross-feature canned-queue brittleness fixture builder** (Phase 3 carry-over).
|
||||||
|
- **Full lifecycle-rollback in regenerate** — Phase 3.5 T83.4 shipped a warning log; proper rollback needs schema-level back-references (`triggered_by_assistant_turn_id` payload field).
|
||||||
|
|||||||
+19
@@ -16,6 +16,7 @@ from chat.db.migrate import apply_migrations
|
|||||||
from chat.eventlog.log import read_events
|
from chat.eventlog.log import read_events
|
||||||
from chat.eventlog.projector import apply_event
|
from chat.eventlog.projector import apply_event
|
||||||
from chat.services.background import BackgroundWorker
|
from chat.services.background import BackgroundWorker
|
||||||
|
from chat.services.embedding_worker import EmbeddingWorker
|
||||||
from chat.services.snapshot import latest_snapshot_path, restore_from_snapshot
|
from chat.services.snapshot import latest_snapshot_path, restore_from_snapshot
|
||||||
|
|
||||||
# Trigger handler registration:
|
# Trigger handler registration:
|
||||||
@@ -31,7 +32,9 @@ from chat.web.drawer import router as drawer_router
|
|||||||
from chat.web.kickoff import router as kickoff_router
|
from chat.web.kickoff import router as kickoff_router
|
||||||
from chat.web.middleware import FirstRunRedirectMiddleware
|
from chat.web.middleware import FirstRunRedirectMiddleware
|
||||||
from chat.web.nav import router as nav_router
|
from chat.web.nav import router as nav_router
|
||||||
|
from chat.web.search import router as search_router
|
||||||
from chat.web.settings import router as settings_router
|
from chat.web.settings import router as settings_router
|
||||||
|
from chat.web.snapshots import router as snapshots_router
|
||||||
from chat.web.sse import router as sse_router
|
from chat.web.sse import router as sse_router
|
||||||
from chat.web.turns import router as turns_router
|
from chat.web.turns import router as turns_router
|
||||||
|
|
||||||
@@ -85,9 +88,23 @@ async def lifespan(app: FastAPI):
|
|||||||
await worker.start()
|
await worker.start()
|
||||||
app.state.background_worker = worker
|
app.state.background_worker = worker
|
||||||
|
|
||||||
|
# T97: separate worker for the async embedding pass. Each
|
||||||
|
# ``memory_written`` enqueues an EmbeddingJob; the worker drains the
|
||||||
|
# queue, calls ``generate_embedding``, and emits ``embedding_indexed``.
|
||||||
|
# Phase 4's pseudo-embedding path is local so the worker doesn't need
|
||||||
|
# an LLM client; we still pass one so the Phase 4.5 swap to a real
|
||||||
|
# model is a one-line change.
|
||||||
|
embedding_worker = EmbeddingWorker(
|
||||||
|
conn_factory=lambda: open_db(settings.db_path),
|
||||||
|
client=_factory(),
|
||||||
|
)
|
||||||
|
await embedding_worker.start()
|
||||||
|
app.state.embedding_worker = embedding_worker
|
||||||
|
|
||||||
try:
|
try:
|
||||||
yield
|
yield
|
||||||
finally:
|
finally:
|
||||||
|
await embedding_worker.stop()
|
||||||
await worker.stop()
|
await worker.stop()
|
||||||
|
|
||||||
|
|
||||||
@@ -122,9 +139,11 @@ async def http_exception_handler(request: Request, exc: StarletteHTTPException):
|
|||||||
app.include_router(bots_router)
|
app.include_router(bots_router)
|
||||||
app.include_router(kickoff_router)
|
app.include_router(kickoff_router)
|
||||||
app.include_router(settings_router)
|
app.include_router(settings_router)
|
||||||
|
app.include_router(snapshots_router)
|
||||||
app.include_router(nav_router)
|
app.include_router(nav_router)
|
||||||
app.include_router(chat_router)
|
app.include_router(chat_router)
|
||||||
app.include_router(drawer_router)
|
app.include_router(drawer_router)
|
||||||
|
app.include_router(search_router)
|
||||||
app.include_router(sse_router)
|
app.include_router(sse_router)
|
||||||
app.include_router(turns_router)
|
app.include_router(turns_router)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
-- Embeddings stored as JSON arrays (pure-Python cosine at query time).
|
||||||
|
-- Phase 4.5+ may swap to sqlite-vec when the host Python supports
|
||||||
|
-- loadable extensions; the schema is intentionally simple to make that
|
||||||
|
-- migration straightforward.
|
||||||
|
CREATE TABLE embeddings (
|
||||||
|
memory_id INTEGER PRIMARY KEY,
|
||||||
|
vector_json TEXT NOT NULL, -- JSON array of floats, length = dim
|
||||||
|
model TEXT NOT NULL,
|
||||||
|
dim INTEGER NOT NULL,
|
||||||
|
indexed_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||||
|
FOREIGN KEY (memory_id) REFERENCES memories(id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX embeddings_model_idx ON embeddings(model);
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
CREATE TABLE branches (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL UNIQUE,
|
||||||
|
origin_event_id INTEGER NOT NULL,
|
||||||
|
head_event_id INTEGER NOT NULL,
|
||||||
|
chat_id TEXT,
|
||||||
|
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||||
|
is_active INTEGER NOT NULL DEFAULT 0
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Exactly one row may have is_active = 1 at any time.
|
||||||
|
CREATE UNIQUE INDEX branches_active_idx ON branches(is_active) WHERE is_active = 1;
|
||||||
|
|
||||||
|
-- Bootstrap the main branch. origin_event_id=0 + head_event_id=0 are
|
||||||
|
-- placeholder seeds; the orchestrator updates head as new events land.
|
||||||
|
INSERT INTO branches (name, origin_event_id, head_event_id, is_active)
|
||||||
|
VALUES ('main', 0, 0, 1);
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
-- 0014_phase45_schema.sql — Phase 4.5 Wave 2 schema bump (T109).
|
||||||
|
--
|
||||||
|
-- Two schema concerns are bundled into this migration:
|
||||||
|
--
|
||||||
|
-- 1. ``embeddings.memory_id`` FK should ideally carry ``ON DELETE CASCADE``
|
||||||
|
-- (T88 review nit). DEFERRED to Phase 5: ``embeddings`` rows are only ever
|
||||||
|
-- deleted when the parent ``memories`` row is deleted, and ``memories``
|
||||||
|
-- rows are never deleted today (memory hide is a soft flag; the surgical
|
||||||
|
-- ``deindex_event`` path operates on ``event_log`` and does NOT cascade
|
||||||
|
-- to projection rows). The CASCADE constraint therefore can't fire under
|
||||||
|
-- current usage — adding the SQLite table-rebuild dance (rename, recreate,
|
||||||
|
-- copy, drop, reindex) for a defensive constraint is unwarranted bloat
|
||||||
|
-- in a polish wave. Revisit during the broader Phase 5 migration cleanup
|
||||||
|
-- when other table reshapes make the rebuild worthwhile.
|
||||||
|
--
|
||||||
|
-- 2. Add ``memories.event_id`` (NULLABLE INTEGER, references ``event_log.id``)
|
||||||
|
-- so cross-chat search results can deep-link back to the originating
|
||||||
|
-- turn (foundation for T111). The column is nullable so historical
|
||||||
|
-- memory rows projected before 0014 ran continue to round-trip cleanly;
|
||||||
|
-- new rows are populated by the ``memory_written`` projector handler
|
||||||
|
-- from the projecting event's id. This is a pure additive change — no
|
||||||
|
-- backfill is performed. Older rows simply read NULL until/unless a
|
||||||
|
-- later migration backfills them; T111 surfaces are coded to accept
|
||||||
|
-- NULL gracefully (no deep-link rendered).
|
||||||
|
ALTER TABLE memories ADD COLUMN event_id INTEGER REFERENCES event_log(id);
|
||||||
@@ -22,6 +22,8 @@ from a fallback.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
|
|
||||||
from chat.llm.classify import classify
|
from chat.llm.classify import classify
|
||||||
@@ -39,7 +41,7 @@ class AddresseeDecision(BaseModel):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
addressee_id: str
|
addressee_id: str
|
||||||
confidence: str = "medium" # "high" | "medium" | "low"
|
confidence: Literal["high", "medium", "low"] = "medium"
|
||||||
reason: str = ""
|
reason: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,107 @@
|
|||||||
|
"""Branching service (T94, Phase 4).
|
||||||
|
|
||||||
|
Wraps branches state with validation + event emission. Phase 4 ships
|
||||||
|
the data model and creation/switching APIs; the read-side filter
|
||||||
|
(event readers consulting is_active) is a Phase 4.5+ follow-up — for
|
||||||
|
now branches are metadata-only and the existing event readers remain
|
||||||
|
branch-agnostic. The drawer UI (T98) drives create/switch via these
|
||||||
|
helpers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
from chat.state.branches import get_branch, list_branches, active_branch # noqa: F401
|
||||||
|
|
||||||
|
|
||||||
|
def branch_from_event(
|
||||||
|
conn: Connection,
|
||||||
|
*,
|
||||||
|
name: str,
|
||||||
|
origin_event_id: int,
|
||||||
|
chat_id: str | None = None,
|
||||||
|
) -> int:
|
||||||
|
"""Create a new named branch forking from origin_event_id.
|
||||||
|
|
||||||
|
Emits a branch_created event. Returns the new branch's row id.
|
||||||
|
Raises ValueError if name already exists or origin_event_id doesn't
|
||||||
|
correspond to a real event."""
|
||||||
|
if not name or not name.strip():
|
||||||
|
raise ValueError("branch name must be non-empty")
|
||||||
|
|
||||||
|
if get_branch(conn, name) is not None:
|
||||||
|
raise ValueError(f"branch {name!r} already exists")
|
||||||
|
|
||||||
|
# Validate origin_event_id is a real event id (or 0 for the bootstrap case
|
||||||
|
# which only main uses).
|
||||||
|
if origin_event_id < 0:
|
||||||
|
raise ValueError(f"origin_event_id must be >= 0, got {origin_event_id}")
|
||||||
|
if origin_event_id > 0:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT 1 FROM event_log WHERE id = ?", (origin_event_id,)
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"origin_event_id {origin_event_id} does not exist in event_log"
|
||||||
|
)
|
||||||
|
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="branch_created",
|
||||||
|
payload={
|
||||||
|
"name": name,
|
||||||
|
"origin_event_id": origin_event_id,
|
||||||
|
"head_event_id": origin_event_id, # head starts at origin
|
||||||
|
"chat_id": chat_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
branch = get_branch(conn, name)
|
||||||
|
if branch is None:
|
||||||
|
# Should be unreachable if append_and_apply worked.
|
||||||
|
raise RuntimeError(f"branch {name!r} not found after creation")
|
||||||
|
return branch["id"]
|
||||||
|
|
||||||
|
|
||||||
|
def switch_active_branch(conn: Connection, *, name: str) -> None:
|
||||||
|
"""Make the named branch active. Emits branch_switched."""
|
||||||
|
if get_branch(conn, name) is None:
|
||||||
|
raise ValueError(f"branch {name!r} does not exist")
|
||||||
|
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="branch_switched",
|
||||||
|
payload={"name": name},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def list_branches_with_metadata(
|
||||||
|
conn: Connection, chat_id: str | None = None
|
||||||
|
) -> list[dict]:
|
||||||
|
"""List branches with computed event_count metadata.
|
||||||
|
|
||||||
|
event_count = head_event_id - origin_event_id + 1 (when both are set)
|
||||||
|
OR head_event_id (when origin is 0, e.g., main branch)
|
||||||
|
OR 0 (when head <= origin, which is the bootstrap state)
|
||||||
|
"""
|
||||||
|
branches = list_branches(conn, chat_id)
|
||||||
|
enriched = []
|
||||||
|
for b in branches:
|
||||||
|
origin = b["origin_event_id"]
|
||||||
|
head = b["head_event_id"]
|
||||||
|
if head < origin:
|
||||||
|
event_count = 0
|
||||||
|
elif origin == 0:
|
||||||
|
event_count = head
|
||||||
|
else:
|
||||||
|
event_count = head - origin + 1
|
||||||
|
enriched.append({**b, "event_count": event_count})
|
||||||
|
return enriched
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"branch_from_event",
|
||||||
|
"switch_active_branch",
|
||||||
|
"list_branches_with_metadata",
|
||||||
|
]
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
"""Cross-chat search service (T93, Phase 4).
|
||||||
|
|
||||||
|
FTS5-based search across ALL owners and ALL chats. Used by the
|
||||||
|
top-bar search UX (T100) for "where did I last see this character
|
||||||
|
mention X?" queries. NO witness filter -- this is intentionally a
|
||||||
|
power-user surface that surfaces memories across POVs.
|
||||||
|
|
||||||
|
Mirrors the FTS5 access pattern of ``chat.state.memory.search_memories``
|
||||||
|
but drops both the ``owner_id = ?`` and the per-witness predicates so a
|
||||||
|
single query can sweep every chat in the database. The composite
|
||||||
|
re-rank is also dropped: callers want raw BM25 ordering for the
|
||||||
|
"highest match strength wins" semantics expected of a global search box.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
|
||||||
|
def search_all_memories(
|
||||||
|
conn: Connection,
|
||||||
|
*,
|
||||||
|
query: str,
|
||||||
|
k: int = 20,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Search FTS5 across all owners and chats.
|
||||||
|
|
||||||
|
Returns rows with ``{memory_id, owner_id, chat_id, scene_id,
|
||||||
|
pov_summary, significance, ts, fts_rank}``, sorted by FTS5 BM25
|
||||||
|
rank ascending (lower rank = stronger match, surfaced first).
|
||||||
|
|
||||||
|
The ``memories`` table has no ``ts`` column; we expose ``created_at``
|
||||||
|
(the projector-side row insertion timestamp) under that key so the
|
||||||
|
UI does not have to know the storage name.
|
||||||
|
|
||||||
|
An empty / whitespace-only ``query`` short-circuits to ``[]`` to
|
||||||
|
avoid an FTS5 ``MATCH ''`` syntax error and to keep the top-bar
|
||||||
|
"no input yet" state from triggering a full-table scan.
|
||||||
|
"""
|
||||||
|
if not query or not query.strip():
|
||||||
|
return []
|
||||||
|
|
||||||
|
# FTS5 MATCH against the same ``memories_fts`` virtual table that
|
||||||
|
# backs ``state.memory.search_memories``; the JOIN pulls metadata
|
||||||
|
# from the content table because the FTS index only stores
|
||||||
|
# ``pov_summary``. ORDER BY rank ASC because BM25 in FTS5 returns
|
||||||
|
# negative scores where lower is better.
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT m.id, m.owner_id, m.chat_id, m.scene_id, "
|
||||||
|
" m.pov_summary, m.significance, m.created_at, "
|
||||||
|
" memories_fts.rank "
|
||||||
|
"FROM memories_fts "
|
||||||
|
"JOIN memories m ON m.id = memories_fts.rowid "
|
||||||
|
"WHERE memories_fts MATCH ? "
|
||||||
|
"ORDER BY memories_fts.rank ASC "
|
||||||
|
"LIMIT ?",
|
||||||
|
(query.strip(), k),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"memory_id": r[0],
|
||||||
|
"owner_id": r[1],
|
||||||
|
"chat_id": r[2],
|
||||||
|
"scene_id": r[3],
|
||||||
|
"pov_summary": r[4],
|
||||||
|
"significance": r[5],
|
||||||
|
"ts": r[6],
|
||||||
|
"fts_rank": r[7],
|
||||||
|
}
|
||||||
|
for r in rows
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["search_all_memories"]
|
||||||
@@ -0,0 +1,147 @@
|
|||||||
|
"""Delete-impact computation service (T95, Phase 4).
|
||||||
|
|
||||||
|
Walks event_log forward from a target event_id and produces an ImpactReport
|
||||||
|
describing what would be removed if rewind-to-target were invoked. Pure
|
||||||
|
computation — does NOT mutate the database. Used by T98's drawer surgical-
|
||||||
|
delete UI to render an 'are you sure?' modal before invoking the actual
|
||||||
|
rewind path (chat/services/rewind.py).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import json
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
|
||||||
|
class DeletedItem(BaseModel):
|
||||||
|
kind: str
|
||||||
|
description: str
|
||||||
|
target_id: int | str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
class ImpactReport(BaseModel):
|
||||||
|
target_event_id: int
|
||||||
|
cascading: list[DeletedItem] = Field(default_factory=list)
|
||||||
|
notes: list[str] = Field(default_factory=list)
|
||||||
|
|
||||||
|
|
||||||
|
def _excerpt(text: str, n: int = 60) -> str:
|
||||||
|
text = (text or "").strip().replace("\n", " ")
|
||||||
|
return text if len(text) <= n else text[: n - 1] + "…"
|
||||||
|
|
||||||
|
|
||||||
|
def compute_delete_impact(
|
||||||
|
conn: Connection,
|
||||||
|
*,
|
||||||
|
target_event_id: int,
|
||||||
|
) -> ImpactReport:
|
||||||
|
"""Compute the cascading impact of rewinding to target_event_id."""
|
||||||
|
# Verify target exists.
|
||||||
|
target_row = conn.execute(
|
||||||
|
"SELECT id, kind, payload_json FROM event_log WHERE id = ?",
|
||||||
|
(target_event_id,),
|
||||||
|
).fetchone()
|
||||||
|
if target_row is None:
|
||||||
|
return ImpactReport(
|
||||||
|
target_event_id=target_event_id,
|
||||||
|
cascading=[],
|
||||||
|
notes=[f"target event_id {target_event_id} not found"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Walk forward: every event with id >= target_event_id is in scope.
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, kind, payload_json FROM event_log "
|
||||||
|
"WHERE id >= ? ORDER BY id ASC",
|
||||||
|
(target_event_id,),
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
cascading: list[DeletedItem] = []
|
||||||
|
notes: list[str] = []
|
||||||
|
scene_close_present = False
|
||||||
|
regenerated_from = None
|
||||||
|
|
||||||
|
for row_id, kind, payload_json in rows:
|
||||||
|
try:
|
||||||
|
payload = json.loads(payload_json) if payload_json else {}
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
payload = {}
|
||||||
|
|
||||||
|
if kind == "memory_written":
|
||||||
|
cascading.append(
|
||||||
|
DeletedItem(
|
||||||
|
kind=kind,
|
||||||
|
description=f"memory: {_excerpt(payload.get('pov_summary', ''))}",
|
||||||
|
target_id=payload.get("memory_id"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif kind == "edge_update":
|
||||||
|
src = payload.get("source_id", "?")
|
||||||
|
tgt = payload.get("target_id", "?")
|
||||||
|
cascading.append(
|
||||||
|
DeletedItem(
|
||||||
|
kind=kind,
|
||||||
|
description=f"edge update: {src} -> {tgt}",
|
||||||
|
target_id=f"{src}->{tgt}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif kind == "scene_closed":
|
||||||
|
scene_close_present = True
|
||||||
|
cascading.append(
|
||||||
|
DeletedItem(
|
||||||
|
kind=kind,
|
||||||
|
description=f"scene close at {payload.get('closed_at', '?')}",
|
||||||
|
target_id=payload.get("scene_id"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif kind in ("user_turn", "user_turn_edit", "assistant_turn"):
|
||||||
|
speaker = payload.get("speaker_id") or ("you" if kind.startswith("user") else "?")
|
||||||
|
prose = payload.get("prose") or payload.get("text") or ""
|
||||||
|
cascading.append(
|
||||||
|
DeletedItem(
|
||||||
|
kind=kind,
|
||||||
|
description=f"turn {row_id} ({speaker}: {_excerpt(prose, 50)})",
|
||||||
|
target_id=row_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if regenerated_from is None and payload.get("regenerated_from"):
|
||||||
|
regenerated_from = payload["regenerated_from"]
|
||||||
|
elif kind == "manual_edit":
|
||||||
|
target_kind = payload.get("target_kind", "?")
|
||||||
|
cascading.append(
|
||||||
|
DeletedItem(
|
||||||
|
kind=kind,
|
||||||
|
description=f"manual edit: {target_kind}",
|
||||||
|
target_id=payload.get("target_id"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cascading.append(
|
||||||
|
DeletedItem(
|
||||||
|
kind=kind,
|
||||||
|
description=f"{kind} event",
|
||||||
|
target_id=row_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Notes / warnings.
|
||||||
|
notes.append(f"{len(rows)} events would be discarded total")
|
||||||
|
if scene_close_present:
|
||||||
|
notes.append(
|
||||||
|
"scene close events are in scope — closing-scene per-POV summaries "
|
||||||
|
"and group_node updates will be reverted"
|
||||||
|
)
|
||||||
|
if regenerated_from is not None:
|
||||||
|
notes.append(
|
||||||
|
f"target turn was regenerated from event_id {regenerated_from}; "
|
||||||
|
f"the original turn remains intact"
|
||||||
|
)
|
||||||
|
|
||||||
|
return ImpactReport(
|
||||||
|
target_event_id=target_event_id,
|
||||||
|
cascading=cascading,
|
||||||
|
notes=notes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["DeletedItem", "ImpactReport", "compute_delete_impact"]
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
"""Embedding worker (T97, Phase 4).
|
||||||
|
|
||||||
|
Drains a queue of embedding jobs. Each job carries a memory id and the
|
||||||
|
narrative text to embed; the worker calls
|
||||||
|
:func:`chat.services.embeddings.generate_embedding` and emits an
|
||||||
|
``embedding_indexed`` event so the projector lands the vector in the
|
||||||
|
``embeddings`` table.
|
||||||
|
|
||||||
|
Mirrors the :class:`chat.services.background.BackgroundWorker` pattern:
|
||||||
|
single asyncio task, sentinel-based shutdown, exceptions are caught and
|
||||||
|
logged so a flaky embedding call doesn't take down the worker. Each job
|
||||||
|
opens its own SQLite connection via ``conn_factory`` — the request path
|
||||||
|
and the worker do not share connections.
|
||||||
|
|
||||||
|
Featherless concurrency (the 2-conn cap) is respected by virtue of the
|
||||||
|
single-task design: jobs run strictly serially. Phase 4's pseudo-embedding
|
||||||
|
path is local and synchronous so this is largely moot, but the pattern
|
||||||
|
is in place for the Phase 4.5+ real-embedding swap.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from sqlite3 import Connection
|
||||||
|
from typing import Callable
|
||||||
|
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
from chat.services.embeddings import (
|
||||||
|
DEFAULT_EMBEDDING_DIM,
|
||||||
|
DEFAULT_EMBEDDING_MODEL,
|
||||||
|
FALLBACK_EMBEDDING_MODEL,
|
||||||
|
generate_embedding,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class EmbeddingJob:
|
||||||
|
"""One unit of work for the embedding worker.
|
||||||
|
|
||||||
|
``memory_id`` is the row to attach the vector to; ``text`` is the
|
||||||
|
narrative text to embed (typically ``memories.pov_summary``).
|
||||||
|
"""
|
||||||
|
|
||||||
|
memory_id: int
|
||||||
|
text: str
|
||||||
|
|
||||||
|
|
||||||
|
class EmbeddingWorker:
|
||||||
|
"""asyncio.Queue-backed single-worker task for embedding generation.
|
||||||
|
|
||||||
|
Started on app startup; ``stop()`` enqueues a sentinel and awaits
|
||||||
|
the task so any in-flight job has a chance to finish. Pending jobs
|
||||||
|
after the sentinel are dropped on shutdown.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
conn_factory: Callable[[], Connection],
|
||||||
|
client, # LLMClient | None — unused on the pseudo path.
|
||||||
|
model: str = DEFAULT_EMBEDDING_MODEL,
|
||||||
|
dim: int = DEFAULT_EMBEDDING_DIM,
|
||||||
|
enabled: bool = True,
|
||||||
|
) -> None:
|
||||||
|
self._queue: asyncio.Queue[EmbeddingJob | None] = asyncio.Queue()
|
||||||
|
self._conn_factory = conn_factory
|
||||||
|
self._client = client
|
||||||
|
self._model = model
|
||||||
|
self._dim = dim
|
||||||
|
self._task: asyncio.Task | None = None
|
||||||
|
self.enabled = enabled
|
||||||
|
|
||||||
|
def enqueue(self, job: EmbeddingJob) -> None:
|
||||||
|
if not self.enabled:
|
||||||
|
return
|
||||||
|
self._queue.put_nowait(job)
|
||||||
|
|
||||||
|
async def start(self) -> None:
|
||||||
|
if self._task is None:
|
||||||
|
self._task = asyncio.create_task(self._run())
|
||||||
|
|
||||||
|
async def stop(self) -> None:
|
||||||
|
if self._task is None:
|
||||||
|
return
|
||||||
|
self._queue.put_nowait(None) # sentinel
|
||||||
|
await self._task
|
||||||
|
self._task = None
|
||||||
|
|
||||||
|
async def _run(self) -> None:
|
||||||
|
while True:
|
||||||
|
job = await self._queue.get()
|
||||||
|
if job is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
await self._process(job)
|
||||||
|
except Exception as exc: # noqa: BLE001 — worker must not die
|
||||||
|
log.warning(
|
||||||
|
"embedding worker failed for memory_id=%s: %s",
|
||||||
|
job.memory_id,
|
||||||
|
exc,
|
||||||
|
exc_info=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _process(self, job: EmbeddingJob) -> None:
|
||||||
|
result = await generate_embedding(
|
||||||
|
self._client,
|
||||||
|
text=job.text,
|
||||||
|
model=self._model,
|
||||||
|
dim=self._dim,
|
||||||
|
)
|
||||||
|
if result.model == FALLBACK_EMBEDDING_MODEL:
|
||||||
|
# Don't index a fallback (zero) vector — the backfill script
|
||||||
|
# can retry later once a real embedding is available.
|
||||||
|
log.debug(
|
||||||
|
"embedding worker skipping fallback result for memory_id=%s",
|
||||||
|
job.memory_id,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
with self._conn_factory() as conn:
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": job.memory_id,
|
||||||
|
"model": result.model,
|
||||||
|
"dim": result.dim,
|
||||||
|
"vector": result.vector,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["EmbeddingJob", "EmbeddingWorker"]
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
"""Embedding generation service (T91, Phase 4).
|
||||||
|
|
||||||
|
Wraps the embedding API call. For Phase 4's first cut we ship a
|
||||||
|
deterministic local pseudo-embedding (hash-derived) so the vector
|
||||||
|
retrieval pipeline can land without an external embedding endpoint
|
||||||
|
or heavy local dependency. Phase 4.5+ swaps to a real model — the
|
||||||
|
EmbeddingResult shape stays the same, only the generator changes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import logging
|
||||||
|
import math
|
||||||
|
import struct
|
||||||
|
|
||||||
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
from chat.llm.client import LLMClient
|
||||||
|
|
||||||
|
|
||||||
|
_log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
DEFAULT_EMBEDDING_DIM = 384
|
||||||
|
DEFAULT_EMBEDDING_MODEL = "pseudo-sha256-384"
|
||||||
|
FALLBACK_EMBEDDING_MODEL = "fallback"
|
||||||
|
|
||||||
|
|
||||||
|
class EmbeddingResult(BaseModel):
|
||||||
|
vector: list[float]
|
||||||
|
model: str
|
||||||
|
dim: int
|
||||||
|
|
||||||
|
|
||||||
|
def _pseudo_embed(text: str, dim: int = DEFAULT_EMBEDDING_DIM) -> list[float]:
|
||||||
|
"""Deterministic pseudo-embedding for Phase 4 first cut.
|
||||||
|
|
||||||
|
Hashes the text with SHA-256, then expands by re-hashing each
|
||||||
|
successive block with the previous block + a counter — this gives
|
||||||
|
``dim * 4`` bytes of fresh entropy per input rather than naively
|
||||||
|
repeating the 32-byte digest (which would collapse the vector onto
|
||||||
|
only 8 unique floats and make distinct inputs cosine-similar).
|
||||||
|
|
||||||
|
Bytes are unpacked as little-endian int32s and rescaled to [-1, 1]
|
||||||
|
so we sidestep the float32 NaN/denormal values that ``struct.unpack
|
||||||
|
'f'`` would otherwise produce on raw hash bytes. The result is
|
||||||
|
unit-normalized so cosine similarity reduces to a dot product.
|
||||||
|
|
||||||
|
NOT semantically meaningful — just consistent for testing the
|
||||||
|
pipeline. Phase 4.5 should swap to a real embedding model.
|
||||||
|
"""
|
||||||
|
needed = dim * 4 # 4 bytes per int32
|
||||||
|
seed = text.encode("utf-8")
|
||||||
|
chunks: list[bytes] = []
|
||||||
|
counter = 0
|
||||||
|
while sum(len(c) for c in chunks) < needed:
|
||||||
|
block = hashlib.sha256(seed + counter.to_bytes(4, "big")).digest()
|
||||||
|
chunks.append(block)
|
||||||
|
counter += 1
|
||||||
|
full = b"".join(chunks)[:needed]
|
||||||
|
ints = struct.unpack(f"<{dim}i", full)
|
||||||
|
# Map int32 to roughly [-1, 1] — exact bound doesn't matter since we
|
||||||
|
# normalize, but keeps values numerically tame.
|
||||||
|
raw = [x / 2147483648.0 for x in ints]
|
||||||
|
norm = math.sqrt(sum(x * x for x in raw)) or 1.0
|
||||||
|
return [x / norm for x in raw]
|
||||||
|
|
||||||
|
|
||||||
|
async def generate_embedding(
|
||||||
|
client: LLMClient,
|
||||||
|
*,
|
||||||
|
text: str,
|
||||||
|
model: str = DEFAULT_EMBEDDING_MODEL,
|
||||||
|
dim: int = DEFAULT_EMBEDDING_DIM,
|
||||||
|
timeout_s: float = 30.0,
|
||||||
|
) -> EmbeddingResult:
|
||||||
|
"""Generate an embedding for the given text.
|
||||||
|
|
||||||
|
Phase 4 default uses a deterministic local pseudo-embedding. If
|
||||||
|
the LLMClient grows an ``embed(...)`` method in Phase 4.5, this
|
||||||
|
wrapper will route to it when ``model != "pseudo-sha256-384"``.
|
||||||
|
|
||||||
|
Falls back to a zero vector with ``model="fallback"`` on any
|
||||||
|
failure (callers detect the sentinel and skip indexing). For the
|
||||||
|
pseudo path, failure is structurally impossible — it's pure local
|
||||||
|
computation.
|
||||||
|
"""
|
||||||
|
if not text or not text.strip():
|
||||||
|
# Empty input — return fallback so caller doesn't index empty rows.
|
||||||
|
return EmbeddingResult(
|
||||||
|
vector=[0.0] * dim, model=FALLBACK_EMBEDDING_MODEL, dim=dim
|
||||||
|
)
|
||||||
|
|
||||||
|
if model == DEFAULT_EMBEDDING_MODEL:
|
||||||
|
# Pure-local pseudo path — no LLMClient call.
|
||||||
|
return EmbeddingResult(vector=_pseudo_embed(text, dim), model=model, dim=dim)
|
||||||
|
|
||||||
|
# Future: real embedding via client.embed(...). Phase 4.5 work.
|
||||||
|
# For Phase 4, any non-default model falls through to fallback —
|
||||||
|
# warn so misconfigured callers (e.g., a real-model swap that isn't
|
||||||
|
# wired up yet) don't silently degrade to a zero vector.
|
||||||
|
_log.warning(
|
||||||
|
"generate_embedding: non-default model %r returned fallback "
|
||||||
|
"(model client.embed() not yet implemented in Phase 4.5+); "
|
||||||
|
"downstream search will degrade silently. Configure a supported model.",
|
||||||
|
model,
|
||||||
|
)
|
||||||
|
return EmbeddingResult(
|
||||||
|
vector=[0.0] * dim, model=FALLBACK_EMBEDDING_MODEL, dim=dim
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"DEFAULT_EMBEDDING_DIM",
|
||||||
|
"DEFAULT_EMBEDDING_MODEL",
|
||||||
|
"FALLBACK_EMBEDDING_MODEL",
|
||||||
|
"EmbeddingResult",
|
||||||
|
"generate_embedding",
|
||||||
|
]
|
||||||
@@ -13,6 +13,14 @@ Phase 1 simplifications (per plan §11.1, T27 will refine):
|
|||||||
pass overwrites via a follow-up event.
|
pass overwrites via a follow-up event.
|
||||||
- Witness flags are hard-coded ``[you=1, host=1, guest=0]``. Phase 2 will
|
- Witness flags are hard-coded ``[you=1, host=1, guest=0]``. Phase 2 will
|
||||||
derive them from ``chat.guest_bot_id`` once a guest can be present.
|
derive them from ``chat.guest_bot_id`` once a guest can be present.
|
||||||
|
|
||||||
|
T97 (Phase 4): each successful memory write also enqueues an
|
||||||
|
:class:`~chat.services.embedding_worker.EmbeddingJob` on the
|
||||||
|
lifespan-managed embedding worker, so the just-written memory gets a
|
||||||
|
vector indexed out-of-band. The hook is opt-in via the ``app`` kwarg —
|
||||||
|
callers without a FastAPI app handle (e.g. one-off scripts, isolated
|
||||||
|
unit tests) simply don't enqueue, and the backfill script can pick up
|
||||||
|
those rows later.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -20,62 +28,7 @@ from __future__ import annotations
|
|||||||
from sqlite3 import Connection
|
from sqlite3 import Connection
|
||||||
|
|
||||||
from chat.eventlog.log import append_and_apply
|
from chat.eventlog.log import append_and_apply
|
||||||
|
from chat.services.embedding_worker import EmbeddingJob
|
||||||
|
|
||||||
def record_turn_memory(
|
|
||||||
conn: Connection,
|
|
||||||
*,
|
|
||||||
chat_id: str,
|
|
||||||
host_bot_id: str,
|
|
||||||
narrative_text: str,
|
|
||||||
scene_id: int | None = None,
|
|
||||||
chat_clock_at: str | None = None,
|
|
||||||
source: str = "direct",
|
|
||||||
significance: int = 1,
|
|
||||||
) -> tuple[int, int | None]:
|
|
||||||
"""Append a ``memory_written`` event for the host bot's POV of this turn.
|
|
||||||
|
|
||||||
Uses :func:`chat.eventlog.log.append_and_apply` (not raw
|
|
||||||
:func:`append_event`) so the new memory row is projected immediately
|
|
||||||
without re-running prior non-idempotent handlers (e.g. ``edge_update``
|
|
||||||
deltas).
|
|
||||||
|
|
||||||
Returns ``(event_id, memory_id)``. ``event_id`` is the row id of the
|
|
||||||
just-appended ``memory_written`` event in ``event_log``. ``memory_id``
|
|
||||||
is the autoincrement PK of the corresponding ``memories`` row — these
|
|
||||||
are *different* numbers (event_log and memories use independent
|
|
||||||
rowid sequences) so callers needing to update significance or pin
|
|
||||||
state must use ``memory_id``. Falls back to ``None`` if the projected
|
|
||||||
row can't be located, which shouldn't happen but keeps the return
|
|
||||||
shape stable.
|
|
||||||
"""
|
|
||||||
payload: dict = {
|
|
||||||
"owner_id": host_bot_id,
|
|
||||||
"chat_id": chat_id,
|
|
||||||
"pov_summary": narrative_text,
|
|
||||||
"witness_you": 1,
|
|
||||||
"witness_host": 1,
|
|
||||||
"witness_guest": 0,
|
|
||||||
"source": source,
|
|
||||||
"reliability": 1.0,
|
|
||||||
"significance": significance,
|
|
||||||
"pinned": 0,
|
|
||||||
"auto_pinned": 0,
|
|
||||||
}
|
|
||||||
if scene_id is not None:
|
|
||||||
payload["scene_id"] = scene_id
|
|
||||||
if chat_clock_at is not None:
|
|
||||||
payload["chat_clock_at"] = chat_clock_at
|
|
||||||
|
|
||||||
event_id = append_and_apply(conn, kind="memory_written", payload=payload)
|
|
||||||
row = conn.execute(
|
|
||||||
"SELECT id FROM memories "
|
|
||||||
"WHERE owner_id = ? AND chat_id = ? "
|
|
||||||
"ORDER BY id DESC LIMIT 1",
|
|
||||||
(host_bot_id, chat_id),
|
|
||||||
).fetchone()
|
|
||||||
memory_id = row[0] if row else None
|
|
||||||
return event_id, memory_id
|
|
||||||
|
|
||||||
|
|
||||||
def _write_one_memory(
|
def _write_one_memory(
|
||||||
@@ -91,9 +44,16 @@ def _write_one_memory(
|
|||||||
chat_clock_at: str | None,
|
chat_clock_at: str | None,
|
||||||
source: str,
|
source: str,
|
||||||
significance: int,
|
significance: int,
|
||||||
|
app=None,
|
||||||
) -> tuple[int, int | None]:
|
) -> tuple[int, int | None]:
|
||||||
"""Append a single ``memory_written`` event for ``owner_id`` and return
|
"""Append a single ``memory_written`` event for ``owner_id`` and return
|
||||||
``(event_id, memory_id)`` for the projected row."""
|
``(event_id, memory_id)`` for the projected row.
|
||||||
|
|
||||||
|
When ``app`` is provided and ``app.state.embedding_worker`` exists,
|
||||||
|
enqueue an :class:`EmbeddingJob` for the freshly-projected memory id
|
||||||
|
(T97). Skipped silently if the worker is absent or the projected row
|
||||||
|
can't be located — the backfill script handles missing-vector rows.
|
||||||
|
"""
|
||||||
payload: dict = {
|
payload: dict = {
|
||||||
"owner_id": owner_id,
|
"owner_id": owner_id,
|
||||||
"chat_id": chat_id,
|
"chat_id": chat_id,
|
||||||
@@ -120,6 +80,23 @@ def _write_one_memory(
|
|||||||
(owner_id, chat_id),
|
(owner_id, chat_id),
|
||||||
).fetchone()
|
).fetchone()
|
||||||
memory_id = row[0] if row else None
|
memory_id = row[0] if row else None
|
||||||
|
|
||||||
|
# T97: enqueue an embedding job for the just-written memory. The
|
||||||
|
# worker drains the queue out-of-band and emits an
|
||||||
|
# ``embedding_indexed`` event when the vector is ready. ``getattr``
|
||||||
|
# keeps this a no-op for callers without a wired-up app (scripts,
|
||||||
|
# tests) — the backfill script handles those rows.
|
||||||
|
if memory_id is not None and narrative_text and narrative_text.strip():
|
||||||
|
worker = (
|
||||||
|
getattr(app.state, "embedding_worker", None)
|
||||||
|
if app is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
if worker is not None:
|
||||||
|
worker.enqueue(
|
||||||
|
EmbeddingJob(memory_id=memory_id, text=narrative_text)
|
||||||
|
)
|
||||||
|
|
||||||
return event_id, memory_id
|
return event_id, memory_id
|
||||||
|
|
||||||
|
|
||||||
@@ -134,17 +111,38 @@ def record_turn_memory_for_present(
|
|||||||
chat_clock_at: str | None = None,
|
chat_clock_at: str | None = None,
|
||||||
source: str = "direct",
|
source: str = "direct",
|
||||||
significance: int = 1,
|
significance: int = 1,
|
||||||
|
you_present: bool = True,
|
||||||
|
app=None,
|
||||||
) -> dict[str, tuple[int, int | None]]:
|
) -> dict[str, tuple[int, int | None]]:
|
||||||
"""Write a ``memory_written`` event for each present bot witness.
|
"""Single entry-point for per-turn memory writes (T84).
|
||||||
|
|
||||||
Host is always written. Guest is written iff ``guest_bot_id is not
|
Writes one ``memory_written`` event per present bot witness. Host is
|
||||||
None``. Witness flags are ``[you=1, host=1, guest=1]`` when a guest
|
always written. Guest is written iff ``guest_bot_id is not None``.
|
||||||
is present, ``[you=1, host=1, guest=0]`` otherwise.
|
|
||||||
|
Witness flags depend on ``you_present``:
|
||||||
|
|
||||||
|
- ``you_present=True`` (default — Phase 1/2/3 you-scenes): the user
|
||||||
|
is a witness. Mask is ``[you=1, host=1, guest=1]`` when a guest is
|
||||||
|
present, ``[you=1, host=1, guest=0]`` otherwise.
|
||||||
|
- ``you_present=False`` (Phase 3 meanwhile scenes): the user is
|
||||||
|
absent. Mask is ``[you=0, host=1, guest=1]`` for both bots. Both
|
||||||
|
``host_bot_id`` and ``guest_bot_id`` are required — a meanwhile
|
||||||
|
scene by definition has both bots, so passing ``guest_bot_id=None``
|
||||||
|
with ``you_present=False`` is a programming error and raises
|
||||||
|
:class:`ValueError`.
|
||||||
|
|
||||||
|
When ``app`` is provided, each per-witness write also enqueues an
|
||||||
|
:class:`EmbeddingJob` on ``app.state.embedding_worker`` (T97).
|
||||||
|
|
||||||
Returns a mapping ``{bot_id: (event_id, memory_id)}`` so callers can
|
Returns a mapping ``{bot_id: (event_id, memory_id)}`` so callers can
|
||||||
look up the freshly-projected memory id per owner without re-querying
|
look up the freshly-projected memory id per owner without re-querying
|
||||||
the database.
|
the database.
|
||||||
"""
|
"""
|
||||||
|
if not you_present and guest_bot_id is None:
|
||||||
|
raise ValueError("you_present=False requires guest_bot_id")
|
||||||
|
|
||||||
|
witness_you = 1 if you_present else 0
|
||||||
|
witness_host = 1
|
||||||
witness_guest = 1 if guest_bot_id is not None else 0
|
witness_guest = 1 if guest_bot_id is not None else 0
|
||||||
|
|
||||||
result: dict[str, tuple[int, int | None]] = {}
|
result: dict[str, tuple[int, int | None]] = {}
|
||||||
@@ -153,13 +151,14 @@ def record_turn_memory_for_present(
|
|||||||
owner_id=host_bot_id,
|
owner_id=host_bot_id,
|
||||||
chat_id=chat_id,
|
chat_id=chat_id,
|
||||||
narrative_text=narrative_text,
|
narrative_text=narrative_text,
|
||||||
witness_you=1,
|
witness_you=witness_you,
|
||||||
witness_host=1,
|
witness_host=witness_host,
|
||||||
witness_guest=witness_guest,
|
witness_guest=witness_guest,
|
||||||
scene_id=scene_id,
|
scene_id=scene_id,
|
||||||
chat_clock_at=chat_clock_at,
|
chat_clock_at=chat_clock_at,
|
||||||
source=source,
|
source=source,
|
||||||
significance=significance,
|
significance=significance,
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
if guest_bot_id is not None:
|
if guest_bot_id is not None:
|
||||||
result[guest_bot_id] = _write_one_memory(
|
result[guest_bot_id] = _write_one_memory(
|
||||||
@@ -167,13 +166,14 @@ def record_turn_memory_for_present(
|
|||||||
owner_id=guest_bot_id,
|
owner_id=guest_bot_id,
|
||||||
chat_id=chat_id,
|
chat_id=chat_id,
|
||||||
narrative_text=narrative_text,
|
narrative_text=narrative_text,
|
||||||
witness_you=1,
|
witness_you=witness_you,
|
||||||
witness_host=1,
|
witness_host=witness_host,
|
||||||
witness_guest=1,
|
witness_guest=1,
|
||||||
scene_id=scene_id,
|
scene_id=scene_id,
|
||||||
chat_clock_at=chat_clock_at,
|
chat_clock_at=chat_clock_at,
|
||||||
source=source,
|
source=source,
|
||||||
significance=significance,
|
significance=significance,
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@@ -189,47 +189,25 @@ def record_meanwhile_memory(
|
|||||||
chat_clock_at: str | None = None,
|
chat_clock_at: str | None = None,
|
||||||
source: str = "direct",
|
source: str = "direct",
|
||||||
significance: int = 1,
|
significance: int = 1,
|
||||||
|
app=None,
|
||||||
) -> dict[str, tuple[int, int | None]]:
|
) -> dict[str, tuple[int, int | None]]:
|
||||||
"""Write per-POV ``memory_written`` events for a meanwhile turn (T64).
|
"""Backward-compat thin wrapper for meanwhile memory writes (T64, T84).
|
||||||
|
|
||||||
A meanwhile scene runs entirely between host + guest, with "you"
|
Equivalent to calling :func:`record_turn_memory_for_present` with
|
||||||
absent. Both bots are present witnesses, so each one gets a row with
|
``you_present=False``. Kept so existing call sites in
|
||||||
witness flags ``[you=0, host=1, guest=1]`` — different from the
|
:mod:`chat.web.meanwhile` continue to work without churn. New code
|
||||||
normal-turn ``record_turn_memory_for_present`` shape, which assumes
|
should prefer the unified entry-point directly.
|
||||||
the user is always a witness (``witness_you=1``).
|
|
||||||
|
|
||||||
The ``guest_bot_id`` is required (a meanwhile scene by definition
|
|
||||||
has both bots) — callers passing ``None`` is a programming error.
|
|
||||||
|
|
||||||
Returns ``{bot_id: (event_id, memory_id)}`` mirroring
|
|
||||||
:func:`record_turn_memory_for_present` so downstream queues
|
|
||||||
(significance scoring) can pull memory ids without re-querying.
|
|
||||||
"""
|
"""
|
||||||
result: dict[str, tuple[int, int | None]] = {}
|
return record_turn_memory_for_present(
|
||||||
result[host_bot_id] = _write_one_memory(
|
|
||||||
conn,
|
conn,
|
||||||
owner_id=host_bot_id,
|
|
||||||
chat_id=chat_id,
|
chat_id=chat_id,
|
||||||
|
host_bot_id=host_bot_id,
|
||||||
|
guest_bot_id=guest_bot_id,
|
||||||
narrative_text=narrative_text,
|
narrative_text=narrative_text,
|
||||||
witness_you=0,
|
|
||||||
witness_host=1,
|
|
||||||
witness_guest=1,
|
|
||||||
scene_id=scene_id,
|
scene_id=scene_id,
|
||||||
chat_clock_at=chat_clock_at,
|
chat_clock_at=chat_clock_at,
|
||||||
source=source,
|
source=source,
|
||||||
significance=significance,
|
significance=significance,
|
||||||
|
you_present=False,
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
result[guest_bot_id] = _write_one_memory(
|
|
||||||
conn,
|
|
||||||
owner_id=guest_bot_id,
|
|
||||||
chat_id=chat_id,
|
|
||||||
narrative_text=narrative_text,
|
|
||||||
witness_you=0,
|
|
||||||
witness_host=1,
|
|
||||||
witness_guest=1,
|
|
||||||
scene_id=scene_id,
|
|
||||||
chat_clock_at=chat_clock_at,
|
|
||||||
source=source,
|
|
||||||
significance=significance,
|
|
||||||
)
|
|
||||||
return result
|
|
||||||
|
|||||||
@@ -379,8 +379,15 @@ def _witness_role_for(speaker_bot_id: str, host_bot_id: str | None) -> str:
|
|||||||
pinned the contract on ``search_memories``; this helper applies it
|
pinned the contract on ``search_memories``; this helper applies it
|
||||||
at the call site so a guest-as-speaker doesn't silently retrieve
|
at the call site so a guest-as-speaker doesn't silently retrieve
|
||||||
memories under the wrong POV mask.
|
memories under the wrong POV mask.
|
||||||
|
|
||||||
|
When ``host_bot_id`` is ``None`` (degenerate case from a half-seeded
|
||||||
|
chat or Phase-1 path), the speaker is treated as the host so the
|
||||||
|
query falls back to the host POV mask rather than silently masking
|
||||||
|
the speaker's own memories as a guest.
|
||||||
"""
|
"""
|
||||||
return "host" if speaker_bot_id == host_bot_id else "guest"
|
if host_bot_id is None or speaker_bot_id == host_bot_id:
|
||||||
|
return "host"
|
||||||
|
return "guest"
|
||||||
|
|
||||||
|
|
||||||
def _resolve_addressee(
|
def _resolve_addressee(
|
||||||
|
|||||||
+196
-112
@@ -68,7 +68,9 @@ Phase 2.5 changes:
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
import json
|
import json
|
||||||
|
import logging
|
||||||
from sqlite3 import Connection
|
from sqlite3 import Connection
|
||||||
|
|
||||||
from chat.config import Settings
|
from chat.config import Settings
|
||||||
@@ -79,6 +81,10 @@ from chat.services.interjection import detect_interjection
|
|||||||
from chat.services.memory_write import record_turn_memory_for_present
|
from chat.services.memory_write import record_turn_memory_for_present
|
||||||
from chat.services.multi_state_update import compute_state_updates_for_present
|
from chat.services.multi_state_update import compute_state_updates_for_present
|
||||||
from chat.services.prompt import assemble_narrative_prompt
|
from chat.services.prompt import assemble_narrative_prompt
|
||||||
|
from chat.services.turn_common import (
|
||||||
|
gather_prior_edges,
|
||||||
|
read_recent_dialogue,
|
||||||
|
)
|
||||||
from chat.state.edges import get_edge
|
from chat.state.edges import get_edge
|
||||||
from chat.state.entities import get_bot, get_you
|
from chat.state.entities import get_bot, get_you
|
||||||
from chat.state.events import list_active_events
|
from chat.state.events import list_active_events
|
||||||
@@ -86,6 +92,8 @@ from chat.state.world import active_scene, get_chat
|
|||||||
from chat.web.pubsub import publish
|
from chat.web.pubsub import publish
|
||||||
from chat.web.render import render_turn_html
|
from chat.web.render import render_turn_html
|
||||||
|
|
||||||
|
_log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
async def regenerate_assistant_turn(
|
async def regenerate_assistant_turn(
|
||||||
conn: Connection,
|
conn: Connection,
|
||||||
@@ -95,6 +103,7 @@ async def regenerate_assistant_turn(
|
|||||||
chat_id: str,
|
chat_id: str,
|
||||||
original_assistant_event_id: int,
|
original_assistant_event_id: int,
|
||||||
edited_user_prose: str | None = None,
|
edited_user_prose: str | None = None,
|
||||||
|
app=None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Regenerate the assistant turn linked to ``original_assistant_event_id``.
|
"""Regenerate the assistant turn linked to ``original_assistant_event_id``.
|
||||||
|
|
||||||
@@ -104,6 +113,19 @@ async def regenerate_assistant_turn(
|
|||||||
|
|
||||||
Raises :class:`ValueError` when the chat or the assistant_turn event
|
Raises :class:`ValueError` when the chat or the assistant_turn event
|
||||||
cannot be found — the FastAPI route translates this to 404.
|
cannot be found — the FastAPI route translates this to 404.
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
**Lifecycle-rollback limitation (T83.4, Phase 4 follow-up).**
|
||||||
|
When the superseded turn already produced lifecycle transitions
|
||||||
|
(``event_started`` / ``event_completed`` / ``event_cancelled``),
|
||||||
|
this function does NOT roll those rows back before re-running
|
||||||
|
``detect_event_transitions`` against the regenerated text. A
|
||||||
|
regenerate-after-completion can therefore double-emit promotion
|
||||||
|
artifacts if the new text re-completes the same event. Phase 3.5
|
||||||
|
only documents the gap and emits a WARNING log naming the
|
||||||
|
affected event_log ids; the actual undo pass is invasive
|
||||||
|
(re-projection / inverse-handler dispatch) and is deferred to
|
||||||
|
Phase 4. See the ``# T83.4`` block below for the warning emit.
|
||||||
"""
|
"""
|
||||||
chat = get_chat(conn, chat_id)
|
chat = get_chat(conn, chat_id)
|
||||||
if chat is None:
|
if chat is None:
|
||||||
@@ -136,6 +158,44 @@ async def regenerate_assistant_turn(
|
|||||||
original_assistant_payload = json.loads(row[0])
|
original_assistant_payload = json.loads(row[0])
|
||||||
original_user_turn_id = original_assistant_payload.get("user_turn_id")
|
original_user_turn_id = original_assistant_payload.get("user_turn_id")
|
||||||
|
|
||||||
|
# T83.4: scan for downstream lifecycle transitions emitted by the
|
||||||
|
# superseded turn — they're not being rolled back (see method
|
||||||
|
# docstring). Heuristic: any ``event_started`` / ``event_completed``
|
||||||
|
# / ``event_cancelled`` event_log row with id strictly greater than
|
||||||
|
# the original assistant_turn's id was emitted as part of (or after)
|
||||||
|
# that turn's processing. Lifecycle events don't carry ``chat_id``
|
||||||
|
# in their payload (their payload references an ``event_id`` FK to
|
||||||
|
# the ``events`` table, which holds chat_id), so we join through
|
||||||
|
# ``events`` to scope to this chat.
|
||||||
|
#
|
||||||
|
# A WARNING log surfaces the affected event ids so operators can
|
||||||
|
# spot double-emit cases until the Phase 4 rollback pass lands.
|
||||||
|
unrolled_lifecycle = conn.execute(
|
||||||
|
"SELECT el.id, el.kind FROM event_log AS el "
|
||||||
|
"JOIN events AS ev "
|
||||||
|
" ON ev.event_id = json_extract(el.payload_json, '$.event_id') "
|
||||||
|
"WHERE el.kind IN ("
|
||||||
|
" 'event_started', 'event_completed', 'event_cancelled'"
|
||||||
|
" ) "
|
||||||
|
" AND ev.chat_id = ? "
|
||||||
|
" AND el.id > ? "
|
||||||
|
"ORDER BY el.id ASC",
|
||||||
|
(chat_id, original_assistant_event_id),
|
||||||
|
).fetchall()
|
||||||
|
if unrolled_lifecycle:
|
||||||
|
# T90.2: phrased as "at-or-after turn <id>" rather than "from
|
||||||
|
# superseded turn" because regenerating an OLDER turn lists
|
||||||
|
# intervening-turn transitions that legitimately stand on their
|
||||||
|
# own — those weren't authored by the superseded turn itself.
|
||||||
|
_log.warning(
|
||||||
|
"regenerate_assistant_turn: %d lifecycle transition(s) "
|
||||||
|
"at-or-after turn %s are NOT being rolled back (Phase 4 "
|
||||||
|
"follow-up). Affected event ids: %s",
|
||||||
|
len(unrolled_lifecycle),
|
||||||
|
original_assistant_event_id,
|
||||||
|
[r[0] for r in unrolled_lifecycle],
|
||||||
|
)
|
||||||
|
|
||||||
# 1a. Look up any sibling interjection beat in the same turn group
|
# 1a. Look up any sibling interjection beat in the same turn group
|
||||||
# (T73.2). The original group is (primary + optional interjection),
|
# (T73.2). The original group is (primary + optional interjection),
|
||||||
# both pinned to the same ``user_turn_id``. The interjection has a
|
# both pinned to the same ``user_turn_id``. The interjection has a
|
||||||
@@ -143,6 +203,13 @@ async def regenerate_assistant_turn(
|
|||||||
# the silent witness (the bot that wasn't the primary addressee).
|
# the silent witness (the bot that wasn't the primary addressee).
|
||||||
# Filter on ``superseded_by IS NULL`` so prior regenerates of this
|
# Filter on ``superseded_by IS NULL`` so prior regenerates of this
|
||||||
# group don't reappear as siblings.
|
# group don't reappear as siblings.
|
||||||
|
#
|
||||||
|
# T83.3: push the chat_id filter into SQL via ``json_extract`` so
|
||||||
|
# the query doesn't scan every assistant_turn row across the whole
|
||||||
|
# database. ``LIMIT 50`` bounds worst-case work even when chat_id
|
||||||
|
# isn't selective (e.g. a single chat with many turns) — we only
|
||||||
|
# need the one matching sibling. Mirrors the SQL pattern in
|
||||||
|
# ``chat.web.meanwhile._last_meanwhile_speaker``.
|
||||||
original_interjection_event_id: int | None = None
|
original_interjection_event_id: int | None = None
|
||||||
original_interjection_payload: dict | None = None
|
original_interjection_payload: dict | None = None
|
||||||
if original_user_turn_id is not None:
|
if original_user_turn_id is not None:
|
||||||
@@ -150,8 +217,11 @@ async def regenerate_assistant_turn(
|
|||||||
"SELECT id, payload_json FROM event_log "
|
"SELECT id, payload_json FROM event_log "
|
||||||
"WHERE kind = 'assistant_turn' "
|
"WHERE kind = 'assistant_turn' "
|
||||||
" AND id != ? "
|
" AND id != ? "
|
||||||
" AND superseded_by IS NULL",
|
" AND superseded_by IS NULL "
|
||||||
(original_assistant_event_id,),
|
" AND json_extract(payload_json, '$.chat_id') = ? "
|
||||||
|
"ORDER BY id DESC "
|
||||||
|
"LIMIT 50",
|
||||||
|
(original_assistant_event_id, chat_id),
|
||||||
)
|
)
|
||||||
for sib_id, sib_payload_json in sibling_cur.fetchall():
|
for sib_id, sib_payload_json in sibling_cur.fetchall():
|
||||||
sib_payload = json.loads(sib_payload_json)
|
sib_payload = json.loads(sib_payload_json)
|
||||||
@@ -208,33 +278,30 @@ async def regenerate_assistant_turn(
|
|||||||
# assistant_turn explicitly (we haven't superseded it yet — that
|
# assistant_turn explicitly (we haven't superseded it yet — that
|
||||||
# update lands at the end so the new event_id is known) and use the
|
# update lands at the end so the new event_id is known) and use the
|
||||||
# standard ``superseded_by IS NULL AND hidden = 0`` filter so any
|
# standard ``superseded_by IS NULL AND hidden = 0`` filter so any
|
||||||
# prior regenerates also drop out.
|
# prior regenerates also drop out. T83.2: shared helper handles the
|
||||||
|
# SQL + filtering; we post-process to map speaker ids to display
|
||||||
|
# names for the prompt.
|
||||||
you_entity = get_you(conn) or {"name": "you", "persona": ""}
|
you_entity = get_you(conn) or {"name": "you", "persona": ""}
|
||||||
you_name = you_entity.get("name", "you")
|
you_name = you_entity.get("name", "you")
|
||||||
cur = conn.execute(
|
raw_recent = read_recent_dialogue(
|
||||||
"SELECT id, kind, payload_json FROM event_log "
|
conn,
|
||||||
"WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn') "
|
chat_id,
|
||||||
" AND id != ? "
|
limit=20,
|
||||||
" AND superseded_by IS NULL AND hidden = 0 "
|
exclude_event_id=original_assistant_event_id,
|
||||||
"ORDER BY id DESC LIMIT 20",
|
|
||||||
(original_assistant_event_id,),
|
|
||||||
)
|
)
|
||||||
rows = list(reversed(cur.fetchall()))
|
|
||||||
recent: list[dict] = []
|
recent: list[dict] = []
|
||||||
for _eid, kind, payload_json in rows:
|
for entry in raw_recent:
|
||||||
p = json.loads(payload_json)
|
spk = entry.get("speaker", "bot")
|
||||||
if p.get("chat_id") != chat_id:
|
if spk == "you":
|
||||||
|
recent.append({"speaker": you_name, "text": entry.get("text", "")})
|
||||||
continue
|
continue
|
||||||
if kind in ("user_turn", "user_turn_edit"):
|
if spk == host_bot_id:
|
||||||
recent.append({"speaker": you_name, "text": p.get("prose", "")})
|
|
||||||
else:
|
|
||||||
spk = p.get("speaker_id", "bot")
|
|
||||||
spk_name = host_bot.get("name", "bot")
|
spk_name = host_bot.get("name", "bot")
|
||||||
if spk == host_bot_id:
|
elif guest_bot is not None and spk == guest_bot.get("id"):
|
||||||
spk_name = host_bot.get("name", "bot")
|
spk_name = guest_bot.get("name", "bot")
|
||||||
elif guest_bot is not None and spk == guest_bot.get("id"):
|
else:
|
||||||
spk_name = guest_bot.get("name", "bot")
|
spk_name = host_bot.get("name", "bot")
|
||||||
recent.append({"speaker": spk_name, "text": p.get("text", "")})
|
recent.append({"speaker": spk_name, "text": entry.get("text", "")})
|
||||||
|
|
||||||
# 4. Assemble the narrative prompt. ``recent`` already excludes the
|
# 4. Assemble the narrative prompt. ``recent`` already excludes the
|
||||||
# current user prose, which we pass through ``user_turn_prose``.
|
# current user prose, which we pass through ``user_turn_prose``.
|
||||||
@@ -250,19 +317,37 @@ async def regenerate_assistant_turn(
|
|||||||
guest_id=guest_bot_id,
|
guest_id=guest_bot_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
# 5. Stream the new narrative.
|
# 5. Stream the new narrative. T83.1: register the streaming Task in
|
||||||
|
# the chat-keyed in-flight registry so POST /chats/<id>/turns/cancel
|
||||||
|
# can call ``.cancel()`` on a mid-regenerate stream. We import the
|
||||||
|
# underscore name from turns.py deliberately — same single-process
|
||||||
|
# registry the cancel route reads, mirrors the meanwhile registration
|
||||||
|
# pattern in chat/web/meanwhile.py.
|
||||||
|
from chat.web.turns import _in_flight_tasks # noqa: PLC0415
|
||||||
|
|
||||||
accumulated: list[str] = []
|
accumulated: list[str] = []
|
||||||
async for chunk in client.stream(
|
|
||||||
messages,
|
async def _stream_primary() -> None:
|
||||||
model=settings.narrative_model,
|
async for chunk in client.stream(
|
||||||
max_tokens=settings.narrative_max_tokens,
|
messages,
|
||||||
temperature=settings.narrative_temperature,
|
model=settings.narrative_model,
|
||||||
):
|
max_tokens=settings.narrative_max_tokens,
|
||||||
accumulated.append(chunk)
|
temperature=settings.narrative_temperature,
|
||||||
await publish(
|
):
|
||||||
chat_id,
|
accumulated.append(chunk)
|
||||||
{"event": "token", "text": chunk, "speaker_id": speaker_bot_id},
|
await publish(
|
||||||
)
|
chat_id,
|
||||||
|
{"event": "token", "text": chunk, "speaker_id": speaker_bot_id},
|
||||||
|
)
|
||||||
|
|
||||||
|
stream_task = asyncio.create_task(_stream_primary())
|
||||||
|
_in_flight_tasks[chat_id] = stream_task
|
||||||
|
try:
|
||||||
|
await stream_task
|
||||||
|
finally:
|
||||||
|
# Always unregister so a subsequent turn / regenerate can register
|
||||||
|
# a fresh task. Mirrors the cleanup in turns.py::post_turn.
|
||||||
|
_in_flight_tasks.pop(chat_id, None)
|
||||||
new_text = "".join(accumulated)
|
new_text = "".join(accumulated)
|
||||||
|
|
||||||
# 6. Append the new assistant_turn event. ``user_turn_id`` points at
|
# 6. Append the new assistant_turn event. ``user_turn_id`` points at
|
||||||
@@ -301,7 +386,10 @@ async def regenerate_assistant_turn(
|
|||||||
speaker_bot.get("name", "bot") if speaker_bot is not None else "bot"
|
speaker_bot.get("name", "bot") if speaker_bot is not None else "bot"
|
||||||
)
|
)
|
||||||
new_turn_html = render_turn_html(
|
new_turn_html = render_turn_html(
|
||||||
speaker_name_for_render, new_text, role="bot"
|
speaker_name_for_render,
|
||||||
|
new_text,
|
||||||
|
role="bot",
|
||||||
|
event_id=new_assistant_event_id,
|
||||||
)
|
)
|
||||||
await publish(
|
await publish(
|
||||||
chat_id,
|
chat_id,
|
||||||
@@ -327,6 +415,7 @@ async def regenerate_assistant_turn(
|
|||||||
narrative_text=new_text,
|
narrative_text=new_text,
|
||||||
scene_id=scene["id"] if scene else None,
|
scene_id=scene["id"] if scene else None,
|
||||||
chat_clock_at=chat.get("time"),
|
chat_clock_at=chat.get("time"),
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
|
|
||||||
last_at = chat.get("time")
|
last_at = chat.get("time")
|
||||||
@@ -354,17 +443,8 @@ async def regenerate_assistant_turn(
|
|||||||
present_names[guest_bot_id] = guest_bot.get("name", "bot")
|
present_names[guest_bot_id] = guest_bot.get("name", "bot")
|
||||||
personas[guest_bot_id] = guest_bot.get("persona") or ""
|
personas[guest_bot_id] = guest_bot.get("persona") or ""
|
||||||
|
|
||||||
prior_edges: dict[tuple[str, str], dict] = {}
|
# T83.2: shared helper builds the directed-pair edge dict.
|
||||||
for src in present_ids:
|
prior_edges = gather_prior_edges(conn, present_ids)
|
||||||
for tgt in present_ids:
|
|
||||||
if src == tgt:
|
|
||||||
continue
|
|
||||||
edge = get_edge(conn, src, tgt) or {
|
|
||||||
"affinity": 50,
|
|
||||||
"trust": 50,
|
|
||||||
"summary": "",
|
|
||||||
}
|
|
||||||
prior_edges[(src, tgt)] = edge
|
|
||||||
|
|
||||||
state_updates = await compute_state_updates_for_present(
|
state_updates = await compute_state_updates_for_present(
|
||||||
client,
|
client,
|
||||||
@@ -453,34 +533,27 @@ async def regenerate_assistant_turn(
|
|||||||
)
|
)
|
||||||
|
|
||||||
if decision.should_interject:
|
if decision.should_interject:
|
||||||
# Re-read recent so the just-appended primary is in the prompt.
|
# Re-read recent so the just-appended primary is in the
|
||||||
interject_cur = conn.execute(
|
# prompt. T83.2: shared helper + the same id->name mapping
|
||||||
"SELECT id, kind, payload_json FROM event_log "
|
# as the primary read above.
|
||||||
"WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn') "
|
raw_interject = read_recent_dialogue(conn, chat_id, limit=20)
|
||||||
" AND superseded_by IS NULL AND hidden = 0 "
|
|
||||||
"ORDER BY id DESC LIMIT 20",
|
|
||||||
)
|
|
||||||
interject_rows = list(reversed(interject_cur.fetchall()))
|
|
||||||
interject_recent: list[dict] = []
|
interject_recent: list[dict] = []
|
||||||
for _eid, kind, payload_json in interject_rows:
|
for entry in raw_interject:
|
||||||
p = json.loads(payload_json)
|
spk = entry.get("speaker", "bot")
|
||||||
if p.get("chat_id") != chat_id:
|
if spk == "you":
|
||||||
|
interject_recent.append(
|
||||||
|
{"speaker": you_name, "text": entry.get("text", "")}
|
||||||
|
)
|
||||||
continue
|
continue
|
||||||
if kind in ("user_turn", "user_turn_edit"):
|
if spk == host_bot_id:
|
||||||
interject_recent.append(
|
spk_name = host_bot.get("name", "bot")
|
||||||
{"speaker": you_name, "text": p.get("prose", "")}
|
elif spk == guest_bot.get("id"):
|
||||||
)
|
spk_name = guest_bot.get("name", "bot")
|
||||||
else:
|
else:
|
||||||
spk = p.get("speaker_id", "bot")
|
spk_name = "bot"
|
||||||
if spk == host_bot_id:
|
interject_recent.append(
|
||||||
spk_name = host_bot.get("name", "bot")
|
{"speaker": spk_name, "text": entry.get("text", "")}
|
||||||
elif spk == guest_bot.get("id"):
|
)
|
||||||
spk_name = guest_bot.get("name", "bot")
|
|
||||||
else:
|
|
||||||
spk_name = "bot"
|
|
||||||
interject_recent.append(
|
|
||||||
{"speaker": spk_name, "text": p.get("text", "")}
|
|
||||||
)
|
|
||||||
if interject_recent and interject_recent[-1].get("speaker") == you_name:
|
if interject_recent and interject_recent[-1].get("speaker") == you_name:
|
||||||
interject_recent = interject_recent[:-1]
|
interject_recent = interject_recent[:-1]
|
||||||
|
|
||||||
@@ -497,21 +570,32 @@ async def regenerate_assistant_turn(
|
|||||||
)
|
)
|
||||||
|
|
||||||
interject_accumulated: list[str] = []
|
interject_accumulated: list[str] = []
|
||||||
async for chunk in client.stream(
|
|
||||||
interject_messages,
|
async def _stream_interjection() -> None:
|
||||||
model=settings.narrative_model,
|
async for chunk in client.stream(
|
||||||
max_tokens=settings.narrative_max_tokens,
|
interject_messages,
|
||||||
temperature=settings.narrative_temperature,
|
model=settings.narrative_model,
|
||||||
):
|
max_tokens=settings.narrative_max_tokens,
|
||||||
interject_accumulated.append(chunk)
|
temperature=settings.narrative_temperature,
|
||||||
await publish(
|
):
|
||||||
chat_id,
|
interject_accumulated.append(chunk)
|
||||||
{
|
await publish(
|
||||||
"event": "token",
|
chat_id,
|
||||||
"text": chunk,
|
{
|
||||||
"speaker_id": silent_witness_id,
|
"event": "token",
|
||||||
},
|
"text": chunk,
|
||||||
)
|
"speaker_id": silent_witness_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# T83.1: register the interjection sub-stream in the same
|
||||||
|
# in-flight registry so /turns/cancel collapses it too.
|
||||||
|
interject_task = asyncio.create_task(_stream_interjection())
|
||||||
|
_in_flight_tasks[chat_id] = interject_task
|
||||||
|
try:
|
||||||
|
await interject_task
|
||||||
|
finally:
|
||||||
|
_in_flight_tasks.pop(chat_id, None)
|
||||||
interject_text = "".join(interject_accumulated)
|
interject_text = "".join(interject_accumulated)
|
||||||
|
|
||||||
new_interjection_event_id = append_event(
|
new_interjection_event_id = append_event(
|
||||||
@@ -541,7 +625,10 @@ async def regenerate_assistant_turn(
|
|||||||
# Broadcast a replace event so connected tabs swap the prior
|
# Broadcast a replace event so connected tabs swap the prior
|
||||||
# interjection node in-place (mirrors T73.1's primary swap).
|
# interjection node in-place (mirrors T73.1's primary swap).
|
||||||
interject_html = render_turn_html(
|
interject_html = render_turn_html(
|
||||||
silent_witness.get("name", "bot"), interject_text, role="bot"
|
silent_witness.get("name", "bot"),
|
||||||
|
interject_text,
|
||||||
|
role="bot",
|
||||||
|
event_id=new_interjection_event_id,
|
||||||
)
|
)
|
||||||
await publish(
|
await publish(
|
||||||
chat_id,
|
chat_id,
|
||||||
@@ -563,6 +650,7 @@ async def regenerate_assistant_turn(
|
|||||||
narrative_text=interject_text,
|
narrative_text=interject_text,
|
||||||
scene_id=scene["id"] if scene else None,
|
scene_id=scene["id"] if scene else None,
|
||||||
chat_clock_at=chat.get("time"),
|
chat_clock_at=chat.get("time"),
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Re-run the multi-pair state-update with the post-interjection
|
# Re-run the multi-pair state-update with the post-interjection
|
||||||
@@ -573,17 +661,8 @@ async def regenerate_assistant_turn(
|
|||||||
"text": interject_text,
|
"text": interject_text,
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
prior_edges_post: dict[tuple[str, str], dict] = {}
|
# T83.2: shared helper handles the directed-pair edge dict.
|
||||||
for src in present_ids:
|
prior_edges_post = gather_prior_edges(conn, present_ids)
|
||||||
for tgt in present_ids:
|
|
||||||
if src == tgt:
|
|
||||||
continue
|
|
||||||
edge = get_edge(conn, src, tgt) or {
|
|
||||||
"affinity": 50,
|
|
||||||
"trust": 50,
|
|
||||||
"summary": "",
|
|
||||||
}
|
|
||||||
prior_edges_post[(src, tgt)] = edge
|
|
||||||
|
|
||||||
state_updates_post = await compute_state_updates_for_present(
|
state_updates_post = await compute_state_updates_for_present(
|
||||||
client,
|
client,
|
||||||
@@ -620,23 +699,28 @@ async def regenerate_assistant_turn(
|
|||||||
(new_assistant_event_id, original_interjection_event_id),
|
(new_assistant_event_id, original_interjection_event_id),
|
||||||
)
|
)
|
||||||
|
|
||||||
# 10. Event-lifecycle detection (Phase 3, T61). Mirrors the post_turn
|
# 9a. Event-lifecycle detection (Phase 3, T61). T83.5 cosmetic
|
||||||
# block: classify whether any active events transitioned in the
|
# ordering: mirrors ``chat.web.turns.post_turn``'s 8a block — runs
|
||||||
# regenerated narrative and append the corresponding event_started /
|
# AFTER the interjection branch (and AFTER the post-interjection
|
||||||
|
# state-update + memory passes) so the classifier sees the same
|
||||||
|
# narrative-text input post_turn does. Numbering uses ``9a`` to
|
||||||
|
# match post_turn's ``8a`` shape (the interjection branch is step 9
|
||||||
|
# in regenerate vs step 8 in post_turn; lifecycle is the immediate
|
||||||
|
# follow-on in both). Behaviour identical to the prior ``step 10``
|
||||||
|
# placement — the block was already structurally last in regenerate
|
||||||
|
# because there's no scene-close pass here.
|
||||||
|
#
|
||||||
|
# Classify whether any active events transitioned in the regenerated
|
||||||
|
# narrative and append the corresponding event_started /
|
||||||
# event_completed / event_cancelled. ``promote_completed_event``
|
# event_completed / event_cancelled. ``promote_completed_event``
|
||||||
# runs inline after a completion so promotion artifacts land in the
|
# runs inline after a completion so promotion artifacts land in the
|
||||||
# same regenerate path.
|
# same regenerate path.
|
||||||
#
|
#
|
||||||
# Phase 3.5 follow-up: when a regenerate replaces a turn that had
|
# T83.4 follow-up: when a regenerate replaces a turn that had
|
||||||
# already produced event transitions, those original transitions are
|
# already produced event transitions, those original transitions
|
||||||
# NOT undone here. The superseded ``assistant_turn`` group keeps its
|
# are NOT undone here (Phase 4 work). A WARNING log earlier in this
|
||||||
# prior ``event_started`` / ``event_completed`` events in the log
|
# function names the affected event_log ids — see the T83.4 block
|
||||||
# (they remain projected onto the events table). Phase 3.5 will add
|
# near the function entry.
|
||||||
# an "undo lifecycle" step to roll back the prior transitions before
|
|
||||||
# re-classifying the regenerated text. For v3 we accept that a
|
|
||||||
# regenerate-after-completion will double-emit promotion artifacts
|
|
||||||
# if the new text re-completes the same event — narratively rare,
|
|
||||||
# and a true fix needs the lifecycle-undo pass.
|
|
||||||
new_active_events = list_active_events(conn, chat_id)
|
new_active_events = list_active_events(conn, chat_id)
|
||||||
if new_active_events:
|
if new_active_events:
|
||||||
lifecycle_decision = await detect_event_transitions(
|
lifecycle_decision = await detect_event_transitions(
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ keeps moving.
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import logging
|
||||||
import uuid
|
import uuid
|
||||||
from datetime import datetime, timezone
|
from datetime import datetime, timezone
|
||||||
from sqlite3 import Connection
|
from sqlite3 import Connection
|
||||||
@@ -39,6 +40,8 @@ from chat.eventlog.log import append_and_apply
|
|||||||
from chat.llm.classify import classify
|
from chat.llm.classify import classify
|
||||||
from chat.llm.client import LLMClient
|
from chat.llm.client import LLMClient
|
||||||
|
|
||||||
|
_log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class ScenePOVSummary(BaseModel):
|
class ScenePOVSummary(BaseModel):
|
||||||
"""Classifier output: one witness's view of a closing scene.
|
"""Classifier output: one witness's view of a closing scene.
|
||||||
@@ -123,7 +126,11 @@ async def summarize_scene(
|
|||||||
|
|
||||||
|
|
||||||
def _read_recent_dialogue(
|
def _read_recent_dialogue(
|
||||||
conn: Connection, chat_id: str, *, limit: int = 50
|
conn: Connection,
|
||||||
|
chat_id: str,
|
||||||
|
*,
|
||||||
|
limit: int = 50,
|
||||||
|
since_event_id: int | None = None,
|
||||||
) -> list[dict]:
|
) -> list[dict]:
|
||||||
"""Pull the last ``limit`` user/assistant turns for ``chat_id``.
|
"""Pull the last ``limit`` user/assistant turns for ``chat_id``.
|
||||||
|
|
||||||
@@ -132,14 +139,29 @@ def _read_recent_dialogue(
|
|||||||
the most recent turns of the chat. Superseded and hidden rows are
|
the most recent turns of the chat. Superseded and hidden rows are
|
||||||
filtered out so regenerated turns (T29) don't bleed into the
|
filtered out so regenerated turns (T29) don't bleed into the
|
||||||
summary.
|
summary.
|
||||||
|
|
||||||
|
T80.2: ``since_event_id`` clamps the result to event_log rows whose
|
||||||
|
``id >= since_event_id`` so callers needing a scene-scoped view (e.g.
|
||||||
|
thread detection on close) don't pull turns that landed before the
|
||||||
|
closing scene's ``scene_opened`` event.
|
||||||
"""
|
"""
|
||||||
cur = conn.execute(
|
if since_event_id is None:
|
||||||
"SELECT kind, payload_json FROM event_log "
|
cur = conn.execute(
|
||||||
"WHERE kind IN ('user_turn', 'assistant_turn') "
|
"SELECT kind, payload_json FROM event_log "
|
||||||
" AND superseded_by IS NULL AND hidden = 0 "
|
"WHERE kind IN ('user_turn', 'assistant_turn') "
|
||||||
"ORDER BY id DESC LIMIT ?",
|
" AND superseded_by IS NULL AND hidden = 0 "
|
||||||
(limit,),
|
"ORDER BY id DESC LIMIT ?",
|
||||||
)
|
(limit,),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT kind, payload_json FROM event_log "
|
||||||
|
"WHERE kind IN ('user_turn', 'assistant_turn') "
|
||||||
|
" AND superseded_by IS NULL AND hidden = 0 "
|
||||||
|
" AND id >= ? "
|
||||||
|
"ORDER BY id DESC LIMIT ?",
|
||||||
|
(since_event_id, limit),
|
||||||
|
)
|
||||||
rows = list(reversed(cur.fetchall()))
|
rows = list(reversed(cur.fetchall()))
|
||||||
out: list[dict] = []
|
out: list[dict] = []
|
||||||
for kind, payload_json in rows:
|
for kind, payload_json in rows:
|
||||||
@@ -158,6 +180,65 @@ def _read_recent_dialogue(
|
|||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _scene_opened_event_id(
|
||||||
|
conn: Connection, chat_id: str, scene_id: int
|
||||||
|
) -> int | None:
|
||||||
|
"""Return the event_log id of the ``scene_opened`` (or
|
||||||
|
``meanwhile_scene_started``) event that created scene row
|
||||||
|
``scene_id``. Used by T80.2 to lower-bound dialogue reads to a
|
||||||
|
single scene's transcript.
|
||||||
|
|
||||||
|
``meanwhile_scene_started`` carries an explicit ``scene_id`` so we
|
||||||
|
match on that directly. ``scene_opened`` doesn't, so we walk the
|
||||||
|
chat's scene rows in id order and zip against the chat's scene-open
|
||||||
|
events in id order — the projector creates one scene row per
|
||||||
|
scene-open event, so positions correspond.
|
||||||
|
|
||||||
|
Returns ``None`` when no matching event is found; callers should
|
||||||
|
treat that as "fall back to chat-wide" rather than over-filter.
|
||||||
|
"""
|
||||||
|
# Fast path for meanwhile children (explicit scene_id in payload).
|
||||||
|
for ev_id, payload_json in conn.execute(
|
||||||
|
"SELECT id, payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'meanwhile_scene_started' "
|
||||||
|
" AND superseded_by IS NULL AND hidden = 0",
|
||||||
|
).fetchall():
|
||||||
|
try:
|
||||||
|
p = json.loads(payload_json)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
if p.get("chat_id") == chat_id and p.get("scene_id") == scene_id:
|
||||||
|
return ev_id
|
||||||
|
# Fallback for parent you-scenes: zip chat-scoped scene-open events
|
||||||
|
# against chat-scoped scene rows in id order.
|
||||||
|
chat_scene_ids = [
|
||||||
|
r[0]
|
||||||
|
for r in conn.execute(
|
||||||
|
"SELECT id FROM scenes WHERE chat_id = ? ORDER BY id ASC",
|
||||||
|
(chat_id,),
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
if scene_id not in chat_scene_ids:
|
||||||
|
return None
|
||||||
|
chat_open_evs: list[int] = []
|
||||||
|
for ev_id, _kind, payload_json in conn.execute(
|
||||||
|
"SELECT id, kind, payload_json FROM event_log "
|
||||||
|
"WHERE kind IN ('scene_opened', 'meanwhile_scene_started') "
|
||||||
|
" AND superseded_by IS NULL AND hidden = 0 "
|
||||||
|
"ORDER BY id ASC",
|
||||||
|
).fetchall():
|
||||||
|
try:
|
||||||
|
p = json.loads(payload_json)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
continue
|
||||||
|
if p.get("chat_id") == chat_id:
|
||||||
|
chat_open_evs.append(ev_id)
|
||||||
|
idx = chat_scene_ids.index(scene_id)
|
||||||
|
if idx < len(chat_open_evs):
|
||||||
|
return chat_open_evs[idx]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
async def _summarize_and_apply_for_witness(
|
async def _summarize_and_apply_for_witness(
|
||||||
conn: Connection,
|
conn: Connection,
|
||||||
client: LLMClient,
|
client: LLMClient,
|
||||||
@@ -213,7 +294,11 @@ async def _summarize_and_apply_for_witness(
|
|||||||
# Empty default -> skip the memory rewrite; the seeded
|
# Empty default -> skip the memory rewrite; the seeded
|
||||||
# per-turn pov_summary stays in place.
|
# per-turn pov_summary stays in place.
|
||||||
continue
|
continue
|
||||||
new_value = pov.summary + key_quotes_suffix
|
# T80.1: a prior close may have already appended a Key quotes
|
||||||
|
# suffix to this row's pov_summary. Strip it here so the fresh
|
||||||
|
# rewrite replaces the existing suffix rather than stacking a
|
||||||
|
# second one on top.
|
||||||
|
new_value = _strip_key_quotes_suffix(pov.summary) + key_quotes_suffix
|
||||||
append_and_apply(
|
append_and_apply(
|
||||||
conn,
|
conn,
|
||||||
kind="manual_edit",
|
kind="manual_edit",
|
||||||
@@ -263,6 +348,31 @@ async def _summarize_and_apply_for_witness(
|
|||||||
return pov
|
return pov
|
||||||
|
|
||||||
|
|
||||||
|
# T80.1: header marker shared by the suffix builder and the
|
||||||
|
# witness-write strip step. Any text starting with this marker is treated
|
||||||
|
# as a previously-appended Key quotes suffix and stripped before reuse so
|
||||||
|
# repeated scene closes don't compose recursive bloat.
|
||||||
|
_KEY_QUOTES_HEADER = "\n\nKey quotes:\n"
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_key_quotes_suffix(text: str) -> str:
|
||||||
|
"""Remove a previously-appended Key quotes suffix from ``text``.
|
||||||
|
|
||||||
|
Returns ``text`` unchanged when the marker is absent, or the prefix
|
||||||
|
up to (but not including) the marker when present. Used in two
|
||||||
|
places: (1) when sourcing quote text from a memory row that may
|
||||||
|
already carry the suffix from a prior close, and (2) when computing
|
||||||
|
the per-POV rewrite's prior_value so the new write replaces — rather
|
||||||
|
than stacks on — the old suffix.
|
||||||
|
"""
|
||||||
|
if not text:
|
||||||
|
return text
|
||||||
|
idx = text.find(_KEY_QUOTES_HEADER)
|
||||||
|
if idx >= 0:
|
||||||
|
return text[:idx]
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
def _build_key_quotes_suffix(conn: Connection, scene_id: int) -> str:
|
def _build_key_quotes_suffix(conn: Connection, scene_id: int) -> str:
|
||||||
"""If the scene's max-turn-significance is >= 2, build the
|
"""If the scene's max-turn-significance is >= 2, build the
|
||||||
"Key quotes:" suffix from the top-3 highest-significance memory rows
|
"Key quotes:" suffix from the top-3 highest-significance memory rows
|
||||||
@@ -274,6 +384,10 @@ def _build_key_quotes_suffix(conn: Connection, scene_id: int) -> str:
|
|||||||
per-turn narrative seeded by T21, since this helper is called BEFORE
|
per-turn narrative seeded by T21, since this helper is called BEFORE
|
||||||
the per-POV rewrite. Texts are truncated to 200 chars to bound
|
the per-POV rewrite. Texts are truncated to 200 chars to bound
|
||||||
memory row growth across many witnesses.
|
memory row growth across many witnesses.
|
||||||
|
|
||||||
|
T80.1: candidate text is run through :func:`_strip_key_quotes_suffix`
|
||||||
|
first so a re-close (whose source memories already carry a suffix from
|
||||||
|
the prior close) doesn't quote a quote.
|
||||||
"""
|
"""
|
||||||
row = conn.execute(
|
row = conn.execute(
|
||||||
"SELECT MAX(significance) FROM memories WHERE scene_id = ?",
|
"SELECT MAX(significance) FROM memories WHERE scene_id = ?",
|
||||||
@@ -288,7 +402,7 @@ def _build_key_quotes_suffix(conn: Connection, scene_id: int) -> str:
|
|||||||
(scene_id,),
|
(scene_id,),
|
||||||
)
|
)
|
||||||
quotes = [
|
quotes = [
|
||||||
(r[0] or "")[:200]
|
_strip_key_quotes_suffix(r[0] or "")[:200]
|
||||||
for r in cur.fetchall()
|
for r in cur.fetchall()
|
||||||
]
|
]
|
||||||
if not quotes:
|
if not quotes:
|
||||||
@@ -454,20 +568,35 @@ async def apply_scene_close_summary(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
# T58.2: thread detection on close. Reuses the dialogue we already
|
# T58.2: thread detection on close. Failure-tolerant: classify()
|
||||||
# gathered for per-POV summarization — same {speaker, text} shape
|
# returns the empty default on retry-exhaustion, and the broad except
|
||||||
# detect_threads expects. Failure-tolerant: classify() returns the
|
# below protects the close pipeline from any other classifier/mock
|
||||||
# empty default on retry-exhaustion, and the broad except below
|
# flap.
|
||||||
# protects the close pipeline from any other classifier/mock flap.
|
#
|
||||||
|
# T80.2: thread detection runs against a SCENE-SCOPED transcript,
|
||||||
|
# not the chat-wide last-50 turns used by the per-POV summaries.
|
||||||
|
# Mis-attributing threads when scene boundaries fall inside the last
|
||||||
|
# 50 turns would otherwise close threads opened in a prior scene.
|
||||||
|
scene_open_ev_id = _scene_opened_event_id(conn, chat_id, scene_id)
|
||||||
|
if scene_open_ev_id is not None:
|
||||||
|
scene_dialogue = _read_recent_dialogue(
|
||||||
|
conn, chat_id, since_event_id=scene_open_ev_id
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
scene_dialogue = dialogue
|
||||||
try:
|
try:
|
||||||
thread_result = await detect_threads(
|
thread_result = await detect_threads(
|
||||||
client,
|
client,
|
||||||
classifier_model=classifier_model,
|
classifier_model=classifier_model,
|
||||||
scene_transcript=dialogue,
|
scene_transcript=scene_dialogue,
|
||||||
open_threads=list_open_threads(conn, chat_id),
|
open_threads=list_open_threads(conn, chat_id),
|
||||||
timeout_s=timeout_s,
|
timeout_s=timeout_s,
|
||||||
)
|
)
|
||||||
except Exception:
|
except Exception as exc:
|
||||||
|
# T80.3: log the swallowed exception at DEBUG so a
|
||||||
|
# programmer-error flap (e.g. wrong kwarg name) surfaces in
|
||||||
|
# local logs without breaking the close pipeline.
|
||||||
|
_log.debug("detect_threads failed: %s", exc, exc_info=True)
|
||||||
from chat.services.thread_detection import ThreadDetectionResult
|
from chat.services.thread_detection import ThreadDetectionResult
|
||||||
|
|
||||||
thread_result = ThreadDetectionResult()
|
thread_result = ThreadDetectionResult()
|
||||||
@@ -495,12 +624,20 @@ async def apply_scene_close_summary(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
elif cand.action == "close" and cand.existing_thread_id:
|
elif cand.action == "close" and cand.existing_thread_id:
|
||||||
|
# T80.4: chat-clock time, not wall clock — the rest of the
|
||||||
|
# close pipeline (memories, edges, scene_closed payloads)
|
||||||
|
# uses chat["time"] so threads must agree. Falls back to
|
||||||
|
# UTC now only when the chat row has no clock yet (defensive
|
||||||
|
# — chat_state always seeds "time" via chat_created).
|
||||||
|
chat_clock_at = chat.get("time") or datetime.now(
|
||||||
|
timezone.utc
|
||||||
|
).isoformat()
|
||||||
append_and_apply(
|
append_and_apply(
|
||||||
conn,
|
conn,
|
||||||
kind="thread_closed",
|
kind="thread_closed",
|
||||||
payload={
|
payload={
|
||||||
"thread_id": cand.existing_thread_id,
|
"thread_id": cand.existing_thread_id,
|
||||||
"closed_at": datetime.now(timezone.utc).isoformat(),
|
"closed_at": chat_clock_at,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
@@ -96,6 +96,7 @@ async def narrate_skip(
|
|||||||
model=narrative_model,
|
model=narrative_model,
|
||||||
max_tokens=200,
|
max_tokens=200,
|
||||||
temperature=0.7,
|
temperature=0.7,
|
||||||
|
timeout_s=timeout_s,
|
||||||
)
|
)
|
||||||
text = (result or "").strip()
|
text = (result or "").strip()
|
||||||
if not text:
|
if not text:
|
||||||
|
|||||||
@@ -0,0 +1,131 @@
|
|||||||
|
"""Shared helpers for turn flows (T83.2).
|
||||||
|
|
||||||
|
Both ``chat.web.turns.post_turn`` and
|
||||||
|
``chat.services.regenerate.regenerate_assistant_turn`` need to:
|
||||||
|
|
||||||
|
1. Pull a chronological tail of user-side and assistant_turn events for
|
||||||
|
prompt assembly + state-update inputs.
|
||||||
|
2. Build a directed-edge dict over a fixed set of "present" entity ids
|
||||||
|
for the multi-pair state-update pass (with the schema 50/50 default
|
||||||
|
filled in for missing rows).
|
||||||
|
|
||||||
|
Before T83.2 each call site had its own copy of these blocks. The two
|
||||||
|
copies drifted on details (T73.1 added ``user_turn_edit`` handling to
|
||||||
|
turns.py; regenerate.py had a slightly different recent-window query).
|
||||||
|
This module is the single source so a future change to either lands in
|
||||||
|
both flows by construction.
|
||||||
|
|
||||||
|
Note on overlap with ``chat.services.scene_summarize._read_recent_dialogue``:
|
||||||
|
that helper has a ``since_event_id`` clamp (T80.2 thread-detection
|
||||||
|
scope) and intentionally does NOT include ``user_turn_edit`` events —
|
||||||
|
its callers want the *original* prose, not edits. Deduplicating it
|
||||||
|
into here would either (a) require a new flag on the shared helper for
|
||||||
|
``user_turn_edit`` inclusion, or (b) silently change scene_summarize's
|
||||||
|
read shape. Both feel more invasive than the duplication is bad, so
|
||||||
|
that helper is left alone for now.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from chat.state.edges import get_edge
|
||||||
|
|
||||||
|
|
||||||
|
def read_recent_dialogue(
|
||||||
|
conn: Connection,
|
||||||
|
chat_id: str,
|
||||||
|
*,
|
||||||
|
limit: int = 50,
|
||||||
|
exclude_event_id: int | None = None,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Pull the last ``limit`` user-side / assistant_turn events for
|
||||||
|
``chat_id`` as ``[{"speaker": <id-or-"you">, "text": <prose>}]``,
|
||||||
|
chronologically ordered (oldest first).
|
||||||
|
|
||||||
|
Filters: ``superseded_by IS NULL AND hidden = 0`` — regenerated
|
||||||
|
rows drop out so the timeline reflects the current state. Includes
|
||||||
|
``user_turn``, ``user_turn_edit`` (T29 edited prose substitutes for
|
||||||
|
the original — the original is marked superseded above), and
|
||||||
|
``assistant_turn`` rows.
|
||||||
|
|
||||||
|
``exclude_event_id`` is an optional event_log id to skip — used by
|
||||||
|
regenerate to drop the original assistant_turn from its prompt
|
||||||
|
context window before that row has been marked superseded (the
|
||||||
|
supersede UPDATE lands at the end so the new event_id is known).
|
||||||
|
|
||||||
|
T90.1: the chat_id filter is pushed into SQL via ``json_extract`` so
|
||||||
|
``LIMIT N`` always returns N rows scoped to the requested chat. The
|
||||||
|
previous implementation filtered chat_id post-fetch in Python, which
|
||||||
|
let foreign-chat rows fill the LIMIT and yield fewer than N relevant
|
||||||
|
rows in busy multi-chat databases.
|
||||||
|
"""
|
||||||
|
if exclude_event_id is None:
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT id, kind, payload_json FROM event_log "
|
||||||
|
"WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn') "
|
||||||
|
" AND superseded_by IS NULL AND hidden = 0 "
|
||||||
|
" AND json_extract(payload_json, '$.chat_id') = ? "
|
||||||
|
"ORDER BY id DESC LIMIT ?",
|
||||||
|
(chat_id, limit),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT id, kind, payload_json FROM event_log "
|
||||||
|
"WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn') "
|
||||||
|
" AND id != ? "
|
||||||
|
" AND superseded_by IS NULL AND hidden = 0 "
|
||||||
|
" AND json_extract(payload_json, '$.chat_id') = ? "
|
||||||
|
"ORDER BY id DESC LIMIT ?",
|
||||||
|
(exclude_event_id, chat_id, limit),
|
||||||
|
)
|
||||||
|
rows = list(reversed(cur.fetchall()))
|
||||||
|
out: list[dict] = []
|
||||||
|
for row_id, kind, payload_json in rows:
|
||||||
|
p = json.loads(payload_json)
|
||||||
|
if kind in ("user_turn", "user_turn_edit"):
|
||||||
|
out.append(
|
||||||
|
{
|
||||||
|
"speaker": "you",
|
||||||
|
"text": p.get("prose", ""),
|
||||||
|
"event_id": row_id,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
out.append(
|
||||||
|
{
|
||||||
|
"speaker": p.get("speaker_id", "bot"),
|
||||||
|
"text": p.get("text", ""),
|
||||||
|
"event_id": row_id,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def gather_prior_edges(
|
||||||
|
conn: Connection, present_ids: list[str]
|
||||||
|
) -> dict[tuple[str, str], dict]:
|
||||||
|
"""Build ``{(src, tgt): {affinity, trust, summary}}`` for every
|
||||||
|
directed pair where both ``src`` and ``tgt`` are in ``present_ids``
|
||||||
|
and ``src != tgt``.
|
||||||
|
|
||||||
|
Missing rows fall back to the schema default 50/50 baseline (mirrors
|
||||||
|
the Phase 1 single-pair flow). Used by post_turn and regenerate to
|
||||||
|
seed the multi-pair state-update classifier.
|
||||||
|
"""
|
||||||
|
prior_edges: dict[tuple[str, str], dict] = {}
|
||||||
|
for src in present_ids:
|
||||||
|
for tgt in present_ids:
|
||||||
|
if src == tgt:
|
||||||
|
continue
|
||||||
|
edge = get_edge(conn, src, tgt) or {
|
||||||
|
"affinity": 50,
|
||||||
|
"trust": 50,
|
||||||
|
"summary": "",
|
||||||
|
}
|
||||||
|
prior_edges[(src, tgt)] = edge
|
||||||
|
return prior_edges
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["read_recent_dialogue", "gather_prior_edges"]
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
"""Vector search service (T92, Phase 4).
|
||||||
|
|
||||||
|
Pure-Python cosine similarity over the embeddings table. Phase 4 ships
|
||||||
|
this without sqlite-vec because the host Python build doesn't support
|
||||||
|
loadable extensions. For single-user scale (< few thousand memories
|
||||||
|
per owner), iterating in Python is sub-millisecond.
|
||||||
|
|
||||||
|
Phase 4.5+ may swap to sqlite-vec when the host Python supports
|
||||||
|
enable_load_extension; the public API stays stable.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import math
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from chat.state.embeddings import list_embeddings_for_owner
|
||||||
|
|
||||||
|
|
||||||
|
_VALID_WITNESS_ROLES = {"you", "host", "guest"}
|
||||||
|
|
||||||
|
|
||||||
|
def _cosine_similarity(a: list[float], b: list[float]) -> float:
|
||||||
|
"""Cosine similarity. Assumes both vectors are non-zero."""
|
||||||
|
if len(a) != len(b):
|
||||||
|
return 0.0
|
||||||
|
dot = sum(x * y for x, y in zip(a, b))
|
||||||
|
norm_a = math.sqrt(sum(x * x for x in a)) or 1.0
|
||||||
|
norm_b = math.sqrt(sum(x * x for x in b)) or 1.0
|
||||||
|
return dot / (norm_a * norm_b)
|
||||||
|
|
||||||
|
|
||||||
|
def vector_search(
|
||||||
|
conn: Connection,
|
||||||
|
*,
|
||||||
|
owner_id: str,
|
||||||
|
witness_role: str, # "you" | "host" | "guest"
|
||||||
|
query_vector: list[float],
|
||||||
|
k: int = 4,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Return top-K memories by cosine similarity to query_vector,
|
||||||
|
witness-filtered for the viewer's POV. Returns rows with
|
||||||
|
{memory_id, pov_summary, significance, score} sorted by score
|
||||||
|
DESC. Empty list if no embeddings indexed for this owner.
|
||||||
|
"""
|
||||||
|
if witness_role not in _VALID_WITNESS_ROLES:
|
||||||
|
raise ValueError(
|
||||||
|
f"witness_role must be one of {_VALID_WITNESS_ROLES}, got {witness_role!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = list_embeddings_for_owner(conn, owner_id)
|
||||||
|
if not rows:
|
||||||
|
return []
|
||||||
|
|
||||||
|
# Witness-filter by the requesting role.
|
||||||
|
witness_key = f"witness_{witness_role}"
|
||||||
|
filtered = [r for r in rows if r.get(witness_key) == 1]
|
||||||
|
if not filtered:
|
||||||
|
return []
|
||||||
|
|
||||||
|
scored: list[tuple[float, dict]] = []
|
||||||
|
for row in filtered:
|
||||||
|
score = _cosine_similarity(query_vector, row["vector"])
|
||||||
|
scored.append(
|
||||||
|
(
|
||||||
|
score,
|
||||||
|
{
|
||||||
|
"memory_id": row["memory_id"],
|
||||||
|
"pov_summary": row["pov_summary"],
|
||||||
|
"significance": row["significance"],
|
||||||
|
"score": score,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
scored.sort(key=lambda t: t[0], reverse=True)
|
||||||
|
return [item for _, item in scored[:k]]
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["vector_search"]
|
||||||
@@ -0,0 +1,164 @@
|
|||||||
|
"""Branches projector + readers (T89, Phase 4).
|
||||||
|
|
||||||
|
A branch is a named fork of the event log. The 'main' branch is bootstrapped
|
||||||
|
by migration 0013 with is_active=1. Subsequent branches reference an
|
||||||
|
origin_event_id (the event they forked from). Phase 4 enables creation
|
||||||
|
and switching; the read-side filter (event readers consulting is_active)
|
||||||
|
is a Phase 4.5 follow-up — for now branches are metadata-only and the
|
||||||
|
existing event readers remain branch-agnostic.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from chat.eventlog.projector import on
|
||||||
|
from chat.eventlog.log import Event
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@on("branch_created")
|
||||||
|
def _apply_branch_created(conn: Connection, e: Event) -> None:
|
||||||
|
"""Insert a new branch row with is_active=0. Idempotent via INSERT OR IGNORE."""
|
||||||
|
p = e.payload
|
||||||
|
conn.execute(
|
||||||
|
"INSERT OR IGNORE INTO branches "
|
||||||
|
"(name, origin_event_id, head_event_id, chat_id, is_active) "
|
||||||
|
"VALUES (?, ?, ?, ?, 0)",
|
||||||
|
(
|
||||||
|
p["name"],
|
||||||
|
int(p["origin_event_id"]),
|
||||||
|
int(p.get("head_event_id", p["origin_event_id"])),
|
||||||
|
p.get("chat_id"),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@on("branch_switched")
|
||||||
|
def _apply_branch_switched(conn: Connection, e: Event) -> None:
|
||||||
|
"""Set is_active=1 on the named branch and is_active=0 on all others.
|
||||||
|
|
||||||
|
Atomic via two UPDATEs ordered to avoid the unique-active-index race.
|
||||||
|
|
||||||
|
If the named branch does not exist, a warning is emitted and the
|
||||||
|
is_active flags are still cleared (preserving prior behavior — the
|
||||||
|
second UPDATE simply matches no rows). Callers should validate the
|
||||||
|
name upstream; this guard surfaces accidental mismatches in the log.
|
||||||
|
"""
|
||||||
|
p = e.payload
|
||||||
|
name = p["name"]
|
||||||
|
# Warn (don't raise) if the target branch is missing. The existing
|
||||||
|
# outcome — zero active branches — is preserved; this just makes the
|
||||||
|
# condition observable instead of silent.
|
||||||
|
exists = conn.execute(
|
||||||
|
"SELECT 1 FROM branches WHERE name = ? LIMIT 1",
|
||||||
|
(name,),
|
||||||
|
).fetchone()
|
||||||
|
if exists is None:
|
||||||
|
logger.warning(
|
||||||
|
"branch_switched to unknown branch name %r; no branch will be active",
|
||||||
|
name,
|
||||||
|
)
|
||||||
|
# Clear ALL is_active flags first (avoids the unique-index trip).
|
||||||
|
conn.execute("UPDATE branches SET is_active = 0 WHERE is_active = 1")
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE branches SET is_active = 1 WHERE name = ?",
|
||||||
|
(name,),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@on("branch_head_updated")
|
||||||
|
def _apply_branch_head_updated(conn: Connection, e: Event) -> None:
|
||||||
|
"""Update head_event_id on the named branch."""
|
||||||
|
p = e.payload
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE branches SET head_event_id = ? WHERE name = ?",
|
||||||
|
(int(p["head_event_id"]), p["name"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_branch(conn: Connection, name: str) -> dict | None:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT id, name, origin_event_id, head_event_id, chat_id, "
|
||||||
|
" created_at, is_active "
|
||||||
|
"FROM branches WHERE name = ?",
|
||||||
|
(name,),
|
||||||
|
).fetchone()
|
||||||
|
if not row:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"id": row[0],
|
||||||
|
"name": row[1],
|
||||||
|
"origin_event_id": row[2],
|
||||||
|
"head_event_id": row[3],
|
||||||
|
"chat_id": row[4],
|
||||||
|
"created_at": row[5],
|
||||||
|
"is_active": bool(row[6]),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def list_branches(conn: Connection, chat_id: str | None = None) -> list[dict]:
|
||||||
|
"""Return branch rows, optionally scoped to a chat.
|
||||||
|
|
||||||
|
When ``chat_id`` is provided the filter is ``chat_id = ? OR chat_id IS NULL``,
|
||||||
|
so global (null-chat) branches are returned in *every* per-chat scope. This
|
||||||
|
is intentional: the bootstrapped ``"main"`` branch (and any future
|
||||||
|
null-chat branches) are global by design — they belong to no single chat
|
||||||
|
and should appear alongside per-chat branches in any chat-scoped listing.
|
||||||
|
Callers that want only per-chat branches should filter the result on
|
||||||
|
``chat_id is not None``.
|
||||||
|
"""
|
||||||
|
if chat_id is None:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, name, origin_event_id, head_event_id, chat_id, "
|
||||||
|
" created_at, is_active "
|
||||||
|
"FROM branches ORDER BY id ASC"
|
||||||
|
).fetchall()
|
||||||
|
else:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, name, origin_event_id, head_event_id, chat_id, "
|
||||||
|
" created_at, is_active "
|
||||||
|
"FROM branches WHERE chat_id = ? OR chat_id IS NULL "
|
||||||
|
"ORDER BY id ASC",
|
||||||
|
(chat_id,),
|
||||||
|
).fetchall()
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"id": r[0],
|
||||||
|
"name": r[1],
|
||||||
|
"origin_event_id": r[2],
|
||||||
|
"head_event_id": r[3],
|
||||||
|
"chat_id": r[4],
|
||||||
|
"created_at": r[5],
|
||||||
|
"is_active": bool(r[6]),
|
||||||
|
}
|
||||||
|
for r in rows
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def active_branch(conn: Connection) -> dict | None:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT id, name, origin_event_id, head_event_id, chat_id, "
|
||||||
|
" created_at, is_active "
|
||||||
|
"FROM branches WHERE is_active = 1"
|
||||||
|
).fetchone()
|
||||||
|
if not row:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"id": row[0],
|
||||||
|
"name": row[1],
|
||||||
|
"origin_event_id": row[2],
|
||||||
|
"head_event_id": row[3],
|
||||||
|
"chat_id": row[4],
|
||||||
|
"created_at": row[5],
|
||||||
|
"is_active": bool(row[6]),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"get_branch",
|
||||||
|
"list_branches",
|
||||||
|
"active_branch",
|
||||||
|
]
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
"""Embeddings projector + readers (T88, Phase 4).
|
||||||
|
|
||||||
|
Embeddings are stored as JSON-serialized float arrays in a regular
|
||||||
|
SQLite table. Cosine similarity is computed in Python at query time
|
||||||
|
(see chat/services/vector_search.py / T92). This deliberately avoids
|
||||||
|
the sqlite-vec extension dependency — the host Python build doesn't
|
||||||
|
support enable_load_extension. Phase 4.5+ may revisit if memory counts
|
||||||
|
grow beyond pure-Python feasibility (~few thousand per query).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
import json
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from chat.eventlog.projector import on
|
||||||
|
from chat.eventlog.log import Event
|
||||||
|
|
||||||
|
|
||||||
|
@on("embedding_indexed")
|
||||||
|
def _apply_embedding_indexed(conn: Connection, e: Event) -> None:
|
||||||
|
"""Insert or replace the embedding for a memory.
|
||||||
|
|
||||||
|
Idempotent: re-projection or re-indexing replaces the prior vector.
|
||||||
|
"""
|
||||||
|
p = e.payload
|
||||||
|
vector = p["vector"]
|
||||||
|
conn.execute(
|
||||||
|
"INSERT OR REPLACE INTO embeddings "
|
||||||
|
"(memory_id, vector_json, model, dim, indexed_at) "
|
||||||
|
"VALUES (?, ?, ?, ?, datetime('now'))",
|
||||||
|
(
|
||||||
|
int(p["memory_id"]),
|
||||||
|
json.dumps(list(vector)),
|
||||||
|
p["model"],
|
||||||
|
int(p.get("dim") or len(vector)),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@on("embedding_deindexed")
|
||||||
|
def _apply_embedding_deindexed(conn: Connection, e: Event) -> None:
|
||||||
|
"""Remove the embedding for a memory (used by reset cascade)."""
|
||||||
|
p = e.payload
|
||||||
|
conn.execute(
|
||||||
|
"DELETE FROM embeddings WHERE memory_id = ?",
|
||||||
|
(int(p["memory_id"]),),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_embedding(conn: Connection, memory_id: int) -> dict | None:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT memory_id, vector_json, model, dim, indexed_at "
|
||||||
|
"FROM embeddings WHERE memory_id = ?",
|
||||||
|
(memory_id,),
|
||||||
|
).fetchone()
|
||||||
|
if not row:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"memory_id": row[0],
|
||||||
|
"vector": json.loads(row[1]),
|
||||||
|
"model": row[2],
|
||||||
|
"dim": row[3],
|
||||||
|
"indexed_at": row[4],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def list_embeddings_for_owner(conn: Connection, owner_id: str) -> list[dict]:
|
||||||
|
"""Return all embeddings for memories owned by ``owner_id``.
|
||||||
|
|
||||||
|
Used by vector search at query time (T92). The join carries the
|
||||||
|
fields the cosine ranker needs to assemble result rows without a
|
||||||
|
second round-trip: the POV summary text, significance, and witness
|
||||||
|
flags. The ``memories`` table has no separate ``text`` column —
|
||||||
|
``pov_summary`` is the canonical narrative text per
|
||||||
|
``chat/services/memory_write.py``.
|
||||||
|
"""
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT e.memory_id, e.vector_json, e.model, e.dim, "
|
||||||
|
" m.pov_summary, m.significance, "
|
||||||
|
" m.witness_you, m.witness_host, m.witness_guest "
|
||||||
|
"FROM embeddings e "
|
||||||
|
"JOIN memories m ON m.id = e.memory_id "
|
||||||
|
"WHERE m.owner_id = ?",
|
||||||
|
(owner_id,),
|
||||||
|
).fetchall()
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"memory_id": r[0],
|
||||||
|
"vector": json.loads(r[1]),
|
||||||
|
"model": r[2],
|
||||||
|
"dim": r[3],
|
||||||
|
"pov_summary": r[4],
|
||||||
|
"significance": r[5],
|
||||||
|
"witness_you": r[6],
|
||||||
|
"witness_host": r[7],
|
||||||
|
"witness_guest": r[8],
|
||||||
|
}
|
||||||
|
for r in rows
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"get_embedding",
|
||||||
|
"list_embeddings_for_owner",
|
||||||
|
]
|
||||||
@@ -30,6 +30,20 @@ T72.3 adds a per-flag witness toggle:
|
|||||||
``{"flag": "you"|"host"|"guest", "value": 0|1}`` and ``prior_value``
|
``{"flag": "you"|"host"|"guest", "value": 0|1}`` and ``prior_value``
|
||||||
mirrors the same shape so an inverse edit can restore the flag.
|
mirrors the same shape so an inverse edit can restore the flag.
|
||||||
|
|
||||||
|
T98.3 adds a hide-from-view toggle:
|
||||||
|
- ``turn_hidden`` — flip ``event_log.hidden`` on a single turn row.
|
||||||
|
Hidden turns are filtered by ``read_recent_dialogue`` (see
|
||||||
|
:mod:`chat.services.turn_common`) so they vanish from the prompt
|
||||||
|
without being deleted from the log. ``target_id`` is the integer
|
||||||
|
``event_log.id`` of the turn; ``new_value`` is ``{"hidden": 0|1}``
|
||||||
|
and ``prior_value`` mirrors the shape so an inverse edit restores it.
|
||||||
|
|
||||||
|
T98.5 finishes the v1 drawer surface with two chat-scope text edits:
|
||||||
|
- ``chat_narrative_anchor`` and ``chat_weather`` — string overwrites of
|
||||||
|
the matching ``chat_state`` columns. ``target_id`` is the chat id
|
||||||
|
(``chats.id``); ``new_value`` is the new string and ``prior_value``
|
||||||
|
carries the previous content for §6.4 reversibility.
|
||||||
|
|
||||||
Pin toggles intentionally use the existing ``memory_pin_changed`` event
|
Pin toggles intentionally use the existing ``memory_pin_changed`` event
|
||||||
(registered in :mod:`chat.state.memory`) rather than ``manual_edit`` so
|
(registered in :mod:`chat.state.memory`) rather than ``manual_edit`` so
|
||||||
the projection writes both ``pinned`` and ``auto_pinned`` atomically.
|
the projection writes both ``pinned`` and ``auto_pinned`` atomically.
|
||||||
@@ -138,5 +152,29 @@ def _apply_manual_edit(conn: Connection, e: Event) -> None:
|
|||||||
f"UPDATE memories SET witness_{flag} = ? WHERE id = ?",
|
f"UPDATE memories SET witness_{flag} = ? WHERE id = ?",
|
||||||
(1 if int(new_value["value"]) else 0, int(target_id)),
|
(1 if int(new_value["value"]) else 0, int(target_id)),
|
||||||
)
|
)
|
||||||
|
elif kind == "turn_hidden":
|
||||||
|
# T98.3: hide-from-view toggle on a turn (event_log row). Sets
|
||||||
|
# ``event_log.hidden`` so :func:`read_recent_dialogue` (which
|
||||||
|
# filters ``hidden = 0``) drops the row from the prompt window
|
||||||
|
# without deleting it from the log. ``new_value`` is
|
||||||
|
# ``{"hidden": 0|1}``.
|
||||||
|
hidden_int = 1 if int(new_value.get("hidden", 0)) else 0
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE event_log SET hidden = ? WHERE id = ?",
|
||||||
|
(hidden_int, int(target_id)),
|
||||||
|
)
|
||||||
|
elif kind == "chat_narrative_anchor":
|
||||||
|
# T98.5: string overwrite of ``chat_state.narrative_anchor`` for
|
||||||
|
# the chat keyed by ``target_id``.
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE chat_state SET narrative_anchor = ? WHERE chat_id = ?",
|
||||||
|
(str(new_value), str(target_id)),
|
||||||
|
)
|
||||||
|
elif kind == "chat_weather":
|
||||||
|
# T98.5: string overwrite of ``chat_state.weather``.
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE chat_state SET weather = ? WHERE chat_id = ?",
|
||||||
|
(str(new_value), str(target_id)),
|
||||||
|
)
|
||||||
# Unknown target_kind: silently no-op for v1. Future kinds (activity
|
# Unknown target_kind: silently no-op for v1. Future kinds (activity
|
||||||
# fields, etc.) extend the dispatch above.
|
# fields, etc.) extend the dispatch above.
|
||||||
|
|||||||
+231
-11
@@ -13,13 +13,18 @@ def _row_to_dict(conn: Connection, row: tuple) -> dict:
|
|||||||
|
|
||||||
@on("memory_written")
|
@on("memory_written")
|
||||||
def _apply_memory_written(conn: Connection, e: Event) -> None:
|
def _apply_memory_written(conn: Connection, e: Event) -> None:
|
||||||
|
# T109 (schema 0014): persist the projecting event's id on the memory
|
||||||
|
# row so cross-chat search results can deep-link back to the
|
||||||
|
# originating turn (T111). Older memory rows projected before 0014
|
||||||
|
# ran read NULL here — the column is nullable for that reason.
|
||||||
p = e.payload
|
p = e.payload
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO memories ("
|
"INSERT INTO memories ("
|
||||||
"owner_id, chat_id, scene_id, pov_summary, "
|
"owner_id, chat_id, scene_id, pov_summary, "
|
||||||
"witness_you, witness_host, witness_guest, "
|
"witness_you, witness_host, witness_guest, "
|
||||||
"chat_clock_at, source, reliability, significance, pinned, auto_pinned"
|
"chat_clock_at, source, reliability, significance, pinned, auto_pinned, "
|
||||||
") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
"event_id"
|
||||||
|
") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||||
(
|
(
|
||||||
p["owner_id"],
|
p["owner_id"],
|
||||||
p["chat_id"],
|
p["chat_id"],
|
||||||
@@ -34,6 +39,7 @@ def _apply_memory_written(conn: Connection, e: Event) -> None:
|
|||||||
int(p.get("significance", 1)),
|
int(p.get("significance", 1)),
|
||||||
int(p.get("pinned", 0)),
|
int(p.get("pinned", 0)),
|
||||||
int(p.get("auto_pinned", 0)),
|
int(p.get("auto_pinned", 0)),
|
||||||
|
e.id,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -102,6 +108,34 @@ _RECENCY_WEIGHT = 0.5
|
|||||||
# a higher-is-better score by a positive constant per the spec wording.
|
# a higher-is-better score by a positive constant per the spec wording.
|
||||||
SIGNIFICANCE_RANK_BIAS = 0.5
|
SIGNIFICANCE_RANK_BIAS = 0.5
|
||||||
|
|
||||||
|
# T96 (Phase 4): reciprocal-rank-fusion constant used when ``search_memories``
|
||||||
|
# is given a ``query_vector`` and must merge FTS + vector candidate lists. The
|
||||||
|
# value 60 is the canonical RRF constant from Cormack et al. ("Reciprocal Rank
|
||||||
|
# Fusion outperforms Condorcet and Individual Rank Learning Methods", SIGIR
|
||||||
|
# 2009): large enough to dampen the head of either ranking so that a strong
|
||||||
|
# top-1 in ranking A doesn't crowd out a moderate top-3 in ranking B, but
|
||||||
|
# small enough that the position-1/position-2 gap still matters.
|
||||||
|
RRF_CONST = 60
|
||||||
|
|
||||||
|
|
||||||
|
def _max_event_id(conn: Connection, owner_id: str) -> int:
|
||||||
|
"""Return the largest ``memories.id`` for ``owner_id`` (1 if none exist).
|
||||||
|
|
||||||
|
Used as the recency-boost denominator by both ``_composite_rerank`` and
|
||||||
|
``_rrf_fuse_and_rerank`` (T104). The row id is a monotonic recency proxy
|
||||||
|
— newer memories have larger ids — so dividing by the per-owner max keeps
|
||||||
|
the boost in [0, 1] regardless of how many memories the owner has.
|
||||||
|
|
||||||
|
Returns 1 (not 0) when the owner has no rows so callers can divide by
|
||||||
|
the result without a guard. The "no memories" case never actually hits
|
||||||
|
this helper because the FTS query above would have returned no rows,
|
||||||
|
but the safe default keeps the helper trivially reusable.
|
||||||
|
"""
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT MAX(id) FROM memories WHERE owner_id = ?", (owner_id,)
|
||||||
|
).fetchone()
|
||||||
|
return row[0] if row and row[0] else 1
|
||||||
|
|
||||||
|
|
||||||
def search_memories(
|
def search_memories(
|
||||||
conn: Connection,
|
conn: Connection,
|
||||||
@@ -109,6 +143,8 @@ def search_memories(
|
|||||||
witness_role: str,
|
witness_role: str,
|
||||||
query: str,
|
query: str,
|
||||||
k: int = 4,
|
k: int = 4,
|
||||||
|
*,
|
||||||
|
query_vector: list[float] | None = None,
|
||||||
) -> list[dict]:
|
) -> list[dict]:
|
||||||
"""FTS5 search over pov_summary, scoped by owner and witness role.
|
"""FTS5 search over pov_summary, scoped by owner and witness role.
|
||||||
|
|
||||||
@@ -125,6 +161,41 @@ def search_memories(
|
|||||||
so that stronger candidates yield smaller composite scores; the result is
|
so that stronger candidates yield smaller composite scores; the result is
|
||||||
sorted ascending and truncated to ``k``. The unmodified ``fts_rank`` and a
|
sorted ascending and truncated to ``k``. The unmodified ``fts_rank`` and a
|
||||||
debug-friendly ``composite_score`` are kept on each returned dict.
|
debug-friendly ``composite_score`` are kept on each returned dict.
|
||||||
|
|
||||||
|
The result ordering applies TWO independent significance boosts:
|
||||||
|
|
||||||
|
* **SQL-side** — ``ORDER BY (rank - significance * SIGNIFICANCE_RANK_BIAS)``
|
||||||
|
pushes higher-significance memories ahead in the FTS5 candidate set so
|
||||||
|
the over-fetch already prefers them for tied / near-tied BM25 ranks
|
||||||
|
(T57, §11.1).
|
||||||
|
* **Python-side** — a composite re-rank with ``_SIGNIFICANCE_WEIGHT``
|
||||||
|
reinforces the ordering after candidate retrieval, alongside the
|
||||||
|
recency boost above.
|
||||||
|
|
||||||
|
PHASE 4 EXTENSION (T96): when ``query_vector`` is provided, fuses FTS and
|
||||||
|
vector hits via reciprocal-rank fusion (RRF):
|
||||||
|
|
||||||
|
fusion_score = 1/(RRF_CONST + fts_rank) + 1/(RRF_CONST + vec_rank)
|
||||||
|
|
||||||
|
where ``fts_rank`` and ``vec_rank`` are the 0-indexed positions of the
|
||||||
|
memory in each candidate list. Each candidate gets the sum of its
|
||||||
|
reciprocal ranks across both rankings; memories appearing in only one
|
||||||
|
ranking still get a partial score (the other term is dropped). Both
|
||||||
|
candidate lists are over-fetched at ``k * 2`` so a memory dominant in
|
||||||
|
only one channel has a fair chance to surface. The Python-side
|
||||||
|
significance + recency re-rank is then applied as a final pass to
|
||||||
|
break ties in favour of more important / more recent memories.
|
||||||
|
|
||||||
|
When ``query_vector`` is None: FTS-only behaviour unchanged — all
|
||||||
|
Phase 1-3.5 callers see the same row shape and ordering as before.
|
||||||
|
|
||||||
|
**Row-shape contract (T104):** every returned dict carries an
|
||||||
|
``fts_rank`` key. For FTS hits this is the BM25 score (a negative float,
|
||||||
|
lower-is-better). For *vector-only* hits surfaced by the fused path —
|
||||||
|
rows that matched the query embedding but did NOT match FTS — the
|
||||||
|
``fts_rank`` value is ``None``. Downstream consumers must accept
|
||||||
|
``None`` here; do not assume ``fts_rank`` is always numeric. The
|
||||||
|
``composite_score`` is always a float on every returned row.
|
||||||
"""
|
"""
|
||||||
if witness_role not in _VALID_WITNESS_ROLES:
|
if witness_role not in _VALID_WITNESS_ROLES:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
@@ -138,7 +209,10 @@ def search_memories(
|
|||||||
select_list = ", ".join(f"m.{c}" for c in cols)
|
select_list = ", ".join(f"m.{c}" for c in cols)
|
||||||
# Over-fetch from FTS so the Python-side re-rank has room to reorder
|
# Over-fetch from FTS so the Python-side re-rank has room to reorder
|
||||||
# results that BM25 alone would have demoted past the top-k boundary.
|
# results that BM25 alone would have demoted past the top-k boundary.
|
||||||
over_fetch = max(k * 4, 20)
|
# When fusing with a vector ranking, we still over-fetch (k*2 from each
|
||||||
|
# channel) so memories that are weak in FTS but strong in vector — and
|
||||||
|
# vice versa — make it into the merge pool.
|
||||||
|
over_fetch = max(k * 2, 20) if query_vector is not None else max(k * 4, 20)
|
||||||
sql = (
|
sql = (
|
||||||
f"SELECT {select_list}, memories_fts.rank AS fts_rank "
|
f"SELECT {select_list}, memories_fts.rank AS fts_rank "
|
||||||
"FROM memories_fts "
|
"FROM memories_fts "
|
||||||
@@ -155,15 +229,38 @@ def search_memories(
|
|||||||
)
|
)
|
||||||
cur = conn.execute(sql, (owner_id, query, SIGNIFICANCE_RANK_BIAS, over_fetch))
|
cur = conn.execute(sql, (owner_id, query, SIGNIFICANCE_RANK_BIAS, over_fetch))
|
||||||
rows = cur.fetchall()
|
rows = cur.fetchall()
|
||||||
if not rows:
|
|
||||||
return []
|
|
||||||
|
|
||||||
# Recency normalises against the current max id for this owner so the
|
# FTS-only path: preserve pre-T96 behaviour exactly.
|
||||||
# boost magnitude is bounded regardless of dataset size.
|
if query_vector is None:
|
||||||
max_id_row = conn.execute(
|
if not rows:
|
||||||
"SELECT MAX(id) FROM memories WHERE owner_id = ?", (owner_id,)
|
return []
|
||||||
).fetchone()
|
return _composite_rerank(conn, cols, rows, owner_id, k)
|
||||||
max_id = max_id_row[0] if max_id_row and max_id_row[0] else 1
|
|
||||||
|
# Fused path: combine FTS candidates with vector candidates via RRF.
|
||||||
|
return _rrf_fuse_and_rerank(
|
||||||
|
conn,
|
||||||
|
cols=cols,
|
||||||
|
fts_rows=rows,
|
||||||
|
owner_id=owner_id,
|
||||||
|
witness_role=witness_role,
|
||||||
|
query_vector=query_vector,
|
||||||
|
k=k,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _composite_rerank(
|
||||||
|
conn: Connection,
|
||||||
|
cols: list[str],
|
||||||
|
rows: list[tuple],
|
||||||
|
owner_id: str,
|
||||||
|
k: int,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Apply the significance + recency composite re-rank to FTS rows.
|
||||||
|
|
||||||
|
Extracted from ``search_memories`` so the no-vector path stays a single
|
||||||
|
call and the fused path can re-use the same boost formulae after RRF.
|
||||||
|
"""
|
||||||
|
max_id = _max_event_id(conn, owner_id)
|
||||||
|
|
||||||
result_cols = cols + ["fts_rank"]
|
result_cols = cols + ["fts_rank"]
|
||||||
enriched: list[dict] = []
|
enriched: list[dict] = []
|
||||||
@@ -177,3 +274,126 @@ def search_memories(
|
|||||||
|
|
||||||
enriched.sort(key=lambda x: x["composite_score"])
|
enriched.sort(key=lambda x: x["composite_score"])
|
||||||
return enriched[:k]
|
return enriched[:k]
|
||||||
|
|
||||||
|
|
||||||
|
def _rrf_fuse_and_rerank(
|
||||||
|
conn: Connection,
|
||||||
|
*,
|
||||||
|
cols: list[str],
|
||||||
|
fts_rows: list[tuple],
|
||||||
|
owner_id: str,
|
||||||
|
witness_role: str,
|
||||||
|
query_vector: list[float],
|
||||||
|
k: int,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Merge FTS + vector candidates via reciprocal-rank fusion, then apply
|
||||||
|
the existing significance + recency boost as a final tie-breaker.
|
||||||
|
|
||||||
|
RRF formula (Cormack et al. 2009)::
|
||||||
|
|
||||||
|
fusion_score = sum over rankings r of 1 / (RRF_CONST + rank_r)
|
||||||
|
|
||||||
|
where ``rank_r`` is the 0-indexed position of the memory in ranking r.
|
||||||
|
"Missing from a ranking" is handled by SKIPPING the term for that
|
||||||
|
ranking — i.e. that channel contributes 0 to the sum, which preserves
|
||||||
|
the fairness property: a memory that only appears in one ranking is
|
||||||
|
not penalised relative to itself, just relative to memories that
|
||||||
|
appeared in both. This matches the canonical RRF presentation.
|
||||||
|
|
||||||
|
The final composite score subtracted from the *negated* fusion score
|
||||||
|
is::
|
||||||
|
|
||||||
|
composite = -fusion - sig_boost - recency_boost
|
||||||
|
|
||||||
|
Sorted ascending, smaller-is-better — the same ordering convention as
|
||||||
|
the FTS-only path so the Python-side significance + recency boosts
|
||||||
|
apply as a clean tie-breaker without inverting any sign.
|
||||||
|
"""
|
||||||
|
# Lazy import to avoid a hard module-level cycle: vector_search reads
|
||||||
|
# from chat.state.embeddings, which is itself a sibling of this module.
|
||||||
|
from chat.services.vector_search import vector_search
|
||||||
|
|
||||||
|
fts_rank_by_id: dict[int, int] = {}
|
||||||
|
fts_row_by_id: dict[int, tuple] = {}
|
||||||
|
id_idx = cols.index("id")
|
||||||
|
for rank, row in enumerate(fts_rows):
|
||||||
|
memory_id = row[id_idx]
|
||||||
|
fts_rank_by_id[memory_id] = rank
|
||||||
|
fts_row_by_id[memory_id] = row
|
||||||
|
|
||||||
|
# Over-fetch the vector channel symmetrically so each channel gets a
|
||||||
|
# fair shot at surfacing its strongest candidates.
|
||||||
|
vec_over_fetch = max(k * 2, 20)
|
||||||
|
vec_hits = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id=owner_id,
|
||||||
|
witness_role=witness_role,
|
||||||
|
query_vector=query_vector,
|
||||||
|
k=vec_over_fetch,
|
||||||
|
)
|
||||||
|
vec_rank_by_id: dict[int, int] = {
|
||||||
|
hit["memory_id"]: rank for rank, hit in enumerate(vec_hits)
|
||||||
|
}
|
||||||
|
|
||||||
|
# If the vector channel returned nothing (no embeddings indexed), the
|
||||||
|
# fused path collapses cleanly to the FTS-only path. No error, no
|
||||||
|
# surprise zero-hit return.
|
||||||
|
if not vec_rank_by_id and not fts_row_by_id:
|
||||||
|
return []
|
||||||
|
if not vec_rank_by_id:
|
||||||
|
return _composite_rerank(conn, cols, fts_rows, owner_id, k)
|
||||||
|
|
||||||
|
# For any vector-only hits we don't have a full memory row for yet,
|
||||||
|
# fetch them in a single round-trip. The FTS row carries an ``fts_rank``
|
||||||
|
# column at the end; vector-only rows get ``None`` there.
|
||||||
|
missing_ids = [mid for mid in vec_rank_by_id if mid not in fts_row_by_id]
|
||||||
|
select_list = ", ".join(cols)
|
||||||
|
if missing_ids:
|
||||||
|
placeholders = ",".join("?" * len(missing_ids))
|
||||||
|
cur = conn.execute(
|
||||||
|
f"SELECT {select_list} FROM memories WHERE id IN ({placeholders})",
|
||||||
|
missing_ids,
|
||||||
|
)
|
||||||
|
for row in cur.fetchall():
|
||||||
|
# Pad with a None for the trailing ``fts_rank`` slot so the row
|
||||||
|
# shape matches FTS rows downstream.
|
||||||
|
fts_row_by_id[row[id_idx]] = tuple(row) + (None,)
|
||||||
|
|
||||||
|
# Compute fusion score per candidate. Missing-from-ranking terms are
|
||||||
|
# simply omitted from the sum.
|
||||||
|
all_ids = set(fts_rank_by_id) | set(vec_rank_by_id)
|
||||||
|
fusion_by_id: dict[int, float] = {}
|
||||||
|
for mid in all_ids:
|
||||||
|
score = 0.0
|
||||||
|
if mid in fts_rank_by_id:
|
||||||
|
score += 1.0 / (RRF_CONST + fts_rank_by_id[mid])
|
||||||
|
if mid in vec_rank_by_id:
|
||||||
|
score += 1.0 / (RRF_CONST + vec_rank_by_id[mid])
|
||||||
|
fusion_by_id[mid] = score
|
||||||
|
|
||||||
|
# Final composite re-rank: significance + recency boosts on top of the
|
||||||
|
# negated fusion score so the sort direction matches the FTS-only path.
|
||||||
|
max_id = _max_event_id(conn, owner_id)
|
||||||
|
|
||||||
|
result_cols = cols + ["fts_rank"]
|
||||||
|
enriched: list[dict] = []
|
||||||
|
for mid in all_ids:
|
||||||
|
row = fts_row_by_id.get(mid)
|
||||||
|
if row is None:
|
||||||
|
# Defensive: a vector hit with no memory row would be a logic
|
||||||
|
# bug (vector_search joins memories), so just skip it rather
|
||||||
|
# than crash the whole search.
|
||||||
|
continue
|
||||||
|
d = dict(zip(result_cols, row))
|
||||||
|
sig_boost = _SIGNIFICANCE_WEIGHT * (d.get("significance") or 0)
|
||||||
|
recency_boost = _RECENCY_WEIGHT * ((d.get("id") or 0) / max_id)
|
||||||
|
fusion = fusion_by_id[mid]
|
||||||
|
# Sort ascending, smaller-is-better → negate fusion so a larger
|
||||||
|
# fusion score yields a smaller composite. Significance and recency
|
||||||
|
# boosts then act as tie-breakers exactly like the FTS-only path.
|
||||||
|
d["fusion_score"] = fusion
|
||||||
|
d["composite_score"] = -fusion - sig_boost - recency_boost
|
||||||
|
enriched.append(d)
|
||||||
|
|
||||||
|
enriched.sort(key=lambda x: x["composite_score"])
|
||||||
|
return enriched[:k]
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
{# T110.3: delete-impact modal partial.
|
||||||
|
|
||||||
|
Rendered from :func:`chat.web.drawer.delete_preview` via a Jinja2
|
||||||
|
TemplateResponse so HTML autoescape covers user-controllable fields
|
||||||
|
(item.kind, item.description, notes) automatically — the prior
|
||||||
|
f-string assembly required explicit html.escape() calls (T110.2)
|
||||||
|
which become redundant under autoescape.
|
||||||
|
|
||||||
|
Inputs:
|
||||||
|
``chat_id`` — the URL chat id (used to build the confirm form action).
|
||||||
|
``impact`` — an :class:`~chat.services.delete_impact.ImpactReport`.
|
||||||
|
#}
|
||||||
|
<div class="delete-impact-modal">
|
||||||
|
<h3>Delete event {{ impact.target_event_id }}?</h3>
|
||||||
|
<p>This will discard {{ impact.cascading|length }} events. Cascade:</p>
|
||||||
|
<ul class="delete-impact-cascade">
|
||||||
|
{% if impact.cascading %}
|
||||||
|
{% for item in impact.cascading %}
|
||||||
|
<li><strong>{{ item.kind }}</strong>: {{ item.description }}</li>
|
||||||
|
{% endfor %}
|
||||||
|
{% else %}
|
||||||
|
<li>none</li>
|
||||||
|
{% endif %}
|
||||||
|
</ul>
|
||||||
|
<ul class="delete-impact-notes">
|
||||||
|
{% for note in impact.notes %}
|
||||||
|
<li>{{ note }}</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
<form hx-post="/chats/{{ chat_id }}/drawer/turn/delete/{{ impact.target_event_id }}"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<button type="submit">Confirm delete</button>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
@@ -16,6 +16,26 @@
|
|||||||
<p class="muted">No active container.</p>
|
<p class="muted">No active container.</p>
|
||||||
{% endif %}
|
{% endif %}
|
||||||
<p>Time: {{ chat.time }}</p>
|
<p>Time: {{ chat.time }}</p>
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/chat/narrative-anchor"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<label>
|
||||||
|
Narrative anchor:
|
||||||
|
<input type="text" name="new_value" maxlength="500"
|
||||||
|
value="{{ chat.narrative_anchor or '' }}">
|
||||||
|
</label>
|
||||||
|
<button type="submit">Save</button>
|
||||||
|
</form>
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/chat/weather"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<label>
|
||||||
|
Weather:
|
||||||
|
<input type="text" name="new_value" maxlength="500"
|
||||||
|
value="{{ chat.weather or '' }}">
|
||||||
|
</label>
|
||||||
|
<button type="submit">Save</button>
|
||||||
|
</form>
|
||||||
{% if scene %}
|
{% if scene %}
|
||||||
<form class="inline-edit"
|
<form class="inline-edit"
|
||||||
hx-post="/chats/{{ chat.id }}/drawer/scene/close"
|
hx-post="/chats/{{ chat.id }}/drawer/scene/close"
|
||||||
@@ -414,6 +434,140 @@
|
|||||||
{% endif %}
|
{% endif %}
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<section class="drawer-section">
|
||||||
|
<h3>Branches</h3>
|
||||||
|
{% if branches %}
|
||||||
|
<ul class="branch-list">
|
||||||
|
{% for b in branches %}
|
||||||
|
<li class="branch-row{% if b.is_active %} branch-active{% endif %}">
|
||||||
|
<strong>{{ b.name }}</strong>
|
||||||
|
{% if b.is_active %}<span class="muted"> (active)</span>{% endif %}
|
||||||
|
<span class="muted"> · {{ b.event_count }} events</span>
|
||||||
|
{% if not b.is_active %}
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/branch/switch"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<input type="hidden" name="name" value="{{ b.name }}">
|
||||||
|
<button type="submit">Switch</button>
|
||||||
|
</form>
|
||||||
|
{% endif %}
|
||||||
|
</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
{% else %}
|
||||||
|
<p class="muted">No branches yet.</p>
|
||||||
|
{% endif %}
|
||||||
|
<details>
|
||||||
|
<summary>Create branch</summary>
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/branch/create"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<label>
|
||||||
|
Name:
|
||||||
|
<input type="text" name="name" required
|
||||||
|
placeholder="e.g. experiment_a">
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
Origin event id:
|
||||||
|
<input type="number" name="origin_event_id" required min="0">
|
||||||
|
</label>
|
||||||
|
<button type="submit">Create</button>
|
||||||
|
</form>
|
||||||
|
</details>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="drawer-section">
|
||||||
|
<h3>Recent turns</h3>
|
||||||
|
{% if recent_turns %}
|
||||||
|
<ul class="recent-turns-list">
|
||||||
|
{% for t in recent_turns %}
|
||||||
|
<li class="turn-row{% if t.hidden %} turn-hidden{% endif %}">
|
||||||
|
<span class="muted">#{{ t.event_id }} {{ t.kind }}</span>
|
||||||
|
<strong>{{ t.speaker }}:</strong>
|
||||||
|
{{ t.excerpt }}{% if t.excerpt|length >= 120 %}…{% endif %}
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/turn/hide/{{ t.event_id }}"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<input type="hidden" name="hidden" value="{{ 0 if t.hidden else 1 }}">
|
||||||
|
<label>
|
||||||
|
<input type="checkbox" {% if t.hidden %}checked{% endif %}
|
||||||
|
onchange="this.form.requestSubmit()">
|
||||||
|
hide from view
|
||||||
|
</label>
|
||||||
|
</form>
|
||||||
|
</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
{% else %}
|
||||||
|
<p class="muted">No turns yet.</p>
|
||||||
|
{% endif %}
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section class="drawer-section">
|
||||||
|
<h3>Significance review</h3>
|
||||||
|
{% set total_mem = significance_distribution.values()|sum %}
|
||||||
|
{% if total_mem %}
|
||||||
|
<ul class="significance-distribution">
|
||||||
|
{% for level in [0, 1, 2, 3] %}
|
||||||
|
{% set count = significance_distribution[level] %}
|
||||||
|
{% set marker = ['·','•','★','★★'][level] %}
|
||||||
|
{% set pct = (100 * count / total_mem)|round(0, 'floor')|int if total_mem else 0 %}
|
||||||
|
<li class="sig-bar sig-{{ level }}">
|
||||||
|
<span class="sig-label">{{ marker }} ({{ level }})</span>
|
||||||
|
<span class="sig-bar-fill" style="width: {{ pct }}%"></span>
|
||||||
|
<span class="sig-count">{{ count }}</span>
|
||||||
|
</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
{% else %}
|
||||||
|
<p class="muted">No memories yet.</p>
|
||||||
|
{% endif %}
|
||||||
|
{% if recent_memories %}
|
||||||
|
<details>
|
||||||
|
<summary>Edit significance (recent memories)</summary>
|
||||||
|
<ul class="significance-edit-list">
|
||||||
|
{% for m in recent_memories %}
|
||||||
|
<li>
|
||||||
|
<span class="sig sig-{{ m.significance }}">{{ ['·','•','★','★★'][m.significance|default(0)] }}</span>
|
||||||
|
{{ m.pov_summary[:80] }}{% if m.pov_summary|length > 80 %}…{% endif %}
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/memory/{{ m.id }}/significance"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<label>
|
||||||
|
Significance:
|
||||||
|
<input type="range" name="significance" min="0" max="3"
|
||||||
|
value="{{ m.significance|default(0) }}"
|
||||||
|
oninput="this.nextElementSibling.value = this.value">
|
||||||
|
<output>{{ m.significance|default(0) }}</output>
|
||||||
|
</label>
|
||||||
|
<button type="submit">Save</button>
|
||||||
|
</form>
|
||||||
|
</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
</details>
|
||||||
|
{% endif %}
|
||||||
|
{# T110.4: bulk significance re-rate. Move every memory in this chat
|
||||||
|
at level_from to level_to with one manual_edit event per row, so
|
||||||
|
the audit trail stays per-memory. #}
|
||||||
|
<details class="bulk-significance">
|
||||||
|
<summary>Bulk re-rate significance</summary>
|
||||||
|
<form class="inline-edit"
|
||||||
|
hx-post="/chats/{{ chat.id }}/drawer/memory/significance/bulk"
|
||||||
|
hx-target="#drawer" hx-swap="innerHTML">
|
||||||
|
<label>
|
||||||
|
From:
|
||||||
|
<input type="number" name="level_from" min="0" max="3" value="0" required>
|
||||||
|
</label>
|
||||||
|
<label>
|
||||||
|
To:
|
||||||
|
<input type="number" name="level_to" min="0" max="3" value="1" required>
|
||||||
|
</label>
|
||||||
|
<button type="submit">Re-rate all</button>
|
||||||
|
</form>
|
||||||
|
</details>
|
||||||
|
</section>
|
||||||
|
|
||||||
<section class="drawer-section">
|
<section class="drawer-section">
|
||||||
<h3>Pinned memories ({{ pinned|length }} / {{ pin_cap }})</h3>
|
<h3>Pinned memories ({{ pinned|length }} / {{ pin_cap }})</h3>
|
||||||
{% if pinned %}
|
{% if pinned %}
|
||||||
|
|||||||
@@ -17,7 +17,7 @@
|
|||||||
<p class="muted">No turns yet. Start typing below.</p>
|
<p class="muted">No turns yet. Start typing below.</p>
|
||||||
{% else %}
|
{% else %}
|
||||||
{% for turn in turns %}
|
{% for turn in turns %}
|
||||||
<div class="turn turn-{{ turn.role }}">
|
<div{% if turn.event_id is not none %} id="turn-{{ turn.event_id }}"{% endif %} class="turn turn-{{ turn.role }}">
|
||||||
<strong>{{ turn.speaker }}</strong>
|
<strong>{{ turn.speaker }}</strong>
|
||||||
{{ turn.text|render_prose|safe }}
|
{{ turn.text|render_prose|safe }}
|
||||||
</div>
|
</div>
|
||||||
@@ -119,6 +119,39 @@ document.querySelector('.drawer-toggle')?.addEventListener('click', (e) => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// T86: live-swap regenerated turns. The backend (chat/services/
|
||||||
|
// regenerate.py) broadcasts a ``turn_html_replace`` SSE frame after
|
||||||
|
// appending the new assistant_turn — JSON payload of shape
|
||||||
|
// ``{data: <html>, turn_id: <new_id>, supersedes_id: <old_id>}``.
|
||||||
|
// We replace the prior turn's DOM node in-place when we can locate
|
||||||
|
// it by id, otherwise fall back to appending so a tab opened mid-
|
||||||
|
// regenerate still shows the new turn. The renderer
|
||||||
|
// (chat/web/render.py::render_turn_html) and the Jinja loop above
|
||||||
|
// both stamp ``id="turn-<event_id>"`` on each turn DIV, so the
|
||||||
|
// primary in-place swap path is the live one — the append fallback
|
||||||
|
// only kicks in when a tab opened AFTER the regenerate started (no
|
||||||
|
// prior turn DOM node to replace).
|
||||||
|
shell.addEventListener('htmx:sseMessage', (e) => {
|
||||||
|
if (e.detail.type !== 'turn_html_replace') return;
|
||||||
|
let data;
|
||||||
|
try { data = JSON.parse(e.detail.data); } catch (_) { return; }
|
||||||
|
const html = (data && data.data) || '';
|
||||||
|
const trimmed = html.trim();
|
||||||
|
if (!trimmed) return;
|
||||||
|
const oldNode = document.getElementById('turn-' + data.supersedes_id);
|
||||||
|
if (oldNode) {
|
||||||
|
const tmpl = document.createElement('template');
|
||||||
|
tmpl.innerHTML = trimmed;
|
||||||
|
const newNode = tmpl.content.firstChild;
|
||||||
|
if (newNode) oldNode.replaceWith(newNode);
|
||||||
|
} else {
|
||||||
|
// Fallback: append if the prior turn isn't in the DOM (e.g. user
|
||||||
|
// opened the tab AFTER the regenerate started, or the renderer
|
||||||
|
// hasn't yet stamped per-turn ids — see comment above).
|
||||||
|
timeline.insertAdjacentHTML('beforeend', trimmed);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
// SSE connection lost — show a banner and unlock so the user can
|
// SSE connection lost — show a banner and unlock so the user can
|
||||||
// retry. The server commits the partial as truncated when its
|
// retry. The server commits the partial as truncated when its
|
||||||
// request.is_disconnected() poll trips (T19).
|
// request.is_disconnected() poll trips (T19).
|
||||||
|
|||||||
@@ -5,8 +5,16 @@
|
|||||||
<ul>
|
<ul>
|
||||||
<li><a href="/chats" class="{% if active_nav == 'chats' %}active{% endif %}">Chats</a></li>
|
<li><a href="/chats" class="{% if active_nav == 'chats' %}active{% endif %}">Chats</a></li>
|
||||||
<li><a href="/bots" class="{% if active_nav == 'bots' %}active{% endif %}">Bots</a></li>
|
<li><a href="/bots" class="{% if active_nav == 'bots' %}active{% endif %}">Bots</a></li>
|
||||||
|
<li><a href="/snapshots" class="{% if active_nav == 'snapshots' %}active{% endif %}">Snapshots</a></li>
|
||||||
<li><a href="/settings" class="{% if active_nav == 'settings' %}active{% endif %}">Settings</a></li>
|
<li><a href="/settings" class="{% if active_nav == 'settings' %}active{% endif %}">Settings</a></li>
|
||||||
</ul>
|
</ul>
|
||||||
|
{# T100: cross-chat search box. GET /search so the URL is shareable
|
||||||
|
and back-button friendly; the results page itself re-renders this
|
||||||
|
form with the query pre-filled. #}
|
||||||
|
<form class="rail-search" action="/search" method="get" role="search">
|
||||||
|
<input type="search" name="q" placeholder="Search" aria-label="Search memories">
|
||||||
|
<button type="submit">Go</button>
|
||||||
|
</form>
|
||||||
</nav>
|
</nav>
|
||||||
<main class="content">
|
<main class="content">
|
||||||
{% block content %}{% endblock %}
|
{% block content %}{% endblock %}
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
{% extends "layout.html" %}
|
||||||
|
{% block title %}Search - chat{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<header class="page-header">
|
||||||
|
<h1>Search</h1>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<form class="search-page-form" action="/search" method="get">
|
||||||
|
<input type="text" name="q" value="{{ query|default('', true) }}"
|
||||||
|
placeholder="Search memories across all chats" autofocus>
|
||||||
|
<button type="submit">Search</button>
|
||||||
|
</form>
|
||||||
|
|
||||||
|
{% if not query %}
|
||||||
|
{# Empty-state placeholder: the top-bar form submits to /search even
|
||||||
|
with no input, so this page must render cleanly with no query. #}
|
||||||
|
<p class="muted search-empty">Enter a query to search memories across all chats.</p>
|
||||||
|
{% elif not results %}
|
||||||
|
<p class="muted">No matches for “{{ query }}”.</p>
|
||||||
|
{% else %}
|
||||||
|
<ul class="search-results">
|
||||||
|
{% for r in results %}
|
||||||
|
<li class="search-result">
|
||||||
|
<a class="search-result-link" href="/chats/{{ r.chat_id }}">
|
||||||
|
<div class="search-result-meta muted">
|
||||||
|
<strong>{{ r.owner_name }}</strong>
|
||||||
|
<span>· {{ r.chat_id }}</span>
|
||||||
|
{% if r.chat_name %}<span>· {{ r.chat_name }}</span>{% endif %}
|
||||||
|
{% if r.scene_label %}<span>· scene {{ r.scene_label }}</span>{% endif %}
|
||||||
|
</div>
|
||||||
|
<div class="search-result-summary">{{ r.pov_summary }}</div>
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
{% endfor %}
|
||||||
|
</ul>
|
||||||
|
{% endif %}
|
||||||
|
{% endblock %}
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
{% extends "layout.html" %}
|
||||||
|
{% block title %}Snapshots - chat{% endblock %}
|
||||||
|
{% block content %}
|
||||||
|
<header class="page-header">
|
||||||
|
<h1>Snapshots</h1>
|
||||||
|
<form method="post" action="/snapshots/take" class="inline-edit">
|
||||||
|
<button type="submit">Take snapshot now</button>
|
||||||
|
</form>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
{% if preview %}
|
||||||
|
<section class="snapshot-preview">
|
||||||
|
<h2>Preview: {{ preview.snapshot_id }}</h2>
|
||||||
|
<dl>
|
||||||
|
<dt>kind</dt><dd>{{ preview.kind }}</dd>
|
||||||
|
<dt>filename</dt><dd>{{ preview.filename }}</dd>
|
||||||
|
<dt>file size (bytes)</dt><dd>{{ preview.file_size_bytes }}</dd>
|
||||||
|
<dt>snapshot last_event_id</dt><dd>{{ preview.last_event_id }}</dd>
|
||||||
|
<dt>current event_log max id</dt><dd>{{ preview.current_event_log_max_id }}</dd>
|
||||||
|
<dt>events since snapshot</dt><dd>{{ preview.event_delta }}</dd>
|
||||||
|
<dt>events stored in snapshot</dt><dd>{{ preview.event_log_rows_in_snapshot }}</dd>
|
||||||
|
</dl>
|
||||||
|
</section>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
|
{% if snapshots %}
|
||||||
|
<table class="snapshot-list">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>ID</th>
|
||||||
|
<th>Kind</th>
|
||||||
|
<th>Created (UTC)</th>
|
||||||
|
<th>Size (bytes)</th>
|
||||||
|
<th>last_event_id</th>
|
||||||
|
<th>Actions</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{% for snap in snapshots %}
|
||||||
|
<tr>
|
||||||
|
<td>{{ snap.snapshot_id }}</td>
|
||||||
|
<td>{{ snap.kind }}</td>
|
||||||
|
<td>{{ snap.created_at }}</td>
|
||||||
|
<td>{{ snap.file_size_bytes }}</td>
|
||||||
|
<td>{{ snap.last_event_id if snap.last_event_id is not none else '?' }}</td>
|
||||||
|
<td>
|
||||||
|
<a href="/snapshots/{{ snap.snapshot_id }}/preview?kind={{ snap.kind }}">Preview</a>
|
||||||
|
<details class="snapshot-row-restore">
|
||||||
|
<summary>Restore</summary>
|
||||||
|
<form method="post" action="/snapshots/restore/{{ snap.snapshot_id }}" class="inline-edit">
|
||||||
|
<input type="hidden" name="kind" value="{{ snap.kind }}">
|
||||||
|
<label>Type "{{ snap.snapshot_id }}" to confirm:
|
||||||
|
<input type="text" name="confirm_id" required>
|
||||||
|
</label>
|
||||||
|
<button type="submit">Restore from this snapshot</button>
|
||||||
|
</form>
|
||||||
|
</details>
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
{% endfor %}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
{% else %}
|
||||||
|
<p class="muted">No snapshots yet. Use "Take snapshot now" to create one.</p>
|
||||||
|
{% endif %}
|
||||||
|
{% endblock %}
|
||||||
+20
-2
@@ -52,12 +52,30 @@ async def chat_detail(chat_id: str, request: Request, conn=Depends(get_conn)):
|
|||||||
raw_turns = _read_recent_dialogue(conn, chat_id, limit=200)
|
raw_turns = _read_recent_dialogue(conn, chat_id, limit=200)
|
||||||
turns: list[dict] = []
|
turns: list[dict] = []
|
||||||
for t in raw_turns:
|
for t in raw_turns:
|
||||||
|
# event_id is forwarded so the Jinja loop can stamp
|
||||||
|
# ``id="turn-<event_id>"`` on each rendered turn — the
|
||||||
|
# ``turn_html_replace`` SSE handler in chat.html relies on this
|
||||||
|
# id to swap a regenerated turn in-place (T86 follow-up).
|
||||||
if t["speaker"] == "you":
|
if t["speaker"] == "you":
|
||||||
turns.append({"role": "you", "speaker": "you", "text": t["text"]})
|
turns.append(
|
||||||
|
{
|
||||||
|
"role": "you",
|
||||||
|
"speaker": "you",
|
||||||
|
"text": t["text"],
|
||||||
|
"event_id": t.get("event_id"),
|
||||||
|
}
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
bot = get_bot(conn, t["speaker"])
|
bot = get_bot(conn, t["speaker"])
|
||||||
label = bot["name"] if bot else t["speaker"]
|
label = bot["name"] if bot else t["speaker"]
|
||||||
turns.append({"role": "bot", "speaker": label, "text": t["text"]})
|
turns.append(
|
||||||
|
{
|
||||||
|
"role": "bot",
|
||||||
|
"speaker": label,
|
||||||
|
"text": t["text"],
|
||||||
|
"event_id": t.get("event_id"),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
return TEMPLATES.TemplateResponse(
|
return TEMPLATES.TemplateResponse(
|
||||||
request,
|
request,
|
||||||
|
|||||||
+469
-8
@@ -36,7 +36,14 @@ from fastapi.responses import HTMLResponse
|
|||||||
from fastapi.templating import Jinja2Templates
|
from fastapi.templating import Jinja2Templates
|
||||||
|
|
||||||
from chat.eventlog.log import append_and_apply
|
from chat.eventlog.log import append_and_apply
|
||||||
|
from chat.services.branching import (
|
||||||
|
branch_from_event,
|
||||||
|
list_branches_with_metadata,
|
||||||
|
switch_active_branch,
|
||||||
|
)
|
||||||
|
from chat.services.delete_impact import compute_delete_impact
|
||||||
from chat.services.relationship_seed import seed_inter_bot_edges
|
from chat.services.relationship_seed import seed_inter_bot_edges
|
||||||
|
from chat.services.rewind import execute_rewind
|
||||||
from chat.services.scene_summarize import apply_scene_close_summary
|
from chat.services.scene_summarize import apply_scene_close_summary
|
||||||
from chat.state.edges import get_edge
|
from chat.state.edges import get_edge
|
||||||
from chat.state.entities import get_bot, get_you, list_bots
|
from chat.state.entities import get_bot, get_you, list_bots
|
||||||
@@ -48,6 +55,7 @@ from chat.state.world import active_scene, get_activity, get_chat, get_container
|
|||||||
from chat.web.bots import get_conn
|
from chat.web.bots import get_conn
|
||||||
from chat.web.kickoff import get_llm_client
|
from chat.web.kickoff import get_llm_client
|
||||||
from chat.web.skip import (
|
from chat.web.skip import (
|
||||||
|
ChatNotFoundError,
|
||||||
_now_iso,
|
_now_iso,
|
||||||
process_elision_skip,
|
process_elision_skip,
|
||||||
process_jump_skip,
|
process_jump_skip,
|
||||||
@@ -168,6 +176,63 @@ async def drawer(chat_id: str, request: Request, conn=Depends(get_conn)):
|
|||||||
active_events = list_active_events(conn, chat_id)
|
active_events = list_active_events(conn, chat_id)
|
||||||
open_threads = list_open_threads(conn, chat_id)
|
open_threads = list_open_threads(conn, chat_id)
|
||||||
|
|
||||||
|
# T98.3: recent turns (user_turn / assistant_turn) for the hide-from-view
|
||||||
|
# panel. Includes ``hidden`` rows so the user can un-hide them — the
|
||||||
|
# filter on the read side (read_recent_dialogue) is what drops hidden
|
||||||
|
# rows from the prompt; the drawer panel always shows everything.
|
||||||
|
turn_rows = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT id, kind, payload_json, hidden
|
||||||
|
FROM event_log
|
||||||
|
WHERE kind IN ('user_turn', 'assistant_turn', 'user_turn_edit')
|
||||||
|
AND superseded_by IS NULL
|
||||||
|
ORDER BY id DESC
|
||||||
|
LIMIT ?
|
||||||
|
""",
|
||||||
|
(RECENT_LIMIT,),
|
||||||
|
).fetchall()
|
||||||
|
recent_turns: list[dict] = []
|
||||||
|
for row in turn_rows:
|
||||||
|
try:
|
||||||
|
payload = json.loads(row[2]) if row[2] else {}
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
payload = {}
|
||||||
|
if payload.get("chat_id") != chat_id:
|
||||||
|
continue
|
||||||
|
text = payload.get("prose") or payload.get("text") or ""
|
||||||
|
speaker = payload.get("speaker_id") or (
|
||||||
|
"you" if row[1].startswith("user") else "?"
|
||||||
|
)
|
||||||
|
recent_turns.append(
|
||||||
|
{
|
||||||
|
"event_id": int(row[0]),
|
||||||
|
"kind": row[1],
|
||||||
|
"speaker": speaker,
|
||||||
|
"excerpt": (text or "").replace("\n", " ")[:120],
|
||||||
|
"hidden": bool(row[3]),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# T98.1: branch metadata (every chat sees the global branch list — branches
|
||||||
|
# may be chat-scoped or global, so :func:`list_branches_with_metadata`
|
||||||
|
# returns both flavours and the template highlights the active one).
|
||||||
|
branches = list_branches_with_metadata(conn, chat_id)
|
||||||
|
|
||||||
|
# T98.2: significance distribution across this chat's memories. Powers
|
||||||
|
# the "Significance review" panel — a small histogram letting authors
|
||||||
|
# spot lopsided buckets (e.g. nothing significant=3 yet) and triage by
|
||||||
|
# editing individual memory significance values.
|
||||||
|
sig_rows = conn.execute(
|
||||||
|
"SELECT significance, COUNT(*) FROM memories "
|
||||||
|
"WHERE chat_id = ? GROUP BY significance ORDER BY significance",
|
||||||
|
(chat_id,),
|
||||||
|
).fetchall()
|
||||||
|
significance_distribution = {int(r[0]): int(r[1]) for r in sig_rows}
|
||||||
|
# Ensure every bucket 0..3 is present so the bar-chart template can
|
||||||
|
# render a stable axis even when a level has zero rows.
|
||||||
|
for level in (0, 1, 2, 3):
|
||||||
|
significance_distribution.setdefault(level, 0)
|
||||||
|
|
||||||
return TEMPLATES.TemplateResponse(
|
return TEMPLATES.TemplateResponse(
|
||||||
request,
|
request,
|
||||||
"_drawer.html",
|
"_drawer.html",
|
||||||
@@ -195,6 +260,9 @@ async def drawer(chat_id: str, request: Request, conn=Depends(get_conn)):
|
|||||||
"pin_cap": PIN_CAP,
|
"pin_cap": PIN_CAP,
|
||||||
"active_events": active_events,
|
"active_events": active_events,
|
||||||
"open_threads": open_threads,
|
"open_threads": open_threads,
|
||||||
|
"branches": branches,
|
||||||
|
"significance_distribution": significance_distribution,
|
||||||
|
"recent_turns": recent_turns,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -343,6 +411,64 @@ async def edit_memory_significance(
|
|||||||
return await drawer(chat_id, request, conn)
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/memory/significance/bulk",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def bulk_re_rate_significance(
|
||||||
|
chat_id: str,
|
||||||
|
request: Request,
|
||||||
|
level_from: int = Form(...),
|
||||||
|
level_to: int = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""T110.4: bulk re-rate every memory in this chat at ``level_from``
|
||||||
|
to ``level_to``.
|
||||||
|
|
||||||
|
Fans out into one ``manual_edit`` event per matching memory rather
|
||||||
|
than a single bulk event so the §6.4 audit trail stays per-row —
|
||||||
|
each affected memory carries its own ``prior_value -> new_value``
|
||||||
|
snapshot, so an inverse edit can restore an individual row without
|
||||||
|
needing to inspect a bulk payload's member list. The drawer's
|
||||||
|
significance-distribution panel surfaces the new buckets on the
|
||||||
|
refreshed partial.
|
||||||
|
|
||||||
|
Both levels are clamped to 0..3 (matching ``edit_memory_significance``)
|
||||||
|
and a no-op (``level_from == level_to``) is rejected with 400 so a
|
||||||
|
misclick can't pad the event log with empty edits.
|
||||||
|
"""
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
|
||||||
|
lf = max(0, min(3, int(level_from)))
|
||||||
|
lt = max(0, min(3, int(level_to)))
|
||||||
|
if lf == lt:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"level_from and level_to must differ (both = {lf})",
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id FROM memories WHERE chat_id = ? AND significance = ? "
|
||||||
|
"ORDER BY id ASC",
|
||||||
|
(chat_id, lf),
|
||||||
|
).fetchall()
|
||||||
|
for row in rows:
|
||||||
|
memory_id = int(row[0])
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="manual_edit",
|
||||||
|
payload={
|
||||||
|
"target_kind": "memory_significance",
|
||||||
|
"target_id": memory_id,
|
||||||
|
"prior_value": lf,
|
||||||
|
"new_value": lt,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
@router.post(
|
@router.post(
|
||||||
"/chats/{chat_id}/drawer/memory/{memory_id}/pin",
|
"/chats/{chat_id}/drawer/memory/{memory_id}/pin",
|
||||||
response_class=HTMLResponse,
|
response_class=HTMLResponse,
|
||||||
@@ -992,14 +1118,14 @@ async def skip_elision(
|
|||||||
chat_id=chat_id,
|
chat_id=chat_id,
|
||||||
new_time=new_time,
|
new_time=new_time,
|
||||||
landing_state_hint=landing_state_hint,
|
landing_state_hint=landing_state_hint,
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
|
except ChatNotFoundError as exc:
|
||||||
|
# Missing chat row: typed exception (T81) replaces the prior
|
||||||
|
# ``str(exc).startswith("chat not found")`` prefix sniff.
|
||||||
|
raise HTTPException(status_code=404, detail=str(exc))
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
# ``process_elision_skip`` raises on missing-chat or malformed /
|
# Input-validation failure (malformed or backwards new_time).
|
||||||
# backwards new_time. The drawer used to 404 / 400 these
|
|
||||||
# separately — preserve the 404-vs-400 split by sniffing the
|
|
||||||
# error message so existing tests keep passing without changes.
|
|
||||||
if str(exc).startswith("chat not found"):
|
|
||||||
raise HTTPException(status_code=404, detail=str(exc))
|
|
||||||
raise HTTPException(status_code=400, detail=str(exc))
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
return await drawer(chat_id, request, conn)
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
@@ -1036,10 +1162,14 @@ async def skip_jump(
|
|||||||
new_time=new_time,
|
new_time=new_time,
|
||||||
notable_prose=notable_prose,
|
notable_prose=notable_prose,
|
||||||
reset_activity=reset_flag,
|
reset_activity=reset_flag,
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
|
except ChatNotFoundError as exc:
|
||||||
|
# Missing chat row: typed exception (T81) replaces the prior
|
||||||
|
# ``str(exc).startswith("chat not found")`` prefix sniff.
|
||||||
|
raise HTTPException(status_code=404, detail=str(exc))
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
if str(exc).startswith("chat not found"):
|
# Input-validation failure (malformed or backwards new_time).
|
||||||
raise HTTPException(status_code=404, detail=str(exc))
|
|
||||||
raise HTTPException(status_code=400, detail=str(exc))
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
return await drawer(chat_id, request, conn)
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
@@ -1075,3 +1205,334 @@ async def close_thread(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
return await drawer(chat_id, request, conn)
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
# --- T98.1 branching UI --------------------------------------------------
|
||||||
|
#
|
||||||
|
# Three POST endpoints wired to the Phase 4 :mod:`chat.services.branching`
|
||||||
|
# helpers. The drawer's "Branches" panel exposes:
|
||||||
|
#
|
||||||
|
# * Create from a free-form ``origin_event_id``.
|
||||||
|
# * Switch the active branch by name.
|
||||||
|
# * Convenience "branch from this turn" against a per-turn event_id (the
|
||||||
|
# chat surface stamps ``id="turn-<event_id>"`` on every turn so users can
|
||||||
|
# pick the right one without copying ids by hand).
|
||||||
|
#
|
||||||
|
# All three return the refreshed drawer partial; failures from the service
|
||||||
|
# layer (duplicate name, unknown branch, invalid origin) surface as 400 so
|
||||||
|
# HTMX displays the inline error.
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/branch/create",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def create_branch(
|
||||||
|
chat_id: str,
|
||||||
|
request: Request,
|
||||||
|
name: str = Form(...),
|
||||||
|
origin_event_id: int = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
try:
|
||||||
|
branch_from_event(
|
||||||
|
conn,
|
||||||
|
name=name,
|
||||||
|
origin_event_id=int(origin_event_id),
|
||||||
|
chat_id=chat_id,
|
||||||
|
)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/branch/switch",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def switch_branch(
|
||||||
|
chat_id: str,
|
||||||
|
request: Request,
|
||||||
|
name: str = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
try:
|
||||||
|
switch_active_branch(conn, name=name)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/chats/{chat_id}/drawer/turn/delete-preview/{event_id}",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def delete_preview(
|
||||||
|
chat_id: str,
|
||||||
|
event_id: int,
|
||||||
|
request: Request,
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""Render an :class:`ImpactReport` for ``event_id`` as a small modal.
|
||||||
|
|
||||||
|
Read-only — :func:`compute_delete_impact` does not mutate the
|
||||||
|
database. The modal contains a confirmation form posting to
|
||||||
|
:func:`delete_turn` below; HTMX swaps the fragment into a modal
|
||||||
|
target on the chat page.
|
||||||
|
"""
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
|
||||||
|
report = compute_delete_impact(conn, target_event_id=int(event_id))
|
||||||
|
|
||||||
|
# T110.3: render via the ``_delete_impact_modal.html`` Jinja partial
|
||||||
|
# so HTML autoescape covers user-controllable fields (item.kind,
|
||||||
|
# item.description, notes) automatically. The prior implementation
|
||||||
|
# built the modal HTML via raw f-string concatenation and required
|
||||||
|
# explicit ``html.escape()`` calls (T110.2) on each interpolated
|
||||||
|
# field; under autoescape those calls become redundant. Mirrors the
|
||||||
|
# rewind-preview style in :func:`chat.web.turns.rewind_preview`.
|
||||||
|
return TEMPLATES.TemplateResponse(
|
||||||
|
request,
|
||||||
|
"_delete_impact_modal.html",
|
||||||
|
{"chat_id": chat_id, "impact": report},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/turn/delete/{event_id}",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def delete_turn(
|
||||||
|
chat_id: str,
|
||||||
|
event_id: int,
|
||||||
|
request: Request,
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""Delete a turn (and everything after) by invoking the existing rewind path.
|
||||||
|
|
||||||
|
The :func:`chat.services.rewind.execute_rewind` API takes
|
||||||
|
``after_event_id``: it removes events with id strictly greater than
|
||||||
|
that argument. To make ``event_id`` itself disappear we pass
|
||||||
|
``after_event_id = event_id - 1`` — a thin adapter, not a
|
||||||
|
re-implementation of rewind.
|
||||||
|
|
||||||
|
A snapshot is taken before truncation (inside ``execute_rewind``)
|
||||||
|
so the user can recover via the snapshot index.
|
||||||
|
|
||||||
|
T110.1 guards ``event_id <= 0``: a stale tab or hand-crafted request
|
||||||
|
posting ``event_id=0`` would otherwise compute ``after_event_id=-1``
|
||||||
|
and silently truncate the entire log. ``id`` is auto-assigned by
|
||||||
|
SQLite starting at 1 so any caller's "real" id is always >= 1; a
|
||||||
|
zero or negative value can only mean a client bug, surfaced as 400.
|
||||||
|
"""
|
||||||
|
if int(event_id) <= 0:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"event_id must be a positive integer, got {event_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
|
||||||
|
settings = request.app.state.settings
|
||||||
|
execute_rewind(
|
||||||
|
db_path=settings.db_path,
|
||||||
|
data_dir=settings.data_dir,
|
||||||
|
after_event_id=int(event_id) - 1,
|
||||||
|
)
|
||||||
|
# ``conn`` is now stale (the rewind opened its own connection and
|
||||||
|
# truncated/reprojected). Re-render the drawer through a fresh open
|
||||||
|
# so the partial reflects the truncated state.
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
|
||||||
|
with open_db(settings.db_path) as fresh:
|
||||||
|
return await drawer(chat_id, request, fresh)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/turn/hide/{event_id}",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def hide_turn(
|
||||||
|
chat_id: str,
|
||||||
|
event_id: int,
|
||||||
|
request: Request,
|
||||||
|
hidden: int = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""Toggle ``event_log.hidden`` on a turn via the ``turn_hidden``
|
||||||
|
``manual_edit`` projector branch.
|
||||||
|
|
||||||
|
The route validates the target is an actual turn-shaped row in this
|
||||||
|
chat (so a stray click on the chat panel can't hide a system event)
|
||||||
|
and snapshots the prior ``hidden`` value for §6.4 reversibility.
|
||||||
|
"""
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT kind, payload_json, hidden FROM event_log WHERE id = ?",
|
||||||
|
(int(event_id),),
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=404, detail=f"event not found: {event_id}"
|
||||||
|
)
|
||||||
|
if row[0] not in ("user_turn", "assistant_turn", "user_turn_edit"):
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"event {event_id} is not a turn (kind={row[0]})",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
payload = json.loads(row[1]) if row[1] else {}
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
payload = {}
|
||||||
|
if payload.get("chat_id") != chat_id:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=404,
|
||||||
|
detail=f"event {event_id} not in chat {chat_id}",
|
||||||
|
)
|
||||||
|
|
||||||
|
prior_hidden = 1 if int(row[2]) else 0
|
||||||
|
new_hidden = 1 if int(hidden) else 0
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="manual_edit",
|
||||||
|
payload={
|
||||||
|
"target_kind": "turn_hidden",
|
||||||
|
"target_id": int(event_id),
|
||||||
|
"prior_value": {"hidden": prior_hidden},
|
||||||
|
"new_value": {"hidden": new_hidden},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
# --- T98.5 chat narrative anchor + weather ----------------------------
|
||||||
|
#
|
||||||
|
# Audit (T98.5) found two §6.4 fields without drawer affordances despite
|
||||||
|
# both being prose strings stored on ``chat_state``: ``narrative_anchor``
|
||||||
|
# (the "Day 1" / "morning of the gala" hint above the chat clock) and
|
||||||
|
# ``weather``. Both land via the existing ``manual_edit`` projector with
|
||||||
|
# new branches added in :mod:`chat.state.manual_edit`. The container
|
||||||
|
# ``properties_json`` blob is more invasive — bounded JSON edits aren't
|
||||||
|
# wired through manual_edit and the drawer never surfaces multiple
|
||||||
|
# containers at once, so it stays out of v1.
|
||||||
|
|
||||||
|
CHAT_NARRATIVE_ANCHOR_MAX = 500
|
||||||
|
CHAT_WEATHER_MAX = 500
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/chat/narrative-anchor",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def edit_chat_narrative_anchor(
|
||||||
|
chat_id: str,
|
||||||
|
request: Request,
|
||||||
|
new_value: str = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
if len(new_value) > CHAT_NARRATIVE_ANCHOR_MAX:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=(
|
||||||
|
f"narrative_anchor exceeds {CHAT_NARRATIVE_ANCHOR_MAX} chars "
|
||||||
|
f"(got {len(new_value)})"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
prior = chat.get("narrative_anchor") or ""
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="manual_edit",
|
||||||
|
payload={
|
||||||
|
"target_kind": "chat_narrative_anchor",
|
||||||
|
"target_id": chat_id,
|
||||||
|
"prior_value": prior,
|
||||||
|
"new_value": new_value,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/chat/weather",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def edit_chat_weather(
|
||||||
|
chat_id: str,
|
||||||
|
request: Request,
|
||||||
|
new_value: str = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
if len(new_value) > CHAT_WEATHER_MAX:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=(
|
||||||
|
f"weather exceeds {CHAT_WEATHER_MAX} chars "
|
||||||
|
f"(got {len(new_value)})"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
prior = chat.get("weather") or ""
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="manual_edit",
|
||||||
|
payload={
|
||||||
|
"target_kind": "chat_weather",
|
||||||
|
"target_id": chat_id,
|
||||||
|
"prior_value": prior,
|
||||||
|
"new_value": new_value,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post(
|
||||||
|
"/chats/{chat_id}/drawer/branch/from-turn/{event_id}",
|
||||||
|
response_class=HTMLResponse,
|
||||||
|
)
|
||||||
|
async def branch_from_turn(
|
||||||
|
chat_id: str,
|
||||||
|
event_id: int,
|
||||||
|
request: Request,
|
||||||
|
name: str = Form(...),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""Convenience: branch from a specific turn event.
|
||||||
|
|
||||||
|
Identical to :func:`create_branch` except ``origin_event_id`` is
|
||||||
|
encoded in the URL — the chat surface renders one such form per turn
|
||||||
|
so users can fork mid-conversation without authoring an event id by
|
||||||
|
hand.
|
||||||
|
"""
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise HTTPException(status_code=404, detail=f"chat not found: {chat_id}")
|
||||||
|
try:
|
||||||
|
branch_from_event(
|
||||||
|
conn,
|
||||||
|
name=name,
|
||||||
|
origin_event_id=int(event_id),
|
||||||
|
chat_id=chat_id,
|
||||||
|
)
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
|
return await drawer(chat_id, request, conn)
|
||||||
|
|||||||
@@ -131,6 +131,7 @@ async def process_meanwhile_turn(
|
|||||||
*,
|
*,
|
||||||
chat_id: str,
|
chat_id: str,
|
||||||
prose: str,
|
prose: str,
|
||||||
|
app=None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Run one meanwhile turn end-to-end.
|
"""Run one meanwhile turn end-to-end.
|
||||||
|
|
||||||
@@ -314,6 +315,7 @@ async def process_meanwhile_turn(
|
|||||||
narrative_text=text,
|
narrative_text=text,
|
||||||
scene_id=scene_id,
|
scene_id=scene_id,
|
||||||
chat_clock_at=chat.get("time"),
|
chat_clock_at=chat.get("time"),
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
|
|
||||||
# 9. Post-turn state-update — exactly 2 directed pairs over the
|
# 9. Post-turn state-update — exactly 2 directed pairs over the
|
||||||
@@ -378,7 +380,12 @@ async def process_meanwhile_turn(
|
|||||||
"truncated": truncated,
|
"truncated": truncated,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
turn_html = _render_turn_html(speaker_bot["name"], text, role="bot")
|
turn_html = _render_turn_html(
|
||||||
|
speaker_bot["name"],
|
||||||
|
text,
|
||||||
|
role="bot",
|
||||||
|
event_id=assistant_event_id,
|
||||||
|
)
|
||||||
await publish(chat_id, {"event": "turn_html", "data": turn_html})
|
await publish(chat_id, {"event": "turn_html", "data": turn_html})
|
||||||
|
|
||||||
if cancelled:
|
if cancelled:
|
||||||
|
|||||||
+15
-2
@@ -84,7 +84,13 @@ def render_prose(text: str) -> str:
|
|||||||
return "".join(f"<p>{p}</p>" for p in paragraphs)
|
return "".join(f"<p>{p}</p>" for p in paragraphs)
|
||||||
|
|
||||||
|
|
||||||
def render_turn_html(speaker: str, text: str, role: str = "bot") -> str:
|
def render_turn_html(
|
||||||
|
speaker: str,
|
||||||
|
text: str,
|
||||||
|
role: str = "bot",
|
||||||
|
*,
|
||||||
|
event_id: int | None = None,
|
||||||
|
) -> str:
|
||||||
"""Render a full transcript turn as ``<div class="turn …">…</div>``.
|
"""Render a full transcript turn as ``<div class="turn …">…</div>``.
|
||||||
|
|
||||||
Used by both the SSE fragment publisher in :mod:`chat.web.turns`
|
Used by both the SSE fragment publisher in :mod:`chat.web.turns`
|
||||||
@@ -94,12 +100,19 @@ def render_turn_html(speaker: str, text: str, role: str = "bot") -> str:
|
|||||||
``role`` selects the CSS class (``turn-you`` vs ``turn-bot``); the
|
``role`` selects the CSS class (``turn-you`` vs ``turn-bot``); the
|
||||||
speaker label and role name are HTML-escaped defensively even though
|
speaker label and role name are HTML-escaped defensively even though
|
||||||
they currently come from trusted server-side state.
|
they currently come from trusted server-side state.
|
||||||
|
|
||||||
|
``event_id`` (T86 follow-up) stamps ``id="turn-<event_id>"`` on the
|
||||||
|
wrapper div so the chat-page ``turn_html_replace`` SSE handler can
|
||||||
|
locate the prior turn node by id and swap it in-place. When omitted
|
||||||
|
the id attribute is dropped so SSE-only fragments without a stable
|
||||||
|
event id (legacy callers) still render cleanly.
|
||||||
"""
|
"""
|
||||||
speaker_html = html.escape(speaker)
|
speaker_html = html.escape(speaker)
|
||||||
role_html = html.escape(role)
|
role_html = html.escape(role)
|
||||||
body_html = render_prose(text)
|
body_html = render_prose(text)
|
||||||
|
id_attr = f' id="turn-{int(event_id)}"' if event_id is not None else ""
|
||||||
return (
|
return (
|
||||||
f'<div class="turn turn-{role_html}">'
|
f'<div{id_attr} class="turn turn-{role_html}">'
|
||||||
f"<strong>{speaker_html}</strong>"
|
f"<strong>{speaker_html}</strong>"
|
||||||
f"{body_html}"
|
f"{body_html}"
|
||||||
f"</div>"
|
f"</div>"
|
||||||
|
|||||||
@@ -0,0 +1,216 @@
|
|||||||
|
"""T100 (Phase 4): cross-chat search UX route.
|
||||||
|
|
||||||
|
Wraps T93's :func:`chat.services.cross_chat_search.search_all_memories`
|
||||||
|
in a small read-only HTML surface so the top-bar search input has
|
||||||
|
somewhere to land. The route does no filtering of its own beyond the
|
||||||
|
empty-query fast-path that T93 already implements; ranking, owner
|
||||||
|
scope, and witness scope all live in the service layer.
|
||||||
|
|
||||||
|
For each match we hydrate just enough metadata to render a row:
|
||||||
|
* the owner bot's display name (so users see "BOTA" not "bot_a"),
|
||||||
|
* the originating ``chat_id`` (the link target — there's no per-turn
|
||||||
|
anchor today because memories don't carry an ``event_id`` column,
|
||||||
|
so we deep-link to the chat as a whole),
|
||||||
|
* the originating scene title when one exists,
|
||||||
|
* and the ``pov_summary`` itself.
|
||||||
|
|
||||||
|
T106 (Phase 4.5): hydration is batched. Pre-T106 the route called
|
||||||
|
``get_bot``/``get_chat``/``get_scene`` once per result row — N+1 with
|
||||||
|
``DEFAULT_SEARCH_K=50`` meaning up to 150 individual SELECTs per page
|
||||||
|
load. We now collect distinct ids first and fan-in via three
|
||||||
|
``WHERE id IN (...)`` queries, then map back per row.
|
||||||
|
|
||||||
|
We deliberately keep this module synchronous and template-only — no
|
||||||
|
HTMX swaps, no JSON API — because the search box is a "leave the
|
||||||
|
current chat to look something up" surface, not an inline drawer.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from sqlite3 import Connection
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, Request
|
||||||
|
from fastapi.responses import HTMLResponse
|
||||||
|
from fastapi.templating import Jinja2Templates
|
||||||
|
|
||||||
|
from chat.services.cross_chat_search import search_all_memories
|
||||||
|
from chat.state.entities import get_bot
|
||||||
|
from chat.state.world import get_chat, get_scene
|
||||||
|
from chat.web.bots import get_conn
|
||||||
|
|
||||||
|
TEMPLATES = Jinja2Templates(
|
||||||
|
directory=str(Path(__file__).resolve().parent.parent / "templates")
|
||||||
|
)
|
||||||
|
|
||||||
|
#: Maximum cross-chat FTS matches surfaced per ``/search`` page load.
|
||||||
|
#: Extracted as a module-level constant (T106) so the cap is tunable
|
||||||
|
#: without touching the route body. ``search_all_memories`` itself
|
||||||
|
#: defaults to a smaller ``k=20``; we override here because the
|
||||||
|
#: top-bar search is a "scan everything I've seen" surface, not an
|
||||||
|
#: inline drawer.
|
||||||
|
DEFAULT_SEARCH_K = 50
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_bots_by_ids(conn: Connection, ids: set[str]) -> dict[str, dict]:
|
||||||
|
"""Batched sibling of :func:`chat.state.entities.get_bot`.
|
||||||
|
|
||||||
|
Inlined here (not exported from ``state.entities``) to keep T106's
|
||||||
|
scope confined to ``search.py`` per the Phase 4.5 plan. Returns
|
||||||
|
``{bot_id: bot_dict}`` for every id present in ``ids``; ids with
|
||||||
|
no matching row are simply absent from the map (the caller falls
|
||||||
|
back to the raw id string the same way it did pre-T106).
|
||||||
|
|
||||||
|
Empty ``ids`` short-circuits to ``{}`` because SQLite rejects
|
||||||
|
``WHERE id IN ()`` as a syntax error.
|
||||||
|
"""
|
||||||
|
if not ids:
|
||||||
|
return {}
|
||||||
|
placeholders = ",".join("?" * len(ids))
|
||||||
|
cols = [c[1] for c in conn.execute("PRAGMA table_info(bots)").fetchall()]
|
||||||
|
rows = conn.execute(
|
||||||
|
f"SELECT * FROM bots WHERE id IN ({placeholders})",
|
||||||
|
tuple(ids),
|
||||||
|
).fetchall()
|
||||||
|
out: dict[str, dict] = {}
|
||||||
|
for row in rows:
|
||||||
|
d = dict(zip(cols, row))
|
||||||
|
d["voice_samples"] = json.loads(d.pop("voice_samples_json"))
|
||||||
|
d["traits"] = json.loads(d.pop("traits_json"))
|
||||||
|
out[d["id"]] = d
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_chats_by_ids(conn: Connection, ids: set[str]) -> dict[str, dict]:
|
||||||
|
"""Batched sibling of :func:`chat.state.world.get_chat`.
|
||||||
|
|
||||||
|
Mirrors that helper's ``chats``/``chat_state`` JOIN so the returned
|
||||||
|
dicts have the same shape (``narrative_anchor``, ``time``,
|
||||||
|
``weather``, ``active_scene_id``, etc.). Empty ``ids`` returns
|
||||||
|
``{}`` to dodge the ``IN ()`` syntax error.
|
||||||
|
"""
|
||||||
|
if not ids:
|
||||||
|
return {}
|
||||||
|
placeholders = ",".join("?" * len(ids))
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT c.id, c.host_bot_id, c.guest_bot_id, c.created_at, "
|
||||||
|
" s.time, s.weather, s.active_scene_id, s.narrative_anchor "
|
||||||
|
f"FROM chats c JOIN chat_state s ON s.chat_id = c.id "
|
||||||
|
f"WHERE c.id IN ({placeholders})",
|
||||||
|
tuple(ids),
|
||||||
|
).fetchall()
|
||||||
|
return {
|
||||||
|
row[0]: {
|
||||||
|
"id": row[0],
|
||||||
|
"host_bot_id": row[1],
|
||||||
|
"guest_bot_id": row[2],
|
||||||
|
"created_at": row[3],
|
||||||
|
"time": row[4],
|
||||||
|
"weather": row[5],
|
||||||
|
"active_scene_id": row[6],
|
||||||
|
"narrative_anchor": row[7],
|
||||||
|
}
|
||||||
|
for row in rows
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_scenes_by_ids(conn: Connection, ids: set[int]) -> dict[int, dict]:
|
||||||
|
"""Batched sibling of :func:`chat.state.world.get_scene`.
|
||||||
|
|
||||||
|
Returns ``{scene_id: scene_dict}`` with ``participants`` already
|
||||||
|
JSON-decoded so callers see the same shape as the per-row helper.
|
||||||
|
Empty ``ids`` returns ``{}``.
|
||||||
|
"""
|
||||||
|
if not ids:
|
||||||
|
return {}
|
||||||
|
placeholders = ",".join("?" * len(ids))
|
||||||
|
cols = [c[1] for c in conn.execute("PRAGMA table_info(scenes)").fetchall()]
|
||||||
|
rows = conn.execute(
|
||||||
|
f"SELECT * FROM scenes WHERE id IN ({placeholders})",
|
||||||
|
tuple(ids),
|
||||||
|
).fetchall()
|
||||||
|
out: dict[int, dict] = {}
|
||||||
|
for row in rows:
|
||||||
|
d = dict(zip(cols, row))
|
||||||
|
d["participants"] = json.loads(d.pop("participants_json"))
|
||||||
|
out[d["id"]] = d
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/search", response_class=HTMLResponse)
|
||||||
|
async def search(request: Request, q: str = "", conn=Depends(get_conn)):
|
||||||
|
"""Render ``search.html`` with up to :data:`DEFAULT_SEARCH_K` matches.
|
||||||
|
|
||||||
|
``q`` is intentionally allowed to be empty — that path renders the
|
||||||
|
page's "enter a query" placeholder rather than a 400, because the
|
||||||
|
top-bar form submits to this URL even with an empty input. T93's
|
||||||
|
service short-circuits whitespace-only queries to ``[]`` so there
|
||||||
|
is no FTS5 ``MATCH ''`` syntax error to guard against here.
|
||||||
|
|
||||||
|
Hydration (T106) is batched: rather than calling ``get_bot`` /
|
||||||
|
``get_chat`` / ``get_scene`` per row (worst case 3 * k individual
|
||||||
|
SELECTs), we collect distinct ids and issue one ``IN (...)`` query
|
||||||
|
per entity kind, then map back during the row build. ``get_bot``
|
||||||
|
et al. remain imported for test-time monkeypatching but are no
|
||||||
|
longer invoked on the hot path.
|
||||||
|
"""
|
||||||
|
raw_results = (
|
||||||
|
search_all_memories(conn, query=q, k=DEFAULT_SEARCH_K) if q else []
|
||||||
|
)
|
||||||
|
|
||||||
|
# Collect distinct ids up front so the IN-list queries dedupe (a
|
||||||
|
# popular bot or scene shows up many times across the result set).
|
||||||
|
bot_ids: set[str] = {r["owner_id"] for r in raw_results if r["owner_id"]}
|
||||||
|
chat_ids: set[str] = {r["chat_id"] for r in raw_results if r["chat_id"]}
|
||||||
|
scene_ids: set[int] = {
|
||||||
|
r["scene_id"] for r in raw_results if r["scene_id"]
|
||||||
|
}
|
||||||
|
|
||||||
|
bots_by_id = _fetch_bots_by_ids(conn, bot_ids)
|
||||||
|
chats_by_id = _fetch_chats_by_ids(conn, chat_ids)
|
||||||
|
scenes_by_id = _fetch_scenes_by_ids(conn, scene_ids)
|
||||||
|
|
||||||
|
# Hydrate display fields per row from the batched maps. We do this
|
||||||
|
# in the route (not the service) so the service stays a pure FTS
|
||||||
|
# shim that other UIs can reuse.
|
||||||
|
results = []
|
||||||
|
for row in raw_results:
|
||||||
|
bot = bots_by_id.get(row["owner_id"])
|
||||||
|
chat = chats_by_id.get(row["chat_id"])
|
||||||
|
scene = (
|
||||||
|
scenes_by_id.get(row["scene_id"]) if row["scene_id"] else None
|
||||||
|
)
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"memory_id": row["memory_id"],
|
||||||
|
"owner_id": row["owner_id"],
|
||||||
|
"owner_name": bot["name"] if bot else row["owner_id"],
|
||||||
|
"chat_id": row["chat_id"],
|
||||||
|
"chat_name": (
|
||||||
|
chat.get("narrative_anchor") if chat else None
|
||||||
|
),
|
||||||
|
"scene_id": row["scene_id"],
|
||||||
|
# Scenes have no ``title`` column today; surface the
|
||||||
|
# ``started_at`` timestamp as a human-friendly label
|
||||||
|
# when a scene is set, otherwise leave it blank.
|
||||||
|
"scene_label": (
|
||||||
|
scene.get("started_at") if scene else None
|
||||||
|
),
|
||||||
|
"pov_summary": row["pov_summary"],
|
||||||
|
"significance": row["significance"],
|
||||||
|
"ts": row["ts"],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
return TEMPLATES.TemplateResponse(
|
||||||
|
request,
|
||||||
|
"search.html",
|
||||||
|
{
|
||||||
|
"query": q,
|
||||||
|
"results": results,
|
||||||
|
"active_nav": "search",
|
||||||
|
},
|
||||||
|
)
|
||||||
+24
-6
@@ -36,6 +36,17 @@ from chat.state.entities import get_bot, get_you
|
|||||||
from chat.state.world import get_activity, get_chat
|
from chat.state.world import get_activity, get_chat
|
||||||
|
|
||||||
|
|
||||||
|
class ChatNotFoundError(Exception):
|
||||||
|
"""Raised when a ``chat_id`` doesn't resolve to a chat row.
|
||||||
|
|
||||||
|
Distinguishes the missing-chat case from generic input-validation
|
||||||
|
failures (which still raise :class:`ValueError`). HTTP callers map
|
||||||
|
this to ``404`` and ``ValueError`` to ``400`` — replacing the
|
||||||
|
earlier ``str(exc).startswith("chat not found")`` prefix sniff
|
||||||
|
(T81) with a typed dispatch.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
def _parse_iso_time(value: str) -> datetime | None:
|
def _parse_iso_time(value: str) -> datetime | None:
|
||||||
"""Permissive ISO 8601 parser shared with the drawer routes (T59).
|
"""Permissive ISO 8601 parser shared with the drawer routes (T59).
|
||||||
|
|
||||||
@@ -80,6 +91,7 @@ async def process_elision_skip(
|
|||||||
chat_id: str,
|
chat_id: str,
|
||||||
new_time: str,
|
new_time: str,
|
||||||
landing_state_hint: str = "",
|
landing_state_hint: str = "",
|
||||||
|
app=None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Run an elision skip end-to-end.
|
"""Run an elision skip end-to-end.
|
||||||
|
|
||||||
@@ -93,13 +105,14 @@ async def process_elision_skip(
|
|||||||
..., "assistant_event_id": ...}`` so callers can introspect the
|
..., "assistant_event_id": ...}`` so callers can introspect the
|
||||||
generated turn (e.g. for SSE rebroadcast or test assertions).
|
generated turn (e.g. for SSE rebroadcast or test assertions).
|
||||||
|
|
||||||
Raises ``ValueError`` on validation failure or when the chat row
|
Raises :class:`ChatNotFoundError` when the chat row is missing
|
||||||
can't be located (the drawer maps it to ``HTTP 400`` / ``404``
|
(HTTP ``404``) and ``ValueError`` on input-validation failure
|
||||||
respectively; the natural-language path follows the same shape).
|
(HTTP ``400``). Splitting the two lets the drawer route dispatch
|
||||||
|
on type instead of sniffing the error string (T81).
|
||||||
"""
|
"""
|
||||||
chat = get_chat(conn, chat_id)
|
chat = get_chat(conn, chat_id)
|
||||||
if chat is None:
|
if chat is None:
|
||||||
raise ValueError(f"chat not found: {chat_id}")
|
raise ChatNotFoundError(f"chat not found: {chat_id}")
|
||||||
|
|
||||||
_validate_new_time(chat, new_time)
|
_validate_new_time(chat, new_time)
|
||||||
|
|
||||||
@@ -163,6 +176,7 @@ async def process_jump_skip(
|
|||||||
new_time: str,
|
new_time: str,
|
||||||
notable_prose: str = "",
|
notable_prose: str = "",
|
||||||
reset_activity: bool = False,
|
reset_activity: bool = False,
|
||||||
|
app=None,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Run a jump skip end-to-end.
|
"""Run a jump skip end-to-end.
|
||||||
|
|
||||||
@@ -178,11 +192,13 @@ async def process_jump_skip(
|
|||||||
Returns ``{"assistant_text": ..., "speaker_id": ..., "skip_event_id":
|
Returns ``{"assistant_text": ..., "speaker_id": ..., "skip_event_id":
|
||||||
..., "assistant_event_id": ...}``.
|
..., "assistant_event_id": ...}``.
|
||||||
|
|
||||||
Raises ``ValueError`` on validation failure (caller maps to ``400``).
|
Raises :class:`ChatNotFoundError` on missing chat (caller maps to
|
||||||
|
``404``) and ``ValueError`` on input-validation failure (caller maps
|
||||||
|
to ``400``).
|
||||||
"""
|
"""
|
||||||
chat = get_chat(conn, chat_id)
|
chat = get_chat(conn, chat_id)
|
||||||
if chat is None:
|
if chat is None:
|
||||||
raise ValueError(f"chat not found: {chat_id}")
|
raise ChatNotFoundError(f"chat not found: {chat_id}")
|
||||||
|
|
||||||
_validate_new_time(chat, new_time)
|
_validate_new_time(chat, new_time)
|
||||||
|
|
||||||
@@ -240,6 +256,7 @@ async def process_jump_skip(
|
|||||||
chat_clock_at=new_time,
|
chat_clock_at=new_time,
|
||||||
source="synthesized",
|
source="synthesized",
|
||||||
significance=mem.significance,
|
significance=mem.significance,
|
||||||
|
app=app,
|
||||||
)
|
)
|
||||||
|
|
||||||
narration = await narrate_skip(
|
narration = await narrate_skip(
|
||||||
@@ -280,6 +297,7 @@ def _now_iso() -> str:
|
|||||||
|
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"ChatNotFoundError",
|
||||||
"process_elision_skip",
|
"process_elision_skip",
|
||||||
"process_jump_skip",
|
"process_jump_skip",
|
||||||
"_now_iso",
|
"_now_iso",
|
||||||
|
|||||||
@@ -0,0 +1,216 @@
|
|||||||
|
"""Snapshot UX routes (T99).
|
||||||
|
|
||||||
|
Surfaces the existing snapshot service (``chat/services/snapshot.py``)
|
||||||
|
through HTML so the user can see, take, restore, and preview snapshots
|
||||||
|
without dropping to a shell.
|
||||||
|
|
||||||
|
Routes:
|
||||||
|
|
||||||
|
* ``GET /snapshots`` list all snapshots (both kinds)
|
||||||
|
* ``POST /snapshots/take`` take a periodic snapshot now
|
||||||
|
* ``POST /snapshots/restore/{id}`` restore (requires matching ``confirm_id`` and ``kind``)
|
||||||
|
* ``GET /snapshots/{id}/preview`` show metadata + delta vs current
|
||||||
|
|
||||||
|
The ``snapshot_id`` is the filename stem (the UTC timestamp written by
|
||||||
|
:func:`chat.services.snapshot.take_snapshot`) — there's no separate UUID,
|
||||||
|
and the timestamp filename is already unique per snapshot kind. Both
|
||||||
|
periodic and rewind snapshots share the same id space lookup-wise, so
|
||||||
|
the restore + preview routes require ``kind`` as a form/query param to
|
||||||
|
disambiguate (a missing/empty ``kind`` is a 400, not a silent default).
|
||||||
|
|
||||||
|
Note on ``created_at`` mtime drift: the listing's ``created_at`` comes
|
||||||
|
from the file's mtime, not the encoded filename timestamp. ``cp -p``
|
||||||
|
preserves mtime, but plain ``cp`` resets it to "now" — so a copied
|
||||||
|
snapshot can show a misleading ``created_at`` while its filename still
|
||||||
|
reflects the original UTC capture time.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Depends, Form, HTTPException, Request
|
||||||
|
from fastapi.responses import HTMLResponse, RedirectResponse
|
||||||
|
from fastapi.templating import Jinja2Templates
|
||||||
|
|
||||||
|
from chat.services.snapshot import (
|
||||||
|
restore_from_snapshot,
|
||||||
|
take_snapshot,
|
||||||
|
)
|
||||||
|
from chat.web.bots import get_conn
|
||||||
|
|
||||||
|
TEMPLATES = Jinja2Templates(
|
||||||
|
directory=str(Path(__file__).resolve().parent.parent / "templates")
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
SNAPSHOT_KINDS = ("periodic", "rewind")
|
||||||
|
|
||||||
|
|
||||||
|
def _list_all_snapshots(data_dir: Path) -> list[dict]:
|
||||||
|
"""Walk ``data/snapshots/{kind}/`` for both kinds and collect metadata.
|
||||||
|
|
||||||
|
Each entry exposes the fields the template needs: ``snapshot_id``
|
||||||
|
(filename stem), ``kind``, ``created_at`` (file mtime as ISO), the
|
||||||
|
on-disk ``file_size_bytes``, and the snapshot's stored
|
||||||
|
``last_event_id`` (parsed from the JSON body — small enough that
|
||||||
|
listing isn't a performance concern for the handful of files we keep).
|
||||||
|
"""
|
||||||
|
rows: list[dict] = []
|
||||||
|
for kind in SNAPSHOT_KINDS:
|
||||||
|
snap_dir = data_dir / "snapshots" / kind
|
||||||
|
if not snap_dir.exists():
|
||||||
|
continue
|
||||||
|
for path in sorted(snap_dir.glob("*.json")):
|
||||||
|
try:
|
||||||
|
dump = json.loads(path.read_text())
|
||||||
|
last_event_id = dump.get("last_event_id", 0)
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
# Corrupt or unreadable files still get listed so the
|
||||||
|
# user can see and delete them; just don't crash here.
|
||||||
|
last_event_id = None
|
||||||
|
stat = path.stat()
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"snapshot_id": path.stem,
|
||||||
|
"kind": kind,
|
||||||
|
"created_at": datetime.fromtimestamp(
|
||||||
|
stat.st_mtime, tz=timezone.utc
|
||||||
|
).isoformat(),
|
||||||
|
"file_size_bytes": stat.st_size,
|
||||||
|
"last_event_id": last_event_id,
|
||||||
|
"filename": path.name,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
# Newest first for display.
|
||||||
|
rows.sort(key=lambda r: r["created_at"], reverse=True)
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _require_kind(kind: str) -> str:
|
||||||
|
"""Reject missing/empty/unknown ``kind`` with 400.
|
||||||
|
|
||||||
|
Defaulting silently to ``"periodic"`` made rewind-snapshot lookups
|
||||||
|
appear as 404s, which is confusing — make the client always state
|
||||||
|
the kind explicitly.
|
||||||
|
"""
|
||||||
|
if not kind or kind not in SNAPSHOT_KINDS:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"kind must be one of {SNAPSHOT_KINDS}",
|
||||||
|
)
|
||||||
|
return kind
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_snapshot_path(
|
||||||
|
data_dir: Path, snapshot_id: str, kind: str
|
||||||
|
) -> Path:
|
||||||
|
"""Map an ``(id, kind)`` pair to the on-disk file, or 404."""
|
||||||
|
_require_kind(kind)
|
||||||
|
path = data_dir / "snapshots" / kind / f"{snapshot_id}.json"
|
||||||
|
if not path.exists():
|
||||||
|
raise HTTPException(status_code=404, detail="snapshot not found")
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/snapshots", response_class=HTMLResponse)
|
||||||
|
async def snapshots_list(request: Request):
|
||||||
|
settings = request.app.state.settings
|
||||||
|
rows = _list_all_snapshots(settings.data_dir)
|
||||||
|
return TEMPLATES.TemplateResponse(
|
||||||
|
request,
|
||||||
|
"snapshots.html",
|
||||||
|
{"snapshots": rows, "active_nav": "snapshots"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/snapshots/take")
|
||||||
|
async def snapshots_take(request: Request, conn=Depends(get_conn)):
|
||||||
|
"""Take a periodic snapshot now.
|
||||||
|
|
||||||
|
We use ``kind="periodic"`` for manual snapshots since they're
|
||||||
|
user-initiated checkpoints, not pre-rewind safety dumps. They count
|
||||||
|
against the 5-snapshot retention but that's fine — manual ones are
|
||||||
|
the most recent so they're the last to be pruned.
|
||||||
|
"""
|
||||||
|
settings = request.app.state.settings
|
||||||
|
take_snapshot(conn, data_dir=settings.data_dir, kind="periodic")
|
||||||
|
return RedirectResponse(url="/snapshots", status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/snapshots/restore/{snapshot_id}")
|
||||||
|
async def snapshots_restore(
|
||||||
|
snapshot_id: str,
|
||||||
|
request: Request,
|
||||||
|
confirm_id: str = Form(""),
|
||||||
|
kind: str = Form(""),
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""Hard-confirm restore: ``confirm_id`` must equal the path id.
|
||||||
|
|
||||||
|
Mismatched confirm → 400 (without touching the DB). On match, the
|
||||||
|
existing :func:`restore_from_snapshot` clears projected tables and
|
||||||
|
re-loads them from the dump.
|
||||||
|
|
||||||
|
``kind`` is required (must be ``"periodic"`` or ``"rewind"``) — a
|
||||||
|
missing or empty value 400s rather than silently defaulting.
|
||||||
|
"""
|
||||||
|
_require_kind(kind)
|
||||||
|
if confirm_id != snapshot_id:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail="confirm_id does not match snapshot id",
|
||||||
|
)
|
||||||
|
settings = request.app.state.settings
|
||||||
|
path = _resolve_snapshot_path(settings.data_dir, snapshot_id, kind)
|
||||||
|
restore_from_snapshot(conn, path)
|
||||||
|
return RedirectResponse(url="/snapshots", status_code=303)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/snapshots/{snapshot_id}/preview", response_class=HTMLResponse)
|
||||||
|
async def snapshots_preview(
|
||||||
|
snapshot_id: str,
|
||||||
|
request: Request,
|
||||||
|
kind: str = "",
|
||||||
|
conn=Depends(get_conn),
|
||||||
|
):
|
||||||
|
"""Show snapshot metadata + a basic delta against the current event log.
|
||||||
|
|
||||||
|
Phase 4 keeps this simple: the snapshot's ``last_event_id`` plus the
|
||||||
|
current ``MAX(event_log.id)`` is enough to tell the user how far the
|
||||||
|
log has moved on. A richer per-table diff is a Phase 4.5+ concern.
|
||||||
|
|
||||||
|
``kind`` is required — see :func:`snapshots_restore`.
|
||||||
|
"""
|
||||||
|
_require_kind(kind)
|
||||||
|
settings = request.app.state.settings
|
||||||
|
path = _resolve_snapshot_path(settings.data_dir, snapshot_id, kind)
|
||||||
|
dump = json.loads(path.read_text())
|
||||||
|
last_event_id = dump.get("last_event_id", 0)
|
||||||
|
|
||||||
|
cur = conn.execute("SELECT MAX(id) FROM event_log")
|
||||||
|
row = cur.fetchone()
|
||||||
|
current_max_id = row[0] if row[0] is not None else 0
|
||||||
|
|
||||||
|
stat = path.stat()
|
||||||
|
return TEMPLATES.TemplateResponse(
|
||||||
|
request,
|
||||||
|
"snapshots.html",
|
||||||
|
{
|
||||||
|
"snapshots": _list_all_snapshots(settings.data_dir),
|
||||||
|
"active_nav": "snapshots",
|
||||||
|
"preview": {
|
||||||
|
"snapshot_id": snapshot_id,
|
||||||
|
"kind": kind,
|
||||||
|
"filename": path.name,
|
||||||
|
"file_size_bytes": stat.st_size,
|
||||||
|
"last_event_id": last_event_id,
|
||||||
|
"current_event_log_max_id": current_max_id,
|
||||||
|
"event_delta": current_max_id - last_event_id,
|
||||||
|
"event_log_rows_in_snapshot": len(dump.get("event_log", [])),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
+117
-52
@@ -64,10 +64,17 @@ from chat.services.event_promotion import promote_completed_event
|
|||||||
from chat.services.interjection import detect_interjection
|
from chat.services.interjection import detect_interjection
|
||||||
from chat.services.memory_write import record_turn_memory_for_present
|
from chat.services.memory_write import record_turn_memory_for_present
|
||||||
from chat.services.multi_state_update import compute_state_updates_for_present
|
from chat.services.multi_state_update import compute_state_updates_for_present
|
||||||
from chat.services.prompt import assemble_narrative_prompt
|
from chat.services.prompt import (
|
||||||
|
assemble_narrative_prompt,
|
||||||
|
consume_pending_meanwhile_digests,
|
||||||
|
)
|
||||||
from chat.services.rewind import compute_rewind_preview, execute_rewind
|
from chat.services.rewind import compute_rewind_preview, execute_rewind
|
||||||
from chat.services.scene_close import detect_scene_close
|
from chat.services.scene_close import detect_scene_close
|
||||||
from chat.services.scene_summarize import apply_scene_close_summary
|
from chat.services.scene_summarize import apply_scene_close_summary
|
||||||
|
from chat.services.turn_common import (
|
||||||
|
gather_prior_edges,
|
||||||
|
read_recent_dialogue,
|
||||||
|
)
|
||||||
from chat.services.turn_parse import ParsedTurn, parse_turn
|
from chat.services.turn_parse import ParsedTurn, parse_turn
|
||||||
from chat.state.edges import get_edge
|
from chat.state.edges import get_edge
|
||||||
from chat.state.entities import get_bot, get_you
|
from chat.state.entities import get_bot, get_you
|
||||||
@@ -79,7 +86,11 @@ from chat.web.kickoff import get_llm_client
|
|||||||
from chat.web.meanwhile import process_meanwhile_turn
|
from chat.web.meanwhile import process_meanwhile_turn
|
||||||
from chat.web.pubsub import publish
|
from chat.web.pubsub import publish
|
||||||
from chat.web.render import render_turn_html as _render_turn_html
|
from chat.web.render import render_turn_html as _render_turn_html
|
||||||
from chat.web.skip import _parse_iso_time, process_elision_skip
|
from chat.web.skip import (
|
||||||
|
ChatNotFoundError,
|
||||||
|
_parse_iso_time,
|
||||||
|
process_elision_skip,
|
||||||
|
)
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
@@ -106,38 +117,13 @@ def _strip_ooc_for_prompt(parsed: ParsedTurn) -> str:
|
|||||||
def _read_recent_dialogue(conn, chat_id: str, limit: int = 200) -> list[dict]:
|
def _read_recent_dialogue(conn, chat_id: str, limit: int = 200) -> list[dict]:
|
||||||
"""Return user-side and assistant_turn events for ``chat_id``.
|
"""Return user-side and assistant_turn events for ``chat_id``.
|
||||||
|
|
||||||
Includes ``user_turn``, ``user_turn_edit`` (T29 edited prose), and
|
T83.2: thin delegate over
|
||||||
``assistant_turn``. Ordered oldest-first; superseded/hidden rows are
|
:func:`chat.services.turn_common.read_recent_dialogue` so post_turn
|
||||||
skipped so regenerated turns (T29) drop out of the rendered timeline.
|
and regenerate share one implementation. The wrapper survives so
|
||||||
Each entry is shaped ``{"speaker": <id-or-"you">, "text": <prose>}``
|
the chat-detail template and other callers in this module don't all
|
||||||
for the prompt assembler and the chat-detail template.
|
have to update at once.
|
||||||
"""
|
"""
|
||||||
cur = conn.execute(
|
return read_recent_dialogue(conn, chat_id, limit=limit)
|
||||||
"SELECT id, kind, payload_json FROM event_log "
|
|
||||||
"WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn') "
|
|
||||||
" AND superseded_by IS NULL AND hidden = 0 "
|
|
||||||
"ORDER BY id DESC LIMIT ?",
|
|
||||||
(limit,),
|
|
||||||
)
|
|
||||||
rows = cur.fetchall()
|
|
||||||
rows.reverse() # back to chronological order
|
|
||||||
out: list[dict] = []
|
|
||||||
for _row_id, kind, payload_json in rows:
|
|
||||||
p = json.loads(payload_json)
|
|
||||||
if p.get("chat_id") != chat_id:
|
|
||||||
continue
|
|
||||||
if kind in ("user_turn", "user_turn_edit"):
|
|
||||||
# Edited prose substitutes for the original user_turn (the
|
|
||||||
# original is marked superseded_by and filtered above).
|
|
||||||
out.append({"speaker": "you", "text": p.get("prose", "")})
|
|
||||||
else:
|
|
||||||
out.append(
|
|
||||||
{
|
|
||||||
"speaker": p.get("speaker_id", "bot"),
|
|
||||||
"text": p.get("text", ""),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _detect_addressee_id(
|
def _detect_addressee_id(
|
||||||
@@ -204,17 +190,8 @@ def _gather_state_update_inputs(
|
|||||||
present_names[guest_bot["id"]] = guest_bot["name"]
|
present_names[guest_bot["id"]] = guest_bot["name"]
|
||||||
personas[guest_bot["id"]] = guest_bot.get("persona") or ""
|
personas[guest_bot["id"]] = guest_bot.get("persona") or ""
|
||||||
|
|
||||||
prior_edges: dict[tuple[str, str], dict] = {}
|
# T83.2: directed-edge gather is shared with regenerate.py.
|
||||||
for src in present_ids:
|
prior_edges = gather_prior_edges(conn, present_ids)
|
||||||
for tgt in present_ids:
|
|
||||||
if src == tgt:
|
|
||||||
continue
|
|
||||||
edge = get_edge(conn, src, tgt) or {
|
|
||||||
"affinity": 50,
|
|
||||||
"trust": 50,
|
|
||||||
"summary": "",
|
|
||||||
}
|
|
||||||
prior_edges[(src, tgt)] = edge
|
|
||||||
return present_ids, present_names, personas, prior_edges
|
return present_ids, present_names, personas, prior_edges
|
||||||
|
|
||||||
|
|
||||||
@@ -271,6 +248,7 @@ async def post_turn(
|
|||||||
settings,
|
settings,
|
||||||
chat_id=chat_id,
|
chat_id=chat_id,
|
||||||
prose=prose,
|
prose=prose,
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise HTTPException(status_code=400, detail=str(exc))
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
@@ -310,6 +288,49 @@ async def post_turn(
|
|||||||
)
|
)
|
||||||
|
|
||||||
if intent == "skip_elision":
|
if intent == "skip_elision":
|
||||||
|
# T82.2: run scene-close detection on the user's prose BEFORE
|
||||||
|
# the skip controller fires. Prose like "fade out, skip an hour"
|
||||||
|
# carries both a close signal and a skip directive; we want the
|
||||||
|
# close summary to capture the closing scene's final beat (and
|
||||||
|
# promote per-POV memories) before the time advances. Order
|
||||||
|
# matters: scene close -> skip narration -> time advance.
|
||||||
|
#
|
||||||
|
# When there's no active scene (or the prose carries no close
|
||||||
|
# signal) ``detect_scene_close`` returns the safe
|
||||||
|
# ``should_close=False`` default and we drop straight to the
|
||||||
|
# skip controller — same behavior as today, no extra cost.
|
||||||
|
skip_scene = active_scene(conn, chat_id)
|
||||||
|
if skip_scene is not None:
|
||||||
|
container = None
|
||||||
|
if skip_scene.get("container_id") is not None:
|
||||||
|
container = get_container(conn, skip_scene["container_id"])
|
||||||
|
container_name = container["name"] if container else "unknown"
|
||||||
|
close_decision = await detect_scene_close(
|
||||||
|
client,
|
||||||
|
model=settings.classifier_model,
|
||||||
|
prose=prose,
|
||||||
|
current_container_name=container_name,
|
||||||
|
)
|
||||||
|
if close_decision.should_close:
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="scene_closed",
|
||||||
|
payload={
|
||||||
|
"scene_id": skip_scene["id"],
|
||||||
|
"ended_at": chat.get("time"),
|
||||||
|
"significance": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model=settings.classifier_model,
|
||||||
|
chat_id=chat_id,
|
||||||
|
scene_id=skip_scene["id"],
|
||||||
|
host_bot_id=host_bot["id"],
|
||||||
|
timeout_s=settings.classifier_timeout_s,
|
||||||
|
)
|
||||||
|
|
||||||
# Derive ``new_time`` from the chat clock. Phase 3 stub: bump by
|
# Derive ``new_time`` from the chat clock. Phase 3 stub: bump by
|
||||||
# 1 hour. The drawer's elision form is the structured path when
|
# 1 hour. The drawer's elision form is the structured path when
|
||||||
# the author wants a specific landing time; here the goal is
|
# the author wants a specific landing time; here the goal is
|
||||||
@@ -332,12 +353,17 @@ async def post_turn(
|
|||||||
new_time=new_time,
|
new_time=new_time,
|
||||||
landing_state_hint=getattr(parsed, "landing_state_hint", "")
|
landing_state_hint=getattr(parsed, "landing_state_hint", "")
|
||||||
or "",
|
or "",
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
|
except ChatNotFoundError as exc:
|
||||||
|
# Defensive: chat existence is checked above, so this only
|
||||||
|
# fires on a TOCTOU race where the chat row is deleted
|
||||||
|
# mid-request. T81 split the typed missing-chat case out of
|
||||||
|
# the generic ValueError so we keep the 404 mapping here.
|
||||||
|
raise HTTPException(status_code=404, detail=str(exc))
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
# The controller raises on missing chat / bad new_time.
|
# Bad new_time is a stub-derivation bug rather than user
|
||||||
# Missing chat is already handled above (we'd have 404'd);
|
# input — surface as 400 with the controller message.
|
||||||
# a bad new_time here is a stub-derivation bug rather than
|
|
||||||
# user input — surface as 400 with the controller message.
|
|
||||||
raise HTTPException(status_code=400, detail=str(exc))
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
return Response(status_code=204)
|
return Response(status_code=204)
|
||||||
|
|
||||||
@@ -459,7 +485,11 @@ async def post_turn(
|
|||||||
|
|
||||||
# 7. Append the assistant_turn with the final text. (See note above on
|
# 7. Append the assistant_turn with the final text. (See note above on
|
||||||
# why we skip ``project`` for these transcript-only event kinds.)
|
# why we skip ``project`` for these transcript-only event kinds.)
|
||||||
append_event(
|
# Capture the returned event id so we can stamp ``id="turn-<n>"`` on
|
||||||
|
# the SSE-emitted HTML fragment — the chat-page ``turn_html_replace``
|
||||||
|
# handler relies on the id to swap regenerated turns in-place
|
||||||
|
# (T86 follow-up).
|
||||||
|
primary_assistant_event_id = append_event(
|
||||||
conn,
|
conn,
|
||||||
kind="assistant_turn",
|
kind="assistant_turn",
|
||||||
payload={
|
payload={
|
||||||
@@ -484,6 +514,7 @@ async def post_turn(
|
|||||||
narrative_text=primary_text,
|
narrative_text=primary_text,
|
||||||
scene_id=scene["id"] if scene else None,
|
scene_id=scene["id"] if scene else None,
|
||||||
chat_clock_at=chat.get("time"),
|
chat_clock_at=chat.get("time"),
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
|
|
||||||
# 7b. Post-turn state-update pass (Requirements §3.4 / T40). All
|
# 7b. Post-turn state-update pass (Requirements §3.4 / T40). All
|
||||||
@@ -559,6 +590,7 @@ async def post_turn(
|
|||||||
interjection_text: str | None = None
|
interjection_text: str | None = None
|
||||||
interjection_speaker_id: str | None = None
|
interjection_speaker_id: str | None = None
|
||||||
interjection_truncated = False
|
interjection_truncated = False
|
||||||
|
interjection_event_id: int | None = None
|
||||||
if (
|
if (
|
||||||
guest_bot is not None
|
guest_bot is not None
|
||||||
and not cancelled
|
and not cancelled
|
||||||
@@ -646,7 +678,9 @@ async def post_turn(
|
|||||||
|
|
||||||
interjection_text = "".join(interject_accumulated)
|
interjection_text = "".join(interject_accumulated)
|
||||||
|
|
||||||
append_event(
|
# Capture the event id (T86 follow-up) so the SSE fragment
|
||||||
|
# below carries ``id="turn-<n>"`` for in-place swap.
|
||||||
|
interjection_event_id = append_event(
|
||||||
conn,
|
conn,
|
||||||
kind="assistant_turn",
|
kind="assistant_turn",
|
||||||
payload={
|
payload={
|
||||||
@@ -715,6 +749,7 @@ async def post_turn(
|
|||||||
narrative_text=interjection_text,
|
narrative_text=interjection_text,
|
||||||
scene_id=scene["id"] if scene else None,
|
scene_id=scene["id"] if scene else None,
|
||||||
chat_clock_at=chat.get("time"),
|
chat_clock_at=chat.get("time"),
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
|
|
||||||
# T74.2: enqueue a significance pass for the interjection
|
# T74.2: enqueue a significance pass for the interjection
|
||||||
@@ -838,6 +873,20 @@ async def post_turn(
|
|||||||
# mid-stream still meant to close the scene — the cancelled bot
|
# mid-stream still meant to close the scene — the cancelled bot
|
||||||
# beat doesn't invalidate that intent. Pinned by
|
# beat doesn't invalidate that intent. Pinned by
|
||||||
# test_cancelled_turn_still_closes_scene_when_user_prose_signals_close.
|
# test_cancelled_turn_still_closes_scene_when_user_prose_signals_close.
|
||||||
|
#
|
||||||
|
# T108 NOTE — the in-memory append order is correct, but the cancel
|
||||||
|
# path re-raises ``CancelledError`` at the end of ``post_turn``
|
||||||
|
# (see step 11 below). The ``open_db`` dependency teardown skips
|
||||||
|
# ``conn.commit()`` when the consumer raises, which means in
|
||||||
|
# production a genuine cancel currently rolls back ALL post-cancel
|
||||||
|
# writes — including this scene_closed event, the truncated
|
||||||
|
# assistant_turn record, edge updates, and per-POV summaries. The
|
||||||
|
# T74.3 regression test passes only because of a missing
|
||||||
|
# ``import asyncio`` in the test module: the inline mock raises
|
||||||
|
# ``NameError`` instead of ``CancelledError``, which is caught by
|
||||||
|
# the ``except Exception:`` branch and leaves ``cancelled=False``,
|
||||||
|
# so the function returns 204 normally and the commit fires. This
|
||||||
|
# is a transactional bug deferred for triage (T108 report).
|
||||||
if scene is not None and prose.strip():
|
if scene is not None and prose.strip():
|
||||||
container = None
|
container = None
|
||||||
if scene.get("container_id") is not None:
|
if scene.get("container_id") is not None:
|
||||||
@@ -878,6 +927,15 @@ async def post_turn(
|
|||||||
timeout_s=settings.classifier_timeout_s,
|
timeout_s=settings.classifier_timeout_s,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# 9a. Consume any pending meanwhile digests now that the assistant_turn
|
||||||
|
# (which surfaced them in its prompt via T65's helper) has landed. The
|
||||||
|
# spec's "first you-turn AFTER meanwhile close consumes the digest"
|
||||||
|
# semantics are preserved by running this AFTER scene-close detection
|
||||||
|
# — anything pending right now belongs to the prompt we just answered,
|
||||||
|
# so it's safe to mark consumed and the NEXT turn starts clean.
|
||||||
|
# Idempotent: re-calling produces zero events when nothing's pending.
|
||||||
|
consume_pending_meanwhile_digests(conn, chat_id)
|
||||||
|
|
||||||
# 10. Broadcast a JSON completion event (for JS consumers) and an HTML
|
# 10. Broadcast a JSON completion event (for JS consumers) and an HTML
|
||||||
# fragment event (for HTMX SSE swap-into-timeline). One pair per
|
# fragment event (for HTMX SSE swap-into-timeline). One pair per
|
||||||
# written assistant_turn so the timeline ends up with both the
|
# written assistant_turn so the timeline ends up with both the
|
||||||
@@ -892,7 +950,10 @@ async def post_turn(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
primary_html = _render_turn_html(
|
primary_html = _render_turn_html(
|
||||||
addressee_bot["name"], primary_text, role="bot"
|
addressee_bot["name"],
|
||||||
|
primary_text,
|
||||||
|
role="bot",
|
||||||
|
event_id=primary_assistant_event_id,
|
||||||
)
|
)
|
||||||
await publish(
|
await publish(
|
||||||
chat_id, {"event": "turn_html", "data": primary_html}
|
chat_id, {"event": "turn_html", "data": primary_html}
|
||||||
@@ -916,7 +977,10 @@ async def post_turn(
|
|||||||
},
|
},
|
||||||
)
|
)
|
||||||
interject_html = _render_turn_html(
|
interject_html = _render_turn_html(
|
||||||
interject_speaker_name, interjection_text, role="bot"
|
interject_speaker_name,
|
||||||
|
interjection_text,
|
||||||
|
role="bot",
|
||||||
|
event_id=interjection_event_id,
|
||||||
)
|
)
|
||||||
await publish(
|
await publish(
|
||||||
chat_id, {"event": "turn_html", "data": interject_html}
|
chat_id, {"event": "turn_html", "data": interject_html}
|
||||||
@@ -1046,6 +1110,7 @@ async def regenerate_turn(
|
|||||||
chat_id=chat_id,
|
chat_id=chat_id,
|
||||||
original_assistant_event_id=event_id,
|
original_assistant_event_id=event_id,
|
||||||
edited_user_prose=edited_prose,
|
edited_user_prose=edited_prose,
|
||||||
|
app=request.app,
|
||||||
)
|
)
|
||||||
except ValueError as e:
|
except ValueError as e:
|
||||||
raise HTTPException(status_code=404, detail=str(e))
|
raise HTTPException(status_code=404, detail=str(e))
|
||||||
|
|||||||
@@ -520,6 +520,8 @@ Written per witness when a scene closes. Different details, different interpreta
|
|||||||
|
|
||||||
### Phase 4 — polish
|
### Phase 4 — polish
|
||||||
|
|
||||||
|
**Status: shipped 2026-04-27** (T88–T102, 15 tasks across 8 waves; +70 tests). See "Phase 4 status" in CLAUDE.md for the per-task breakdown. Vector retrieval shipped via pure-Python cosine over a JSON-blob embeddings table (sqlite-vec deferred — host Python lacks loadable extensions); branching is data-model + drawer UI; significance review, hide-from-view soft delete, surgical delete with cascade preview, snapshot UX, and cross-chat search all surface from the drawer or top-bar.
|
||||||
|
|
||||||
- Vector retrieval (sqlite-vss or sqlite-vec).
|
- Vector retrieval (sqlite-vss or sqlite-vec).
|
||||||
- Branching UI.
|
- Branching UI.
|
||||||
- Drawer-edit on every field.
|
- Drawer-edit on every field.
|
||||||
|
|||||||
@@ -0,0 +1,709 @@
|
|||||||
|
# Roleplay Engine — Phase 3.5 Cleanup Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use `superpowers-extended-cc:executing-plans` to implement this plan task-by-task. Use the parallel-dispatch pattern documented under "Parallel-Execution Strategy" for waves that fan out to multiple subagents.
|
||||||
|
|
||||||
|
**Goal:** Burn down the combined Phase 2.6/3 + Phase 3.5/4 backlog tracked in [`CLAUDE.md`](../../CLAUDE.md). 17 follow-up items consolidated into 12 tasks (file-disjoint where possible) across 7 waves so several can run in parallel.
|
||||||
|
|
||||||
|
**Architecture:** No new architecture, no schema migrations, no new event kinds. Every change here is either a polish on an existing service/route, a refactor for DRY/typing/observability, or a UI affordance for state that already exists (frontend `turn_html_replace` consumer).
|
||||||
|
|
||||||
|
**Tech Stack:** Same as Phase 3. No new dependencies.
|
||||||
|
|
||||||
|
**Source-of-truth references:**
|
||||||
|
|
||||||
|
- Backlog list: [`CLAUDE.md`](../../CLAUDE.md) §"Phase 2.6 / 3 backlog" (7 items, 6 actionable + 1 deferred) + §"Phase 3.5 / 4 backlog" (~13 items grouped by source review) = 17 items net.
|
||||||
|
- Conventions: [`CLAUDE.md`](../../CLAUDE.md) §"Behavioral defaults" + §"Phase 3 status".
|
||||||
|
- Phase 2.5 cleanup plan (style, file-bundling pattern): [2026-04-26-v2.5-phase2.5-cleanup.md](2026-04-26-v2.5-phase2.5-cleanup.md).
|
||||||
|
|
||||||
|
When a task says "see §X", that's the requirements doc unless stated otherwise.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Pre-flight
|
||||||
|
|
||||||
|
**Branch:** create `phase-3.5` from the latest `main` after Phase 3 has merged (it has — main is at `753cec3`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git checkout main && git pull && git checkout -b phase-3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
**Schema baseline:** Phase 3 leaves the DB at version 11. Phase 3.5 adds **no migrations**. Schema-version assertion in `tests/test_world.py` stays at 11.
|
||||||
|
|
||||||
|
**Pinned non-negotiables (carried forward):**
|
||||||
|
|
||||||
|
- State changes go through the event log. Use `append_and_apply(conn, kind, payload)` for the live path; `apply_event` only after a fresh `append_event` returning the new id.
|
||||||
|
- Witness filter every memory read at SQL level (hard `WHERE` constraint; never a soft signal).
|
||||||
|
- Edges are directed; `botA → botB` and `botB → botA` are independent records.
|
||||||
|
- Per-POV scene summaries — never write omniscient narration.
|
||||||
|
- TDD: every task starts with a failing test (or a regression test pinning existing contract before refactor).
|
||||||
|
- One commit per task minimum. Tasks that bundle multiple backlog items SHOULD split commits within the task — one commit per item — so review can bisect cleanly.
|
||||||
|
|
||||||
|
**Verification before claiming done:** Use `superpowers-extended-cc:verification-before-completion` — run the test command, paste actual output. Don't assume green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Backlog item → task mapping
|
||||||
|
|
||||||
|
17 items consolidated into 12 tasks by **file ownership** (so each wave's tasks stay file-disjoint). Bundled tasks split commits internally.
|
||||||
|
|
||||||
|
| # | Backlog item | Source | Task |
|
||||||
|
|---|--------------|--------|------|
|
||||||
|
| 1 | `narrate_skip` `timeout_s` not piped through | T53 review | **T76** |
|
||||||
|
| 2 | `AddresseeDecision.confidence` should be `Literal[...]` | T74 review (Phase 2.5 carry-over) | **T77** |
|
||||||
|
| 3 | `search_memories` docstring missing SQL-bias note | T57 review | **T78** |
|
||||||
|
| 4 | `_witness_role_for` defensive coding for `host_bot_id is None` | T71 review (Phase 2.5 carry-over) | **T79** |
|
||||||
|
| 5 | Scene close re-close suffix bloat risk | T58 review | **T80** |
|
||||||
|
| 6 | Thread detection transcript scoping (chat-wide vs scene-scoped) | T58 review | **T80** |
|
||||||
|
| 7 | Swallowed `Exception` in `detect_threads` try/except — log at debug | T58 review | **T80** |
|
||||||
|
| 8 | Scene close `closed_at` clock divergence | T58 review | **T80** |
|
||||||
|
| 9 | T58 test coverage gaps (truncation, update/close, fallback) | T58 review | **T80** |
|
||||||
|
| 10 | Error-message prefix sniff for 404 vs 400 routing in skip routes | T62 review | **T81** |
|
||||||
|
| 11 | `consume_pending_meanwhile_digests` not wired into `post_turn` | T66 integration tests | **T82** |
|
||||||
|
| 12 | Skip command bypasses scene close detection | T62 review | **T82** |
|
||||||
|
| 13 | Cancel/stop hook for in-flight regenerate streams | T73 review (Phase 2.5 carry-over) | **T83** |
|
||||||
|
| 14 | DRY: regenerate vs post_turn (recent-dialogue + prior-edges duplication) | T73 review (Phase 2.5 carry-over) | **T83** |
|
||||||
|
| 15 | Sibling-discovery query optimization in regenerate.py | T73 review (Phase 2.5 carry-over) | **T83** |
|
||||||
|
| 16 | Regenerate doesn't roll back lifecycle transitions from superseded turn | T61 review | **T83** |
|
||||||
|
| 17 | Asymmetry in event-detection ordering (turns.py vs regenerate.py) | T61 review | **T83** |
|
||||||
|
| 18 | `record_meanwhile_memory` / `record_turn_memory_for_present` unified API | T64 review | **T84** |
|
||||||
|
| 19 | `participants_json` JSON-build audit (other state modules) | T63 review | **T85** |
|
||||||
|
| 20 | Stop-button cancellation route-level coverage for meanwhile turns | T64 review | **T85** |
|
||||||
|
| 21 | Frontend handler for `turn_html_replace` SSE event | T73.1 review (Phase 2.5 carry-over) | **T86** |
|
||||||
|
| — | Docs sweep — remove shipped items, capture residuals | (this plan) | **T87** |
|
||||||
|
|
||||||
|
**Deferred (not in this plan):**
|
||||||
|
|
||||||
|
- **Cross-feature canned-queue brittleness** (Wave 6b cross-feature): would require a structured-fixture refactor across 3+ test files. Tracking but not in scope here — open as its own work if it surfaces a real test-maintenance pain.
|
||||||
|
- **Scene-close-on-cancel UX revisit** (T74.3): pinned to existing behavior; no action needed unless real play surfaces a regression.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parallel-Execution Strategy
|
||||||
|
|
||||||
|
Same pattern as Phases 2.5 and 3. Seven waves: parallel within each wave (file-disjoint), serial across waves.
|
||||||
|
|
||||||
|
### How to dispatch a wave in parallel
|
||||||
|
|
||||||
|
Use the **Agent tool with `isolation: "worktree"`** so each subagent gets its own git worktree. (If the controlling session's working directory is **not** the chat repo, create worktrees manually with `git worktree add .worktrees/<wave>-<task> -b <wave>/<task> phase-3.5` from inside the chat repo and pass the worktree path explicitly into each subagent prompt.)
|
||||||
|
|
||||||
|
In a single message, dispatch all tasks in the wave:
|
||||||
|
|
||||||
|
```
|
||||||
|
Agent({
|
||||||
|
description: "Wave 1 — T76 narrate_skip timeout_s",
|
||||||
|
subagent_type: "general-purpose",
|
||||||
|
isolation: "worktree",
|
||||||
|
prompt: "<full task text from below>",
|
||||||
|
})
|
||||||
|
Agent({ ...T77... })
|
||||||
|
Agent({ ...T78... })
|
||||||
|
Agent({ ...T79... })
|
||||||
|
```
|
||||||
|
|
||||||
|
### After a wave completes
|
||||||
|
|
||||||
|
1. Each subagent returns its worktree path and commit SHA(s).
|
||||||
|
2. **Run a spec + code-quality reviewer subagent on each completed task.** Combined review acceptable for purely mechanical fixes (T76–T79). Separate spec + quality reviewers for bundled tasks (T80, T82, T83).
|
||||||
|
3. **Merge the wave into `phase-3.5`** in any order (file-disjointness guarantees no conflict). Use `--no-ff`.
|
||||||
|
4. **Run the full test suite** on the merged `phase-3.5`. If red, the wave's mutual-independence assumption was violated — bisect, fix, re-merge.
|
||||||
|
5. **Push `phase-3.5`** to gitea.
|
||||||
|
6. Optionally clean up worktrees: `git worktree remove .worktrees/<branch>` and `git branch -D <branch>`.
|
||||||
|
|
||||||
|
### Conflict prevention checklist
|
||||||
|
|
||||||
|
For each parallel wave, verify the **Files** sections of all tasks have **no overlapping paths**. Hot files in this plan: `chat/web/turns.py` (T82 only), `chat/services/regenerate.py` (T83 only), `chat/web/skip.py` + `chat/web/drawer.py` (T81 only). Each is owned by exactly one task.
|
||||||
|
|
||||||
|
### Failure recovery
|
||||||
|
|
||||||
|
If one subagent fails: cancel it, merge the others' successful work, re-dispatch the failed task as a single follow-up. Don't block the wave.
|
||||||
|
|
||||||
|
### Why each wave is parallel-safe
|
||||||
|
|
||||||
|
| Wave | Tasks | Hot files touched | Disjoint? |
|
||||||
|
|------|-------|-------------------|-----------|
|
||||||
|
| 1 | T76, T77, T78, T79 | 4 different services/state files; no overlap | ✅ |
|
||||||
|
| 2 | T80 | `chat/services/scene_summarize.py` | (single task) |
|
||||||
|
| 3 | T81 | `chat/web/skip.py` + `chat/web/drawer.py` (typed exception) | (single task) |
|
||||||
|
| 4 | T82 | `chat/web/turns.py` | (single task) |
|
||||||
|
| 5 | T83 | `chat/services/regenerate.py` (+ optional new shared-helpers module) | (single task) |
|
||||||
|
| 6 | T84, T85, T86 | `chat/services/memory_write.py` (T84); audit + tests (T85); frontend HTML/JS (T86) | ✅ |
|
||||||
|
| 7 | T87 | `CLAUDE.md` | (single task) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Wave 1 ─┬─ T76: narrate_skip timeout_s plumbed through (skip_narration.py)
|
||||||
|
├─ T77: AddresseeDecision.confidence as Literal (addressee.py)
|
||||||
|
├─ T78: search_memories docstring SQL-bias note (memory.py)
|
||||||
|
└─ T79: _witness_role_for defensive None handling (prompt.py)
|
||||||
|
|
||||||
|
Wave 2 ─── T80: scene_summarize.py polish bundle (5 T58 items)
|
||||||
|
|
||||||
|
Wave 3 ─── T81: typed exception for skip controllers (skip.py + drawer.py)
|
||||||
|
|
||||||
|
Wave 4 ─── T82: turns.py wiring (consume_pending_meanwhile_digests + skip-bypass scene close)
|
||||||
|
|
||||||
|
Wave 5 ─── T83: regenerate.py polish bundle (cancel hook + DRY + sibling query +
|
||||||
|
lifecycle rollback + ordering asymmetry)
|
||||||
|
|
||||||
|
Wave 6 ─┬─ T84: memory_write.py unified record-memory API
|
||||||
|
├─ T85: JSON-build audit + meanwhile stop-button route-level test
|
||||||
|
└─ T86: frontend turn_html_replace SSE handler
|
||||||
|
|
||||||
|
Wave 7 ─── T87: docs sweep — prune CLAUDE.md backlogs, capture residuals
|
||||||
|
```
|
||||||
|
|
||||||
|
Critical path: 7 sequential merge points. Total tasks: 12. Wall-clock parallelism advantage: Waves 1 and 6 dispatch concurrently (4-way and 3-way respectively). Waves 2, 3, 4, 5, 7 are single-task by hot-file constraint.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 1 — Independent small fixes (parallel, 4 tasks)
|
||||||
|
|
||||||
|
All 4 tasks are tiny (1-line + tests). Fully file-disjoint.
|
||||||
|
|
||||||
|
### Task 76: `narrate_skip` timeout_s plumbed through
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/skip_narration.py`
|
||||||
|
- Modify: `tests/test_skip_narration.py` (add 1 test)
|
||||||
|
|
||||||
|
**Spec:** `narrate_skip(timeout_s=60.0)` accepts the parameter but doesn't pass it to `client.generate(...)`. Per T53 review — fix:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Inside narrate_skip:
|
||||||
|
result = await client.generate(
|
||||||
|
[
|
||||||
|
Message(role="system", content=system),
|
||||||
|
Message(role="user", content=user),
|
||||||
|
],
|
||||||
|
model=narrative_model,
|
||||||
|
max_tokens=200,
|
||||||
|
temperature=0.7,
|
||||||
|
timeout_s=timeout_s, # NEW: pipe through
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
If the underlying `client.generate` doesn't honor `timeout_s` (Featherless client signature accepts `**params` so it'll ride through harmlessly), this is a no-op at runtime but documents intent. Read `chat/llm/featherless.py::FeatherlessClient.generate` to confirm.
|
||||||
|
|
||||||
|
**Test added:** `test_narrate_skip_passes_timeout_through` — capture the kwargs passed to `client.generate` via a custom mock that records them; assert `timeout_s` is present with the expected value.
|
||||||
|
|
||||||
|
**Commit:** `fix: plumb narrate_skip timeout_s through to client.generate (T76)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 77: `AddresseeDecision.confidence` as Literal type
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/addressee.py`
|
||||||
|
- Modify: `tests/test_addressee.py` (existing tests should still pass; add 1 test for the typed validation)
|
||||||
|
|
||||||
|
**Spec:** Per T74 review (Phase 2.5 carry-over). `AddresseeDecision.confidence` is currently `str` with a comment noting valid values. Tighten to `Literal["high", "medium", "low"]`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
class AddresseeDecision(BaseModel):
|
||||||
|
addressee_id: str
|
||||||
|
confidence: Literal["high", "medium", "low"] = "medium"
|
||||||
|
reason: str = ""
|
||||||
|
```
|
||||||
|
|
||||||
|
Pydantic will reject classifier output with values outside the literal set, falling back via the `default=` in `classify(...)`. The existing fallback path returns `confidence="low"` so no behavior change for legitimate calls.
|
||||||
|
|
||||||
|
**Test added:** `test_invalid_confidence_value_falls_back_to_default` — feed canned classifier output with `"confidence": "VERY_HIGH"`; assert the result is the default fallback (Pydantic validation failed → `classify` falls back).
|
||||||
|
|
||||||
|
**Commit:** `fix: AddresseeDecision.confidence as Literal[high|medium|low] (T77)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 78: `search_memories` docstring mentions SQL-bias
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/state/memory.py` — extend the `search_memories` docstring with a one-liner about `SIGNIFICANCE_RANK_BIAS`.
|
||||||
|
- No test changes needed (docstring-only).
|
||||||
|
|
||||||
|
**Spec:** Per T57 review. The current `search_memories` docstring describes only the Python-side composite re-rank with `_SIGNIFICANCE_WEIGHT`. T57 added a SQL-side `SIGNIFICANCE_RANK_BIAS` constant. Add to the docstring:
|
||||||
|
|
||||||
|
```
|
||||||
|
... (existing prose) ...
|
||||||
|
|
||||||
|
The result ordering applies TWO independent significance boosts:
|
||||||
|
- SQL-side: ``ORDER BY (rank - significance * SIGNIFICANCE_RANK_BIAS)``
|
||||||
|
pushes higher-significance memories ahead in the FTS5 candidate set.
|
||||||
|
- Python-side: a composite re-rank with `_SIGNIFICANCE_WEIGHT` reinforces
|
||||||
|
the ordering after candidate retrieval.
|
||||||
|
```
|
||||||
|
|
||||||
|
**Commit:** `docs: search_memories docstring mentions SQL-side significance bias (T78)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 79: `_witness_role_for` defensive None handling
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/prompt.py`
|
||||||
|
- Modify: `tests/test_prompt.py` (add 1 test)
|
||||||
|
|
||||||
|
**Spec:** Per T71 review (Phase 2.5 carry-over). Current `_witness_role_for(speaker_bot_id, host_bot_id)` returns `"host" if speaker_bot_id == host_bot_id else "guest"`. When `host_bot_id` is `None` (degenerate case — can happen if a chat row is half-seeded in a test), this returns `"guest"` even though the speaker is logically the host. Defensive fix:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _witness_role_for(speaker_bot_id: str, host_bot_id: str | None) -> str:
|
||||||
|
"""..."""
|
||||||
|
if host_bot_id is None or speaker_bot_id == host_bot_id:
|
||||||
|
return "host"
|
||||||
|
return "guest"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test added:** `test_witness_role_for_none_host_returns_host` — call helper with `host_bot_id=None`; assert returns `"host"`.
|
||||||
|
|
||||||
|
**Commit:** `fix: _witness_role_for defensive None handling (T79)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 2 — `scene_summarize.py` polish bundle (single task)
|
||||||
|
|
||||||
|
T80 bundles 5 backlog items (4 fixes + 1 test gap) all touching `chat/services/scene_summarize.py` and its test file. Single task by hot-file constraint; split into 5 commits internally for clean review bisection.
|
||||||
|
|
||||||
|
### Task 80: scene_summarize.py polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/scene_summarize.py`
|
||||||
|
- Modify: `tests/test_per_pov_summary.py`
|
||||||
|
|
||||||
|
**Spec:** Five sub-fixes per the T58 review.
|
||||||
|
|
||||||
|
#### 80.1 — Re-close suffix bloat guard
|
||||||
|
|
||||||
|
**Problem:** `_build_key_quotes_suffix` reads from `memories.pov_summary`. If a scene close runs twice (replay, manual re-close, idempotent retry), the second pass reads the rewritten text plus the previous "Key quotes:" suffix and appends a second one — recursive bloat.
|
||||||
|
|
||||||
|
**Fix (recommended option):** source quotes from `event_log` `assistant_turn`/`user_turn` text instead of `memories.pov_summary`. The event-log text is immutable per turn, so re-close produces identical key-quote text.
|
||||||
|
|
||||||
|
Implementation: in `_build_key_quotes_suffix(conn, scene_id)`, replace the SELECT from `memories` with a JOIN that pulls turn text from `event_log` filtered by scene_id (via `payload_json` JSON extraction), keeping the significance-from-memories filter.
|
||||||
|
|
||||||
|
If event-log queries prove too expensive or fragile, fall back to: at the start of `_build_key_quotes_suffix`, strip any existing `"\n\nKey quotes:\n"` suffix from each `pov_summary` before computing the new one. Document the choice in a code comment.
|
||||||
|
|
||||||
|
**Test added:** `test_scene_close_re_run_does_not_double_suffix` — close a scene with significance ≥ 2; assert each pov_summary contains "Key quotes:" exactly ONCE. Then call `apply_scene_close_summary` again on the same scene; assert each pov_summary STILL contains "Key quotes:" exactly ONCE (no second append).
|
||||||
|
|
||||||
|
**Commit:** `fix: guard scene close key-quote suffix against re-close bloat (T80.1)`.
|
||||||
|
|
||||||
|
#### 80.2 — Thread detection transcript scoping
|
||||||
|
|
||||||
|
**Problem:** `_read_recent_dialogue` returns chat-wide history with no `scene_id` filter. Feeding chat-wide history to `detect_threads` will misattribute threads to the closing scene when a scene boundary falls inside the last 50 turns.
|
||||||
|
|
||||||
|
**Fix:** add a `scene_started_at` lookup at the top of `apply_scene_close_summary`. Filter the transcript to turns where the event_log row's `created_at` (or `chat_clock_at`) is `>= scene_started_at`. Read `chat/state/world.py::get_scene` for the started_at column.
|
||||||
|
|
||||||
|
If `get_scene` doesn't expose `started_at`, query directly: `SELECT started_at FROM scenes WHERE id = ?`.
|
||||||
|
|
||||||
|
**Test added:** `test_thread_detection_uses_scene_scoped_transcript` — seed a scene with 3 turns. Then close the scene. Then start a new scene and seed 3 more turns. Then close the second scene. Mock `detect_threads` to capture its `scene_transcript` arg. Assert the second close's transcript contains ONLY the 3 second-scene turns (not the first scene's turns).
|
||||||
|
|
||||||
|
**Commit:** `fix: scope thread detection transcript to closing scene (T80.2)`.
|
||||||
|
|
||||||
|
#### 80.3 — Log swallowed exceptions in `detect_threads` try/except
|
||||||
|
|
||||||
|
**Problem:** T58's `try/except Exception:` around `detect_threads` swallows programmer errors silently.
|
||||||
|
|
||||||
|
**Fix:** add a `logging.getLogger(__name__).debug("detect_threads failed: %s", exc, exc_info=True)` (or appropriate level) inside the except block. Use the standard library `logging` module (it's already configured by Phase 1's `chat.app`).
|
||||||
|
|
||||||
|
**Test added:** `test_detect_threads_failure_is_logged` — patch `detect_threads` to raise `RuntimeError("test")`. Use `caplog` (pytest fixture) to capture log output. Assert the log contains the error message.
|
||||||
|
|
||||||
|
**Commit:** `fix: log swallowed exceptions in detect_threads try/except (T80.3)`.
|
||||||
|
|
||||||
|
#### 80.4 — Scene close `closed_at` chat-clock semantics
|
||||||
|
|
||||||
|
**Problem:** T58 uses `datetime.now(timezone.utc).isoformat()` for `closed_at` on emitted events instead of chat-clock time. Diverges from chat-clock semantics elsewhere.
|
||||||
|
|
||||||
|
**Fix:** read `chat["time"]` at the top of `apply_scene_close_summary` (already loaded for per-POV writes via `chat_clock_at`). Pass `chat_clock_at` through to `thread_closed` events' `closed_at` field instead of `datetime.now(...)`.
|
||||||
|
|
||||||
|
**Test added:** `test_thread_closed_uses_chat_clock_time` — seed a chat with `chat["time"] = "2026-04-26T10:00:00+00:00"`. Mock `detect_threads` to return one `ThreadCandidate(action="close", existing_thread_id="thr_x")`. Close the scene. Inspect the emitted `thread_closed` event payload — assert `closed_at == "2026-04-26T10:00:00+00:00"`, NOT a `datetime.now(...)` value.
|
||||||
|
|
||||||
|
**Commit:** `fix: thread_closed uses chat-clock time, not wall clock (T80.4)`.
|
||||||
|
|
||||||
|
#### 80.5 — T58 test coverage gaps
|
||||||
|
|
||||||
|
**Spec:** Per T58 review, these specific scenarios lack coverage. Add tests:
|
||||||
|
|
||||||
|
1. `test_key_quote_truncation_at_200_chars` — seed a memory with significance 2 and a 500-char `pov_summary`. Close scene. Assert the key-quote bullet for that memory is exactly 200 chars (or 199 + ellipsis, depending on the truncation impl).
|
||||||
|
|
||||||
|
2. `test_thread_detection_update_candidate_emits_thread_updated` — mock `detect_threads` to return `ThreadCandidate(action="update", existing_thread_id="thr_x", summary="updated")`. Close. Assert a `thread_updated` event landed.
|
||||||
|
|
||||||
|
3. `test_thread_detection_close_candidate_emits_thread_closed` — same setup with `action="close"`. Assert a `thread_closed` event landed.
|
||||||
|
|
||||||
|
(The `try/except` fallback test from 80.3 covers the exception path; no separate test needed.)
|
||||||
|
|
||||||
|
**Commit:** `test: T58 coverage gaps (truncation, update/close paths) (T80.5)`.
|
||||||
|
|
||||||
|
#### Verification gates for T80
|
||||||
|
|
||||||
|
- All 5 sub-fix commits cleanly split per `git show`.
|
||||||
|
- All 13 existing per_pov_summary tests still pass.
|
||||||
|
- 5+ new tests pass.
|
||||||
|
- Full suite green.
|
||||||
|
- `git diff phase-3.5 --stat` shows ONLY `chat/services/scene_summarize.py` and `tests/test_per_pov_summary.py`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 3 — Typed exception for skip controllers (single task)
|
||||||
|
|
||||||
|
### Task 81: `ChatNotFoundError` for skip route routing
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/web/skip.py` — define a typed exception and raise it instead of generic `ValueError("chat not found")`.
|
||||||
|
- Modify: `chat/web/drawer.py` — replace string-prefix sniff with `except ChatNotFoundError: 404`.
|
||||||
|
- Modify: `tests/test_drawer_events_threads_skip.py` (add 1 test).
|
||||||
|
|
||||||
|
**Spec:** Per T62 review. Drawer skip routes use `str(exc).startswith("chat not found")` to distinguish 404 from 400. Fragile if error wording changes. Replace with a typed exception subclass.
|
||||||
|
|
||||||
|
```python
|
||||||
|
# chat/web/skip.py top:
|
||||||
|
class ChatNotFoundError(Exception):
|
||||||
|
"""Raised when a chat_id doesn't resolve. Maps to 404 in HTTP routes."""
|
||||||
|
|
||||||
|
# Inside process_elision_skip / process_jump_skip:
|
||||||
|
chat = get_chat(conn, chat_id)
|
||||||
|
if chat is None:
|
||||||
|
raise ChatNotFoundError(f"chat {chat_id!r} not found")
|
||||||
|
|
||||||
|
# Other validation failures still raise ValueError (mapped to 400).
|
||||||
|
```
|
||||||
|
|
||||||
|
```python
|
||||||
|
# chat/web/drawer.py skip routes:
|
||||||
|
try:
|
||||||
|
result = await process_elision_skip(...)
|
||||||
|
except ChatNotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="chat not found")
|
||||||
|
except ValueError as exc:
|
||||||
|
raise HTTPException(status_code=400, detail=str(exc))
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test added:** `test_skip_route_raises_404_via_typed_exception` — POST to `/chats/nonexistent/drawer/skip/elision`; assert response 404. (Existing tests should still pass — they check 404 for missing chats and 400 for validation errors.)
|
||||||
|
|
||||||
|
**Commit:** `fix: typed ChatNotFoundError replaces string-prefix sniff in skip routes (T81)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 4 — `turns.py` wiring (single task)
|
||||||
|
|
||||||
|
### Task 82: post_turn meanwhile-digest consumption + skip-command scene close
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/web/turns.py`
|
||||||
|
- Modify: `tests/test_turn_flow.py` (add 2 tests)
|
||||||
|
- Modify: `tests/test_phase3_integration.py` (T66 test 4 currently calls `consume_pending_meanwhile_digests` directly — update it to assert the helper is now invoked AUTOMATICALLY by post_turn)
|
||||||
|
|
||||||
|
**Spec:** Two related fixes both touching `post_turn`.
|
||||||
|
|
||||||
|
#### 82.1 — Wire `consume_pending_meanwhile_digests` into post_turn
|
||||||
|
|
||||||
|
**Problem (T66):** `chat/services/prompt.py::consume_pending_meanwhile_digests` is defined but never called. Meanwhile digests stay pending forever in production.
|
||||||
|
|
||||||
|
**Fix:** call the helper at the END of `post_turn` (after the assistant_turn lands and all classifier passes finish, BEFORE the response returns). The helper appends `meanwhile_digest_consumed` events for each pending digest, marking them as surfaced. Idempotent — re-calling produces zero events.
|
||||||
|
|
||||||
|
Place the call only when the chat has just-surfaced digests. The simplest pattern: always call after the primary turn — the helper short-circuits when nothing's pending. (Cost: 1 trivial DB query per turn.)
|
||||||
|
|
||||||
|
```python
|
||||||
|
from chat.services.prompt import consume_pending_meanwhile_digests
|
||||||
|
|
||||||
|
# Near the end of post_turn, before publish/return:
|
||||||
|
consume_pending_meanwhile_digests(conn, chat_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test added:** `test_post_turn_consumes_pending_meanwhile_digests` — seed a pending digest. POST a turn. Assert: a `meanwhile_digest_consumed` event landed in event_log AND `list_pending_meanwhile_digests` is now empty.
|
||||||
|
|
||||||
|
**Update existing T66 test 4:** the integration test currently calls `consume_pending_meanwhile_digests` directly to drive the consumption side-effect. After T82, the consumption happens automatically inside post_turn. Remove the explicit call in the test (or assert the count even without calling explicitly).
|
||||||
|
|
||||||
|
**Commit:** `fix: post_turn consumes pending meanwhile digests (T82.1)`.
|
||||||
|
|
||||||
|
#### 82.2 — Skip command runs scene close detection
|
||||||
|
|
||||||
|
**Problem (T62):** A user typing "skip an hour" via natural-language skip causes the skip path to bypass scene close detection. The user's prose may have signaled close intent ("fade out, skip an hour") but the skip path returns 204 without checking.
|
||||||
|
|
||||||
|
**Fix:** in the natural-language skip dispatch (T62 `intent="skip_elision"` branch), call `detect_scene_close` BEFORE the skip controller runs. If close is detected, append `scene_closed` first, then run the skip flow.
|
||||||
|
|
||||||
|
Order matters: scene close → skip narration → time advance. The narration should reflect the new scene start at the new time.
|
||||||
|
|
||||||
|
```python
|
||||||
|
if parsed.intent == "skip_elision":
|
||||||
|
# Run scene close detection first - the user may have signaled close.
|
||||||
|
if scene := active_scene(conn, chat_id):
|
||||||
|
close_decision = await detect_scene_close(client, classifier_model=...,
|
||||||
|
prose=parsed.prose, ...)
|
||||||
|
if close_decision.should_close:
|
||||||
|
append_and_apply(conn, kind="scene_closed", payload={"scene_id": scene["id"], ...})
|
||||||
|
await apply_scene_close_summary(conn, client, ...)
|
||||||
|
# ... existing skip dispatch ...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Test added:** `test_natural_language_skip_with_close_signal_closes_scene` — user prose "fade out, skip to morning". Mock `detect_scene_close` to return True. Mock `narrate_skip`. Assert: `scene_closed` event lands AND `time_skip_elision` event lands AND ordering is `scene_closed` BEFORE `time_skip_elision`.
|
||||||
|
|
||||||
|
**Commit:** `fix: natural-language skip runs scene close detection (T82.2)`.
|
||||||
|
|
||||||
|
#### Verification gates for T82
|
||||||
|
|
||||||
|
- 2 new tests pass.
|
||||||
|
- All existing turn_flow + phase3_integration tests still pass (T66 test 4 may need a small update; document in commit).
|
||||||
|
- Full suite green.
|
||||||
|
- `git diff phase-3.5 --stat`: only `chat/web/turns.py`, `tests/test_turn_flow.py`, `tests/test_phase3_integration.py`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 5 — `regenerate.py` polish bundle (single task)
|
||||||
|
|
||||||
|
T83 bundles 5 regenerate-related backlog items. All touch `chat/services/regenerate.py`. Single task by hot-file constraint; 5 internal commits.
|
||||||
|
|
||||||
|
### Task 83: regenerate.py polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/regenerate.py`
|
||||||
|
- Optional: create `chat/services/turn_common.py` (new shared-helpers module for 83.2 DRY extraction)
|
||||||
|
- Modify: `chat/web/turns.py` (only if 83.2 extracts shared helpers — minimal)
|
||||||
|
- Modify: `tests/test_regenerate.py` (add 5+ tests)
|
||||||
|
|
||||||
|
**Spec:** Five sub-fixes per the T73 + T61 reviews.
|
||||||
|
|
||||||
|
#### 83.1 — Cancel/stop hook for in-flight regenerate streams
|
||||||
|
|
||||||
|
**Problem:** `post_turn` registers stream tasks in `_in_flight_tasks` so `/turns/cancel` cancels them. Regenerate doesn't.
|
||||||
|
|
||||||
|
**Fix:** mirror the registration pattern from `post_turn` (Phase 2.5 T74.4 documented this). Wrap the regenerate stream in `asyncio.create_task(...)`, register in `_in_flight_tasks[chat_id]`, await, unregister in `finally`.
|
||||||
|
|
||||||
|
**Test added:** `test_regenerate_registers_task_in_in_flight_tasks` — mid-stream snapshot pattern (mirror Phase 3 T64.fix-up's `_SnapshotMock`). Assert the chat_id is registered during streaming.
|
||||||
|
|
||||||
|
**Commit:** `feat: regenerate registers stream task in _in_flight_tasks for cancellation (T83.1)`.
|
||||||
|
|
||||||
|
#### 83.2 — DRY: extract recent-dialogue + prior-edges helpers
|
||||||
|
|
||||||
|
**Problem:** Recent-dialogue assembly + prior-edges block are duplicated between `chat/services/regenerate.py` and `chat/web/turns.py`. Diff drift risk.
|
||||||
|
|
||||||
|
**Fix:** create `chat/services/turn_common.py` with:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def read_recent_dialogue(conn, chat_id: str, limit: int = 50) -> list[dict]:
|
||||||
|
"""Pull the last N user_turn + assistant_turn rows for the chat as
|
||||||
|
structured [{speaker, text}, ...]. Filters superseded_by IS NULL
|
||||||
|
AND hidden = 0. Used by post_turn AND regenerate AND scene_summarize."""
|
||||||
|
|
||||||
|
def gather_prior_edges(conn, chat_id: str, present_ids: list[str]) -> dict[tuple, dict]:
|
||||||
|
"""Build {(src, tgt): {affinity, trust, summary}} dict for all directed
|
||||||
|
pairs where both endpoints are in present_ids."""
|
||||||
|
```
|
||||||
|
|
||||||
|
Update `chat/web/turns.py::post_turn` to use the new helpers. Update `chat/services/regenerate.py::regenerate_assistant_turn` to use them too. Existing tests should pass unchanged.
|
||||||
|
|
||||||
|
**Test added:** `tests/test_turn_common.py` (NEW file) with 2-3 tests: dialogue read filters superseded; prior_edges builds correct keys.
|
||||||
|
|
||||||
|
**Commit:** `refactor: extract turn_common helpers from regenerate + turns (T83.2)`.
|
||||||
|
|
||||||
|
#### 83.3 — Sibling-discovery query optimization
|
||||||
|
|
||||||
|
**Problem (T73):** `regenerate.py`'s sibling-assistant-turn lookup scans ALL non-superseded `assistant_turn` rows globally with no `chat_id` predicate.
|
||||||
|
|
||||||
|
**Fix:** add a `json_extract(payload_json, '$.chat_id') = ?` predicate in the SQL query, plus `LIMIT 50` for safety. Mirror Phase 3 T64's `_last_meanwhile_speaker` SQL pattern.
|
||||||
|
|
||||||
|
**Test added:** `test_regenerate_sibling_lookup_scoped_to_chat` — seed two chats both with regenerate-able turn groups. Regenerate in chat A. Assert the sibling lookup didn't return any chat-B rows (verify via SQL query interception or by ensuring cross-chat sibling pollution wouldn't break).
|
||||||
|
|
||||||
|
**Commit:** `perf: scope regenerate sibling-lookup to chat_id (T83.3)`.
|
||||||
|
|
||||||
|
#### 83.4 — Lifecycle-transition rollback on regenerate
|
||||||
|
|
||||||
|
**Problem (T61):** Regenerate doesn't roll back lifecycle transitions from the superseded turn. `event_started`/`event_completed` rows from a superseded turn remain. May double-emit promotion artifacts.
|
||||||
|
|
||||||
|
**Fix (Phase 3.5 first cut — minimal):** at the START of regenerate, scan the superseded turn's downstream events. For any `event_started`/`event_completed`/`event_cancelled` linked to the superseded turn (via `superseded_by` chain on the original assistant_turn AND a back-pointer in the lifecycle event payload), revert the projected state by appending compensating events:
|
||||||
|
|
||||||
|
- Original was `event_started` for evt_x → emit `event_started_undone` (NEW event kind?) OR mark the original superseded.
|
||||||
|
- Original was `event_completed` → similarly.
|
||||||
|
|
||||||
|
**This is too invasive for one Phase 3.5 commit.** Instead, the simpler Phase 3.5 fix: **document the limitation explicitly** in the regenerate flow with a TODO + add an `assert` or warning when regenerate detects prior lifecycle transitions on the superseded turn. Real rollback support is a Phase 4 item.
|
||||||
|
|
||||||
|
**Test added:** `test_regenerate_with_prior_lifecycle_logs_warning` — seed a turn that emits `event_completed`. Regenerate. Assert a warning log entry mentions the un-rolled-back transition.
|
||||||
|
|
||||||
|
**Commit:** `chore: document regenerate lifecycle-rollback limitation (T83.4)`.
|
||||||
|
|
||||||
|
(If the implementer judges that proper rollback IS feasible in the time available, they can implement it instead — but the safe Phase 3.5 default is documentation + warning. The choice is the implementer's; report which path was taken.)
|
||||||
|
|
||||||
|
#### 83.5 — Event-detection ordering symmetry
|
||||||
|
|
||||||
|
**Problem (T61):** `post_turn` runs lifecycle BETWEEN interjection and scene-close; `regenerate.py` runs lifecycle at the END. Benign because regenerate has no scene-close path, but worth tidying.
|
||||||
|
|
||||||
|
**Fix:** move the event-detection block in `regenerate.py` to mirror `post_turn`'s position (between interjection regenerate and the function's tail). Cosmetic — no behavior change.
|
||||||
|
|
||||||
|
**Test:** verify all existing regenerate tests still pass after the move.
|
||||||
|
|
||||||
|
**Commit:** `refactor: regenerate event-detection ordering mirrors post_turn (T83.5)`.
|
||||||
|
|
||||||
|
#### Verification gates for T83
|
||||||
|
|
||||||
|
- 5 internal commits visible.
|
||||||
|
- All 6 existing regenerate tests still pass.
|
||||||
|
- All 10+ existing turn_flow tests still pass.
|
||||||
|
- 5+ new tests pass.
|
||||||
|
- Full suite green.
|
||||||
|
- `git diff phase-3.5 --stat`: `chat/services/regenerate.py`, `chat/services/turn_common.py` (if extracted), `chat/web/turns.py` (only if helper-call sites updated), `tests/test_regenerate.py`, `tests/test_turn_common.py` (if new).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 6 — Final polish (parallel, 3 tasks)
|
||||||
|
|
||||||
|
### Task 84: Unified record-memory API
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/memory_write.py`
|
||||||
|
- Modify: `tests/test_memory_write.py` (add tests)
|
||||||
|
- Modify: `chat/web/meanwhile.py` (call site update — minimal)
|
||||||
|
|
||||||
|
**Spec:** Per T64 review. `record_meanwhile_memory` and `record_turn_memory_for_present` share private `_write_one_memory` but expose two separate public APIs. Unify:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def record_turn_memory(
|
||||||
|
conn,
|
||||||
|
*,
|
||||||
|
chat_id: str,
|
||||||
|
host_bot_id: str,
|
||||||
|
guest_bot_id: str | None,
|
||||||
|
narrative_text: str,
|
||||||
|
scene_id: int | None = None,
|
||||||
|
chat_clock_at: str | None = None,
|
||||||
|
source: str = "direct",
|
||||||
|
significance: int = 1,
|
||||||
|
you_present: bool = True, # NEW: False for meanwhile turns
|
||||||
|
) -> dict[str, tuple[int, int | None]]:
|
||||||
|
"""Single entry-point for memory writes. When you_present is True,
|
||||||
|
witnesses are [you=1, host=1, guest if present]. When False (meanwhile),
|
||||||
|
witnesses are [you=0, host=1, guest=1] and host_bot_id + guest_bot_id
|
||||||
|
are both required."""
|
||||||
|
```
|
||||||
|
|
||||||
|
`record_meanwhile_memory` becomes a thin wrapper (kept for backward compat) that calls `record_turn_memory(..., you_present=False)`. `record_turn_memory_for_present` is renamed/aliased to `record_turn_memory` for the same reason.
|
||||||
|
|
||||||
|
Update `chat/web/meanwhile.py` to call the unified function with `you_present=False`. Verify the existing meanwhile turn tests still pass.
|
||||||
|
|
||||||
|
**Tests added:** 2 new tests in `tests/test_memory_write.py`:
|
||||||
|
- `test_record_turn_memory_you_present_false_writes_meanwhile_witness_mask` — assert witness `[0, 1, 1]`.
|
||||||
|
- `test_record_turn_memory_you_present_true_default_writes_normal_witness_mask` — assert `[1, 1, 0|1]`.
|
||||||
|
|
||||||
|
**Commit:** `refactor: unified record_turn_memory API with you_present kwarg (T84)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 85: JSON-build audit + meanwhile stop-button route-level test
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Audit: read all `chat/state/*.py` for f-string JSON construction. NO PRODUCTION CHANGES unless an issue is found.
|
||||||
|
- Modify (only if issue found): the affected state module.
|
||||||
|
- Modify: `tests/test_meanwhile_turn_flow.py` (add stop-button route-level test).
|
||||||
|
|
||||||
|
**Spec:** Two unrelated minor follow-ups bundled by being small.
|
||||||
|
|
||||||
|
#### 85.1 — JSON-build audit (T63)
|
||||||
|
|
||||||
|
T63 originally used f-string interpolation for `participants_json` (fixed during review). Audit other state modules for the same pattern. Grep:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -rn 'f["\']\[' chat/state/ chat/services/ chat/eventlog/
|
||||||
|
```
|
||||||
|
|
||||||
|
For each f-string that constructs a JSON string from user-controllable data, replace with `json.dumps(...)`. If NONE found (likely — most code already uses `json.dumps`), add a one-line note in the commit body confirming the audit.
|
||||||
|
|
||||||
|
If issues are found, fix them and add tests asserting JSON validity for inputs containing quote/backslash characters.
|
||||||
|
|
||||||
|
**Commit:** `chore: audit JSON-string construction sites; replace any f-string usages with json.dumps (T85.1)` (or "audit confirms no issues").
|
||||||
|
|
||||||
|
#### 85.2 — Meanwhile stop-button route-level test
|
||||||
|
|
||||||
|
T64's fix-up registered stream tasks in `_in_flight_tasks`. There's a unit test pinning registration but no test that drives `/turns/cancel` against a meanwhile turn.
|
||||||
|
|
||||||
|
**Test added:** `test_meanwhile_turn_can_be_cancelled_via_route` — start a meanwhile turn (POST /turns), capture the in-flight task, immediately POST /turns/cancel, assert the task transitions to cancelled state and the assistant_turn payload has `truncated=True`.
|
||||||
|
|
||||||
|
This test may be tricky to write reliably (timing). If the existing `_SnapshotMock` pattern doesn't support mid-stream cancel injection, document the limitation and add a smaller test that just verifies the cancel endpoint works on the chat (asserts no tasks remaining after cancel).
|
||||||
|
|
||||||
|
**Commit:** `test: meanwhile turn cancellation via /turns/cancel route (T85.2)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 86: Frontend `turn_html_replace` SSE handler
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/templates/chat.html` (or wherever the chat page's SSE consumer lives) — add a JS event handler for `turn_html_replace`.
|
||||||
|
- Optional: `chat/static/app.css` (if styling needs adjustment for replaced turns).
|
||||||
|
- Modify: `tests/test_streaming_ux.py` (add 1 test).
|
||||||
|
|
||||||
|
**Spec:** Per Phase 2.5 T73.1 (carry-over). Regenerate's backend broadcasts a `turn_html_replace` SSE event but no live tab swaps the prior turn. Wire the JS:
|
||||||
|
|
||||||
|
```html
|
||||||
|
<!-- In chat.html, near the existing SSE setup: -->
|
||||||
|
<script>
|
||||||
|
const sse = new EventSource('/chats/{{ chat.id }}/events');
|
||||||
|
|
||||||
|
// Existing: turn_html appends new turn HTML.
|
||||||
|
sse.addEventListener('turn_html', (ev) => {
|
||||||
|
const turnsContainer = document.getElementById('turns');
|
||||||
|
turnsContainer.insertAdjacentHTML('beforeend', ev.data);
|
||||||
|
});
|
||||||
|
|
||||||
|
// NEW: turn_html_replace swaps an existing turn's DOM in place.
|
||||||
|
sse.addEventListener('turn_html_replace', (ev) => {
|
||||||
|
const data = JSON.parse(ev.data);
|
||||||
|
// data: {html, turn_id, supersedes_id}
|
||||||
|
const oldNode = document.getElementById(`turn-${data.supersedes_id}`);
|
||||||
|
if (oldNode) {
|
||||||
|
const tmpl = document.createElement('template');
|
||||||
|
tmpl.innerHTML = data.html.trim();
|
||||||
|
oldNode.replaceWith(tmpl.content.firstChild);
|
||||||
|
} else {
|
||||||
|
// Fallback: append if the prior turn isn't in the DOM (edge case).
|
||||||
|
document.getElementById('turns').insertAdjacentHTML('beforeend', data.html);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
```
|
||||||
|
|
||||||
|
(Adapt the actual template structure — read `chat/templates/chat.html` first to see the existing SSE setup. The `turn-{id}` DOM-id pattern needs to be consistent with how the backend renders turn HTML — verify by checking `chat/web/render.py`.)
|
||||||
|
|
||||||
|
**Test added:** Hard to write a true browser-driven test in pytest. The simplest pin: a unit test that asserts the rendered HTML for a regenerate response contains the `turn_html_replace` SSE event AND the event payload contains the expected `supersedes_id`. (Already exists in T73.1 tests — don't duplicate.) Optionally add a Selenium/playwright test if the project has frontend testing infrastructure.
|
||||||
|
|
||||||
|
If pytest-only is the available coverage, document the manual smoke step in CLAUDE.md ("multi-tab regenerate" smoke test) and rely on backend pin tests + manual verification.
|
||||||
|
|
||||||
|
**Commit:** `feat: frontend turn_html_replace SSE handler (T86)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 7 — Docs sweep (single task)
|
||||||
|
|
||||||
|
### Task 87: CLAUDE.md backlog burn-down
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `CLAUDE.md`
|
||||||
|
|
||||||
|
**Spec:** Walk through the Phase 2.6/3 backlog and Phase 3.5/4 backlog sections. For each item shipped in T76–T86, remove from backlog list. Add a new section "Phase 3.5 status" near the existing "Phase 3 status" listing what shipped. Add a new "Phase 3.6 / 4 backlog" section if any new items were discovered during T76–T86 reviews (likely some — the review cycle always surfaces fresh follow-ups).
|
||||||
|
|
||||||
|
If any task during T76–T86 chose to defer a sub-item (e.g., T83.4 chose documentation over rollback impl), keep that sub-item in the new "Phase 3.6+ deferred" section with the rationale.
|
||||||
|
|
||||||
|
Mark §13 Phase 3 deliverables in the requirements doc as **shipped** (already done in T67) — verify and don't re-mark.
|
||||||
|
|
||||||
|
**Commit:** `docs: phase 3.5 status, prune shipped backlog items (T87)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wrap-up
|
||||||
|
|
||||||
|
After Wave 7 lands:
|
||||||
|
|
||||||
|
1. **Run full suite** on `phase-3.5`: should be ~330+ tests passing (315 from Phase 3 + ~15-25 new across the wave).
|
||||||
|
2. **Manual smoke** (recommended before opening the PR):
|
||||||
|
- Multi-tab regenerate: open two tabs on the same chat; click Regenerate on one; verify the other tab swaps the regenerated turn live (T86).
|
||||||
|
- Trigger a meanwhile scene; close it; resume the main chat; verify the digest renders ONCE on the next you-turn AND is consumed automatically (T82.1).
|
||||||
|
- Type "fade out, skip an hour" — verify both scene close + skip fire in correct order (T82.2).
|
||||||
|
- Plan an event, complete it via a turn, then regenerate that turn — observe the warning log mentioning un-rolled-back lifecycle transitions (T83.4).
|
||||||
|
3. **Push `phase-3.5`** to gitea.
|
||||||
|
4. **Open PR** `phase-3.5 → main`.
|
||||||
|
5. **Phase 3.6+ residuals** (track in CLAUDE.md): genuine lifecycle-rollback impl, structured fixture builder for canned-queue tests, scene-close-on-cancel revisit if play-testing surfaces a regression.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes for the controller running this plan
|
||||||
|
|
||||||
|
- **Don't dispatch Wave 5 until Wave 4 is merged AND green.** T82 (turns.py) + T83 (regenerate.py + optionally turns.py for shared-helper extraction) both touch `turns.py`. Sequential ordering prevents merge conflict.
|
||||||
|
- **After each parallel wave**, run a code-review subagent. Combined spec+quality review is acceptable for trivial tasks (T76, T77, T78, T79). Separate spec + quality reviewers for bundled tasks (T80, T82, T83) — each bundles 4-5 sub-fixes.
|
||||||
|
- **Token-spend rough estimate**: Phase 3.5 should be ~50-60% the size of Phase 3 (smaller scope, all reuse, no new schema). Per-task spend similar to Phase 2.5's bundled tasks.
|
||||||
|
- **DO NOT break existing v1/v2/v3 surface contracts.** Every test file that was green at the start of Phase 3.5 must stay green at the end. The cross-feature integration tests (`tests/test_phase3_integration.py`) are particularly load-bearing — if any of T80/T82/T83 break them, that's a regression to investigate, NOT to suppress.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
{
|
||||||
|
"planPath": "docs/plans/2026-04-26-v3.5-phase3.5-cleanup.md",
|
||||||
|
"tasks": [
|
||||||
|
{"id": 76, "subject": "T76: narrate_skip timeout_s plumbed through", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 77, "subject": "T77: AddresseeDecision.confidence as Literal", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 78, "subject": "T78: search_memories docstring SQL-bias note", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 79, "subject": "T79: _witness_role_for defensive None handling", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 80, "subject": "T80: scene_summarize.py polish bundle (re-close suffix + transcript scoping + log + clock + tests)", "status": "pending", "wave": 2, "parallelGroup": null},
|
||||||
|
{"id": 81, "subject": "T81: ChatNotFoundError replaces string-prefix sniff in skip routes", "status": "pending", "wave": 3, "parallelGroup": null},
|
||||||
|
{"id": 82, "subject": "T82: post_turn wiring (consume_pending_meanwhile_digests + skip command scene close)", "status": "pending", "wave": 4, "parallelGroup": null},
|
||||||
|
{"id": 83, "subject": "T83: regenerate.py polish (cancel hook + DRY + sibling query + lifecycle rollback note + ordering)", "status": "pending", "wave": 5, "parallelGroup": null, "blockedBy": [82]},
|
||||||
|
{"id": 84, "subject": "T84: unified record_turn_memory API with you_present kwarg", "status": "pending", "wave": 6, "parallelGroup": "wave-6", "blockedBy": [83]},
|
||||||
|
{"id": 85, "subject": "T85: JSON-build audit + meanwhile stop-button route-level test", "status": "pending", "wave": 6, "parallelGroup": "wave-6", "blockedBy": [83]},
|
||||||
|
{"id": 86, "subject": "T86: frontend turn_html_replace SSE handler", "status": "pending", "wave": 6, "parallelGroup": "wave-6", "blockedBy": [83]},
|
||||||
|
{"id": 87, "subject": "T87: docs sweep — prune CLAUDE.md backlogs, capture residuals", "status": "pending", "wave": 7, "parallelGroup": null, "blockedBy": [84, 85, 86]}
|
||||||
|
],
|
||||||
|
"lastUpdated": "2026-04-26T00:00:00Z",
|
||||||
|
"notes": "12 tasks across 7 waves consolidating 17 backlog items (7 from Phase 2.6/3, 10 from Phase 3.5/4). Wave 1 (4-way parallel) and Wave 6 (3-way parallel) are file-disjoint. Waves 2, 3, 4, 5, 7 are single-task by hot-file constraint. Bundled tasks (T80, T82, T83) split into per-item sub-commits for clean review bisection. No schema migrations — schema baseline stays at version 11. Uses task ids T76-T87 to avoid id collision with prior phases (Phase 1: T0-T35, Phase 2: T36-T48, Phase 3: T49-T67, Phase 2.5: T68-T75)."
|
||||||
|
}
|
||||||
@@ -0,0 +1,832 @@
|
|||||||
|
# Roleplay Engine — Phase 4 Implementation Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use `superpowers-extended-cc:executing-plans` to implement this plan task-by-task. Use the parallel-dispatch pattern documented under "Parallel-Execution Strategy" for parallel waves.
|
||||||
|
|
||||||
|
**Goal:** Land Phase 4 polish per requirements doc §13 + §14: vector retrieval, branching UI, drawer-edit on every field, backup tooling, significance review UI, surgical delete with cascade preview, hide-from-view soft delete, plus cross-chat search and the small Phase 3.6 carry-over fixes.
|
||||||
|
|
||||||
|
**Architecture:** Builds on Phase 3.5's stable base. Two new tables (`embeddings`, `branches`) and one external dependency (sqlite-vec extension). Embedding generation runs as a deferred async job — NOT inline with turns — so the play loop stays fast even when the embedding endpoint is slow. Branching is data-model-only at first (events + selectors); UI grafts on top. Surgical delete + cascade preview reuses the existing rewind-and-supersede plumbing. Cross-chat search piggybacks on the existing FTS5 + (now) vector retrieval.
|
||||||
|
|
||||||
|
**Tech Stack:**
|
||||||
|
|
||||||
|
- **NEW dependency: `sqlite-vec`** (or `sqlite-vss` — Phase 4 picks; recommended `sqlite-vec` for simpler load semantics and active maintenance). Add to `pyproject.toml`.
|
||||||
|
- **Embedding model selection** is part of T91 spec. Recommended default: a small model on Featherless (e.g., `BAAI/bge-small-en-v1.5` if available) or a local CPU-friendly model via `sentence-transformers`. Document choice in CLAUDE.md.
|
||||||
|
- Same as Phase 3 otherwise (Python 3.11+, FastAPI, HTMX, SQLite).
|
||||||
|
|
||||||
|
**Source-of-truth references:**
|
||||||
|
|
||||||
|
- Phase 4 scope: requirements doc §13 "Phase 4 — polish" + §14 "Open / Deferred Decisions".
|
||||||
|
- Behavioral details: §6 (prompt assembly + retrieval), §10 (rewind / regenerate / reset), §11 (compression + significance), §12 (snapshots).
|
||||||
|
- Conventions: [`CLAUDE.md`](../../CLAUDE.md) §"Behavioral defaults" + §"Phase 3 status" + §"Phase 3.5 status".
|
||||||
|
- Phase 3.5 cleanup plan (style, file-bundling pattern): [2026-04-26-v3.5-phase3.5-cleanup.md](2026-04-26-v3.5-phase3.5-cleanup.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Pre-flight
|
||||||
|
|
||||||
|
**Branch:** create `phase-4` from the latest `main` after Phase 3.5 has merged (it has — main is at `1b66a28`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git checkout main && git pull && git checkout -b phase-4
|
||||||
|
```
|
||||||
|
|
||||||
|
**Schema baseline:** Phase 3.5 leaves the DB at version 11. Phase 4 adds two migrations: `0012_embeddings.sql` and `0013_branches.sql`. Final schema version: 13.
|
||||||
|
|
||||||
|
**External dependency setup (BEFORE T88 dispatch):**
|
||||||
|
|
||||||
|
The controlling agent should add `sqlite-vec` to `pyproject.toml` and run `pip install -e .` (or equivalent) so all worktrees pick up the new dependency. Confirm `sqlite_vec` imports cleanly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -c "import sqlite_vec; print(sqlite_vec.__version__)"
|
||||||
|
```
|
||||||
|
|
||||||
|
If `sqlite_vec` isn't on PyPI when this plan executes, fall back to `sqlite-vss` and adapt T88/T92 accordingly. Both expose vector-search SQL via a loadable extension.
|
||||||
|
|
||||||
|
**Pinned non-negotiables (carried forward):**
|
||||||
|
|
||||||
|
- State changes go through the event log. Use `append_and_apply(conn, kind, payload)` for the live path; `apply_event` only after a fresh `append_event` returning the new id.
|
||||||
|
- Witness filter every memory read at SQL level (hard `WHERE` constraint; never a soft signal).
|
||||||
|
- Per-POV scene summaries — never write omniscient narration.
|
||||||
|
- TDD: every task starts with a failing test (or a regression test pinning existing contract before refactor).
|
||||||
|
- One commit per task minimum. Tasks that bundle multiple sub-features SHOULD split commits internally.
|
||||||
|
|
||||||
|
**Verification before claiming done:** Use `superpowers-extended-cc:verification-before-completion` — run the test command, paste actual output. Don't assume green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phase 3.6 carry-overs folded in
|
||||||
|
|
||||||
|
Three small items from Phase 3.6 backlog are bundled into Phase 4's Wave 1 trivial-fixes task (T90):
|
||||||
|
|
||||||
|
1. `read_recent_dialogue` chat-id pushdown into SQL (T80 review nit)
|
||||||
|
2. Lifecycle warning wording in regenerate (T83.4 — "at-or-after turn X" tightening)
|
||||||
|
3. Legacy single-bot `record_turn_memory` consolidation (T84 review nit)
|
||||||
|
|
||||||
|
Three items remain DEFERRED beyond Phase 4 (Phase 4.5 if needed):
|
||||||
|
|
||||||
|
- Scene-close-on-cancel UX revisit (no action unless real play surfaces a regression).
|
||||||
|
- Cross-feature canned-queue brittleness (structured fixture builder for tests — not blocking).
|
||||||
|
- Full lifecycle-rollback in regenerate (warning log already shipped in T83.4; proper rollback needs schema-level back-references, deferred indefinitely).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parallel-Execution Strategy
|
||||||
|
|
||||||
|
Same pattern as Phase 3.5. Eight waves: parallel within each wave (file-disjoint), serial across waves.
|
||||||
|
|
||||||
|
### How to dispatch a wave in parallel
|
||||||
|
|
||||||
|
Use the **Agent tool with `isolation: "worktree"`** so each subagent gets its own git worktree. (If the controlling session's working directory is **not** the chat repo, create worktrees manually with `git worktree add .worktrees/<wave>-<task> -b <wave>/<task> phase-4` from inside the chat repo.)
|
||||||
|
|
||||||
|
Dispatch all tasks in a wave in a single message:
|
||||||
|
|
||||||
|
```
|
||||||
|
Agent({ description: "Wave 1 — T88 embeddings table", prompt: "...", isolation: "worktree" })
|
||||||
|
Agent({ description: "Wave 1 — T89 branches table", ... })
|
||||||
|
Agent({ description: "Wave 1 — T90 phase 3.6 carry-overs", ... })
|
||||||
|
```
|
||||||
|
|
||||||
|
### After a wave completes
|
||||||
|
|
||||||
|
1. Each subagent returns its worktree path and commit SHA(s).
|
||||||
|
2. **Run a spec + code-quality reviewer subagent on each completed task.** Combined review acceptable for trivial tasks (T90 carry-overs); separate spec + quality reviewers for vector-retrieval tasks (T91, T92, T96, T97) since the integration surface is wider.
|
||||||
|
3. **Merge the wave into `phase-4`** in any order (file-disjointness guarantees no conflict). Use `--no-ff`.
|
||||||
|
4. **Run the full test suite** on the merged `phase-4`. If red, the wave's mutual-independence assumption was violated — bisect, fix, re-merge.
|
||||||
|
5. **Push `phase-4`** to gitea.
|
||||||
|
6. Optionally clean up worktrees.
|
||||||
|
|
||||||
|
### Conflict prevention checklist
|
||||||
|
|
||||||
|
For each parallel wave, verify the **Files** sections of all tasks have **no overlapping paths**. Hot files in this plan: `chat/web/drawer.py` + `chat/templates/_drawer.html` (T98 only — bundled), `chat/state/memory.py` (T96 only), `chat/services/memory_write.py` (T90 + T97 — sequential), `chat/web/turns.py` (T98 only via delete affordance — sequential after T96).
|
||||||
|
|
||||||
|
### Why each wave is parallel-safe
|
||||||
|
|
||||||
|
| Wave | Tasks | Hot files touched | Disjoint? |
|
||||||
|
|------|-------|-------------------|-----------|
|
||||||
|
| 1 | T88, T89, T90 | new migrations + new state modules; T90 touches `turn_common.py` + `regenerate.py` + `memory_write.py` (additive only) | ✅ |
|
||||||
|
| 2 | T91, T92, T93 | new service modules (embeddings, vector_search, cross_chat_search) | ✅ |
|
||||||
|
| 3 | T94, T95 | new service modules (branching, delete_impact) | ✅ |
|
||||||
|
| 4 | T96 | `chat/state/memory.py` (combined retrieval ranking) | (single task) |
|
||||||
|
| 5 | T97 | `chat/services/memory_write.py` + new backfill script | (single task) |
|
||||||
|
| 6 | T98 | `chat/web/drawer.py` + `chat/templates/_drawer.html` (drawer Phase 4 bundle) | (single task) |
|
||||||
|
| 7 | T99, T100 | new files: `chat/web/snapshots.py` + `chat/templates/snapshots.html` (T99); `chat/web/search.py` + `chat/templates/search.html` + small chat.html top-bar addition (T100) | ✅ (disjoint) |
|
||||||
|
| 8 | T101, T102 | new test file (T101); CLAUDE.md + design doc (T102) | ✅ |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Wave 1 ─┬─ T88: embeddings table + projector handlers
|
||||||
|
├─ T89: branches table + projector handlers
|
||||||
|
└─ T90: Phase 3.6 carry-overs trio (chat-id SQL pushdown + lifecycle wording + legacy-fn consolidation)
|
||||||
|
|
||||||
|
Wave 2 ─┬─ T91: embedding generation service (Featherless or local)
|
||||||
|
├─ T92: vector search service via sqlite-vec
|
||||||
|
└─ T93: cross-chat search service (FTS over all owners)
|
||||||
|
|
||||||
|
Wave 3 ─┬─ T94: branch_from_event service (event-log fork, branch metadata)
|
||||||
|
└─ T95: delete-impact computation service (cascade preview)
|
||||||
|
|
||||||
|
Wave 4 ─── T96: combined FTS + vector retrieval ranking in search_memories
|
||||||
|
|
||||||
|
Wave 5 ─── T97: memory_write enqueues embedding job + backfill script for existing memories
|
||||||
|
|
||||||
|
Wave 6 ─── T98: drawer Phase 4 bundle — branching UI + significance review + hide-from-view + surgical delete + remaining v1 edits
|
||||||
|
|
||||||
|
Wave 7 ─┬─ T99: snapshot UX (manual trigger, retention display, restore-from-snapshot UI)
|
||||||
|
└─ T100: cross-chat search UX (top-bar input + search results page)
|
||||||
|
|
||||||
|
Wave 8 ─┬─ T101: cross-feature integration tests (vector × branching × delete × snapshot × search)
|
||||||
|
└─ T102: Phase 4 documentation update
|
||||||
|
```
|
||||||
|
|
||||||
|
Critical path: 8 sequential merge points. Total tasks: 15. Parallelism: Waves 1, 2, 3, 7, 8 dispatch concurrently (3-way and 2-way). Waves 4, 5, 6 are single-task by hot-file constraint.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 1 — Schema foundation + Phase 3.6 carry-overs (parallel)
|
||||||
|
|
||||||
|
### Task 88: Embeddings table + projector handlers
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/db/migrations/0012_embeddings.sql`
|
||||||
|
- Create: `chat/state/embeddings.py`
|
||||||
|
- Create: `tests/test_embeddings_state.py`
|
||||||
|
- Modify: `pyproject.toml` (add `sqlite-vec` dependency — controlling agent should pre-install before dispatch; the worktree commits the dependency declaration)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Adds the `embeddings` table that stores per-memory embedding vectors for vector retrieval. Uses `sqlite-vec` virtual-table syntax for cosine-similarity search. Schema:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Load sqlite-vec extension at connection time (handled in chat/db/connection.py).
|
||||||
|
-- Embeddings are stored as blobs in a vec0 virtual table for fast similarity search.
|
||||||
|
|
||||||
|
CREATE VIRTUAL TABLE embeddings USING vec0(
|
||||||
|
memory_id INTEGER PRIMARY KEY,
|
||||||
|
embedding FLOAT[384] -- 384-dim default; adjust per chosen model
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Sidecar table for non-vector metadata (model used, dim, indexed_at).
|
||||||
|
CREATE TABLE embeddings_meta (
|
||||||
|
memory_id INTEGER PRIMARY KEY,
|
||||||
|
model TEXT NOT NULL,
|
||||||
|
dim INTEGER NOT NULL,
|
||||||
|
indexed_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||||
|
FOREIGN KEY (memory_id) REFERENCES memories(id)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
(If `sqlite-vss` is chosen instead, replace `vec0` with `vss0` and adapt the dim declaration. Both have similar Python loading semantics.)
|
||||||
|
|
||||||
|
**`chat/state/embeddings.py`:**
|
||||||
|
|
||||||
|
- `@on("embedding_indexed")` payload `{memory_id, model, dim, vector: list[float]}`. Inserts into both `embeddings` and `embeddings_meta`. Idempotent via `INSERT OR REPLACE` (re-indexing a memory replaces the prior vector).
|
||||||
|
- `@on("embedding_deindexed")` payload `{memory_id}`. Deletes from both tables. Used when a memory is purged via reset/cascade.
|
||||||
|
- Reader `get_embedding_meta(conn, memory_id) -> dict | None` returns the meta row.
|
||||||
|
|
||||||
|
The `chat/db/connection.py` `open_db` helper needs to load the sqlite-vec extension on each connection. Add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import sqlite_vec
|
||||||
|
# Inside open_db, after connection is opened:
|
||||||
|
conn.enable_load_extension(True)
|
||||||
|
sqlite_vec.load(conn)
|
||||||
|
conn.enable_load_extension(False)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is a small modification to `connection.py`. Include it in T88's diff.
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
|
||||||
|
1. `test_embedding_indexed_inserts_row`: append `bot_authored`, `chat_created`, `memory_written` (creates a memory), then `embedding_indexed` with `vector=[0.1] * 384`. Project. Assert `embeddings_meta` row exists for that memory_id with the right model.
|
||||||
|
2. `test_embedding_deindexed_removes_row`: same setup; index then de-index; assert row is gone.
|
||||||
|
3. `test_vector_similarity_search_returns_nearest`: index two memories with distinct vectors; query for nearest neighbor of one vector; assert correct memory_id returned. Uses `sqlite-vec`'s `MATCH '...'` syntax (verify against actual sqlite-vec docs; adapt if needed).
|
||||||
|
|
||||||
|
If running tests requires sqlite-vec to be loaded, the test fixture may need to skip / xfail when the extension isn't installed. Use `pytest.importorskip("sqlite_vec")` at the top of the test file.
|
||||||
|
|
||||||
|
**Commit:** `feat: embeddings table + projector handlers via sqlite-vec (T88)`.
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
|
||||||
|
- Schema version after migration alone: 12. T89 adds 0013, taking final to 13. The schema_version assertion in `tests/test_world.py` updates to 13 in the wave-merge step.
|
||||||
|
- The `connection.py` change is small but cross-cutting — affects every `open_db` call. Verify the existing 343 tests still pass after the change.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 89: Branches table + projector handlers
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/db/migrations/0013_branches.sql`
|
||||||
|
- Create: `chat/state/branches.py`
|
||||||
|
- Create: `tests/test_branches_state.py`
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Adds the `branches` table that records named alternate event-log forks. A branch is metadata: a name, an `origin_event_id` (the event we forked from), and a `head_event_id` (the latest event in this branch). The event log itself is unchanged — the branch table just **labels** linear ranges of event ids.
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE branches (
|
||||||
|
id INTEGER PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL UNIQUE,
|
||||||
|
origin_event_id INTEGER NOT NULL,
|
||||||
|
head_event_id INTEGER NOT NULL,
|
||||||
|
chat_id TEXT,
|
||||||
|
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||||
|
is_active INTEGER NOT NULL DEFAULT 0
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Exactly one row may have is_active = 1 at any time.
|
||||||
|
CREATE UNIQUE INDEX branches_active_idx ON branches(is_active) WHERE is_active = 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
The "main" branch is implicit and bootstrapped by the migration: `INSERT INTO branches (name, origin_event_id, head_event_id, is_active) VALUES ('main', 0, 0, 1);`. Subsequent branches reference an `origin_event_id` (the event that the branch forked from).
|
||||||
|
|
||||||
|
`chat/state/branches.py`:
|
||||||
|
|
||||||
|
- `@on("branch_created")` payload `{name, origin_event_id, chat_id?, head_event_id}`. Inserts a new row with `is_active=0`. Idempotent re-insertion via `INSERT OR IGNORE`.
|
||||||
|
- `@on("branch_switched")` payload `{name}`. Sets `is_active=1` on the named branch and `is_active=0` on all others. Atomic via a single UPDATE.
|
||||||
|
- `@on("branch_head_updated")` payload `{name, head_event_id}`. Updates `head_event_id` on the named branch. Used by the orchestrator when new events extend the branch.
|
||||||
|
- Readers: `get_branch(conn, name)`, `list_branches(conn, chat_id=None)`, `active_branch(conn)`.
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
|
||||||
|
1. `test_branch_created_inserts_row`: append `branch_created` with name="experiment", origin_event_id=42; project; assert `get_branch(conn, "experiment")` returns the row.
|
||||||
|
2. `test_branch_switched_atomic`: seed two branches; switch from one to the other; assert exactly one is active.
|
||||||
|
3. `test_main_branch_bootstrapped_by_migration`: open a fresh DB, apply migrations; assert `active_branch(conn)["name"] == "main"`.
|
||||||
|
|
||||||
|
**Commit:** `feat: branches table + projector handlers (T89)`.
|
||||||
|
|
||||||
|
**Notes:**
|
||||||
|
|
||||||
|
- Schema version after this migration alone: 13. Combined with T88: 13 (since T88 was 12, T89 stacks). Wave-merge bumps `tests/test_world.py` schema_version assertion to 13.
|
||||||
|
- This task does NOT yet teach the orchestrator to consult `is_active` — the existing event_log queries assume a single timeline. T98 (drawer branching UI) will enable user-driven switches, but the actual "follow only the active branch" filter on event reads is a follow-up (Phase 4.5 nit; document in T102 docs sweep).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 90: Phase 3.6 carry-overs trio
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `chat/services/turn_common.py` (push chat_id filter into SQL)
|
||||||
|
- Modify: `chat/services/regenerate.py` (lifecycle warning wording tightening)
|
||||||
|
- Modify: `chat/services/memory_write.py` (consolidate legacy `record_turn_memory` into the unified API or delete it)
|
||||||
|
- Modify: `tests/test_turn_common.py`, `tests/test_regenerate.py`, `tests/test_memory_write.py`
|
||||||
|
|
||||||
|
**Spec:** Three small Phase 3.6 carry-over fixes bundled because each is 1-line + 1-test.
|
||||||
|
|
||||||
|
#### 90.1 — `read_recent_dialogue` chat-id SQL pushdown
|
||||||
|
|
||||||
|
Per T80 review nit. Currently `read_recent_dialogue` filters chat_id post-fetch in Python. Push into SQL for tighter LIMIT semantics:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT id, kind, payload_json
|
||||||
|
FROM event_log
|
||||||
|
WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn')
|
||||||
|
AND superseded_by IS NULL
|
||||||
|
AND hidden = 0
|
||||||
|
AND json_extract(payload_json, '$.chat_id') = ?
|
||||||
|
ORDER BY id DESC
|
||||||
|
LIMIT ?
|
||||||
|
```
|
||||||
|
|
||||||
|
Then the post-fetch loop becomes a simple reverse + slice — no chat_id check needed.
|
||||||
|
|
||||||
|
**Test added:** `test_read_recent_dialogue_limit_respects_chat_scope` — seed two chats with 60 turns each; query chat_a with `limit=50`; assert returned rows are exactly 50 chat_a rows (not 50 cross-chat rows that filter down to <50 after Python).
|
||||||
|
|
||||||
|
**Commit:** `perf: read_recent_dialogue pushes chat-id filter into SQL (T90.1)`.
|
||||||
|
|
||||||
|
#### 90.2 — Lifecycle warning wording tightening
|
||||||
|
|
||||||
|
Per T83.4 review nit. Current warning lists "lifecycle transitions from superseded turn are NOT being rolled back". When user regenerates an OLDER turn (T29 supports this), the warning lists intervening-turn transitions that legitimately stand. Tighten wording to "lifecycle transitions at-or-after turn X" so operators reading logs aren't misled.
|
||||||
|
|
||||||
|
Change is one log message string. Test asserts the new wording appears.
|
||||||
|
|
||||||
|
**Commit:** `chore: clarify regenerate lifecycle warning wording (T90.2)`.
|
||||||
|
|
||||||
|
#### 90.3 — Legacy `record_turn_memory` consolidation
|
||||||
|
|
||||||
|
Per T84 review nit. The original Phase 1 single-bot `record_turn_memory` function still exists alongside the unified `record_turn_memory_for_present`. Either:
|
||||||
|
|
||||||
|
- (a) Remove the legacy function entirely; update any remaining callers to use the unified API.
|
||||||
|
- (b) Convert it to a thin wrapper for backward compat.
|
||||||
|
|
||||||
|
Pick (a) if there are zero remaining callers; (b) if any callers exist. Read the codebase to confirm. The mock-data seed scripts may still use the legacy fn.
|
||||||
|
|
||||||
|
**Commit:** `refactor: consolidate legacy record_turn_memory into unified API (T90.3)`.
|
||||||
|
|
||||||
|
**TDD process for T90:**
|
||||||
|
|
||||||
|
1. Read all 3 affected files + their tests.
|
||||||
|
2. Implement 90.1 with test; commit.
|
||||||
|
3. Implement 90.2 with test; commit.
|
||||||
|
4. Implement 90.3 with test; commit.
|
||||||
|
5. Run full suite — should be 343 + 3 = 346 (or +2 if 90.3 had no behavioral change).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 2 — Embedding & search services (parallel)
|
||||||
|
|
||||||
|
Three new service modules. Fully file-disjoint.
|
||||||
|
|
||||||
|
### Task 91: Embedding generation service
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/services/embeddings.py`
|
||||||
|
- Create: `tests/test_embeddings.py`
|
||||||
|
|
||||||
|
**Spec:** Wraps the embedding API call. Signature:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class EmbeddingResult(BaseModel):
|
||||||
|
vector: list[float]
|
||||||
|
model: str
|
||||||
|
dim: int
|
||||||
|
|
||||||
|
async def generate_embedding(
|
||||||
|
client: LLMClient, # or a separate embedding-specific client
|
||||||
|
*,
|
||||||
|
text: str,
|
||||||
|
model: str,
|
||||||
|
timeout_s: float = 30.0,
|
||||||
|
) -> EmbeddingResult:
|
||||||
|
"""Generate an embedding vector for the given text. Falls back to a
|
||||||
|
zero-vector with model='fallback' on failure (so callers get a deterministic
|
||||||
|
sentinel they can detect and skip indexing)."""
|
||||||
|
```
|
||||||
|
|
||||||
|
**Implementation:** call the embedding endpoint (Featherless OpenAI-compatible `/v1/embeddings`, or a local `sentence-transformers` model). Add a new method `client.embed(text, model)` to `LLMClient` Protocol (and to `MockLLMClient` and `FeatherlessClient`).
|
||||||
|
|
||||||
|
**Embedding model choice:**
|
||||||
|
|
||||||
|
Default to a small CPU-friendly model accessible through the existing Featherless setup:
|
||||||
|
|
||||||
|
- If Featherless has `BAAI/bge-small-en-v1.5` or similar 384-dim model: use that.
|
||||||
|
- If not: fall back to local `sentence-transformers/all-MiniLM-L6-v2` (384-dim, runs CPU). Add `sentence-transformers` to `pyproject.toml`.
|
||||||
|
- Document choice in CLAUDE.md (T102 docs sweep).
|
||||||
|
|
||||||
|
The 384 dim is hardcoded in T88's migration. If a different model with different dim is chosen, update T88's schema accordingly BEFORE T88 dispatches.
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
|
||||||
|
1. `test_generate_embedding_returns_vector_of_correct_dim`: mock embedding response with a 384-element vector; assert returned `vector` length is 384.
|
||||||
|
2. `test_generate_embedding_returns_correct_model_metadata`: assert `result.model` matches the input.
|
||||||
|
3. `test_generate_embedding_falls_back_on_failure`: mock the client to raise; assert the result is a 384-element zero vector with `model="fallback"`.
|
||||||
|
|
||||||
|
**Commit:** `feat: embedding generation service (T91)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 92: Vector search service via sqlite-vec
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/services/vector_search.py`
|
||||||
|
- Create: `tests/test_vector_search.py`
|
||||||
|
|
||||||
|
**Spec:** Wraps sqlite-vec's `MATCH` syntax for cosine-similarity search over the `embeddings` virtual table. Witness-filter aware (joins through `memories` table for the witness check).
|
||||||
|
|
||||||
|
```python
|
||||||
|
def vector_search(
|
||||||
|
conn,
|
||||||
|
*,
|
||||||
|
owner_id: str,
|
||||||
|
witness_role: str, # "you" | "host" | "guest"
|
||||||
|
query_vector: list[float],
|
||||||
|
k: int = 4,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Return top-K memories by cosine similarity to query_vector,
|
||||||
|
witness-filtered for the requesting bot's POV. Returns same row
|
||||||
|
shape as state.memory.search_memories for combined-ranking
|
||||||
|
compatibility."""
|
||||||
|
```
|
||||||
|
|
||||||
|
SQL pattern (sqlite-vec):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT m.id, m.text, m.pov_summary, m.significance, e.distance
|
||||||
|
FROM embeddings e
|
||||||
|
JOIN memories m ON m.id = e.memory_id
|
||||||
|
WHERE e.embedding MATCH ?
|
||||||
|
AND k = ?
|
||||||
|
AND m.owner_id = ?
|
||||||
|
AND m.witness_<role> = 1
|
||||||
|
ORDER BY e.distance ASC
|
||||||
|
LIMIT ?
|
||||||
|
```
|
||||||
|
|
||||||
|
(Adapt to actual sqlite-vec syntax — use `vec0` MATCH semantics. The `witness_<role>` interpolation needs the same allowlist guard pattern as Phase 2.5 T72.3.)
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
|
||||||
|
1. `test_vector_search_returns_nearest_neighbors`: index 5 memories with synthetic vectors; query for nearest 3; assert correct order.
|
||||||
|
2. `test_vector_search_respects_witness_filter`: index a memory with witness `[1, 1, 0]`; query with `witness_role="guest"`; assert empty result.
|
||||||
|
3. `test_vector_search_respects_owner_filter`: index memories for two owners; assert query for owner_a doesn't return owner_b's memories.
|
||||||
|
|
||||||
|
**Commit:** `feat: vector search service via sqlite-vec (T92)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 93: Cross-chat search service
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/services/cross_chat_search.py`
|
||||||
|
- Create: `tests/test_cross_chat_search.py`
|
||||||
|
|
||||||
|
**Spec:** FTS5-based search across ALL chats and all owners (admin-style search; no witness filter). For "where did I last see this person mention X?" queries.
|
||||||
|
|
||||||
|
```python
|
||||||
|
def search_all_memories(
|
||||||
|
conn,
|
||||||
|
*,
|
||||||
|
query: str,
|
||||||
|
k: int = 20,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Search FTS across all owners and chats. Returns rows with
|
||||||
|
{memory_id, owner_id, chat_id, text, pov_summary, scene_id,
|
||||||
|
significance, ts}. Sorted by FTS rank."""
|
||||||
|
```
|
||||||
|
|
||||||
|
This is intentionally NOT witness-filtered — it's a power-user search surface. The UI (T100) prompts the user to acknowledge they're seeing memories across POVs.
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
|
||||||
|
1. `test_search_all_memories_returns_matches_across_owners`: seed 2 owners with overlapping keyword; search; assert both owner's matches appear.
|
||||||
|
2. `test_search_all_memories_orders_by_fts_rank`: seed memories with varying FTS-match strength; assert order.
|
||||||
|
3. `test_search_all_memories_respects_k_limit`.
|
||||||
|
|
||||||
|
**Commit:** `feat: cross-chat search service (FTS5 over all owners) (T93)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 3 — Branching + delete services (parallel)
|
||||||
|
|
||||||
|
Two new service modules. Fully file-disjoint.
|
||||||
|
|
||||||
|
### Task 94: branch_from_event service
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/services/branching.py`
|
||||||
|
- Create: `tests/test_branching.py`
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
```python
|
||||||
|
def branch_from_event(
|
||||||
|
conn,
|
||||||
|
*,
|
||||||
|
name: str,
|
||||||
|
origin_event_id: int,
|
||||||
|
chat_id: str | None = None,
|
||||||
|
) -> int:
|
||||||
|
"""Create a new named branch forking from origin_event_id.
|
||||||
|
Emits a branch_created event. Returns the new branch's row id.
|
||||||
|
Raises ValueError if name already exists."""
|
||||||
|
|
||||||
|
def switch_active_branch(conn, *, name: str) -> None:
|
||||||
|
"""Make the named branch active. Emits branch_switched. Subsequent
|
||||||
|
event reads should consult is_active to filter."""
|
||||||
|
|
||||||
|
def list_branches_with_metadata(conn, chat_id: str | None = None) -> list[dict]:
|
||||||
|
"""List branches with: name, origin_event_id, head_event_id, is_active,
|
||||||
|
event_count (number of events between origin and head, inclusive),
|
||||||
|
created_at."""
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests cover: basic create, duplicate-name raises, switch updates `is_active` exclusively, list returns metadata.
|
||||||
|
|
||||||
|
**Commit:** `feat: branching service (T94)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 95: Delete-impact computation service
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/services/delete_impact.py`
|
||||||
|
- Create: `tests/test_delete_impact.py`
|
||||||
|
|
||||||
|
**Spec:** Computes the cascade impact of deleting a single event_log row (or a turn group: user_turn + assistant_turn + interjection if any). Returns a structured `ImpactReport` for the UI to render.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class DeletedItem(BaseModel):
|
||||||
|
kind: str # "memory" | "edge_update" | "scene_close" | etc.
|
||||||
|
description: str # human-readable
|
||||||
|
target_id: int | str | None
|
||||||
|
|
||||||
|
class ImpactReport(BaseModel):
|
||||||
|
target_event_id: int
|
||||||
|
cascading: list[DeletedItem]
|
||||||
|
notes: list[str] # warnings, e.g. "this turn opened scene_X which has 3 subsequent turns"
|
||||||
|
|
||||||
|
def compute_delete_impact(conn, *, target_event_id: int) -> ImpactReport:
|
||||||
|
"""Walk the event log forward from target_event_id and identify
|
||||||
|
everything that depends on this event: child memory_written events,
|
||||||
|
edge_update events with this turn as source, scene_closed events
|
||||||
|
triggered by this turn, etc. Also identify subsequent turns that
|
||||||
|
REFERENCE this event (regenerated_from chains, etc.).
|
||||||
|
|
||||||
|
Does NOT mutate the database. Pure computation for preview."""
|
||||||
|
```
|
||||||
|
|
||||||
|
The actual delete (truncate + supersede) is the existing rewind path from Phase 1 T31. T95 just builds the preview.
|
||||||
|
|
||||||
|
**Tests:** 4 minimum.
|
||||||
|
|
||||||
|
1. `test_impact_for_simple_turn_lists_memory_and_edges`: seed a chat with a turn that wrote 1 memory + 2 edge_updates. Compute impact. Assert the 3 items appear in `cascading`.
|
||||||
|
2. `test_impact_for_scene_opening_turn_warns_about_subsequent_turns`: seed a turn that opened a scene + 5 subsequent turns. Assert `notes` mentions the dependency.
|
||||||
|
3. `test_impact_for_regenerated_turn_lists_supersede_chain`: seed a turn that's been regenerated (has `superseded_by`). Compute impact for the original. Assert the chain appears.
|
||||||
|
4. `test_impact_does_not_mutate_database`: snapshot event_log before + after; assert byte-identical.
|
||||||
|
|
||||||
|
**Commit:** `feat: delete-impact computation service (T95)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 4 — Combined retrieval ranking (single)
|
||||||
|
|
||||||
|
### Task 96: Combined FTS + vector retrieval ranking
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `chat/state/memory.py` — extend `search_memories` to optionally include vector hits
|
||||||
|
- Modify: `tests/test_memory_search.py` — add 4 tests
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
`search_memories` currently does FTS5 + Python-side significance/recency re-rank. Phase 4 adds:
|
||||||
|
|
||||||
|
- An optional `query_vector: list[float] | None = None` kwarg.
|
||||||
|
- When `query_vector` is provided, run `vector_search` (T92) for top-K-vector candidates.
|
||||||
|
- Merge with FTS top-K candidates via reciprocal-rank fusion (RRF) or a simpler sum-of-ranks scheme — implementer's choice. Document the merge formula.
|
||||||
|
- Final result is top-K from the fused set, with the existing significance + recency boosts applied as a final pass.
|
||||||
|
|
||||||
|
When `query_vector` is None: existing behavior unchanged. Phase 1/2/3 callers that don't pass `query_vector` see no change.
|
||||||
|
|
||||||
|
**Implementation note:** the embedding for the query (the speaker's recent context) must be generated by the caller (Wave 5 T97 wires the prompt-assembly pipeline to call `generate_embedding` on the dialogue tail). T96 only handles the search side — assumes the vector is pre-computed.
|
||||||
|
|
||||||
|
**Tests:** 4 added.
|
||||||
|
|
||||||
|
1. `test_search_memories_without_query_vector_uses_fts_only`: regression — call without `query_vector`; assert the existing FTS+rerank behavior.
|
||||||
|
2. `test_search_memories_with_query_vector_includes_vector_hits`: index 5 memories where 1 is FTS-only-matching, 1 is vector-only-matching, 3 are unrelated. Pass both `query=...` and `query_vector=...`. Assert both the FTS hit and the vector hit appear in results.
|
||||||
|
3. `test_search_memories_fusion_significance_bias_still_applies`: confirm the existing significance bias rerank still works on top of fused results.
|
||||||
|
4. `test_search_memories_fusion_handles_empty_vector_results`: pass a vector for a memory that has no embeddings indexed; assert FTS-only results still come back.
|
||||||
|
|
||||||
|
**Commit:** `feat: combined FTS + vector retrieval ranking (T96)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 5 — Memory write hook + backfill (single)
|
||||||
|
|
||||||
|
### Task 97: Embedding generation hook + backfill script
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `chat/services/memory_write.py` — after each `memory_written` event, enqueue a background embedding job
|
||||||
|
- Create: `chat/services/embedding_worker.py` — async worker that consumes the queue and emits `embedding_indexed` events
|
||||||
|
- Create: `scripts/backfill_embeddings.py` — one-time script that walks all existing memories and embeds them
|
||||||
|
- Modify: `chat/app.py` — wire the embedding worker into the lifespan startup
|
||||||
|
- Modify: `tests/test_memory_write.py` — add 2 tests for the enqueue hook
|
||||||
|
- Create: `tests/test_embedding_worker.py` — 3 tests for the worker drain logic
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
After each successful `memory_written` event, enqueue an embedding job. The worker dequeues and:
|
||||||
|
|
||||||
|
1. Reads the memory text (via `get_memory(conn, memory_id)`).
|
||||||
|
2. Calls `generate_embedding(client, text=memory.text, model=settings.embedding_model)`.
|
||||||
|
3. Appends `embedding_indexed` event with the result. (Skip if `result.model == "fallback"` — leave the memory un-indexed; will retry later via backfill.)
|
||||||
|
|
||||||
|
The worker pattern mirrors Phase 1's `chat/services/significance.py` SignificanceWorker. Reuse its queue + lifecycle pattern.
|
||||||
|
|
||||||
|
**Backfill script:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/python scripts/backfill_embeddings.py [--limit N] [--dry-run]
|
||||||
|
```
|
||||||
|
|
||||||
|
Walks all memories where no `embeddings_meta` row exists. For each, generates an embedding and emits `embedding_indexed`. Useful for the initial migration after Phase 4 lands AND for periodic re-runs if an embedding model changes.
|
||||||
|
|
||||||
|
**Tests:**
|
||||||
|
|
||||||
|
`tests/test_memory_write.py`:
|
||||||
|
1. `test_record_turn_memory_enqueues_embedding_job`: monkeypatch the worker's enqueue method; record_turn_memory_for_present; assert the worker received a job per memory.
|
||||||
|
|
||||||
|
`tests/test_embedding_worker.py`:
|
||||||
|
1. `test_worker_drains_jobs_and_emits_indexed_events`: enqueue 3 jobs with mock embeddings; run worker; assert 3 `embedding_indexed` events landed.
|
||||||
|
2. `test_worker_skips_fallback_results`: mock the embedding service to return a fallback result; assert NO `embedding_indexed` event landed for that job.
|
||||||
|
3. `test_worker_handles_concurrent_jobs_serially`: pin the Featherless 2-conn cap behavior (worker calls embed sequentially under the existing semaphore).
|
||||||
|
|
||||||
|
**Commit (split):**
|
||||||
|
|
||||||
|
- `feat: embedding worker drains queue and emits embedding_indexed events (T97.1)`
|
||||||
|
- `feat: memory_write enqueues embedding job after each memory_written (T97.2)`
|
||||||
|
- `feat: backfill_embeddings script for existing memories (T97.3)`
|
||||||
|
|
||||||
|
**Verification gates:**
|
||||||
|
|
||||||
|
- All Phase 1/2/3/3.5 memory tests still pass (regression critical).
|
||||||
|
- New tests pass.
|
||||||
|
- Manual smoke: run `scripts/backfill_embeddings.py --dry-run` against a seeded DB and verify expected count.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 6 — Drawer Phase 4 bundle (single task)
|
||||||
|
|
||||||
|
### Task 98: Drawer Phase 4 features
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `chat/web/drawer.py` (add many new POST routes and GET extensions)
|
||||||
|
- Modify: `chat/templates/_drawer.html` (add 5 new sections)
|
||||||
|
- Create: `tests/test_drawer_phase4.py`
|
||||||
|
|
||||||
|
**Spec:** Drawer affordances for 5 Phase 4 features. Single task by hot-file constraint; split into 5 commits internally.
|
||||||
|
|
||||||
|
#### 98.1 — Branching UI
|
||||||
|
|
||||||
|
GET drawer extension: `list_branches_with_metadata(conn)` → render in a "Branches" section (active branch highlighted + count of events).
|
||||||
|
|
||||||
|
POST routes:
|
||||||
|
- `/drawer/branch/create` — form `{name, origin_event_id}` → `branch_from_event` service.
|
||||||
|
- `/drawer/branch/switch` — form `{name}` → `switch_active_branch`.
|
||||||
|
- `/drawer/branch/from-turn/{event_id}` — convenience: branch from a specific turn (used by per-turn UI affordance).
|
||||||
|
|
||||||
|
#### 98.2 — Significance review panel
|
||||||
|
|
||||||
|
GET extension: significance distribution per chat (`SELECT significance, COUNT(*) GROUP BY significance`) → render histogram.
|
||||||
|
|
||||||
|
POST route:
|
||||||
|
- `/drawer/memory/significance/{memory_id}` — form `{new_value}` (already supported via T22 `manual_edit` `target_kind=memory_significance`); just add the UI form.
|
||||||
|
|
||||||
|
Bulk re-rate is a Phase 4.5 polish — not in scope here. Just per-memory edit + distribution display.
|
||||||
|
|
||||||
|
#### 98.3 — Hide-from-view toggle
|
||||||
|
|
||||||
|
POST route:
|
||||||
|
- `/drawer/turn/hide/{event_id}` — form `{hidden: bool}` → emits a `manual_edit` with `target_kind="turn_hidden"`.
|
||||||
|
|
||||||
|
NEW `manual_edit` projector branch for `turn_hidden`: sets `event_log.hidden = ?` for the target event. Reuses the existing `hidden` column.
|
||||||
|
|
||||||
|
UI affordance: per-turn checkbox in the chat surface or drawer (per-turn list with hide toggle).
|
||||||
|
|
||||||
|
#### 98.4 — Surgical delete with cascade preview
|
||||||
|
|
||||||
|
GET extension:
|
||||||
|
- `/drawer/turn/delete-preview/{event_id}` → returns the `ImpactReport` (T95) rendered as a modal.
|
||||||
|
|
||||||
|
POST route:
|
||||||
|
- `/drawer/turn/delete/{event_id}` — invokes the rewind-and-truncate path (Phase 1 T31's `rewind_to_turn`) restricted to the target turn group.
|
||||||
|
|
||||||
|
Important: this reuses the existing pre-rewind snapshot path so the action is undoable.
|
||||||
|
|
||||||
|
#### 98.5 — Remaining v1 edits
|
||||||
|
|
||||||
|
Audit: are any v1 fields STILL not editable from the drawer? Phase 2.5 T72.1 added edge_trust/edge_summary/memory_pov_summary/edge_knowledge_facts. T72.3 added witness flags. Anything left?
|
||||||
|
|
||||||
|
Likely candidates: scene `narrative_anchor`, scene `weather`, container `properties` JSON. Add edit forms for any that surface during the audit. If none, this sub-fix is a no-op.
|
||||||
|
|
||||||
|
**Tests:** 8+ in `tests/test_drawer_phase4.py` (one per sub-feature × happy path; plus 1 for the cascade-preview rendering).
|
||||||
|
|
||||||
|
**Commits (5):**
|
||||||
|
|
||||||
|
- `feat: drawer branching UI (T98.1)`
|
||||||
|
- `feat: drawer significance review panel (T98.2)`
|
||||||
|
- `feat: drawer hide-from-view toggle + manual_edit turn_hidden branch (T98.3)`
|
||||||
|
- `feat: drawer surgical delete with cascade preview (T98.4)`
|
||||||
|
- `feat: drawer remaining v1 field edits (T98.5)` (or "no-op audit" if nothing left)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 7 — Snapshot + cross-chat search UX (parallel)
|
||||||
|
|
||||||
|
### Task 99: Snapshot UX
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/web/snapshots.py` (new route module)
|
||||||
|
- Create: `chat/templates/snapshots.html` (snapshot list page)
|
||||||
|
- Modify: `chat/templates/layout.html` (add "Snapshots" nav link)
|
||||||
|
- Create: `tests/test_snapshot_ux.py`
|
||||||
|
|
||||||
|
**Spec:** Surface the existing snapshot infrastructure (Phase 1 T20 wrote snapshots; Phase 4 makes them visible).
|
||||||
|
|
||||||
|
GET `/snapshots` — list all snapshots (periodic + pre-rewind) with metadata: kind, created_at, event_log_size, file_size_bytes.
|
||||||
|
|
||||||
|
POST `/snapshots/take` — manually trigger a snapshot now.
|
||||||
|
|
||||||
|
POST `/snapshots/restore/{snapshot_id}` — restore from snapshot (with hard confirmation).
|
||||||
|
|
||||||
|
GET `/snapshots/{snapshot_id}/preview` — show what's in the snapshot vs. current state.
|
||||||
|
|
||||||
|
**Tests:** 4 minimum (list, take, restore, preview).
|
||||||
|
|
||||||
|
**Commit:** `feat: snapshot UX (manual trigger, list, restore) (T99)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 100: Cross-chat search UX
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `chat/web/search.py` (new route module)
|
||||||
|
- Create: `chat/templates/search.html` (search results page)
|
||||||
|
- Modify: `chat/templates/layout.html` (add top-bar search input)
|
||||||
|
- Create: `tests/test_search_ux.py`
|
||||||
|
|
||||||
|
**Spec:** Top-bar search box submits to `/search?q=...`. Results page shows up to 50 matches across all chats and all owners (uses T93's `search_all_memories`). Each result shows: chat name, owner bot name, scene context, memory text excerpt with FTS highlight, "Open chat at this turn" link.
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
1. Search returns results from multiple chats.
|
||||||
|
2. Empty query returns empty result set.
|
||||||
|
3. Result links navigate to the right chat anchor.
|
||||||
|
|
||||||
|
**Commit:** `feat: cross-chat search UX (top-bar input + results page) (T100)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 8 — Polish (parallel)
|
||||||
|
|
||||||
|
### Task 101: Cross-feature integration tests
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Create: `tests/test_phase4_integration.py`
|
||||||
|
|
||||||
|
**Spec:** End-to-end multi-feature flows. 5 tests minimum.
|
||||||
|
|
||||||
|
1. **Vector retrieval feedback loop**: write a memory → embedding worker indexes it → search retrieves it via vector path.
|
||||||
|
2. **Branch + diverge**: create branch B from turn 10 → switch to B → play 3 new turns → switch back to main → assert main's turn 11+ are still intact.
|
||||||
|
3. **Surgical delete**: compute impact for a turn → confirm → assert event log truncated correctly + pre-rewind snapshot saved.
|
||||||
|
4. **Hide + retrieval**: hide a turn → assert it doesn't appear in `read_recent_dialogue` (existing `hidden = 0` filter) → unhide → assert it reappears.
|
||||||
|
5. **Cross-chat search**: write memories in 3 chats → search for keyword present in all 3 → assert all 3 appear in results.
|
||||||
|
|
||||||
|
**Commit:** `test: phase 4 cross-feature integration coverage (T101)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 102: Phase 4 documentation update
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
|
||||||
|
- Modify: `CLAUDE.md` (add "Phase 4 status" section; update behavioral defaults; add "Phase 4.5 / 5 backlog" with carry-overs)
|
||||||
|
- Modify: `docs/plans/2026-04-26-v1-requirements-design.md` (annotate §13 Phase 4 as **Status: shipped 2026-04-27**)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Mirror the Phase 3 / 3.5 status sections. Document:
|
||||||
|
|
||||||
|
- **Vector retrieval**: sqlite-vec virtual table, embedding worker async pipeline, combined FTS + vector ranking via RRF.
|
||||||
|
- **Branching**: forks the event log; UI in drawer; `is_active` flag plus orchestrator filter (caveat — see backlog if filter not yet wired into all readers).
|
||||||
|
- **Drawer-edit on every field**: branching, significance review, hide-from-view, surgical delete with preview, plus any audit findings.
|
||||||
|
- **Backup tooling**: snapshots panel surfaces existing infra.
|
||||||
|
- **Significance review UI**: distribution + per-memory edit.
|
||||||
|
- **Surgical delete + cascade preview**: piggybacks on rewind path; impact report from T95.
|
||||||
|
- **Hide-from-view soft delete**: `manual_edit` `turn_hidden` branch.
|
||||||
|
- **Cross-chat search**: top-bar + results page over T93's service.
|
||||||
|
|
||||||
|
**Phase 4.5 / 5 backlog candidates** (reflect any discovered during execution):
|
||||||
|
|
||||||
|
- Branching read-side filter — if T89's `is_active` isn't yet consulted by every event reader, this is the work to do.
|
||||||
|
- Bulk significance re-rate (per T98.2 deferral).
|
||||||
|
- Snapshot retention policy UI controls (per Phase 1 T19 deferred).
|
||||||
|
- Auto-pin override UI (per Phase 2 design).
|
||||||
|
- Embedding model swap migration tooling (when changing embedding model, need to re-embed everything).
|
||||||
|
- Vector index optimization (HNSW vs flat — Phase 5 if needed).
|
||||||
|
- Carry-overs that remained deferred from Phase 3.6: scene-close-on-cancel UX revisit, canned-queue brittleness fixture builder, full lifecycle rollback in regenerate.
|
||||||
|
|
||||||
|
**Commit:** `docs: phase 4 status, behavioral defaults, deferred items (T102)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wrap-up
|
||||||
|
|
||||||
|
After Wave 8 lands:
|
||||||
|
|
||||||
|
1. **Run full suite** on `phase-4`: should be ~390+ tests passing (343 from Phase 3.5 + ~50 new).
|
||||||
|
2. **Manual smoke** (recommended before opening the PR):
|
||||||
|
- Run `scripts/backfill_embeddings.py` against a seeded DB to verify vector indexing works.
|
||||||
|
- Search for a phrase that's substring-distinct but semantically similar to a memory; verify vector path returns it (FTS would miss).
|
||||||
|
- Create a branch from an old turn; switch; play a few turns; switch back.
|
||||||
|
- Trigger surgical delete on a turn; verify the impact preview matches what actually gets removed.
|
||||||
|
- Hide a turn; verify it disappears from the chat surface; unhide.
|
||||||
|
- Use top-bar search to find a phrase; verify cross-chat results appear.
|
||||||
|
- Click the "Snapshots" nav link; trigger a manual snapshot; verify it appears.
|
||||||
|
3. **Push `phase-4`** to gitea.
|
||||||
|
4. **Open PR** `phase-4 → main`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes for the controller running this plan
|
||||||
|
|
||||||
|
- **External dependency**: `sqlite-vec` (or `sqlite-vss`) MUST be added to `pyproject.toml` and installed BEFORE Wave 1 dispatches. The migration in T88 expects the extension to be loadable.
|
||||||
|
- **Embedding model choice**: pin in T91 spec before dispatch. The 384 dim is hardcoded in T88's migration; if a different dim is used, update T88 first.
|
||||||
|
- **After each parallel wave**, run a code-review subagent. Combined spec+quality acceptable for trivial tasks (T90 carry-overs); separate spec + quality reviewers for vector-retrieval and integration tasks (T91, T96, T97, T98, T101) — surface area is larger.
|
||||||
|
- **Don't dispatch Wave 5 until Wave 4 merged green.** T97 (memory_write enqueue) calls into the embedding-aware worker; the worker uses T91's `generate_embedding`. Both must be merged into `phase-4` first.
|
||||||
|
- **Don't dispatch Wave 6 until Wave 5 merged green.** T98 (drawer) wires UI affordances over services from earlier waves.
|
||||||
|
- **Token-spend rough estimate**: Phase 4 should be ~70-80% the size of Phase 3 (similar scope, larger per-task because vector + branching are non-trivial). Per-task spend similar to Phase 3's larger tasks (T59, T64).
|
||||||
|
- **DO NOT break existing v1/v2/v3/v3.5 surface contracts.** Every test file that was green at the start of Phase 4 must stay green at the end. The cross-feature integration tests from Phase 3 (`tests/test_phase3_integration.py`) are particularly load-bearing.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"planPath": "docs/plans/2026-04-27-v4-phase4-implementation.md",
|
||||||
|
"tasks": [
|
||||||
|
{"id": 88, "subject": "T88: embeddings table + projector handlers (sqlite-vec)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 89, "subject": "T89: branches table + projector handlers", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 90, "subject": "T90: phase 3.6 carry-overs (chat-id pushdown + lifecycle wording + legacy fn consolidation)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 91, "subject": "T91: embedding generation service", "status": "pending", "wave": 2, "parallelGroup": "wave-2", "blockedBy": [88]},
|
||||||
|
{"id": 92, "subject": "T92: vector search service via sqlite-vec", "status": "pending", "wave": 2, "parallelGroup": "wave-2", "blockedBy": [88]},
|
||||||
|
{"id": 93, "subject": "T93: cross-chat search service (FTS5 over all owners)", "status": "pending", "wave": 2, "parallelGroup": "wave-2"},
|
||||||
|
{"id": 94, "subject": "T94: branch_from_event service", "status": "pending", "wave": 3, "parallelGroup": "wave-3", "blockedBy": [89]},
|
||||||
|
{"id": 95, "subject": "T95: delete-impact computation service", "status": "pending", "wave": 3, "parallelGroup": "wave-3"},
|
||||||
|
{"id": 96, "subject": "T96: combined FTS + vector retrieval ranking in search_memories", "status": "pending", "wave": 4, "parallelGroup": null, "blockedBy": [91, 92]},
|
||||||
|
{"id": 97, "subject": "T97: memory_write enqueues embedding job + backfill script", "status": "pending", "wave": 5, "parallelGroup": null, "blockedBy": [91, 96]},
|
||||||
|
{"id": 98, "subject": "T98: drawer Phase 4 bundle (branching + sig review + hide + surgical delete + remaining edits)", "status": "pending", "wave": 6, "parallelGroup": null, "blockedBy": [94, 95, 97]},
|
||||||
|
{"id": 99, "subject": "T99: snapshot UX (manual trigger + list + restore + preview)", "status": "pending", "wave": 7, "parallelGroup": "wave-7"},
|
||||||
|
{"id": 100, "subject": "T100: cross-chat search UX (top-bar + results page)", "status": "pending", "wave": 7, "parallelGroup": "wave-7", "blockedBy": [93]},
|
||||||
|
{"id": 101, "subject": "T101: cross-feature integration tests (vector × branching × delete × snapshot × search)", "status": "pending", "wave": 8, "parallelGroup": "wave-8", "blockedBy": [98, 99, 100]},
|
||||||
|
{"id": 102, "subject": "T102: Phase 4 documentation update", "status": "pending", "wave": 8, "parallelGroup": "wave-8", "blockedBy": [98, 99, 100]}
|
||||||
|
],
|
||||||
|
"lastUpdated": "2026-04-27T00:00:00Z",
|
||||||
|
"notes": "15 tasks across 8 waves. Adds vector retrieval (sqlite-vec), branching UI, drawer-edit on every field, backup tooling, significance review UI, surgical delete with cascade preview, hide-from-view, and cross-chat search. Phase 3.6 carry-overs (3 small fixes) bundled into T90. External dependency: sqlite-vec must be installed BEFORE Wave 1 dispatch. Embedding model choice (default: 384-dim small model) pinned in T91 spec before dispatch — schema 0012 hardcodes 384 dim. Two new schema migrations (0012 embeddings, 0013 branches), final schema version 13. Uses task ids T88-T102."
|
||||||
|
}
|
||||||
@@ -0,0 +1,724 @@
|
|||||||
|
# Roleplay Engine — Phase 4.5 Cleanup Plan
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: Use `superpowers-extended-cc:executing-plans` to implement this plan task-by-task. Use the parallel-dispatch pattern documented under "Parallel-Execution Strategy" for parallel waves.
|
||||||
|
|
||||||
|
**Goal:** Burn down all 24 items in `CLAUDE.md` §"Phase 4.5 / 5 backlog". Mix of small defensive cleanups (most), three big features (real embedding model swap, branching read-side filter, lifecycle rollback in regenerate), one environment-dependent feature (sqlite-vec swap), and the long-deferred carry-overs (scene-close-on-cancel revisit, structured test-fixture builder).
|
||||||
|
|
||||||
|
**Architecture:** No new architecture. Two new schema migrations (0014 schema polish, 0015 sqlite-vec virtual tables). New external dependency optional (`apsw` if Python rebuild isn't possible). All other changes are polish / refactor / observability.
|
||||||
|
|
||||||
|
**Tech Stack:**
|
||||||
|
|
||||||
|
- Existing — same as Phase 4.
|
||||||
|
- **OPTIONAL:** rebuild Python with `--enable-loadable-sqlite-extensions` OR install `apsw` to enable T115 sqlite-vec swap. T115 is the only task that requires this; the other 13 tasks land without it. If neither is available, T115 is deferred to Phase 5.
|
||||||
|
|
||||||
|
**Source-of-truth references:**
|
||||||
|
|
||||||
|
- Backlog: [`CLAUDE.md`](../../CLAUDE.md) §"Phase 4.5 / 5 backlog" (24 items grouped by review source + deferred).
|
||||||
|
- Phase 3.5 / Phase 2.5 cleanup plans (pattern reference): [2026-04-26-v3.5-phase3.5-cleanup.md](2026-04-26-v3.5-phase3.5-cleanup.md), [2026-04-26-v2.5-phase2.5-cleanup.md](2026-04-26-v2.5-phase2.5-cleanup.md).
|
||||||
|
- Conventions: [`CLAUDE.md`](../../CLAUDE.md) §"Behavioral defaults" + §"Phase 4 status".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Pre-flight
|
||||||
|
|
||||||
|
**Branch:** create `phase-4.5` from the latest `main`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git checkout main && git pull && git checkout -b phase-4.5
|
||||||
|
```
|
||||||
|
|
||||||
|
**Schema baseline:** Phase 4 leaves the DB at version 13. Phase 4.5 adds two migrations: `0014_phase45_schema.sql` (T109) and `0015_vec0_virtual_tables.sql` (T115 — only lands if T115 ships). Final schema version: 14 or 15.
|
||||||
|
|
||||||
|
**Optional pre-flight for T115 (sqlite-vec swap):**
|
||||||
|
|
||||||
|
The host Python build needs `enable_load_extension`. Two options:
|
||||||
|
|
||||||
|
1. **Rebuild Python** via pyenv with `PYTHON_CONFIGURE_OPTS="--enable-loadable-sqlite-extensions" pyenv install 3.12.0 --force` and recreate the venv.
|
||||||
|
2. **Add `apsw`** as a dependency and migrate `chat/db/connection.py` to use `apsw.Connection` (significant refactor — the entire codebase uses stdlib `sqlite3`).
|
||||||
|
|
||||||
|
If neither is acceptable, **defer T115** to Phase 5 and ship Phase 4.5 with 13 tasks instead of 14. The other tasks are unaffected.
|
||||||
|
|
||||||
|
**Pinned non-negotiables (carried forward):**
|
||||||
|
|
||||||
|
- State changes go through the event log. Use `append_and_apply` for the live path.
|
||||||
|
- Witness filter every memory read at SQL level.
|
||||||
|
- TDD: every task starts with a failing test (or a regression test pinning existing contract before refactor).
|
||||||
|
- One commit per task minimum. Bundled tasks split internally.
|
||||||
|
|
||||||
|
**Verification before claiming done:** Use `superpowers-extended-cc:verification-before-completion` — run the test command, paste actual output.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Backlog item → task mapping
|
||||||
|
|
||||||
|
24 items consolidated into 14 tasks by **file ownership**:
|
||||||
|
|
||||||
|
| # | Item | Source | Task |
|
||||||
|
|---|------|--------|------|
|
||||||
|
| 1 | `embeddings` FK lacks `ON DELETE CASCADE` | T88 | **T109** (schema migration) |
|
||||||
|
| 2 | `list_branches(chat_id=...)` global-branch leak — document | T89 | **T103** |
|
||||||
|
| 3 | Branch-switch silently leaves zero active — log warning | T89 | **T103** |
|
||||||
|
| 4 | Real embedding model swap | T91 / deferred | **T112** |
|
||||||
|
| 5 | `timeout_s` fallback-path logging | T91 | **T107** |
|
||||||
|
| 6 | Duplicate `MAX(id)` lookup in retrieval ranking | T96 | **T104** |
|
||||||
|
| 7 | `fts_rank=None` for vector-only rows — document | T96 | **T104** |
|
||||||
|
| 8 | `event_id <= 0` guard in `delete_turn` | T98 | **T110** |
|
||||||
|
| 9 | `html.escape()` on delete-impact modal output | T98 | **T110** |
|
||||||
|
| 10 | Extract delete-impact modal to Jinja partial | T98 | **T110** |
|
||||||
|
| 11 | Hoist `datetime`/`timezone` imports in `snapshots.py` | T99 | **T105** |
|
||||||
|
| 12 | Strict `kind` validation in snapshot routes | T99 | **T105** |
|
||||||
|
| 13 | `created_at` from file mtime — document drift risk | T99 | **T105** |
|
||||||
|
| 14 | Hardcoded `k=50` → module constant | T100 | **T106** |
|
||||||
|
| 15 | N+1 lookups in search results | T100 | **T106** |
|
||||||
|
| 16 | FTS highlighting via `snippet()` | T100 | **T111** |
|
||||||
|
| 17 | Result links chat-level only — add deep-link via memories.event_id | T100 | **T109** + **T111** |
|
||||||
|
| 18 | sqlite-vec swap when host Python supports loadable extensions | deferred | **T115** |
|
||||||
|
| 19 | Branching read-side filter (consult `is_active`) | deferred | **T113** |
|
||||||
|
| 20 | Bulk significance re-rate in drawer | deferred | **T110** |
|
||||||
|
| 21 | Vector index optimization (HNSW) | deferred | **T115** (post-ship note) |
|
||||||
|
| 22 | Scene-close-on-cancel UX revisit | Phase 2.5 carry-over | **T108** |
|
||||||
|
| 23 | Cross-feature canned-queue brittleness fixture builder | Phase 3 carry-over | **T116** |
|
||||||
|
| 24 | Full lifecycle-rollback in regenerate | Phase 3.5 carry-over | **T114** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parallel-Execution Strategy
|
||||||
|
|
||||||
|
Same pattern as Phase 3.5 / Phase 2.5 / Phase 4. Nine waves: parallel within each wave (file-disjoint), serial across waves.
|
||||||
|
|
||||||
|
### How to dispatch a wave in parallel
|
||||||
|
|
||||||
|
Use the **Agent tool with `isolation: "worktree"`**. (If the controlling session's working directory is **not** the chat repo, create worktrees manually with `git worktree add .worktrees/<wave>-<task> -b <wave>/<task> phase-4.5`.)
|
||||||
|
|
||||||
|
### After a wave completes
|
||||||
|
|
||||||
|
1. Each subagent returns its worktree path and commit SHA(s).
|
||||||
|
2. **Run a spec + code-quality reviewer subagent on each completed task.** Combined review acceptable for trivial tasks (T103–T108); separate spec + quality reviewers for big tasks (T112, T113, T114, T115).
|
||||||
|
3. **Merge the wave into `phase-4.5`** in any order (file-disjointness guarantees no conflict). Use `--no-ff`.
|
||||||
|
4. **Run the full test suite** on the merged `phase-4.5`.
|
||||||
|
5. **Push `phase-4.5`** to gitea.
|
||||||
|
6. Optionally clean up worktrees.
|
||||||
|
|
||||||
|
### Conflict prevention checklist
|
||||||
|
|
||||||
|
For each parallel wave, verify the **Files** sections of all tasks have **no overlapping paths**. Hot files in this plan (each owned by exactly one task): `chat/state/memory.py`, `chat/web/drawer.py`, `chat/web/search.py`, `chat/services/regenerate.py`, `chat/services/turn_common.py`, `chat/services/embeddings.py`, `chat/db/migrations/`.
|
||||||
|
|
||||||
|
### Why each wave is parallel-safe
|
||||||
|
|
||||||
|
| Wave | Tasks | Hot files | Disjoint? |
|
||||||
|
|------|-------|-----------|-----------|
|
||||||
|
| 1 | T103, T104, T105, T106, T107, T108 | 6 different files; no overlap | ✅ |
|
||||||
|
| 2 | T109 | new migration + minor projector update | (single task) |
|
||||||
|
| 3 | T110 | `chat/web/drawer.py` (bundle) | (single task) |
|
||||||
|
| 4 | T111 | `chat/services/cross_chat_search.py` + `chat/web/search.py` + template | (single task; depends on T109) |
|
||||||
|
| 5 | T112 | `chat/services/embeddings.py` + `chat/llm/*.py` (Protocol + Featherless + Mock) | (single task) |
|
||||||
|
| 6 | T113 | `chat/services/turn_common.py` + multiple readers (cross-cutting) | (single task) |
|
||||||
|
| 7 | T114 | `chat/services/regenerate.py` + projector handler | (single task) |
|
||||||
|
| 8 | T115 | new migration + `chat/services/vector_search.py` + `chat/db/connection.py` | (single task; environmental) |
|
||||||
|
| 9 | T116, T117, T118 | new test fixture file (T116); new test file (T117); CLAUDE.md (T118) | ✅ |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Task overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Wave 1 ─┬─ T103: branches polish (global-branch doc + branch-switch warning)
|
||||||
|
├─ T104: state/memory.py polish (DRY MAX(id) + fts_rank doc)
|
||||||
|
├─ T105: snapshots.py polish (datetime hoist + kind validation + mtime doc)
|
||||||
|
├─ T106: search.py polish (k constant + N+1 batched lookups)
|
||||||
|
├─ T107: embeddings.py timeout_s fallback-path logging
|
||||||
|
└─ T108: scene-close-on-cancel UX revisit (pin behavior with regression test)
|
||||||
|
|
||||||
|
Wave 2 ─── T109: 0014 schema migration (FK CASCADE + memories.event_id column)
|
||||||
|
|
||||||
|
Wave 3 ─── T110: drawer Phase 4.5 bundle (event_id guard + html.escape + modal partial + bulk sig re-rate)
|
||||||
|
|
||||||
|
Wave 4 ─── T111: search UX enhancements (FTS snippet() highlighting + deep-link via memories.event_id)
|
||||||
|
|
||||||
|
Wave 5 ─── T112: real embedding model swap (LLMClient.embed protocol + Featherless impl + generate_embedding routing + backfill)
|
||||||
|
|
||||||
|
Wave 6 ─── T113: branching read-side filter (event readers consult is_active branch range)
|
||||||
|
|
||||||
|
Wave 7 ─── T114: regenerate lifecycle rollback (back-reference field + compensating events on supersede)
|
||||||
|
|
||||||
|
Wave 8 ─── T115: sqlite-vec swap (vec0 virtual tables + MATCH-based vector_search) [ENVIRONMENTAL — see pre-flight]
|
||||||
|
|
||||||
|
Wave 9 ─┬─ T116: structured test-fixture builder (canned-queue brittleness)
|
||||||
|
├─ T117: Phase 4.5 cross-feature integration tests
|
||||||
|
└─ T118: docs sweep — Phase 4.5 status, prune backlog, capture Phase 5 residuals
|
||||||
|
```
|
||||||
|
|
||||||
|
Critical path: 9 sequential merge points. Total tasks: 14 (or 13 if T115 deferred). Parallelism: Waves 1 (6-way) and 9 (3-way) dispatch concurrently. Waves 2–8 are single-task by hot-file constraint.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 1 — Independent small fixes (parallel, 6 tasks)
|
||||||
|
|
||||||
|
All trivial, file-disjoint. Each is 1-line + 1-test or similar.
|
||||||
|
|
||||||
|
### Task 103: branches polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/state/branches.py`
|
||||||
|
- Modify: `tests/test_branches_state.py`
|
||||||
|
|
||||||
|
**Spec (2 sub-fixes, single commit):**
|
||||||
|
|
||||||
|
1. **Document global-branch leak**: `list_branches(chat_id=...)` filter `chat_id = ? OR chat_id IS NULL` returns global/null-chat branches (like "main") in every chat scope. Add a docstring note explaining this is intentional ("main" is global by design; per-chat branches are scoped).
|
||||||
|
|
||||||
|
2. **Warn on branch-switch to nonexistent name**: in `_apply_branch_switched`, before the SQL UPDATE, check if a branch with the given name exists. If not, emit `logging.getLogger(__name__).warning(...)` rather than silently leaving zero active branches.
|
||||||
|
|
||||||
|
**Test:** `test_branch_switched_unknown_name_warns` — capture log via `caplog`, append `branch_switched` for nonexistent name, assert warning message + no active branch (existing behavior preserved, just observable).
|
||||||
|
|
||||||
|
**Commit:** `chore: branches polish — global-leak docs + unknown-name warning (T103)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 104: state/memory.py polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/state/memory.py`
|
||||||
|
- Modify: `tests/test_memory_search.py` (no new tests; just add docstring assertions if needed)
|
||||||
|
|
||||||
|
**Spec (2 sub-fixes):**
|
||||||
|
|
||||||
|
1. **DRY `MAX(id)` lookup**: `_composite_rerank` (Phase 3.5 T57) and `_rrf_fuse_and_rerank` (Phase 4 T96) both query `SELECT MAX(id) FROM event_log` for the recency boost. Extract a `_max_event_id(conn)` helper.
|
||||||
|
|
||||||
|
2. **`fts_rank=None` documentation**: search_memories docstring should note that vector-only rows have `fts_rank=None`. Downstream consumers must accept None (they currently do, but contract is implicit).
|
||||||
|
|
||||||
|
**Test:** existing tests cover both via the public API; no new test needed unless docstring assertion is desired.
|
||||||
|
|
||||||
|
**Commit:** `chore: memory.py DRY MAX(id) helper + document fts_rank=None contract (T104)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 105: snapshots.py polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/web/snapshots.py`
|
||||||
|
- Modify: `tests/test_snapshot_ux.py` (1 new test)
|
||||||
|
|
||||||
|
**Spec (3 sub-fixes):**
|
||||||
|
|
||||||
|
1. **Hoist `datetime`/`timezone` imports** to module level (currently inside `_list_all_snapshots`).
|
||||||
|
|
||||||
|
2. **Strict `kind` validation in restore/preview routes**: currently `kind` defaults to `"periodic"`. If a rewind snapshot is requested without explicit `kind`, the lookup silently 404s. Reject missing `kind` with a 400 instead of silently defaulting.
|
||||||
|
|
||||||
|
3. **Document `created_at` mtime drift risk** in module docstring: snapshot timestamps come from file mtime, not the encoded filename timestamp. Files copied via `cp -p` preserve mtime; `cp` without `-p` resets it. Add a one-line note.
|
||||||
|
|
||||||
|
**Test:** `test_restore_without_kind_returns_400` — POST `/snapshots/restore/<id>` without `kind`; assert 400.
|
||||||
|
|
||||||
|
**Commit:** `chore: snapshots.py polish — hoisted imports + strict kind + mtime doc (T105)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 106: search.py polish
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/web/search.py`
|
||||||
|
- Modify: `tests/test_search_ux.py` (1 new test)
|
||||||
|
|
||||||
|
**Spec (2 sub-fixes):**
|
||||||
|
|
||||||
|
1. **Hardcoded `k=50` → module constant**: extract `DEFAULT_SEARCH_K = 50` at module level. Tunable without code change at the call site.
|
||||||
|
|
||||||
|
2. **N+1 lookup batching**: GET `/search?q=...` currently calls `get_bot(conn, owner_id)`, `get_chat(conn, chat_id)`, `get_scene(conn, scene_id)` per result row (worst case 50×3 = 150 individual queries). Batch via `WHERE id IN (...)` queries: collect distinct ids first, fetch in 3 batched queries, then map back per row.
|
||||||
|
|
||||||
|
**Test:** `test_search_results_use_batched_lookups` — mock `get_bot`/`get_chat`/`get_scene` and assert each is called once (not per row). OR easier: time the search with 50 results and assert it doesn't degrade linearly with `k`.
|
||||||
|
|
||||||
|
**Commit:** `perf: search.py N+1 batching + k constant extraction (T106)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 107: embeddings.py timeout_s fallback-path logging
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/embeddings.py`
|
||||||
|
- Modify: `tests/test_embeddings.py` (1 new test)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
When `model != DEFAULT_EMBEDDING_MODEL` and falls through to fallback (zero-vector with model="fallback"), log a `warning` so misconfigured callers (e.g., a Phase 4.5+ caller pointing at a real model that doesn't exist) don't silently degrade.
|
||||||
|
|
||||||
|
```python
|
||||||
|
if model != DEFAULT_EMBEDDING_MODEL:
|
||||||
|
_log.warning(
|
||||||
|
"generate_embedding: non-default model %r returned fallback "
|
||||||
|
"(model client.embed() not yet implemented in Phase 4.5+); "
|
||||||
|
"downstream search will degrade silently. Configure a supported model.",
|
||||||
|
model,
|
||||||
|
)
|
||||||
|
return EmbeddingResult(...) # fallback
|
||||||
|
```
|
||||||
|
|
||||||
|
The Phase 4 default path (`model == DEFAULT_EMBEDDING_MODEL` → pseudo-embedding) is silent; only non-default models trigger the warning.
|
||||||
|
|
||||||
|
**Test:** `test_generate_embedding_non_default_model_logs_warning` — call with `model="real-model"`; capture log via `caplog`; assert the warning message appears.
|
||||||
|
|
||||||
|
**Commit:** `chore: embeddings.py warns on fallback for non-default models (T107)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 108: scene-close-on-cancel UX revisit
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `tests/test_turn_flow.py` (extend the existing pin test added in Phase 2.5 T74.3 OR add a new one)
|
||||||
|
- Optionally modify: `chat/web/turns.py` if a real bug surfaces during investigation
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
This carry-over has been pending since Phase 2.5 T74.3. The pinned behavior: scene close fires even when the primary turn is cancelled mid-stream, because `detect_scene_close` consults user prose (fully present at cancel time), not bot output.
|
||||||
|
|
||||||
|
**Action:**
|
||||||
|
|
||||||
|
1. **Re-investigate** by reading the post_turn cancellation path. Confirm the rationale still holds (it should — nothing about the close-detection logic changed in Phase 3 or 4).
|
||||||
|
2. **Strengthen the regression test** in `tests/test_turn_flow.py` (the existing `test_cancelled_turn_still_closes_scene_when_user_prose_signals_close`). Add an assertion that the user prose IS present at the moment scene_close_decision fires (even though the bot output isn't).
|
||||||
|
3. If investigation surfaces an actual UX issue (e.g., the close fires too eagerly on prose like "fade out... actually wait"), this becomes a real fix — but default action is documentation-only.
|
||||||
|
|
||||||
|
**Default outcome:** add a docstring comment to the post_turn close-detection branch explaining the rationale. No behavioral change.
|
||||||
|
|
||||||
|
**Test (extend existing):** assert ordering — `scene_closed` event lands AFTER the user_turn event but BEFORE any potential assistant_turn (which is cancelled). Pin the contract.
|
||||||
|
|
||||||
|
**Commit:** `chore: scene-close-on-cancel — strengthen regression test + document rationale (T108)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 2 — Schema migration (single)
|
||||||
|
|
||||||
|
### Task 109: 0014 schema migration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `chat/db/migrations/0014_phase45_schema.sql`
|
||||||
|
- Modify: `chat/state/memory.py` or `chat/services/memory_write.py` (populate the new `event_id` column on memory_written)
|
||||||
|
- Modify: `tests/test_world.py` (bump schema_version assertion to 14)
|
||||||
|
- Modify: `tests/test_memory_write.py` (assert event_id populated)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Two schema changes bundled into a single migration:
|
||||||
|
|
||||||
|
1. **`embeddings.memory_id` FK gets `ON DELETE CASCADE`** (T88 review nit). SQLite doesn't support `ALTER TABLE ... ALTER COLUMN`, so the standard pattern is: rename old table, create new, copy data, drop old, recreate indices. Alternatively, since this is a new-ish table (Phase 4 added it) and the change is purely defensive, document as "WONTFIX in 4.5; deindex events remain the only deletion path; ON DELETE CASCADE remains a Phase 5 candidate when we do a broader migration cleanup". Choose pragmatically.
|
||||||
|
|
||||||
|
2. **Add `memories.event_id INTEGER` column** (NULL allowed for backward compat) referencing `event_log.id`. This is the foundation for T111's deep-linking from cross-chat search results to specific turns. Migration adds the column; the projector for `memory_written` populates it from the event id when projecting.
|
||||||
|
|
||||||
|
**Production code change:** in the `memory_written` projector handler (in `chat/state/memory.py` or wherever it lives), populate the new `event_id` column with the projecting event's `id`. The `Event` object has `id` available in the projector context.
|
||||||
|
|
||||||
|
**Tests:**
|
||||||
|
|
||||||
|
1. `test_schema_version_after_migration_is_14` (rename + bump from 13).
|
||||||
|
2. `test_memory_written_populates_event_id` — append memory_written; project; query memories table; assert `event_id` is the projecting event's id.
|
||||||
|
3. (Backward compat) older memories from existing seed data have NULL `event_id` — the column is nullable.
|
||||||
|
|
||||||
|
**Commit:** `feat: 0014 schema — embeddings FK CASCADE (deferred or applied) + memories.event_id column (T109)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 3 — Drawer Phase 4.5 bundle (single)
|
||||||
|
|
||||||
|
### Task 110: drawer polish + bulk significance re-rate
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/web/drawer.py`
|
||||||
|
- Modify: `chat/templates/_drawer.html`
|
||||||
|
- Create: `chat/templates/_delete_impact_modal.html` (extracted partial)
|
||||||
|
- Modify: `chat/state/manual_edit.py` (potentially — if bulk re-rate emits a new manual_edit kind)
|
||||||
|
- Modify: `tests/test_drawer_phase4.py` (extend with 4-5 new tests)
|
||||||
|
|
||||||
|
**Spec (4 sub-fixes, 4 commits):**
|
||||||
|
|
||||||
|
1. **`event_id <= 0` guard in `delete_turn`** (T98 nit): currently silently rewinds everything if `event_id` is 0. Add `if event_id <= 0: raise HTTPException(400, "...")`.
|
||||||
|
|
||||||
|
2. **`html.escape()` on delete-impact modal** (T98 nit): the rendered HTML in `compute_delete_impact` output is built via raw f-strings from model-controlled strings. Wrap user-controllable fields with `html.escape()`. Defense-in-depth — currently safe, but if event payload fields ever appear in descriptions, autoescape would prevent XSS.
|
||||||
|
|
||||||
|
3. **Extract delete-impact modal HTML to a Jinja partial**: create `chat/templates/_delete_impact_modal.html`; render via `templates.TemplateResponse(...)` instead of f-string concatenation. Inherits Jinja2 autoescape automatically. Tests use the existing TestClient pattern.
|
||||||
|
|
||||||
|
4. **Bulk significance re-rate** (T98.2 deferral): drawer panel showing memory significance distribution per chat. New POST route `/chats/{chat_id}/drawer/memory/significance/bulk` accepting `{level_from, level_to}` form fields. Updates ALL memories in the chat at `level_from` to `level_to` via a sequence of `manual_edit` events (one per memory — preserves the audit trail).
|
||||||
|
|
||||||
|
**Tests:**
|
||||||
|
|
||||||
|
1. `test_delete_turn_with_event_id_zero_returns_400`.
|
||||||
|
2. `test_delete_impact_modal_uses_jinja_partial` (assert response renders the partial template; verify with `assert b"<div class=\"delete-impact-modal\">" in response.content` or similar).
|
||||||
|
3. `test_delete_impact_modal_escapes_user_controllable_strings` — seed an event with a payload containing `<script>` in a description-bound field; render preview; assert it appears HTML-escaped.
|
||||||
|
4. `test_bulk_significance_re_rate_emits_manual_edit_per_memory` — seed 5 memories at significance 0; bulk re-rate to 2; assert 5 `manual_edit` events landed.
|
||||||
|
|
||||||
|
**Commits (4):**
|
||||||
|
- `fix: drawer delete_turn guards event_id <= 0 (T110.1)`
|
||||||
|
- `fix: drawer delete-impact modal HTML escapes user-controllable fields (T110.2)`
|
||||||
|
- `refactor: drawer delete-impact modal extracted to Jinja partial (T110.3)`
|
||||||
|
- `feat: drawer bulk significance re-rate per chat (T110.4)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 4 — Search UX enhancements (single)
|
||||||
|
|
||||||
|
### Task 111: FTS highlighting + deep-link to turn
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/cross_chat_search.py`
|
||||||
|
- Modify: `chat/web/search.py`
|
||||||
|
- Modify: `chat/templates/search.html`
|
||||||
|
- Modify: `tests/test_search_ux.py`
|
||||||
|
|
||||||
|
**Spec (2 sub-fixes, 2 commits):**
|
||||||
|
|
||||||
|
1. **FTS highlighting via `snippet()`** (T100 nit): replace the `pov_summary` column in `search_all_memories`'s SELECT with `snippet(memories_fts, 0, '<mark>', '</mark>', '…', 32)` to return a highlighted snippet around the match. The template renders this raw via `|safe` (the snippet is built by SQLite from indexed content; the `<mark>` tags are the only HTML, and SQLite escapes any HTML special chars in the source content).
|
||||||
|
|
||||||
|
2. **Deep-link to turn via memories.event_id** (T100 nit + T109 dependency): now that `memories.event_id` exists (from T109), each search result row knows the originating event id. The chat page uses turn-id stamping (Phase 3.5 T86 added `id="turn-{event_id}"`). Build result links as `/chats/{chat_id}#turn-{event_id}`. The chat page DOM scrolls to the anchor on load (browser default).
|
||||||
|
|
||||||
|
**Tests:**
|
||||||
|
|
||||||
|
1. `test_search_results_include_fts_snippet_with_highlight` — seed memory with text containing "rabbit"; search for "rabbit"; assert response body contains `<mark>rabbit</mark>` (or whatever marker the snippet uses).
|
||||||
|
2. `test_search_result_link_includes_turn_anchor` — seed memory with known event_id; search; assert link href contains `#turn-{event_id}`.
|
||||||
|
|
||||||
|
**Commits (2):**
|
||||||
|
- `feat: cross-chat search FTS snippet highlighting (T111.1)`
|
||||||
|
- `feat: cross-chat search deep-links to turn via memories.event_id (T111.2)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 5 — Real embedding model (single)
|
||||||
|
|
||||||
|
### Task 112: Real embedding model swap
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/llm/client.py` (Protocol — add `embed(text, model) -> list[float]` method)
|
||||||
|
- Modify: `chat/llm/featherless.py` (FeatherlessClient — implement `embed` against Featherless `/v1/embeddings` endpoint OR equivalent)
|
||||||
|
- Modify: `chat/llm/mock.py` (MockLLMClient — accept canned embedding vectors)
|
||||||
|
- Modify: `chat/services/embeddings.py` (route non-default model through `client.embed()`)
|
||||||
|
- Modify: `chat/config.py` (add `embedding_model: str` setting; default to current pseudo)
|
||||||
|
- Modify: `scripts/backfill_embeddings.py` (re-embed-all option for model swaps)
|
||||||
|
- Modify: `tests/test_embeddings.py` + `tests/test_llm_mock.py` + `tests/test_featherless.py` (if exists)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Phase 4 ships a deterministic SHA-256 pseudo-embedding (deterministic but semantically meaningless). T112 wires the path for a real embedding model.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
|
||||||
|
1. **Extend `LLMClient` Protocol** with `async def embed(self, text: str, *, model: str) -> list[float]`.
|
||||||
|
|
||||||
|
2. **Implement on FeatherlessClient**: call the Featherless OpenAI-compatible `/v1/embeddings` endpoint:
|
||||||
|
```python
|
||||||
|
response = await self._http.post(
|
||||||
|
"/v1/embeddings",
|
||||||
|
json={"model": model, "input": text},
|
||||||
|
headers={"Authorization": f"Bearer {self._api_key}"},
|
||||||
|
)
|
||||||
|
data = response.json()
|
||||||
|
return data["data"][0]["embedding"]
|
||||||
|
```
|
||||||
|
Handle rate limits (existing 2-conn semaphore covers this).
|
||||||
|
|
||||||
|
3. **Implement on MockLLMClient**: `embed` pops a canned vector from a new `canned_embeddings` queue. Tests configure this queue.
|
||||||
|
|
||||||
|
4. **Update `generate_embedding`**: when `model != DEFAULT_EMBEDDING_MODEL`, call `client.embed(text, model=model)` instead of falling through to fallback. Wrap in try/except — failures fall back to zero vector (existing fallback path).
|
||||||
|
|
||||||
|
5. **Settings**: add `embedding_model: str = "pseudo-sha256-384"` to `Settings`. App reads this at startup; the embedding worker (`chat/services/embedding_worker.py`) passes it through.
|
||||||
|
|
||||||
|
6. **Backfill script**: add `--re-embed-all` flag that walks ALL memories (regardless of existing `embeddings_meta` rows) and re-embeds with the configured model. Useful for swapping models.
|
||||||
|
|
||||||
|
**Tests:**
|
||||||
|
|
||||||
|
1. `test_embed_routes_to_client_when_non_default_model` — mock client with canned vector; call `generate_embedding(model="bge-small-en-v1.5")`; assert vector matches the canned response.
|
||||||
|
2. `test_embed_falls_back_on_client_failure` — mock client to raise; assert returns zero vector with model="fallback".
|
||||||
|
3. `test_mock_llm_client_embed_pops_canned`.
|
||||||
|
4. `test_featherless_embed_calls_correct_endpoint` (if there's an existing featherless test pattern; otherwise mock the HTTP layer).
|
||||||
|
|
||||||
|
**Commits:**
|
||||||
|
- `feat: LLMClient Protocol gains embed() method (T112.1)`
|
||||||
|
- `feat: FeatherlessClient.embed() against /v1/embeddings (T112.2)`
|
||||||
|
- `feat: generate_embedding routes non-default models through client.embed (T112.3)`
|
||||||
|
- `feat: backfill_embeddings --re-embed-all flag for model swaps (T112.4)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 6 — Branching read-side filter (single, BIG)
|
||||||
|
|
||||||
|
### Task 113: Branching read-side filter
|
||||||
|
|
||||||
|
**Files (cross-cutting):**
|
||||||
|
- Modify: `chat/services/turn_common.py::read_recent_dialogue` — filter events to active branch's range
|
||||||
|
- Modify: `chat/services/scene_summarize.py::_read_recent_dialogue` (similar)
|
||||||
|
- Modify: `chat/state/memory.py::search_memories` — memories should be filtered to active branch (memories.event_id from T109 enables this)
|
||||||
|
- Modify: `chat/state/branches.py` — add helper `active_branch_event_ids(conn) -> tuple[int, int]` returning (origin, head)
|
||||||
|
- Add tests across multiple files
|
||||||
|
- Modify: `tests/test_branching.py` — add cross-feature tests
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Phase 4 T89 + T94 shipped branching as metadata-only (the table tracks branches; the drawer UI can switch). But event readers DON'T consult `is_active` — they read the entire event_log. So switching branches has no functional effect.
|
||||||
|
|
||||||
|
T113 wires the filter:
|
||||||
|
|
||||||
|
1. **Helper** `active_branch_event_ids(conn) -> tuple[int, int]`: returns `(origin_event_id, head_event_id)` for the currently active branch. For "main" with origin=0 + head=N, returns `(0, N)` meaning "all events visible".
|
||||||
|
|
||||||
|
2. **Apply filter** in every event reader that returns historical state:
|
||||||
|
- `read_recent_dialogue`: WHERE clause adds `id BETWEEN ? AND ?` (the active branch's range).
|
||||||
|
- `search_memories`: WHERE clause adds `m.event_id BETWEEN ? AND ?` (uses T109's column).
|
||||||
|
- `scene_summarize._read_recent_dialogue`: same as turn_common.
|
||||||
|
- Other readers TBD — grep for `event_log` SELECT patterns and audit each one.
|
||||||
|
|
||||||
|
3. **Branches that diverge**: when branch B is created from event 10 and then accumulates events 11-15 (which only exist on B's timeline), but main also accumulates 11-12, the events overlap by id range. This is OK because event reads filter by `id <= active_branch.head_event_id`. The simpler model: branches share event_log ids globally, but each branch's "head" defines which ids are visible.
|
||||||
|
|
||||||
|
4. **Events written under branch B** carry an implicit branch tag — but the event_log table has no `branch_id` column today. T113 punts on cross-branch event writes (they all land in the global log) and relies on the `head_event_id` filter to scope reads. This is a Phase 4.5+ first cut; full branch-isolated event_log is Phase 5+.
|
||||||
|
|
||||||
|
**Edge cases:**
|
||||||
|
|
||||||
|
- Active branch has `head_event_id = 0` (just created): readers return empty.
|
||||||
|
- No active branch: readers fall through to "all events visible" (defensive).
|
||||||
|
- Switching branches mid-flight: each `read_recent_dialogue` call re-queries `active_branch`, so it's always current. No caching.
|
||||||
|
|
||||||
|
**Tests:** 5+ minimum.
|
||||||
|
|
||||||
|
1. `test_read_recent_dialogue_respects_active_branch_head` — seed 10 events; active branch head = 5; assert only first 5 returned.
|
||||||
|
2. `test_search_memories_respects_active_branch_head` — same.
|
||||||
|
3. `test_branch_switch_changes_visible_events` — switch branches; immediately read; assert different result sets.
|
||||||
|
4. `test_main_branch_with_head_zero_returns_empty` — defensive.
|
||||||
|
5. `test_no_active_branch_falls_through_to_all_events` — defensive.
|
||||||
|
|
||||||
|
**Commit:** `feat: branching read-side filter — event readers consult active branch range (T113)`.
|
||||||
|
|
||||||
|
**This is the largest task in Phase 4.5.** Estimate 200-400 lines across multiple files. Implementer should split commits if it helps clarity (one per affected reader).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 7 — Lifecycle rollback in regenerate (single)
|
||||||
|
|
||||||
|
### Task 114: Lifecycle rollback
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `chat/services/regenerate.py`
|
||||||
|
- Modify: `chat/db/migrations/0014_phase45_schema.sql` (T109's migration) — add column? OR
|
||||||
|
- Add new migration — see decision below
|
||||||
|
- Modify: tests in `tests/test_regenerate.py`
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Phase 3.5 T83.4 shipped a warning log when regenerate detects un-rolled-back lifecycle transitions. T114 implements actual rollback.
|
||||||
|
|
||||||
|
**Schema decision:**
|
||||||
|
|
||||||
|
Option A: extend lifecycle event payloads with `triggered_by_assistant_turn_id` (no schema change needed — just a payload convention). Production code (T61 turn flow) populates it when emitting `event_started`/`event_completed`/`event_cancelled`. Existing rows have NULL — rollback skips them with a debug log.
|
||||||
|
|
||||||
|
Option B: add a column to `event_log` for stronger invariants. Significant migration cost.
|
||||||
|
|
||||||
|
**Recommended:** Option A. Safer, no migration, backward compatible (older events skip rollback). Document in commit body.
|
||||||
|
|
||||||
|
**Rollback semantics:**
|
||||||
|
|
||||||
|
When regenerate detects lifecycle events triggered by the superseded turn:
|
||||||
|
- `event_started` → emit `event_cancelled` (or a NEW `event_started_undone` event kind that reverts status to "planned") with the same event_id.
|
||||||
|
- `event_completed` → emit `event_uncompleted` (NEW event kind that reverts status from "completed" to "active").
|
||||||
|
- `event_cancelled` → emit `event_uncancelled` (reverts to prior status — which we'd need to track; or simpler: emit `event_started` again to restore "active").
|
||||||
|
|
||||||
|
**Simpler approach (recommended):** add ONE new event kind `event_status_reverted` with payload `{event_id, prior_status}`. The projector sets `events.status = prior_status` for the event_id. Rollback emits this event for each affected lifecycle transition, looking up the prior status from the row's history (via event_log scan) or accepting it as a payload field.
|
||||||
|
|
||||||
|
**Production code change:** in `chat/web/turns.py::post_turn` (and `chat/services/regenerate.py`), when emitting `event_started`/`event_completed`/`event_cancelled`, populate `triggered_by_assistant_turn_id: <id>` in the payload. Forward-only — older code doesn't need updating.
|
||||||
|
|
||||||
|
**Tests:** 3 minimum.
|
||||||
|
|
||||||
|
1. `test_regenerate_rolls_back_event_started_from_superseded_turn` — seed an event; play a turn that starts it; regenerate; assert `event_status_reverted` event landed with `prior_status="planned"` and the events row is back to "planned".
|
||||||
|
2. `test_regenerate_rolls_back_event_completed_to_active` — same but completed → active rollback.
|
||||||
|
3. `test_regenerate_skips_events_without_back_reference` — older events without `triggered_by_assistant_turn_id` are not rolled back (debug log). Pin the backward-compat behavior.
|
||||||
|
|
||||||
|
**Commits:**
|
||||||
|
- `feat: lifecycle events carry triggered_by_assistant_turn_id back-reference (T114.1)`
|
||||||
|
- `feat: event_status_reverted event kind + projector handler (T114.2)`
|
||||||
|
- `feat: regenerate rolls back lifecycle transitions on supersede (T114.3)`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 8 — sqlite-vec swap (single, ENVIRONMENTAL)
|
||||||
|
|
||||||
|
### Task 115: sqlite-vec swap (optional)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `chat/db/migrations/0015_vec0_virtual_tables.sql`
|
||||||
|
- Modify: `chat/db/connection.py` (load extension on every connection)
|
||||||
|
- Modify: `chat/services/vector_search.py` (rewrite to use vec0 MATCH instead of pure-Python cosine)
|
||||||
|
- Modify: `chat/state/embeddings.py` (writer needs to populate vec0 table)
|
||||||
|
- Modify: `pyproject.toml` (add `sqlite-vec` dependency)
|
||||||
|
|
||||||
|
**Pre-flight:**
|
||||||
|
|
||||||
|
This task REQUIRES one of:
|
||||||
|
- Python rebuilt with `--enable-loadable-sqlite-extensions` (pyenv reinstall).
|
||||||
|
- `apsw` migration of `chat/db/connection.py`.
|
||||||
|
|
||||||
|
If neither is feasible at the time of execution: SKIP THIS TASK and document the deferral in T118 docs sweep. The other 13 Phase 4.5 tasks ship without it.
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
1. **Migration** `0015_vec0_virtual_tables.sql`:
|
||||||
|
```sql
|
||||||
|
CREATE VIRTUAL TABLE embeddings_vec USING vec0(
|
||||||
|
memory_id INTEGER PRIMARY KEY,
|
||||||
|
embedding FLOAT[384]
|
||||||
|
);
|
||||||
|
-- Backfill from existing JSON embeddings table.
|
||||||
|
INSERT INTO embeddings_vec (memory_id, embedding)
|
||||||
|
SELECT memory_id, vec_f32(vector_json) FROM embeddings;
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **`chat/db/connection.py`** loads `sqlite_vec` extension on every connection:
|
||||||
|
```python
|
||||||
|
import sqlite_vec
|
||||||
|
def open_db(...):
|
||||||
|
conn = sqlite3.connect(...)
|
||||||
|
conn.enable_load_extension(True)
|
||||||
|
sqlite_vec.load(conn)
|
||||||
|
conn.enable_load_extension(False)
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Rewrite `vector_search.py`** to use `embeddings_vec MATCH ?` syntax with `k=?` clause:
|
||||||
|
```sql
|
||||||
|
SELECT m.id, m.pov_summary, m.significance, e.distance
|
||||||
|
FROM embeddings_vec e
|
||||||
|
JOIN memories m ON m.id = e.memory_id
|
||||||
|
WHERE e.embedding MATCH ? AND k = ?
|
||||||
|
AND m.owner_id = ?
|
||||||
|
AND m.witness_<role> = 1
|
||||||
|
ORDER BY e.distance ASC
|
||||||
|
LIMIT ?
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **HNSW note**: vec0 supports both flat (default) and HNSW indexes. T115 ships flat (sufficient for < few thousand memories). Document HNSW upgrade path in CLAUDE.md if memory counts ever grow past pure-Python feasibility.
|
||||||
|
|
||||||
|
5. **Old `embeddings` JSON table**: keep alongside `embeddings_vec` (data redundancy is fine; the JSON table is the source of truth and `embeddings_vec` is the index). Backfill on migration. Keep the `embedding_indexed` projector populating both.
|
||||||
|
|
||||||
|
**Tests:** rewrite `tests/test_vector_search.py` to expect new behavior. Same observable contract — only implementation changes. All 5 existing tests should pass post-swap.
|
||||||
|
|
||||||
|
**Commit:** `feat: sqlite-vec swap (vec0 virtual tables + MATCH-based search) (T115)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 9 — Polish (parallel, 3 tasks)
|
||||||
|
|
||||||
|
### Task 116: Structured test-fixture builder
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `tests/fixtures.py` (or extend `tests/conftest.py`)
|
||||||
|
- Modify: existing test files that use brittle canned-queue arrays (selectively)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Phase 3 carry-over. Tests across `test_turn_flow.py`, `test_meanwhile_turn_flow.py`, `test_phase3_integration.py`, `test_phase4_integration.py` use positional canned-response arrays for `MockLLMClient`. Adding a new classifier call to a code path requires updating canned arrays in many tests.
|
||||||
|
|
||||||
|
**Solution:** structured fixture builder that lets tests declare their classifier expectations by name, not position:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# tests/fixtures.py
|
||||||
|
class CannedQueue:
|
||||||
|
def __init__(self):
|
||||||
|
self._queue = []
|
||||||
|
def parse_turn(self, **fields): ...
|
||||||
|
def state_update(self, **fields): ...
|
||||||
|
def detect_scene_close(self, should_close: bool): ...
|
||||||
|
def detect_event_transitions(self, transitions: list[dict]): ...
|
||||||
|
def summarize_scene(self, summary: str, **fields): ...
|
||||||
|
def detect_threads(self, candidates: list[dict]): ...
|
||||||
|
# ... one method per classifier service
|
||||||
|
def build(self) -> list[str]:
|
||||||
|
return [json.dumps(item) for item in self._queue]
|
||||||
|
```
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def test_post_turn_with_event_transition(...):
|
||||||
|
canned = (
|
||||||
|
CannedQueue()
|
||||||
|
.parse_turn(intent="narrative")
|
||||||
|
.narrative("BotA speaks.") # narrative is a stream, but for simplicity treat it like a canned response
|
||||||
|
.state_update(affinity_delta=0, trust_delta=0)
|
||||||
|
.state_update(affinity_delta=0, trust_delta=0)
|
||||||
|
.detect_event_transitions([{"event_id": "evt_1", "new_status": "completed"}])
|
||||||
|
.detect_scene_close(should_close=False)
|
||||||
|
.build()
|
||||||
|
)
|
||||||
|
mock = MockLLMClient(canned=canned)
|
||||||
|
# ...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Migration scope:** don't migrate ALL existing tests at once — that's a separate massive refactor. Instead, ship the fixture builder + migrate 2-3 representative tests as proof of concept. Document the migration path in the fixture's docstring.
|
||||||
|
|
||||||
|
**Tests:** the fixture builder itself doesn't need extensive testing — it's just a builder. Add 1-2 sanity tests that the JSON output matches expected shapes.
|
||||||
|
|
||||||
|
**Commit:** `test: structured CannedQueue fixture builder for classifier mocks (T116)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 117: Phase 4.5 cross-feature integration tests
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `tests/test_phase45_integration.py`
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
End-to-end multi-feature flows specific to Phase 4.5 changes. 5 tests minimum.
|
||||||
|
|
||||||
|
1. **Real embedding swap + retrieval** — configure `embedding_model="bge-small-en-v1.5"` (mocked); write a memory; backfill or wait for worker; assert vector search returns the memory via `client.embed`-derived vector (not pseudo).
|
||||||
|
|
||||||
|
2. **Branching read-side filter end-to-end** — create a branch from turn 5; switch; play 3 turns on the branch; switch back to main; assert main's recent dialogue is missing the branch turns (read filter respects active branch's head).
|
||||||
|
|
||||||
|
3. **Lifecycle rollback** — start an event via a turn; regenerate that turn; assert lifecycle reverted (event back to "planned").
|
||||||
|
|
||||||
|
4. **Search deep-link** — write memories; search; click a result; verify the chat page renders with the right turn anchored (assert via TestClient response — either the browser anchor OR a server-side scroll-to-anchor mechanism).
|
||||||
|
|
||||||
|
5. **Bulk significance re-rate end-to-end** — seed 5 memories at significance 0; bulk re-rate via drawer; verify significance histogram updates.
|
||||||
|
|
||||||
|
**Commit:** `test: phase 4.5 cross-feature integration coverage (T117)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 118: Phase 4.5 documentation update
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `CLAUDE.md`
|
||||||
|
- Modify: `docs/plans/2026-04-26-v1-requirements-design.md` (annotate §13 Phase 4 entries — though they're already shipped per Phase 4 T102)
|
||||||
|
|
||||||
|
**Spec:**
|
||||||
|
|
||||||
|
Mirror the Phase 3.5 / 2.5 status sections. Document:
|
||||||
|
|
||||||
|
- All shipped items per task (T103–T117).
|
||||||
|
- Empty out the Phase 4.5 / 5 backlog (replace with single "All items shipped" line).
|
||||||
|
- Add new "Phase 5 backlog" section if any Phase 4.5 reviews surfaced new follow-ups.
|
||||||
|
|
||||||
|
**Phase 5 backlog candidates** (default, if no new follow-ups discovered):
|
||||||
|
|
||||||
|
- Vector index optimization (HNSW) when memory counts grow past flat-index feasibility.
|
||||||
|
- Branch-isolated event_log (each branch has its own physical event_log range vs the current shared id space + head filter).
|
||||||
|
- Embedding model swap migration tooling — when changing models, need to re-embed everything; T112 added `--re-embed-all` but a more orchestrated swap (drain old worker, re-seed all memories, swap config) is Phase 5+.
|
||||||
|
- Real-time collaborative branching (multi-user) — out of scope for v1.
|
||||||
|
- Avatars / portraits (multimodality) — deferred indefinitely per design §14.
|
||||||
|
|
||||||
|
**Commit:** `docs: phase 4.5 status, prune backlog, capture phase 5 candidates (T118)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wrap-up
|
||||||
|
|
||||||
|
After Wave 9 lands:
|
||||||
|
|
||||||
|
1. **Run full suite** on `phase-4.5`: should be ~430+ tests passing (413 from Phase 4 + ~20 new across Phase 4.5).
|
||||||
|
2. **Manual smoke** (recommended before opening the PR):
|
||||||
|
- Configure `embedding_model="bge-small-en-v1.5"` (or whatever real model is chosen); restart server; play a turn; verify `embedding_indexed` events use the real model and search returns semantically-relevant memories.
|
||||||
|
- Create a branch, switch, play turns, switch back — verify main's history is unaffected.
|
||||||
|
- Plan an event, complete it via a turn, regenerate that turn — verify event reverts to "planned".
|
||||||
|
- Use cross-chat search; click a result; verify it lands on the right turn in the chat page.
|
||||||
|
- Bulk re-rate a chat's significance distribution.
|
||||||
|
3. **Push `phase-4.5`** to gitea.
|
||||||
|
4. **Open PR** `phase-4.5 → main`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes for the controller running this plan
|
||||||
|
|
||||||
|
- **T115 (sqlite-vec swap)** is environmental. If pre-flight fails (no rebuilt Python, no apsw), defer to Phase 5 and ship Phase 4.5 with 13 tasks. T118 docs sweep should note the deferral.
|
||||||
|
- **T112 (real embedding swap)** assumes Featherless or similar exposes an `/v1/embeddings` endpoint. If not available, document the gap and ship the Protocol + Mock impl only (Featherless impl deferred). The pseudo path remains the default in that case — same as Phase 4.
|
||||||
|
- **T113 (branching read-side filter)** is the riskiest task. Cross-cutting. Land it on a quiet branch, test thoroughly. If integration tests break in unexpected ways, bisect the affected reader and add coverage.
|
||||||
|
- **After each parallel wave**, run a code-review subagent. Combined spec+quality acceptable for trivial tasks (T103–T108); separate spec + quality reviewers for big tasks (T112, T113, T114, T115).
|
||||||
|
- **Token-spend rough estimate**: Phase 4.5 should be ~50% the size of Phase 4 (similar number of tasks, mostly smaller). Big tasks (T112, T113, T114) bring the per-task spend up but parallelism in Wave 1 + Wave 9 brings the wall-clock down.
|
||||||
|
- **DO NOT break existing v1/v2/v3/v3.5/v4 surface contracts.** Every test file that was green at the start of Phase 4.5 must stay green at the end. The cross-feature integration tests (`tests/test_phase4_integration.py`, `tests/test_phase3_integration.py`) are particularly load-bearing.
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"planPath": "docs/plans/2026-04-27-v4.5-phase4.5-cleanup.md",
|
||||||
|
"tasks": [
|
||||||
|
{"id": 103, "subject": "T103: branches polish (global-leak doc + branch-switch warning)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 104, "subject": "T104: state/memory.py polish (DRY MAX(id) + fts_rank doc)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 105, "subject": "T105: snapshots.py polish (datetime hoist + kind validation + mtime doc)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 106, "subject": "T106: search.py polish (k constant + N+1 batched lookups)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 107, "subject": "T107: embeddings.py timeout_s fallback-path logging", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 108, "subject": "T108: scene-close-on-cancel UX revisit (regression test pin + rationale doc)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||||
|
{"id": 109, "subject": "T109: 0014 schema migration (FK CASCADE + memories.event_id column)", "status": "pending", "wave": 2, "parallelGroup": null},
|
||||||
|
{"id": 110, "subject": "T110: drawer Phase 4.5 bundle (event_id guard + html.escape + modal partial + bulk sig re-rate)", "status": "pending", "wave": 3, "parallelGroup": null, "blockedBy": [109]},
|
||||||
|
{"id": 111, "subject": "T111: search UX (FTS snippet highlighting + deep-link via memories.event_id)", "status": "pending", "wave": 4, "parallelGroup": null, "blockedBy": [109]},
|
||||||
|
{"id": 112, "subject": "T112: real embedding model swap (LLMClient.embed protocol + Featherless impl + routing)", "status": "pending", "wave": 5, "parallelGroup": null},
|
||||||
|
{"id": 113, "subject": "T113: branching read-side filter (event readers consult is_active branch range)", "status": "pending", "wave": 6, "parallelGroup": null, "blockedBy": [109]},
|
||||||
|
{"id": 114, "subject": "T114: regenerate lifecycle rollback (back-reference + event_status_reverted)", "status": "pending", "wave": 7, "parallelGroup": null},
|
||||||
|
{"id": 115, "subject": "T115: sqlite-vec swap (vec0 virtual tables + MATCH search) [ENVIRONMENTAL — may defer]", "status": "pending", "wave": 8, "parallelGroup": null},
|
||||||
|
{"id": 116, "subject": "T116: structured CannedQueue test fixture builder", "status": "pending", "wave": 9, "parallelGroup": "wave-9"},
|
||||||
|
{"id": 117, "subject": "T117: phase 4.5 cross-feature integration tests", "status": "pending", "wave": 9, "parallelGroup": "wave-9", "blockedBy": [110, 111, 112, 113, 114]},
|
||||||
|
{"id": 118, "subject": "T118: phase 4.5 docs sweep — prune backlog, capture phase 5 candidates", "status": "pending", "wave": 9, "parallelGroup": "wave-9", "blockedBy": [110, 111, 112, 113, 114]}
|
||||||
|
],
|
||||||
|
"lastUpdated": "2026-04-27T00:00:00Z",
|
||||||
|
"notes": "16 tasks across 9 waves consolidating all 24 items in CLAUDE.md Phase 4.5/5 backlog. Wave 1 (6-way parallel) and Wave 9 (3-way parallel) maximize parallelism. Waves 2-8 are single-task by hot-file constraint. T115 (sqlite-vec swap) requires Python rebuild OR apsw migration — environmental; may defer to Phase 5. Schema baseline 13 -> 14 (T109's 0014) -> optionally 15 (T115's 0015). Big tasks: T112 (real embedding swap), T113 (branching read-side filter — riskiest), T114 (lifecycle rollback). Uses task ids T103-T118."
|
||||||
|
}
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
"""Backfill embeddings for memories that lack them (T97, Phase 4).
|
||||||
|
|
||||||
|
Walks all memories where no row exists in the ``embeddings`` table. For
|
||||||
|
each, calls :func:`chat.services.embeddings.generate_embedding` and emits
|
||||||
|
an ``embedding_indexed`` event so the projector lands the vector.
|
||||||
|
|
||||||
|
Phase 4 ships the deterministic local pseudo-embedding so this script
|
||||||
|
runs synchronously without a network round-trip — the LLMClient argument
|
||||||
|
is not needed on the pseudo path. Phase 4.5+ will need a real client.
|
||||||
|
|
||||||
|
Run from the repo root:
|
||||||
|
.venv/bin/python scripts/backfill_embeddings.py [--limit N] [--dry-run]
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from chat.config import load_settings
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
from chat.services.embeddings import (
|
||||||
|
FALLBACK_EMBEDDING_MODEL,
|
||||||
|
generate_embedding,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Trigger projector handler registration so ``append_and_apply`` lands
|
||||||
|
# the embedding rows correctly.
|
||||||
|
import chat.state.embeddings # noqa: F401
|
||||||
|
import chat.state.entities # noqa: F401
|
||||||
|
import chat.state.memory # noqa: F401
|
||||||
|
import chat.state.world # noqa: F401
|
||||||
|
|
||||||
|
|
||||||
|
async def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument(
|
||||||
|
"--limit",
|
||||||
|
type=int,
|
||||||
|
default=None,
|
||||||
|
help="Cap the number of memories backfilled in this run.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--dry-run",
|
||||||
|
action="store_true",
|
||||||
|
help="Print the count of memories needing embeddings, then exit.",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
settings = load_settings()
|
||||||
|
settings.db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
apply_migrations(settings.db_path)
|
||||||
|
|
||||||
|
with open_db(settings.db_path) as conn:
|
||||||
|
sql = (
|
||||||
|
"SELECT m.id, m.pov_summary FROM memories m "
|
||||||
|
"LEFT JOIN embeddings e ON e.memory_id = m.id "
|
||||||
|
"WHERE e.memory_id IS NULL "
|
||||||
|
"ORDER BY m.id"
|
||||||
|
)
|
||||||
|
if args.limit is not None:
|
||||||
|
sql += f" LIMIT {int(args.limit)}"
|
||||||
|
rows = conn.execute(sql).fetchall()
|
||||||
|
print(f"Found {len(rows)} memories needing embeddings.")
|
||||||
|
if args.dry_run:
|
||||||
|
return
|
||||||
|
|
||||||
|
indexed = 0
|
||||||
|
skipped = 0
|
||||||
|
for memory_id, text in rows:
|
||||||
|
result = await generate_embedding(
|
||||||
|
client=None, # pseudo path: no client needed
|
||||||
|
text=text or "",
|
||||||
|
)
|
||||||
|
if result.model == FALLBACK_EMBEDDING_MODEL:
|
||||||
|
print(f" Skipping memory_id={memory_id} (empty text)")
|
||||||
|
skipped += 1
|
||||||
|
continue
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": memory_id,
|
||||||
|
"model": result.model,
|
||||||
|
"dim": result.dim,
|
||||||
|
"vector": result.vector,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
indexed += 1
|
||||||
|
print(f" Indexed memory_id={memory_id}")
|
||||||
|
print(f"Done. Indexed {indexed}, skipped {skipped}.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
asyncio.run(main())
|
||||||
@@ -97,3 +97,38 @@ async def test_classifier_failure_falls_back_to_host():
|
|||||||
assert result.addressee_id == "bot_a"
|
assert result.addressee_id == "bot_a"
|
||||||
assert result.reason == "fallback"
|
assert result.reason == "fallback"
|
||||||
assert result.confidence == "low"
|
assert result.confidence == "low"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_invalid_confidence_value_falls_back_to_default():
|
||||||
|
"""Pydantic rejects ``confidence`` values outside the literal set
|
||||||
|
(``high`` / ``medium`` / ``low``). After the retry budget is
|
||||||
|
exhausted, classify returns the configured fallback default —
|
||||||
|
here that's ``confidence="low"`` with ``reason="fallback"``.
|
||||||
|
"""
|
||||||
|
canned = [
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"addressee_id": "bot_a",
|
||||||
|
"confidence": "VERY_HIGH",
|
||||||
|
"reason": "out-of-range value",
|
||||||
|
}
|
||||||
|
),
|
||||||
|
"still_bad",
|
||||||
|
"still_bad",
|
||||||
|
]
|
||||||
|
client = MockLLMClient(canned=canned)
|
||||||
|
|
||||||
|
result = await detect_addressee(
|
||||||
|
client,
|
||||||
|
classifier_model="test-model",
|
||||||
|
user_prose="anything",
|
||||||
|
host_id="bot_a",
|
||||||
|
host_name="BotA",
|
||||||
|
guest_id="bot_b",
|
||||||
|
guest_name="BotB",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.addressee_id == "bot_a"
|
||||||
|
assert result.confidence == "low"
|
||||||
|
assert result.reason == "fallback"
|
||||||
|
|||||||
@@ -0,0 +1,176 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
import chat.state.branches # registers handlers
|
||||||
|
from chat.state.branches import active_branch, get_branch, list_branches
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_branch_bootstrapped_by_migration(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
active = active_branch(conn)
|
||||||
|
assert active is not None
|
||||||
|
assert active["name"] == "main"
|
||||||
|
assert active["is_active"] is True
|
||||||
|
assert active["origin_event_id"] == 0
|
||||||
|
assert active["head_event_id"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_created_inserts_row(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_created",
|
||||||
|
payload={
|
||||||
|
"name": "experiment",
|
||||||
|
"origin_event_id": 42,
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
b = get_branch(conn, "experiment")
|
||||||
|
assert b is not None
|
||||||
|
assert b["name"] == "experiment"
|
||||||
|
assert b["origin_event_id"] == 42
|
||||||
|
# head defaults to origin when not specified
|
||||||
|
assert b["head_event_id"] == 42
|
||||||
|
assert b["chat_id"] == "chat_a"
|
||||||
|
assert b["is_active"] is False
|
||||||
|
|
||||||
|
# main remains active
|
||||||
|
active = active_branch(conn)
|
||||||
|
assert active is not None
|
||||||
|
assert active["name"] == "main"
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_switched_atomic(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_created",
|
||||||
|
payload={
|
||||||
|
"name": "experiment",
|
||||||
|
"origin_event_id": 5,
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_switched",
|
||||||
|
payload={"name": "experiment"},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
active = active_branch(conn)
|
||||||
|
assert active is not None
|
||||||
|
assert active["name"] == "experiment"
|
||||||
|
|
||||||
|
main = get_branch(conn, "main")
|
||||||
|
assert main is not None
|
||||||
|
assert main["is_active"] is False
|
||||||
|
|
||||||
|
# switch back
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_switched",
|
||||||
|
payload={"name": "main"},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
active2 = active_branch(conn)
|
||||||
|
assert active2 is not None
|
||||||
|
assert active2["name"] == "main"
|
||||||
|
|
||||||
|
experiment = get_branch(conn, "experiment")
|
||||||
|
assert experiment is not None
|
||||||
|
assert experiment["is_active"] is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_head_updated_changes_head(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_created",
|
||||||
|
payload={
|
||||||
|
"name": "experiment",
|
||||||
|
"origin_event_id": 10,
|
||||||
|
"head_event_id": 10,
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_head_updated",
|
||||||
|
payload={"name": "experiment", "head_event_id": 20},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
b = get_branch(conn, "experiment")
|
||||||
|
assert b is not None
|
||||||
|
assert b["head_event_id"] == 20
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_branches_returns_all(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_created",
|
||||||
|
payload={
|
||||||
|
"name": "experiment",
|
||||||
|
"origin_event_id": 1,
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
names = [b["name"] for b in list_branches(conn)]
|
||||||
|
assert "main" in names
|
||||||
|
assert "experiment" in names
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_switched_unknown_name_warns(tmp_path, caplog):
|
||||||
|
"""Switching to a nonexistent branch logs a warning and leaves no branch active.
|
||||||
|
|
||||||
|
The previous behavior silently cleared is_active flags and applied no UPDATE
|
||||||
|
when the named branch did not exist. T103 makes that condition observable
|
||||||
|
by emitting a warning while preserving the existing (zero-active) outcome.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
with caplog.at_level(logging.WARNING, logger="chat.state.branches"):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="branch_switched",
|
||||||
|
payload={"name": "does_not_exist"},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
# A warning was emitted naming the missing branch.
|
||||||
|
warnings = [
|
||||||
|
r for r in caplog.records
|
||||||
|
if r.levelno == logging.WARNING and r.name == "chat.state.branches"
|
||||||
|
]
|
||||||
|
assert warnings, "expected a warning for unknown branch name"
|
||||||
|
assert any("does_not_exist" in r.getMessage() for r in warnings)
|
||||||
|
|
||||||
|
# Existing behavior preserved: no branch is active after the switch.
|
||||||
|
assert active_branch(conn) is None
|
||||||
|
|
||||||
|
# The unknown name was not inserted as a side effect.
|
||||||
|
assert get_branch(conn, "does_not_exist") is None
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
"""Tests for the branching service (T94, Phase 4)."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
import chat.state.branches # noqa: F401 registers handlers
|
||||||
|
from chat.services.branching import (
|
||||||
|
branch_from_event,
|
||||||
|
list_branches_with_metadata,
|
||||||
|
switch_active_branch,
|
||||||
|
)
|
||||||
|
from chat.state.branches import active_branch, get_branch
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_event(conn) -> int:
|
||||||
|
"""Append a benign event so we have a real event_log row to fork from.
|
||||||
|
|
||||||
|
``user_turn`` is a transcript-only kind with no registered projector
|
||||||
|
handler, so ``append_and_apply`` is a clean no-op on the projector
|
||||||
|
side regardless of what other handlers are imported by the suite.
|
||||||
|
"""
|
||||||
|
return append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={"chat_id": "c1", "text": "hi"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_from_event_creates_branch_via_event(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
seed_id = _seed_event(conn)
|
||||||
|
|
||||||
|
new_id = branch_from_event(
|
||||||
|
conn,
|
||||||
|
name="experiment",
|
||||||
|
origin_event_id=seed_id,
|
||||||
|
chat_id="c1",
|
||||||
|
)
|
||||||
|
assert isinstance(new_id, int) and new_id > 0
|
||||||
|
|
||||||
|
b = get_branch(conn, "experiment")
|
||||||
|
assert b is not None
|
||||||
|
assert b["id"] == new_id
|
||||||
|
assert b["origin_event_id"] == seed_id
|
||||||
|
assert b["head_event_id"] == seed_id
|
||||||
|
assert b["chat_id"] == "c1"
|
||||||
|
assert b["is_active"] is False
|
||||||
|
|
||||||
|
# branch_created event landed in event_log
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'branch_created'"
|
||||||
|
).fetchone()
|
||||||
|
assert row[0] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_from_event_duplicate_name_raises(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
seed_id = _seed_event(conn)
|
||||||
|
branch_from_event(conn, name="dup", origin_event_id=seed_id)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError, match="already exists"):
|
||||||
|
branch_from_event(conn, name="dup", origin_event_id=seed_id)
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_from_event_invalid_origin_raises(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
with pytest.raises(ValueError, match="does not exist"):
|
||||||
|
branch_from_event(conn, name="ghost", origin_event_id=99999)
|
||||||
|
|
||||||
|
|
||||||
|
def test_switch_active_branch_changes_active(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
seed_id = _seed_event(conn)
|
||||||
|
branch_from_event(conn, name="experiment", origin_event_id=seed_id)
|
||||||
|
|
||||||
|
switch_active_branch(conn, name="experiment")
|
||||||
|
active = active_branch(conn)
|
||||||
|
assert active is not None
|
||||||
|
assert active["name"] == "experiment"
|
||||||
|
|
||||||
|
# Switch back to main.
|
||||||
|
switch_active_branch(conn, name="main")
|
||||||
|
active2 = active_branch(conn)
|
||||||
|
assert active2 is not None
|
||||||
|
assert active2["name"] == "main"
|
||||||
|
|
||||||
|
|
||||||
|
def test_switch_active_branch_unknown_name_raises(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
with pytest.raises(ValueError, match="does not exist"):
|
||||||
|
switch_active_branch(conn, name="nope")
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_branches_with_metadata_includes_event_count(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Seed enough events to cover origin=10 and head=15.
|
||||||
|
for _ in range(15):
|
||||||
|
_seed_event(conn)
|
||||||
|
|
||||||
|
# Create the branch at origin=10, then bump its head to 15.
|
||||||
|
branch_from_event(conn, name="exp", origin_event_id=10)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="branch_head_updated",
|
||||||
|
payload={"name": "exp", "head_event_id": 15},
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = {b["name"]: b for b in list_branches_with_metadata(conn)}
|
||||||
|
|
||||||
|
# main: bootstrap state — origin=0, head=0 — event_count == 0.
|
||||||
|
assert rows["main"]["event_count"] == 0
|
||||||
|
# exp: origin=10, head=15 — event_count == 6 (inclusive).
|
||||||
|
assert rows["exp"]["origin_event_id"] == 10
|
||||||
|
assert rows["exp"]["head_event_id"] == 15
|
||||||
|
assert rows["exp"]["event_count"] == 6
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
"""T93 (Phase 4): cross-chat FTS5 search across all owners and chats.
|
||||||
|
|
||||||
|
Verifies that ``chat.services.cross_chat_search.search_all_memories``:
|
||||||
|
* surfaces matches across multiple owner_ids (the per-owner restriction
|
||||||
|
used by ``state.memory.search_memories`` is intentionally absent),
|
||||||
|
* applies no witness filter (admin/power-user surface),
|
||||||
|
* orders results by FTS5 BM25 rank (lower = stronger match, surfaced
|
||||||
|
first), and
|
||||||
|
* honours the ``k`` LIMIT and the empty-query fast-path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
from chat.services.cross_chat_search import search_all_memories
|
||||||
|
import chat.state.memory # noqa: F401 (registers memory_written handler)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed(db, *, memory_specs):
|
||||||
|
"""Apply migrations + project a list of memory_written events."""
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
for spec in memory_specs:
|
||||||
|
payload = {
|
||||||
|
"owner_id": spec.get("owner_id", "bot_a"),
|
||||||
|
"chat_id": spec.get("chat_id", "chat_bot_a"),
|
||||||
|
"pov_summary": spec["pov_summary"],
|
||||||
|
"witness_you": spec.get("witness_you", 1),
|
||||||
|
"witness_host": spec.get("witness_host", 1),
|
||||||
|
"witness_guest": spec.get("witness_guest", 0),
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": spec.get("significance", 1),
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
}
|
||||||
|
append_event(conn, kind="memory_written", payload=payload)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_all_memories_returns_matches_across_owners(tmp_path):
|
||||||
|
"""Cross-owner: a single query must surface memories from every owner.
|
||||||
|
|
||||||
|
The per-owner ``owner_id = ?`` predicate that ``search_memories`` uses
|
||||||
|
is intentionally absent here, so a "rabbit" memory under ``bot_a`` and
|
||||||
|
one under ``bot_b`` should both come back from a single call.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
_seed(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
{
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": "the rabbit darted into the brambles",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"owner_id": "bot_b",
|
||||||
|
"chat_id": "chat_bot_b",
|
||||||
|
"pov_summary": "a white rabbit watched from the hedge",
|
||||||
|
},
|
||||||
|
# Distractor: must not appear for "rabbit".
|
||||||
|
{
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": "the kettle whistled",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_all_memories(conn, query="rabbit")
|
||||||
|
owners = {row["owner_id"] for row in out}
|
||||||
|
assert owners == {"bot_a", "bot_b"}
|
||||||
|
assert len(out) == 2
|
||||||
|
# Returned shape contract.
|
||||||
|
for row in out:
|
||||||
|
assert set(row.keys()) >= {
|
||||||
|
"memory_id",
|
||||||
|
"owner_id",
|
||||||
|
"chat_id",
|
||||||
|
"scene_id",
|
||||||
|
"pov_summary",
|
||||||
|
"significance",
|
||||||
|
"ts",
|
||||||
|
"fts_rank",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_all_memories_orders_by_fts_rank(tmp_path):
|
||||||
|
"""Stronger BM25 match must come first (rank ASC = lower is better)."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
_seed(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
# Single occurrence -> weaker BM25 score.
|
||||||
|
{
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": "a rabbit appeared",
|
||||||
|
},
|
||||||
|
# Triple occurrence in a short row -> stronger BM25 score.
|
||||||
|
{
|
||||||
|
"owner_id": "bot_b",
|
||||||
|
"chat_id": "chat_bot_b",
|
||||||
|
"pov_summary": "rabbit rabbit rabbit",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_all_memories(conn, query="rabbit", k=5)
|
||||||
|
assert len(out) == 2
|
||||||
|
# Stronger match first; fts_rank monotonically non-decreasing
|
||||||
|
# (lower-is-better, so ASC).
|
||||||
|
assert out[0]["pov_summary"] == "rabbit rabbit rabbit"
|
||||||
|
assert out[0]["fts_rank"] <= out[1]["fts_rank"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_all_memories_respects_k_limit(tmp_path):
|
||||||
|
"""LIMIT ? must cap result count even when more matches exist."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
_seed(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
{
|
||||||
|
"owner_id": f"bot_{i}",
|
||||||
|
"chat_id": f"chat_{i}",
|
||||||
|
"pov_summary": f"rabbit sighting number {i}",
|
||||||
|
}
|
||||||
|
for i in range(10)
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_all_memories(conn, query="rabbit", k=3)
|
||||||
|
assert len(out) == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_all_memories_empty_query_returns_empty(tmp_path):
|
||||||
|
"""Empty / whitespace-only query must short-circuit to []."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
_seed(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
{
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": "the rabbit darted into the brambles",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
assert search_all_memories(conn, query="") == []
|
||||||
|
assert search_all_memories(conn, query=" ") == []
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
"""Tests for Task 95 — delete-impact computation service (Phase 4).
|
||||||
|
|
||||||
|
`compute_delete_impact` walks event_log forward from a target event_id and
|
||||||
|
produces an :class:`ImpactReport` describing what would be removed if
|
||||||
|
rewind-to-target were invoked. It is a pure preview — no database mutation.
|
||||||
|
T98's drawer surgical-delete UI uses this to render an "are you sure?"
|
||||||
|
modal before invoking the actual rewind path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.services.delete_impact import compute_delete_impact
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_chat(conn) -> tuple[int, int]:
|
||||||
|
"""Append minimal bot + chat events; return their event ids."""
|
||||||
|
bot_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": "bot_a",
|
||||||
|
"name": "BotA",
|
||||||
|
"persona": "...",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
chat_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_bot_a",
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return bot_id, chat_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_impact_for_simple_turn_lists_memory_and_edges(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_chat(conn)
|
||||||
|
user_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "hey there friend",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "Hi! Good to see you.",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": "You greeted me warmly today.",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"source": "turn",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": 1,
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="edge_update",
|
||||||
|
payload={
|
||||||
|
"source_id": "you",
|
||||||
|
"target_id": "bot_a",
|
||||||
|
"affinity_delta": 0.1,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
report = compute_delete_impact(conn, target_event_id=user_id)
|
||||||
|
|
||||||
|
assert report.target_event_id == user_id
|
||||||
|
kinds = [item.kind for item in report.cascading]
|
||||||
|
# Walk from user_turn forward — user_turn, assistant_turn,
|
||||||
|
# memory_written, edge_update should all be in scope, in order.
|
||||||
|
assert kinds == [
|
||||||
|
"user_turn",
|
||||||
|
"assistant_turn",
|
||||||
|
"memory_written",
|
||||||
|
"edge_update",
|
||||||
|
]
|
||||||
|
# Memory description includes the pov_summary excerpt.
|
||||||
|
mem_item = report.cascading[2]
|
||||||
|
assert "memory:" in mem_item.description
|
||||||
|
assert "greeted" in mem_item.description
|
||||||
|
# Edge description includes both endpoints.
|
||||||
|
edge_item = report.cascading[3]
|
||||||
|
assert "you" in edge_item.description
|
||||||
|
assert "bot_a" in edge_item.description
|
||||||
|
assert edge_item.target_id == "you->bot_a"
|
||||||
|
# Notes mentions total count.
|
||||||
|
assert any("4 events" in n for n in report.notes)
|
||||||
|
|
||||||
|
|
||||||
|
def test_impact_for_scene_opening_turn_warns_about_subsequent(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_chat(conn)
|
||||||
|
early_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={"chat_id": "chat_bot_a", "prose": "the start", "segments": []},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "ok",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": early_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="scene_closed",
|
||||||
|
payload={
|
||||||
|
"scene_id": 1,
|
||||||
|
"closed_at": "2026-04-26T21:00:00+00:00",
|
||||||
|
"significance": 2,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
report = compute_delete_impact(conn, target_event_id=early_id)
|
||||||
|
|
||||||
|
# Scene-close warning fires when one is in scope.
|
||||||
|
assert any("scene close" in n.lower() for n in report.notes)
|
||||||
|
# The scene_closed event also appears as a cascading item.
|
||||||
|
assert any(item.kind == "scene_closed" for item in report.cascading)
|
||||||
|
|
||||||
|
|
||||||
|
def test_impact_for_missing_event_returns_empty_with_note(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_chat(conn)
|
||||||
|
report = compute_delete_impact(conn, target_event_id=999_999)
|
||||||
|
|
||||||
|
assert report.cascading == []
|
||||||
|
assert any("not found" in n for n in report.notes)
|
||||||
|
|
||||||
|
|
||||||
|
def test_impact_does_not_mutate_database(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_chat(conn)
|
||||||
|
user_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={"chat_id": "chat_bot_a", "prose": "hi", "segments": []},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "hello",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Snapshot all event_log rows as a tuple-of-tuples.
|
||||||
|
before = conn.execute(
|
||||||
|
"SELECT id, branch_id, ts, kind, payload_json, superseded_by, "
|
||||||
|
"hidden FROM event_log ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
compute_delete_impact(conn, target_event_id=user_id)
|
||||||
|
|
||||||
|
after = conn.execute(
|
||||||
|
"SELECT id, branch_id, ts, kind, payload_json, superseded_by, "
|
||||||
|
"hidden FROM event_log ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
|
# Byte-identical: nothing inserted, deleted, or updated.
|
||||||
|
assert before == after
|
||||||
|
|
||||||
|
|
||||||
|
def test_impact_includes_regenerated_from_warning(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_chat(conn)
|
||||||
|
original_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "first try",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
regen_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "second try",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": 0,
|
||||||
|
"regenerated_from": original_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
report = compute_delete_impact(conn, target_event_id=regen_id)
|
||||||
|
|
||||||
|
# The regenerated_from note carries the original event id so the user
|
||||||
|
# knows the original turn isn't lost.
|
||||||
|
assert any("regenerated from" in n for n in report.notes)
|
||||||
|
assert any(str(original_id) in n for n in report.notes)
|
||||||
@@ -273,6 +273,43 @@ def test_post_skip_elision_advances_clock_and_emits_narration(client, tmp_path):
|
|||||||
assert tp["speaker_id"] == "bot_a"
|
assert tp["speaker_id"] == "bot_a"
|
||||||
|
|
||||||
|
|
||||||
|
def test_skip_route_404_via_typed_exception_class(client, tmp_path):
|
||||||
|
"""T81: drawer skip routes 404 via :class:`ChatNotFoundError`.
|
||||||
|
|
||||||
|
Pre-T81, the route caught ``ValueError`` and recovered the 404 case
|
||||||
|
by sniffing ``str(exc).startswith("chat not found")`` — fragile if
|
||||||
|
the message ever changed wording. The controller now raises a typed
|
||||||
|
exception so the route dispatches on type. Asserting the 404 from
|
||||||
|
the unseeded chat exercises the typed branch end-to-end; importing
|
||||||
|
the class confirms it's a real subclass of ``Exception`` and not a
|
||||||
|
re-export of ``ValueError`` (which would defeat the type split).
|
||||||
|
"""
|
||||||
|
# Don't seed any chat — the controller hits ``get_chat`` returning
|
||||||
|
# ``None`` and raises ``ChatNotFoundError``. The drawer route then
|
||||||
|
# maps that to ``404`` via the typed handler (no string sniff).
|
||||||
|
_override_llm([])
|
||||||
|
try:
|
||||||
|
response = client.post(
|
||||||
|
"/chats/nonexistent/drawer/skip/elision",
|
||||||
|
data={
|
||||||
|
"landing_state_hint": "x",
|
||||||
|
"new_time": "2026-04-26T20:30:00+00:00",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert response.status_code == 404
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
|
||||||
|
# The exception class itself is importable, distinct from ValueError,
|
||||||
|
# and a proper Exception subclass — pinning the type-based dispatch
|
||||||
|
# so future refactors can't quietly collapse it back to a string sniff.
|
||||||
|
from chat.web.skip import ChatNotFoundError
|
||||||
|
|
||||||
|
assert ChatNotFoundError is not None
|
||||||
|
assert issubclass(ChatNotFoundError, Exception)
|
||||||
|
assert not issubclass(ChatNotFoundError, ValueError)
|
||||||
|
|
||||||
|
|
||||||
def test_post_skip_elision_invalid_time_returns_400(client, tmp_path):
|
def test_post_skip_elision_invalid_time_returns_400(client, tmp_path):
|
||||||
_seed_chat(tmp_path / "test.db")
|
_seed_chat(tmp_path / "test.db")
|
||||||
_override_llm([])
|
_override_llm([])
|
||||||
|
|||||||
@@ -0,0 +1,700 @@
|
|||||||
|
"""T98 (Phase 4): drawer phase-4 bundle.
|
||||||
|
|
||||||
|
Five sub-features extending the chat drawer:
|
||||||
|
|
||||||
|
* T98.1 — branching UI (create / switch / from-turn).
|
||||||
|
* T98.2 — significance-review panel (distribution + significance edits).
|
||||||
|
* T98.3 — hide-from-view toggle (per-turn, via ``manual_edit`` projector
|
||||||
|
branch ``turn_hidden``).
|
||||||
|
* T98.4 — surgical delete with cascade preview (preview modal +
|
||||||
|
rewind execution against a target turn).
|
||||||
|
* T98.5 — remaining v1 edits (chat narrative_anchor + weather).
|
||||||
|
|
||||||
|
Tests follow the T59 pattern in ``tests/test_drawer_events_threads_skip.py``
|
||||||
|
— a TestClient against the real FastAPI app with a per-test temp DB.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from chat.app import app
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.eventlog.log import append_and_apply, append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client(tmp_path, monkeypatch):
|
||||||
|
cfg = tmp_path / "config.toml"
|
||||||
|
cfg.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(cfg))
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(db))
|
||||||
|
with TestClient(app) as c:
|
||||||
|
if hasattr(app.state, "background_worker"):
|
||||||
|
app.state.background_worker.enabled = False
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
def _bot_payload(bot_id: str, name: str) -> dict:
|
||||||
|
return {
|
||||||
|
"id": bot_id,
|
||||||
|
"name": name,
|
||||||
|
"persona": "...",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_chat(db: Path, *, with_scene: bool = True) -> int:
|
||||||
|
"""Seed a chat hosted by ``bot_a``; return the latest event id (chat_created)."""
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(conn, kind="bot_authored", payload=_bot_payload("bot_a", "BotA"))
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="you_authored",
|
||||||
|
payload={"name": "Me", "pronouns": "they/them", "persona": ""},
|
||||||
|
)
|
||||||
|
chat_event_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_bot_a",
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
if with_scene:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="scene_opened",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"container_id": None,
|
||||||
|
"started_at": "2026-04-26T20:00:00+00:00",
|
||||||
|
"participants": ["you", "bot_a"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
return chat_event_id
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T98.1 — branching UI.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_1_create_branch_emits_branch_created_and_renders(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
seed_id = _seed_chat(db)
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
"/chats/chat_bot_a/drawer/branch/create",
|
||||||
|
data={"name": "experiment_a", "origin_event_id": str(seed_id)},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'branch_created'"
|
||||||
|
).fetchone()
|
||||||
|
assert rows[0] == 1
|
||||||
|
from chat.state.branches import get_branch
|
||||||
|
|
||||||
|
b = get_branch(conn, "experiment_a")
|
||||||
|
assert b is not None
|
||||||
|
assert b["origin_event_id"] == seed_id
|
||||||
|
assert b["chat_id"] == "chat_bot_a"
|
||||||
|
|
||||||
|
# Drawer partial lists the new branch.
|
||||||
|
body = response.text
|
||||||
|
assert "<h3>Branches</h3>" in body
|
||||||
|
assert "experiment_a" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_1_switch_branch_marks_active_and_unknown_400s(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
seed_id = _seed_chat(db)
|
||||||
|
|
||||||
|
# Create branch directly via the service so this test focuses on switch.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
from chat.services.branching import branch_from_event
|
||||||
|
|
||||||
|
branch_from_event(
|
||||||
|
conn, name="experiment_b", origin_event_id=seed_id, chat_id="chat_bot_a"
|
||||||
|
)
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
"/chats/chat_bot_a/drawer/branch/switch",
|
||||||
|
data={"name": "experiment_b"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
from chat.state.branches import active_branch
|
||||||
|
|
||||||
|
active = active_branch(conn)
|
||||||
|
assert active is not None
|
||||||
|
assert active["name"] == "experiment_b"
|
||||||
|
|
||||||
|
# Unknown branch -> 400.
|
||||||
|
bad = client.post(
|
||||||
|
"/chats/chat_bot_a/drawer/branch/switch",
|
||||||
|
data={"name": "ghost_branch"},
|
||||||
|
)
|
||||||
|
assert bad.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_1_branch_from_turn_emits_branch_created(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
seed_id = _seed_chat(db)
|
||||||
|
|
||||||
|
# Append an extra turn so we can branch from it specifically.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
turn_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={"chat_id": "chat_bot_a", "prose": "hi", "segments": []},
|
||||||
|
)
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/branch/from-turn/{turn_id}",
|
||||||
|
data={"name": "fork_at_turn"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
from chat.state.branches import get_branch
|
||||||
|
|
||||||
|
b = get_branch(conn, "fork_at_turn")
|
||||||
|
assert b is not None
|
||||||
|
assert b["origin_event_id"] == turn_id
|
||||||
|
assert b["chat_id"] == "chat_bot_a"
|
||||||
|
|
||||||
|
# Duplicate name -> 400 from service ValueError.
|
||||||
|
dup = client.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/branch/from-turn/{turn_id}",
|
||||||
|
data={"name": "fork_at_turn"},
|
||||||
|
)
|
||||||
|
assert dup.status_code == 400
|
||||||
|
assert seed_id < turn_id # sanity: turn is after chat_created
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T98.2 — significance review panel.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_memories_for_significance(db: Path) -> list[int]:
|
||||||
|
"""Seed three memories with significance levels 0, 1, 2. Returns ids.
|
||||||
|
|
||||||
|
Uses ``append_and_apply`` (vs ``append_event`` + a final ``project``)
|
||||||
|
so each row is applied exactly once — the chat row was already
|
||||||
|
materialised by ``_seed_chat`` and a re-projection would conflict
|
||||||
|
on ``chats.id`` UNIQUE.
|
||||||
|
"""
|
||||||
|
ids: list[int] = []
|
||||||
|
with open_db(db) as conn:
|
||||||
|
for sig in (0, 1, 2):
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": f"memory at significance {sig}",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"significance": sig,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id FROM memories WHERE chat_id = 'chat_bot_a' "
|
||||||
|
"ORDER BY id ASC"
|
||||||
|
).fetchall()
|
||||||
|
ids = [int(r[0]) for r in rows]
|
||||||
|
return ids
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_2_distribution_renders_per_significance_bucket(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
_seed_memories_for_significance(db)
|
||||||
|
|
||||||
|
response = client.get("/chats/chat_bot_a/drawer")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
|
||||||
|
# Section heading + bar entries for each significance level.
|
||||||
|
assert "<h3>Significance review</h3>" in body
|
||||||
|
# All four buckets appear by their canonical label even when count=0.
|
||||||
|
assert ">★★ (3)<" in body or "(3)" in body
|
||||||
|
# The distribution markup names each level explicitly.
|
||||||
|
for level in (0, 1, 2, 3):
|
||||||
|
assert f"sig-bar sig-{level}" in body
|
||||||
|
# Three seeded memories (sigs 0, 1, 2) — each has a count = 1 bar.
|
||||||
|
# We don't pin exact text formatting, just verify the per-level bars
|
||||||
|
# are present.
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_2_edit_significance_via_existing_route_lands_manual_edit(
|
||||||
|
client, tmp_path
|
||||||
|
):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
ids = _seed_memories_for_significance(db)
|
||||||
|
|
||||||
|
target_id = ids[0] # initially significance=0
|
||||||
|
response = client.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/memory/{target_id}/significance",
|
||||||
|
data={"significance": "3"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Significance updated in the projected table.
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT significance FROM memories WHERE id = ?", (target_id,)
|
||||||
|
).fetchone()
|
||||||
|
assert int(row[0]) == 3
|
||||||
|
|
||||||
|
# manual_edit landed in the event log with the prior snapshot.
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
log_rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'manual_edit' ORDER BY id DESC LIMIT 1"
|
||||||
|
).fetchone()
|
||||||
|
payload = _json.loads(log_rows[0])
|
||||||
|
assert payload["target_kind"] == "memory_significance"
|
||||||
|
assert int(payload["target_id"]) == target_id
|
||||||
|
assert payload["prior_value"] == 0
|
||||||
|
assert payload["new_value"] == 3
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T98.3 — hide-from-view toggle.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_turns(db: Path) -> tuple[int, int]:
|
||||||
|
"""Append one user_turn + one assistant_turn; return their event ids."""
|
||||||
|
with open_db(db) as conn:
|
||||||
|
user_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "How are you doing today?",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
bot_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "Quite well, thanks for asking!",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return user_id, bot_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_3_hide_turn_flips_event_log_hidden_via_manual_edit(
|
||||||
|
client, tmp_path
|
||||||
|
):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
user_id, bot_id = _seed_turns(db)
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/hide/{user_id}",
|
||||||
|
data={"hidden": "1"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# event_log.hidden flipped to 1.
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT hidden FROM event_log WHERE id = ?", (user_id,)
|
||||||
|
).fetchone()
|
||||||
|
assert int(row[0]) == 1
|
||||||
|
|
||||||
|
# manual_edit landed with the prior snapshot.
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
log = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'manual_edit' ORDER BY id DESC LIMIT 1"
|
||||||
|
).fetchone()
|
||||||
|
payload = _json.loads(log[0])
|
||||||
|
assert payload["target_kind"] == "turn_hidden"
|
||||||
|
assert int(payload["target_id"]) == user_id
|
||||||
|
assert payload["prior_value"] == {"hidden": 0}
|
||||||
|
assert payload["new_value"] == {"hidden": 1}
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_3_hidden_turn_disappears_from_read_recent_dialogue(
|
||||||
|
client, tmp_path
|
||||||
|
):
|
||||||
|
"""Hiding a turn must drop it from the prompt-window read.
|
||||||
|
|
||||||
|
``read_recent_dialogue`` (chat.services.turn_common) filters
|
||||||
|
``hidden = 0`` server-side, so flipping the flag via the drawer
|
||||||
|
route must surface immediately.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
user_id, bot_id = _seed_turns(db)
|
||||||
|
|
||||||
|
# Sanity baseline — both turns visible before the hide.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
from chat.services.turn_common import read_recent_dialogue
|
||||||
|
|
||||||
|
before = read_recent_dialogue(conn, "chat_bot_a", limit=10)
|
||||||
|
before_ids = [t["event_id"] for t in before]
|
||||||
|
assert user_id in before_ids
|
||||||
|
assert bot_id in before_ids
|
||||||
|
|
||||||
|
# Hide the user turn via the drawer route.
|
||||||
|
response = client.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/hide/{user_id}",
|
||||||
|
data={"hidden": "1"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
from chat.services.turn_common import read_recent_dialogue
|
||||||
|
|
||||||
|
after = read_recent_dialogue(conn, "chat_bot_a", limit=10)
|
||||||
|
after_ids = [t["event_id"] for t in after]
|
||||||
|
assert user_id not in after_ids
|
||||||
|
assert bot_id in after_ids # the unhidden bot turn still surfaces
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T98.4 — surgical delete with cascade preview.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_4_delete_preview_returns_impact_report_html(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
user_id, bot_id = _seed_turns(db)
|
||||||
|
|
||||||
|
response = client.get(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/delete-preview/{user_id}"
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
|
||||||
|
# Modal markup with the event id and the cascade list.
|
||||||
|
assert "delete-impact-modal" in body
|
||||||
|
assert f"Delete event {user_id}?" in body
|
||||||
|
assert "delete-impact-cascade" in body
|
||||||
|
# Both turns ride along in the cascade — user_turn at user_id, then
|
||||||
|
# the assistant_turn at bot_id (>= user_id).
|
||||||
|
assert "user_turn" in body
|
||||||
|
assert "assistant_turn" in body
|
||||||
|
# Confirm-form posts to the delete route.
|
||||||
|
assert f"/drawer/turn/delete/{user_id}" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_4_delete_invokes_rewind_and_drops_cascade(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
user_id, bot_id = _seed_turns(db)
|
||||||
|
|
||||||
|
# Append a third turn after the assistant_turn so we can verify the
|
||||||
|
# cascade catches everything from user_id forward.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
extra_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "follow-up",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sanity: all three turn rows exist.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
turn_count = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log "
|
||||||
|
"WHERE kind IN ('user_turn', 'assistant_turn')"
|
||||||
|
).fetchone()[0]
|
||||||
|
assert turn_count == 3
|
||||||
|
|
||||||
|
# Delete from user_id forward.
|
||||||
|
response = client.post(f"/chats/chat_bot_a/drawer/turn/delete/{user_id}")
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
# All three turns are gone — the rewind truncated the log past
|
||||||
|
# user_id - 1, removing user_id, bot_id, and extra_id.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
turn_count = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log "
|
||||||
|
"WHERE kind IN ('user_turn', 'assistant_turn')"
|
||||||
|
).fetchone()[0]
|
||||||
|
assert turn_count == 0
|
||||||
|
for ev_id in (user_id, bot_id, extra_id):
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT 1 FROM event_log WHERE id = ?", (ev_id,)
|
||||||
|
).fetchone()
|
||||||
|
assert row is None, f"event {ev_id} should have been deleted"
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_impact_modal_uses_jinja_partial(client, tmp_path):
|
||||||
|
"""T110.3: the modal HTML is rendered from a Jinja partial
|
||||||
|
(`_delete_impact_modal.html`) rather than f-string concatenation in
|
||||||
|
Python. Verify the partial-rendered shape: the wrapping
|
||||||
|
``delete-impact-modal`` div, the cascade list, and the confirm form.
|
||||||
|
|
||||||
|
The partial inherits Jinja2 autoescape so HTML safety follows
|
||||||
|
automatically — the explicit ``html.escape()`` calls from T110.2
|
||||||
|
become redundant once this lands.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
user_id, _bot_id = _seed_turns(db)
|
||||||
|
|
||||||
|
response = client.get(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/delete-preview/{user_id}"
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
|
||||||
|
# Markup shape that the partial produces. Double-quoted attributes
|
||||||
|
# signal Jinja rendering (the prior f-string used single quotes).
|
||||||
|
assert '<div class="delete-impact-modal">' in body
|
||||||
|
assert '<ul class="delete-impact-cascade">' in body
|
||||||
|
# The confirm form still posts to the same delete route.
|
||||||
|
assert f"/chats/chat_bot_a/drawer/turn/delete/{user_id}" in body
|
||||||
|
assert "Confirm delete" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_impact_modal_escapes_user_controllable_strings(client, tmp_path):
|
||||||
|
"""T110.2: defense-in-depth — fields embedded in the modal HTML come
|
||||||
|
from event payloads (turn prose, scene timestamps, etc.) which are
|
||||||
|
ultimately user-controllable. Wrap them with ``html.escape`` so a
|
||||||
|
payload like ``<script>alert(1)</script>`` renders as inert text and
|
||||||
|
doesn't leak through into the rendered modal as actual markup.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
|
||||||
|
# Seed a user_turn whose prose contains an HTML-script payload. The
|
||||||
|
# modal renders ``description = "turn N (you: <prose excerpt>)"`` so
|
||||||
|
# the prose flows verbatim into the cascade list <li>.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
evil_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "<script>alert('xss')</script>",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
response = client.get(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/delete-preview/{evil_id}"
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
|
||||||
|
# Raw <script> must NOT survive into the rendered HTML. The escaped
|
||||||
|
# form (<script>) is what we want to see instead.
|
||||||
|
assert "<script>alert" not in body
|
||||||
|
assert "<script>alert" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_bulk_significance_re_rate_emits_manual_edit_per_memory(client, tmp_path):
|
||||||
|
"""T110.4: bulk significance re-rate fans out into one
|
||||||
|
``manual_edit`` event per matching memory — preserving the per-row
|
||||||
|
audit trail (and reversibility) instead of collapsing everything
|
||||||
|
into a single bulk event.
|
||||||
|
|
||||||
|
Seed five memories at significance 0, bulk re-rate 0 -> 2, and
|
||||||
|
verify five new ``memory_significance`` ``manual_edit`` rows landed
|
||||||
|
AND every memory now sits at significance 2.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
|
||||||
|
# Five memories at significance 0.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
for i in range(5):
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": f"low-sig memory {i}",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"significance": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Plus one memory at significance 1 to verify the re-rate is
|
||||||
|
# scoped to ``level_from`` and doesn't sweep the whole chat.
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": "already-rated memory",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"significance": 1,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
prior_manual_edits = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'manual_edit'"
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
"/chats/chat_bot_a/drawer/memory/significance/bulk",
|
||||||
|
data={"level_from": "0", "level_to": "2"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Five new manual_edit rows, one per matching memory.
|
||||||
|
new_manual_edits = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'manual_edit'"
|
||||||
|
).fetchone()[0]
|
||||||
|
assert new_manual_edits - prior_manual_edits == 5
|
||||||
|
|
||||||
|
# Every emitted edit is a memory_significance edit with prior=0
|
||||||
|
# and new=2.
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'manual_edit' "
|
||||||
|
"ORDER BY id DESC LIMIT 5"
|
||||||
|
).fetchall()
|
||||||
|
for r in rows:
|
||||||
|
payload = _json.loads(r[0])
|
||||||
|
assert payload["target_kind"] == "memory_significance"
|
||||||
|
assert payload["prior_value"] == 0
|
||||||
|
assert payload["new_value"] == 2
|
||||||
|
|
||||||
|
# Projection caught up — five memories at sig=2, the untouched
|
||||||
|
# one stays at sig=1, none remain at sig=0.
|
||||||
|
dist = dict(
|
||||||
|
conn.execute(
|
||||||
|
"SELECT significance, COUNT(*) FROM memories "
|
||||||
|
"WHERE chat_id = 'chat_bot_a' GROUP BY significance"
|
||||||
|
).fetchall()
|
||||||
|
)
|
||||||
|
assert dist.get(0, 0) == 0
|
||||||
|
assert dist.get(1, 0) == 1
|
||||||
|
assert dist.get(2, 0) == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_delete_turn_with_event_id_zero_returns_400(client, tmp_path):
|
||||||
|
"""T110.1: ``event_id <= 0`` is an obvious client error and must NOT
|
||||||
|
silently rewind the entire log via ``after_event_id = -1``. The route
|
||||||
|
rejects it with 400 so the audit trail stays intact.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
_seed_turns(db)
|
||||||
|
|
||||||
|
# Sanity: events present before the bad request.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
before = conn.execute("SELECT COUNT(*) FROM event_log").fetchone()[0]
|
||||||
|
assert before > 0
|
||||||
|
|
||||||
|
response = client.post("/chats/chat_bot_a/drawer/turn/delete/0")
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
# And the log was NOT truncated.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
after = conn.execute("SELECT COUNT(*) FROM event_log").fetchone()[0]
|
||||||
|
assert after == before
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T98.5 — remaining v1 edits (chat narrative anchor + weather).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_5_edit_chat_narrative_anchor_emits_manual_edit(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
"/chats/chat_bot_a/drawer/chat/narrative-anchor",
|
||||||
|
data={"new_value": "Late evening, after dinner"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT narrative_anchor FROM chat_state WHERE chat_id = ?",
|
||||||
|
("chat_bot_a",),
|
||||||
|
).fetchone()
|
||||||
|
assert row[0] == "Late evening, after dinner"
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
log = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'manual_edit' ORDER BY id DESC LIMIT 1"
|
||||||
|
).fetchone()
|
||||||
|
payload = _json.loads(log[0])
|
||||||
|
assert payload["target_kind"] == "chat_narrative_anchor"
|
||||||
|
assert payload["target_id"] == "chat_bot_a"
|
||||||
|
assert payload["prior_value"] == "Day 1"
|
||||||
|
assert payload["new_value"] == "Late evening, after dinner"
|
||||||
|
|
||||||
|
|
||||||
|
def test_t98_5_edit_chat_weather_emits_manual_edit(client, tmp_path):
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_chat(db)
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
"/chats/chat_bot_a/drawer/chat/weather",
|
||||||
|
data={"new_value": "thunderstorm rolling in"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT weather FROM chat_state WHERE chat_id = ?",
|
||||||
|
("chat_bot_a",),
|
||||||
|
).fetchone()
|
||||||
|
assert row[0] == "thunderstorm rolling in"
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
log = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'manual_edit' ORDER BY id DESC LIMIT 1"
|
||||||
|
).fetchone()
|
||||||
|
payload = _json.loads(log[0])
|
||||||
|
assert payload["target_kind"] == "chat_weather"
|
||||||
|
assert payload["target_id"] == "chat_bot_a"
|
||||||
|
assert payload["prior_value"] == ""
|
||||||
|
assert payload["new_value"] == "thunderstorm rolling in"
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
"""Embedding worker (T97, Phase 4).
|
||||||
|
|
||||||
|
The worker drains a queue of EmbeddingJobs and emits ``embedding_indexed``
|
||||||
|
events. Mirrors test_significance.py's BackgroundWorker tests in shape:
|
||||||
|
seed a memory, enqueue jobs, call ``stop()`` to drain via sentinel, then
|
||||||
|
assert on the projected ``embeddings`` table and the underlying event_log.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
from chat.services.embedding_worker import EmbeddingJob, EmbeddingWorker
|
||||||
|
from chat.services.embeddings import (
|
||||||
|
DEFAULT_EMBEDDING_MODEL,
|
||||||
|
EmbeddingResult,
|
||||||
|
FALLBACK_EMBEDDING_MODEL,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Trigger handler registration for projection.
|
||||||
|
import chat.state.embeddings # noqa: F401
|
||||||
|
import chat.state.entities # noqa: F401
|
||||||
|
import chat.state.memory # noqa: F401
|
||||||
|
import chat.state.world # noqa: F401
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_memories(db_path: Path, count: int) -> list[int]:
|
||||||
|
"""Seed ``count`` memory rows for ``bot_a`` and return their ids."""
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": "bot_a",
|
||||||
|
"name": "BotA",
|
||||||
|
"persona": "...",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_bot_a",
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for i in range(count):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"pov_summary": f"memory text {i}",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": 1,
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
return [
|
||||||
|
r[0]
|
||||||
|
for r in conn.execute(
|
||||||
|
"SELECT id FROM memories WHERE owner_id = 'bot_a' ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_worker_drains_jobs_and_emits_indexed_events(tmp_path):
|
||||||
|
"""Three jobs in -> three ``embedding_indexed`` events out, all
|
||||||
|
projected into the ``embeddings`` table."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
memory_ids = _seed_memories(db, count=3)
|
||||||
|
|
||||||
|
worker = EmbeddingWorker(
|
||||||
|
conn_factory=lambda: open_db(db),
|
||||||
|
client=None, # pseudo path — no client needed
|
||||||
|
)
|
||||||
|
await worker.start()
|
||||||
|
for mid in memory_ids:
|
||||||
|
worker.enqueue(EmbeddingJob(memory_id=mid, text=f"text-{mid}"))
|
||||||
|
await worker.stop()
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Three embedding_indexed events landed.
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'embedding_indexed'"
|
||||||
|
)
|
||||||
|
assert cur.fetchone()[0] == 3
|
||||||
|
# Three rows in the embeddings table — one per memory.
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT memory_id, model, dim FROM embeddings ORDER BY memory_id"
|
||||||
|
)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
assert len(rows) == 3
|
||||||
|
for (mid, model, dim), expected_mid in zip(rows, memory_ids):
|
||||||
|
assert mid == expected_mid
|
||||||
|
assert model == DEFAULT_EMBEDDING_MODEL
|
||||||
|
assert dim > 0
|
||||||
|
|
||||||
|
|
||||||
|
async def test_worker_skips_fallback_results(tmp_path, monkeypatch):
|
||||||
|
"""A fallback EmbeddingResult must NOT produce an embedding_indexed
|
||||||
|
event — backfill can retry later when a real embedding is available."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
memory_ids = _seed_memories(db, count=1)
|
||||||
|
|
||||||
|
async def _fake_generate(client, *, text, model, dim, timeout_s=30.0):
|
||||||
|
return EmbeddingResult(
|
||||||
|
vector=[0.0] * dim, model=FALLBACK_EMBEDDING_MODEL, dim=dim
|
||||||
|
)
|
||||||
|
|
||||||
|
# Patch the symbol the worker resolved at import time.
|
||||||
|
import chat.services.embedding_worker as worker_mod
|
||||||
|
|
||||||
|
monkeypatch.setattr(worker_mod, "generate_embedding", _fake_generate)
|
||||||
|
|
||||||
|
worker = EmbeddingWorker(
|
||||||
|
conn_factory=lambda: open_db(db),
|
||||||
|
client=None,
|
||||||
|
)
|
||||||
|
await worker.start()
|
||||||
|
worker.enqueue(EmbeddingJob(memory_id=memory_ids[0], text="anything"))
|
||||||
|
await worker.stop()
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'embedding_indexed'"
|
||||||
|
)
|
||||||
|
assert cur.fetchone()[0] == 0
|
||||||
|
cur = conn.execute("SELECT COUNT(*) FROM embeddings")
|
||||||
|
assert cur.fetchone()[0] == 0
|
||||||
|
|
||||||
|
|
||||||
|
async def test_worker_handles_concurrent_jobs_serially(tmp_path):
|
||||||
|
"""Five jobs queued back-to-back must process in FIFO order — the
|
||||||
|
single-task design respects the Featherless 2-conn cap (and keeps
|
||||||
|
event_log ordering deterministic)."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
memory_ids = _seed_memories(db, count=5)
|
||||||
|
|
||||||
|
worker = EmbeddingWorker(
|
||||||
|
conn_factory=lambda: open_db(db),
|
||||||
|
client=None,
|
||||||
|
)
|
||||||
|
await worker.start()
|
||||||
|
# Enqueue all five before yielding to the loop — exercises the queue
|
||||||
|
# rather than a one-at-a-time drain.
|
||||||
|
for mid in memory_ids:
|
||||||
|
worker.enqueue(EmbeddingJob(memory_id=mid, text=f"text-{mid}"))
|
||||||
|
await worker.stop()
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Events landed in enqueue order (FIFO).
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT json_extract(payload_json, '$.memory_id') "
|
||||||
|
"FROM event_log WHERE kind = 'embedding_indexed' "
|
||||||
|
"ORDER BY id"
|
||||||
|
)
|
||||||
|
seen = [r[0] for r in cur.fetchall()]
|
||||||
|
assert seen == memory_ids
|
||||||
|
|
||||||
|
# All five embeddings projected.
|
||||||
|
cur = conn.execute("SELECT COUNT(*) FROM embeddings")
|
||||||
|
assert cur.fetchone()[0] == 5
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
"""Tests for the embedding generation service (T91, Phase 4).
|
||||||
|
|
||||||
|
Phase 4's first cut ships a deterministic local pseudo-embedding so the
|
||||||
|
vector retrieval pipeline can land without an external embeddings API
|
||||||
|
or a heavy local model dependency. These tests pin the contract:
|
||||||
|
|
||||||
|
* the result has the right shape (vector length, ``dim`` metadata),
|
||||||
|
* the default ``model`` string is reported back unchanged,
|
||||||
|
* output is byte-identical for the same input (deterministic),
|
||||||
|
* distinct inputs produce distinct vectors (so cosine actually
|
||||||
|
discriminates),
|
||||||
|
* empty / whitespace-only input collapses to the ``"fallback"`` sentinel
|
||||||
|
with a zero vector — callers detect this and skip indexing,
|
||||||
|
* the vector is unit-normalized so cosine similarity behaves.
|
||||||
|
|
||||||
|
The pseudo path doesn't touch the LLMClient, so we pass an empty
|
||||||
|
``MockLLMClient`` — any accidental call into it would raise
|
||||||
|
``IndexError`` and surface as a regression.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import math
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from chat.llm.mock import MockLLMClient
|
||||||
|
from chat.services.embeddings import (
|
||||||
|
DEFAULT_EMBEDDING_DIM,
|
||||||
|
DEFAULT_EMBEDDING_MODEL,
|
||||||
|
FALLBACK_EMBEDDING_MODEL,
|
||||||
|
EmbeddingResult,
|
||||||
|
generate_embedding,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _client() -> MockLLMClient:
|
||||||
|
# Pseudo path never calls the client — empty canned list ensures any
|
||||||
|
# accidental call raises and surfaces the regression loudly.
|
||||||
|
return MockLLMClient(canned=[])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_returns_vector_of_correct_dim():
|
||||||
|
result = await generate_embedding(_client(), text="hello")
|
||||||
|
assert isinstance(result, EmbeddingResult)
|
||||||
|
assert isinstance(result.vector, list)
|
||||||
|
assert len(result.vector) == DEFAULT_EMBEDDING_DIM == 384
|
||||||
|
assert result.dim == 384
|
||||||
|
assert all(isinstance(x, float) for x in result.vector)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_returns_correct_model_metadata():
|
||||||
|
result = await generate_embedding(_client(), text="hello")
|
||||||
|
assert result.model == DEFAULT_EMBEDDING_MODEL == "pseudo-sha256-384"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_is_deterministic():
|
||||||
|
a = await generate_embedding(_client(), text="hello world")
|
||||||
|
b = await generate_embedding(_client(), text="hello world")
|
||||||
|
assert a.vector == b.vector
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_distinct_text_produces_distinct_vectors():
|
||||||
|
a = await generate_embedding(_client(), text="hello world")
|
||||||
|
b = await generate_embedding(_client(), text="totally different content")
|
||||||
|
assert a.vector != b.vector
|
||||||
|
# Sanity-check cosine similarity — both vectors are unit-normalized,
|
||||||
|
# so this reduces to a plain dot product.
|
||||||
|
cosine = sum(x * y for x, y in zip(a.vector, b.vector))
|
||||||
|
assert cosine < 0.99
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_empty_text_returns_fallback():
|
||||||
|
for empty in ("", " ", "\n\t"):
|
||||||
|
result = await generate_embedding(_client(), text=empty)
|
||||||
|
assert result.model == FALLBACK_EMBEDDING_MODEL == "fallback"
|
||||||
|
assert result.dim == DEFAULT_EMBEDDING_DIM
|
||||||
|
assert len(result.vector) == DEFAULT_EMBEDDING_DIM
|
||||||
|
assert all(x == 0.0 for x in result.vector)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_unit_normalized():
|
||||||
|
result = await generate_embedding(_client(), text="some non-empty text")
|
||||||
|
norm_sq = sum(x * x for x in result.vector)
|
||||||
|
assert math.isclose(norm_sq, 1.0, abs_tol=1e-6)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_non_default_model_logs_warning(caplog):
|
||||||
|
"""T107: non-default model falls through to fallback and must warn.
|
||||||
|
|
||||||
|
A Phase 4.5+ caller pointing at a real model that isn't yet wired
|
||||||
|
up would otherwise silently degrade (zero vector → useless cosine).
|
||||||
|
The warning surfaces the misconfiguration in logs.
|
||||||
|
"""
|
||||||
|
caplog.set_level(logging.WARNING, logger="chat.services.embeddings")
|
||||||
|
result = await generate_embedding(_client(), text="hello", model="real-model")
|
||||||
|
|
||||||
|
# Behavior unchanged: still returns the fallback sentinel.
|
||||||
|
assert result.model == FALLBACK_EMBEDDING_MODEL == "fallback"
|
||||||
|
assert all(x == 0.0 for x in result.vector)
|
||||||
|
|
||||||
|
# Warning fired and names the offending model.
|
||||||
|
warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
|
||||||
|
assert any("non-default model" in r.getMessage() for r in warnings)
|
||||||
|
assert any("real-model" in r.getMessage() for r in warnings)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_generate_embedding_default_model_does_not_warn(caplog):
|
||||||
|
"""T107: the silent default path must stay silent."""
|
||||||
|
caplog.set_level(logging.WARNING, logger="chat.services.embeddings")
|
||||||
|
await generate_embedding(_client(), text="hello")
|
||||||
|
warnings = [r for r in caplog.records if r.levelno == logging.WARNING]
|
||||||
|
assert warnings == []
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
import chat.state.memory # registers memory_written handler
|
||||||
|
import chat.state.embeddings # registers embedding handlers
|
||||||
|
from chat.state.embeddings import get_embedding, list_embeddings_for_owner
|
||||||
|
|
||||||
|
|
||||||
|
def _base_memory(**overrides):
|
||||||
|
payload = {
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"scene_id": 1,
|
||||||
|
"pov_summary": "She laughed at his joke about owls.",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"chat_clock_at": "2026-04-26T10:00:00",
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": 1,
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
}
|
||||||
|
payload.update(overrides)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _vec(n: int = 384, base: float = 0.1) -> list[float]:
|
||||||
|
"""Return a length-n float vector with predictable values for assertions."""
|
||||||
|
return [round(base + i * 0.001, 6) for i in range(n)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_embedding_indexed_inserts_row(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(conn, kind="memory_written", payload=_base_memory())
|
||||||
|
project(conn)
|
||||||
|
memory_id = conn.execute("SELECT id FROM memories").fetchone()[0]
|
||||||
|
|
||||||
|
vector = _vec(384, base=0.1)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": memory_id,
|
||||||
|
"vector": vector,
|
||||||
|
"model": "test-model",
|
||||||
|
"dim": 384,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
emb = get_embedding(conn, memory_id)
|
||||||
|
assert emb is not None
|
||||||
|
assert emb["memory_id"] == memory_id
|
||||||
|
assert emb["vector"] == vector
|
||||||
|
assert emb["model"] == "test-model"
|
||||||
|
assert emb["dim"] == 384
|
||||||
|
assert emb["indexed_at"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_embedding_deindexed_removes_row(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(conn, kind="memory_written", payload=_base_memory())
|
||||||
|
project(conn)
|
||||||
|
memory_id = conn.execute("SELECT id FROM memories").fetchone()[0]
|
||||||
|
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": memory_id,
|
||||||
|
"vector": _vec(),
|
||||||
|
"model": "test-model",
|
||||||
|
"dim": 384,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
assert get_embedding(conn, memory_id) is not None
|
||||||
|
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_deindexed",
|
||||||
|
payload={"memory_id": memory_id},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
assert get_embedding(conn, memory_id) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_embedding_indexed_replaces_existing(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
append_event(conn, kind="memory_written", payload=_base_memory())
|
||||||
|
project(conn)
|
||||||
|
memory_id = conn.execute("SELECT id FROM memories").fetchone()[0]
|
||||||
|
|
||||||
|
vec_a = _vec(384, base=0.1)
|
||||||
|
vec_b = _vec(384, base=0.5)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": memory_id,
|
||||||
|
"vector": vec_a,
|
||||||
|
"model": "test-model",
|
||||||
|
"dim": 384,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
first = get_embedding(conn, memory_id)
|
||||||
|
assert first is not None
|
||||||
|
assert first["vector"] == vec_a
|
||||||
|
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": memory_id,
|
||||||
|
"vector": vec_b,
|
||||||
|
"model": "test-model",
|
||||||
|
"dim": 384,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
second = get_embedding(conn, memory_id)
|
||||||
|
assert second is not None
|
||||||
|
assert second["vector"] == vec_b
|
||||||
|
# Still exactly one row for this memory.
|
||||||
|
count = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM embeddings WHERE memory_id = ?", (memory_id,)
|
||||||
|
).fetchone()[0]
|
||||||
|
assert count == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_embeddings_for_owner_returns_joined_rows(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Two memories for bot_a, one for bot_b.
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload=_base_memory(
|
||||||
|
owner_id="bot_a",
|
||||||
|
pov_summary="Alpha memory.",
|
||||||
|
significance=2,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload=_base_memory(
|
||||||
|
owner_id="bot_a",
|
||||||
|
pov_summary="Beta memory.",
|
||||||
|
significance=3,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload=_base_memory(
|
||||||
|
owner_id="bot_b",
|
||||||
|
pov_summary="Gamma memory.",
|
||||||
|
significance=1,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, owner_id FROM memories ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
# Index every memory with a distinct vector so we can check ordering.
|
||||||
|
for i, (mid, _owner) in enumerate(rows):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": mid,
|
||||||
|
"vector": _vec(384, base=0.1 * (i + 1)),
|
||||||
|
"model": "test-model",
|
||||||
|
"dim": 384,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
a_rows = list_embeddings_for_owner(conn, "bot_a")
|
||||||
|
assert len(a_rows) == 2
|
||||||
|
summaries = {r["pov_summary"] for r in a_rows}
|
||||||
|
assert summaries == {"Alpha memory.", "Beta memory."}
|
||||||
|
sigs = {r["significance"] for r in a_rows}
|
||||||
|
assert sigs == {2, 3}
|
||||||
|
for r in a_rows:
|
||||||
|
assert r["model"] == "test-model"
|
||||||
|
assert r["dim"] == 384
|
||||||
|
assert isinstance(r["vector"], list)
|
||||||
|
assert len(r["vector"]) == 384
|
||||||
|
assert r["witness_you"] == 1
|
||||||
|
assert r["witness_host"] == 1
|
||||||
|
assert r["witness_guest"] == 0
|
||||||
|
|
||||||
|
b_rows = list_embeddings_for_owner(conn, "bot_b")
|
||||||
|
assert len(b_rows) == 1
|
||||||
|
assert b_rows[0]["pov_summary"] == "Gamma memory."
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_embedding_returns_none_when_missing(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
assert get_embedding(conn, 999) is None
|
||||||
@@ -570,3 +570,173 @@ def test_meanwhile_turn_registered_in_in_flight_tasks(
|
|||||||
# Post-flight: the entry has been cleaned up so the next turn (or
|
# Post-flight: the entry has been cleaned up so the next turn (or
|
||||||
# the cancel route) doesn't see a stale task.
|
# the cancel route) doesn't see a stale task.
|
||||||
assert "chat_bot_a" not in _in_flight_tasks
|
assert "chat_bot_a" not in _in_flight_tasks
|
||||||
|
|
||||||
|
|
||||||
|
def test_meanwhile_turn_cancellation_via_route(app_state_setup, tmp_path):
|
||||||
|
"""T85.2: a cancellation that fires while a meanwhile beat is
|
||||||
|
streaming truncates the assistant_turn and skips the post-turn
|
||||||
|
memory + state-update writes — the same end-to-end shape the
|
||||||
|
/turns/cancel route produces.
|
||||||
|
|
||||||
|
Drives the cancel by hijacking ``client.stream`` to raise
|
||||||
|
CancelledError on its first iteration — the exact pattern proven
|
||||||
|
by ``test_cancelled_turn_still_closes_scene_when_user_prose_signals_close``
|
||||||
|
in ``tests/test_turn_flow.py``. This mirrors what
|
||||||
|
``cancel_turn`` does in production (``task.cancel()`` schedules a
|
||||||
|
CancelledError on the next await); doing the raise inline avoids
|
||||||
|
the TestClient-loop-reentry problem that prevents driving a second
|
||||||
|
POST mid-stream from the same synchronous test thread, while
|
||||||
|
exercising the same code path: the meanwhile streamer's
|
||||||
|
``except asyncio.CancelledError`` block at meanwhile.py:276 sets
|
||||||
|
``cancelled=True`` + ``truncated=True``, the assistant_turn lands
|
||||||
|
with the partial, and the memory/state-update branch is skipped.
|
||||||
|
|
||||||
|
The ``_in_flight_tasks`` registration that wires the cancel route
|
||||||
|
to the meanwhile streamer is independently pinned by
|
||||||
|
``test_meanwhile_turn_registered_in_in_flight_tasks`` above; this
|
||||||
|
test pins the downstream behavioural shape the registration
|
||||||
|
enables — together they cover the full Stop-button lifecycle for
|
||||||
|
meanwhile beats.
|
||||||
|
|
||||||
|
Behavioural pins:
|
||||||
|
|
||||||
|
* ``assistant_turn`` lands with ``truncated=True``,
|
||||||
|
``meanwhile_scene_id=2``, ``speaker_id="bot_a"``.
|
||||||
|
* No ``memory_written`` events fire (cancel skips per-bot writes).
|
||||||
|
* No post-turn ``edge_update`` events fire (cancel skips state updates).
|
||||||
|
* ``_in_flight_tasks`` is empty post-flight.
|
||||||
|
"""
|
||||||
|
from typing import AsyncIterator, Sequence
|
||||||
|
|
||||||
|
from chat.llm.client import Message
|
||||||
|
from chat.web.turns import _in_flight_tasks
|
||||||
|
|
||||||
|
_seed_meanwhile_chat(tmp_path / "test.db")
|
||||||
|
|
||||||
|
class _CancelOnStreamMock(MockLLMClient):
|
||||||
|
"""Yields CancelledError on first iteration of ``stream`` —
|
||||||
|
simulates ``cancel_turn`` having fired ``task.cancel()`` on the
|
||||||
|
in-flight streaming task. ``generate`` is delegated to the
|
||||||
|
canned-queue base so parse_turn still resolves cleanly.
|
||||||
|
"""
|
||||||
|
|
||||||
|
async def stream(
|
||||||
|
self, messages: Sequence[Message], *, model: str, **params
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
raise asyncio.CancelledError
|
||||||
|
yield # pragma: no cover — keeps this an async generator.
|
||||||
|
|
||||||
|
canned_parse = json.dumps(
|
||||||
|
{"segments": [{"kind": "narration", "text": "they exchange a glance"}]}
|
||||||
|
)
|
||||||
|
# Canned queue: only parse_turn — the narrative slot is never pulled
|
||||||
|
# because stream raises before consuming it, and post-turn
|
||||||
|
# state-update is skipped by the cancel branch.
|
||||||
|
mock = _CancelOnStreamMock(canned=[canned_parse])
|
||||||
|
from chat.web.kickoff import get_llm_client
|
||||||
|
|
||||||
|
app.dependency_overrides[get_llm_client] = lambda: mock
|
||||||
|
try:
|
||||||
|
# The meanwhile controller re-raises CancelledError after the
|
||||||
|
# partial assistant_turn is recorded (meanwhile.py:387). The
|
||||||
|
# outer post_turn route has no catch for CancelledError on the
|
||||||
|
# meanwhile path (turns.py:244-254 only catches ValueError), so
|
||||||
|
# the exception propagates up through Starlette. TestClient
|
||||||
|
# surfaces that as a 500 or a propagated exception depending on
|
||||||
|
# Starlette/asyncio versions; we don't pin the response.
|
||||||
|
try:
|
||||||
|
app_state_setup.post(
|
||||||
|
"/chats/chat_bot_a/turns",
|
||||||
|
data={"prose": "they exchange a glance"},
|
||||||
|
)
|
||||||
|
except BaseException:
|
||||||
|
pass
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
|
||||||
|
with open_db(tmp_path / "test.db") as conn:
|
||||||
|
assistant_rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'assistant_turn' ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
memory_count = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'memory_written'"
|
||||||
|
).fetchone()[0]
|
||||||
|
# Edge updates AFTER the assistant_turn (i.e. excluding seeded ones).
|
||||||
|
max_at_row = conn.execute(
|
||||||
|
"SELECT MAX(id) FROM event_log WHERE kind = 'assistant_turn'"
|
||||||
|
).fetchone()
|
||||||
|
max_at = max_at_row[0] if max_at_row[0] is not None else 0
|
||||||
|
post_turn_edge_updates = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log "
|
||||||
|
"WHERE kind = 'edge_update' AND id > ?",
|
||||||
|
(max_at,),
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
|
# The cancelled assistant_turn was still recorded with truncated=True,
|
||||||
|
# carrying whatever partial text accumulated before cancel propagated
|
||||||
|
# (zero text here since the cancel hits on the first iteration).
|
||||||
|
assert len(assistant_rows) == 1
|
||||||
|
payload = json.loads(assistant_rows[0][0])
|
||||||
|
assert payload["truncated"] is True, payload
|
||||||
|
assert payload["meanwhile_scene_id"] == 2
|
||||||
|
assert payload["speaker_id"] == "bot_a"
|
||||||
|
|
||||||
|
# No per-bot memory writes — cancellation short-circuits the memory
|
||||||
|
# + state-update branch (see chat/web/meanwhile.py:308).
|
||||||
|
assert memory_count == 0
|
||||||
|
|
||||||
|
# No post-turn edge_updates — same short-circuit.
|
||||||
|
assert post_turn_edge_updates == 0
|
||||||
|
|
||||||
|
# Post-flight: registry cleared so the cancel route won't try to
|
||||||
|
# re-cancel a defunct task on a follow-up POST.
|
||||||
|
assert "chat_bot_a" not in _in_flight_tasks
|
||||||
|
|
||||||
|
|
||||||
|
def test_meanwhile_cancel_route_no_op_after_turn_completes(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""T85.2: POST ``/chats/<id>/turns/cancel`` AFTER a meanwhile turn
|
||||||
|
has fully completed is a silent 204 no-op — there is no in-flight
|
||||||
|
task to cancel, the registry is empty, and the route must not error.
|
||||||
|
|
||||||
|
Pins the cancel endpoint's robustness against the common-but-racy
|
||||||
|
sequence where the user clicks Stop just after the stream finished
|
||||||
|
(the SSE channel hasn't yet flipped the client-side ``isStreaming``
|
||||||
|
flag). This is a complement to the snapshot test: the snapshot test
|
||||||
|
pins that the registry IS populated mid-flight, this test pins that
|
||||||
|
it isn't AFTER and that the route copes gracefully.
|
||||||
|
"""
|
||||||
|
from chat.web.turns import _in_flight_tasks
|
||||||
|
|
||||||
|
_seed_meanwhile_chat(tmp_path / "test.db")
|
||||||
|
canned_parse = json.dumps(
|
||||||
|
{"segments": [{"kind": "narration", "text": "they exchange a glance"}]}
|
||||||
|
)
|
||||||
|
canned = [
|
||||||
|
canned_parse,
|
||||||
|
"BotA leans in. *quietly*",
|
||||||
|
_zero_state(),
|
||||||
|
_zero_state(),
|
||||||
|
]
|
||||||
|
mock = _override_llm(canned)
|
||||||
|
try:
|
||||||
|
response = app_state_setup.post(
|
||||||
|
"/chats/chat_bot_a/turns",
|
||||||
|
data={"prose": "they exchange a glance"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 204
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
assert mock._canned == []
|
||||||
|
|
||||||
|
# Registry was cleaned up after the stream completed.
|
||||||
|
assert "chat_bot_a" not in _in_flight_tasks
|
||||||
|
|
||||||
|
# Cancel after-the-fact: 204, no error, registry stays empty.
|
||||||
|
cancel_response = app_state_setup.post(
|
||||||
|
"/chats/chat_bot_a/turns/cancel"
|
||||||
|
)
|
||||||
|
assert cancel_response.status_code == 204
|
||||||
|
assert "chat_bot_a" not in _in_flight_tasks
|
||||||
|
|||||||
@@ -16,6 +16,7 @@ from chat.eventlog.log import append_event
|
|||||||
from chat.eventlog.projector import project
|
from chat.eventlog.projector import project
|
||||||
from chat.state.memory import search_memories
|
from chat.state.memory import search_memories
|
||||||
import chat.state.memory # noqa: F401 (registers memory_written handler)
|
import chat.state.memory # noqa: F401 (registers memory_written handler)
|
||||||
|
import chat.state.embeddings # noqa: F401 (registers embedding_indexed handler)
|
||||||
|
|
||||||
|
|
||||||
def _seed(db, *, memory_specs):
|
def _seed(db, *, memory_specs):
|
||||||
@@ -159,3 +160,216 @@ def test_significance_bias_is_constant_module_level():
|
|||||||
# Must be non-negative -- a negative bias would invert the desired
|
# Must be non-negative -- a negative bias would invert the desired
|
||||||
# "higher significance ranks higher" semantics.
|
# "higher significance ranks higher" semantics.
|
||||||
assert SIGNIFICANCE_RANK_BIAS >= 0
|
assert SIGNIFICANCE_RANK_BIAS >= 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T96 (Phase 4): combined FTS + vector retrieval ranking via reciprocal-rank
|
||||||
|
# fusion. The fused path activates only when ``query_vector`` is provided —
|
||||||
|
# the no-vector path (above) is unchanged.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _one_hot(dim: int, idx: int) -> list[float]:
|
||||||
|
v = [0.0] * dim
|
||||||
|
v[idx] = 1.0
|
||||||
|
return v
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_memories_with_optional_embeddings(db, *, memory_specs):
|
||||||
|
"""Like ``_seed`` but also projects ``embedding_indexed`` events for any
|
||||||
|
spec carrying a ``vector`` key.
|
||||||
|
|
||||||
|
Memory rows are assigned ids in the order their ``memory_written`` events
|
||||||
|
were appended (the ``memories.id`` column is an autoincrementing primary
|
||||||
|
key), so we predict ``memory_id = i + 1`` per spec and append both kinds
|
||||||
|
of events back-to-back BEFORE projecting. Projecting only once keeps the
|
||||||
|
INSERT-based ``memory_written`` handler from duplicating rows.
|
||||||
|
"""
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# First pass: append every memory_written event in order. The DB
|
||||||
|
# assigns autoincrementing ids 1..N matching the order of these
|
||||||
|
# events, so we can pair vectors to memories by index.
|
||||||
|
for spec in memory_specs:
|
||||||
|
payload = {
|
||||||
|
"owner_id": spec.get("owner_id", "bot_a"),
|
||||||
|
"chat_id": spec.get("chat_id", "chat_bot_a"),
|
||||||
|
"pov_summary": spec["pov_summary"],
|
||||||
|
"witness_you": spec.get("witness_you", 1),
|
||||||
|
"witness_host": spec.get("witness_host", 1),
|
||||||
|
"witness_guest": spec.get("witness_guest", 0),
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": spec.get("significance", 1),
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
}
|
||||||
|
append_event(conn, kind="memory_written", payload=payload)
|
||||||
|
# Second pass: append embedding_indexed events for any spec that
|
||||||
|
# supplied a vector, using the predicted memory id.
|
||||||
|
for i, spec in enumerate(memory_specs, start=1):
|
||||||
|
if "vector" not in spec:
|
||||||
|
continue
|
||||||
|
vec = spec["vector"]
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": i,
|
||||||
|
"vector": list(vec),
|
||||||
|
"model": "test-model",
|
||||||
|
"dim": len(vec),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Single projection — avoids the memory_written handler INSERTing
|
||||||
|
# the same row twice on a re-projection.
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_memories_without_query_vector_uses_fts_only(tmp_path):
|
||||||
|
"""Regression: omitting ``query_vector`` keeps the existing FTS-only path.
|
||||||
|
|
||||||
|
Identical seed to ``test_search_higher_significance_ranks_above_lower``
|
||||||
|
but pinned to the no-vector code path explicitly (no kwarg passed).
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
_seed(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
{"pov_summary": "small promise"},
|
||||||
|
{"pov_summary": "huge promise"},
|
||||||
|
{"pov_summary": "tiny promise", "significance": 3},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_memories(conn, "bot_a", "host", "promise", k=3)
|
||||||
|
assert len(out) == 3
|
||||||
|
# The composite re-rank surfaces the high-significance row first.
|
||||||
|
assert out[0]["pov_summary"] == "tiny promise"
|
||||||
|
# Sanity: the row shape still carries ``fts_rank`` + ``composite_score``
|
||||||
|
# like the FTS-only path always has.
|
||||||
|
assert "fts_rank" in out[0]
|
||||||
|
assert "composite_score" in out[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_memories_with_query_vector_includes_vector_hits(tmp_path):
|
||||||
|
"""RRF fuses FTS hits with vector hits — both kinds surface in the result.
|
||||||
|
|
||||||
|
Memory 1 only matches FTS (keyword "rabbit", embedding far from query).
|
||||||
|
Memory 2 only matches the vector (embedding identical to query, no
|
||||||
|
keyword overlap). Memories 3-5 are unrelated. The fused top-K must
|
||||||
|
contain BOTH memory 1 and memory 2.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
dim = 8
|
||||||
|
# Query vector = one-hot at index 0. Memory 2 mirrors it exactly. The
|
||||||
|
# FTS-only memory (memory 1) has NO embedding so it cannot leak into
|
||||||
|
# the vector ranking; the filler memories (3-5) likewise have no
|
||||||
|
# embeddings, so the vector ranking returns memory 2 alone.
|
||||||
|
query_vec = _one_hot(dim, 0)
|
||||||
|
_seed_memories_with_optional_embeddings(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
# Memory 1: FTS-only match. No embedding indexed.
|
||||||
|
{"pov_summary": "rabbit hopped over the fence"},
|
||||||
|
# Memory 2: vector-only match. No keyword overlap with "rabbit".
|
||||||
|
{
|
||||||
|
"pov_summary": "completely unrelated narrative line",
|
||||||
|
"vector": _one_hot(dim, 0),
|
||||||
|
},
|
||||||
|
# Memories 3-5: filler, irrelevant to both channels.
|
||||||
|
{"pov_summary": "lighthouse keeper polished the lens"},
|
||||||
|
{"pov_summary": "they discussed cartography for hours"},
|
||||||
|
{"pov_summary": "she taught him semaphore signals"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_memories(
|
||||||
|
conn,
|
||||||
|
"bot_a",
|
||||||
|
"host",
|
||||||
|
"rabbit",
|
||||||
|
k=4,
|
||||||
|
query_vector=query_vec,
|
||||||
|
)
|
||||||
|
summaries = [r["pov_summary"] for r in out]
|
||||||
|
# FTS-only candidate (memory 1) made it through.
|
||||||
|
assert "rabbit hopped over the fence" in summaries
|
||||||
|
# Vector-only candidate (memory 2) also made it through despite
|
||||||
|
# having no keyword overlap with the query string.
|
||||||
|
assert "completely unrelated narrative line" in summaries
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_memories_fusion_significance_bias_still_applies(tmp_path):
|
||||||
|
"""With two RRF-tied candidates, the higher-significance one ranks first.
|
||||||
|
|
||||||
|
Two memories share the keyword "promise" AND share an identical
|
||||||
|
embedding to the query — so their FTS rank and vector rank are both
|
||||||
|
ties. RRF gives them the same fusion score. The Python-side
|
||||||
|
significance + recency boost must break the tie in favour of the
|
||||||
|
higher-significance memory.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
dim = 4
|
||||||
|
shared_vec = _one_hot(dim, 0)
|
||||||
|
_seed_memories_with_optional_embeddings(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
{
|
||||||
|
"pov_summary": "she made a promise",
|
||||||
|
"significance": 0,
|
||||||
|
"vector": list(shared_vec),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"pov_summary": "she made a promise",
|
||||||
|
"significance": 3,
|
||||||
|
"vector": list(shared_vec),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_memories(
|
||||||
|
conn,
|
||||||
|
"bot_a",
|
||||||
|
"host",
|
||||||
|
"promise",
|
||||||
|
k=2,
|
||||||
|
query_vector=list(shared_vec),
|
||||||
|
)
|
||||||
|
assert len(out) == 2
|
||||||
|
# Higher significance breaks the RRF tie.
|
||||||
|
assert out[0]["significance"] == 3
|
||||||
|
assert out[1]["significance"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_memories_fusion_handles_empty_vector_results(tmp_path):
|
||||||
|
"""Vector path returning [] (no embeddings indexed) must not break FTS.
|
||||||
|
|
||||||
|
No ``embedding_indexed`` events are projected, so ``vector_search``
|
||||||
|
returns an empty list. The function should still return the FTS hits
|
||||||
|
as if ``query_vector`` had not been supplied.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
_seed(
|
||||||
|
db,
|
||||||
|
memory_specs=[
|
||||||
|
{"pov_summary": "the vault held an old promise"},
|
||||||
|
{"pov_summary": "another promise was kept that night"},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = search_memories(
|
||||||
|
conn,
|
||||||
|
"bot_a",
|
||||||
|
"host",
|
||||||
|
"promise",
|
||||||
|
k=4,
|
||||||
|
query_vector=[0.0] * 384, # No embeddings exist for this owner.
|
||||||
|
)
|
||||||
|
# Both FTS hits still come back — no error from the empty vector path.
|
||||||
|
assert len(out) == 2
|
||||||
|
summaries = {r["pov_summary"] for r in out}
|
||||||
|
assert summaries == {
|
||||||
|
"the vault held an old promise",
|
||||||
|
"another promise was kept that night",
|
||||||
|
}
|
||||||
|
|||||||
+201
-3
@@ -22,7 +22,7 @@ from chat.db.migrate import apply_migrations
|
|||||||
from chat.eventlog.log import append_event
|
from chat.eventlog.log import append_event
|
||||||
from chat.eventlog.projector import project
|
from chat.eventlog.projector import project
|
||||||
from chat.llm.mock import MockLLMClient
|
from chat.llm.mock import MockLLMClient
|
||||||
from chat.services.memory_write import record_turn_memory, record_turn_memory_for_present
|
from chat.services.memory_write import record_turn_memory_for_present
|
||||||
import chat.state.entities # noqa: F401 - register handlers
|
import chat.state.entities # noqa: F401 - register handlers
|
||||||
import chat.state.memory # noqa: F401
|
import chat.state.memory # noqa: F401
|
||||||
import chat.state.world # noqa: F401
|
import chat.state.world # noqa: F401
|
||||||
@@ -64,14 +64,19 @@ def test_record_turn_memory_writes_event_and_projects(tmp_path):
|
|||||||
apply_migrations(db)
|
apply_migrations(db)
|
||||||
_seed_minimal(db)
|
_seed_minimal(db)
|
||||||
with open_db(db) as conn:
|
with open_db(db) as conn:
|
||||||
eid, mid = record_turn_memory(
|
# T90.3: legacy ``record_turn_memory`` was removed; the unified
|
||||||
|
# ``record_turn_memory_for_present`` with ``guest_bot_id=None``
|
||||||
|
# produces the same single-bot witness mask [1,1,0].
|
||||||
|
result = record_turn_memory_for_present(
|
||||||
conn,
|
conn,
|
||||||
chat_id="chat_bot_a",
|
chat_id="chat_bot_a",
|
||||||
host_bot_id="bot_a",
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id=None,
|
||||||
narrative_text="BotA looks up. 'You're back late.'",
|
narrative_text="BotA looks up. 'You're back late.'",
|
||||||
scene_id=None,
|
scene_id=None,
|
||||||
chat_clock_at="2026-04-26T20:00:00+00:00",
|
chat_clock_at="2026-04-26T20:00:00+00:00",
|
||||||
)
|
)
|
||||||
|
eid, mid = result["bot_a"]
|
||||||
assert eid > 0
|
assert eid > 0
|
||||||
assert mid is not None and mid > 0
|
assert mid is not None and mid > 0
|
||||||
|
|
||||||
@@ -111,12 +116,15 @@ def test_record_turn_memory_omits_optional_fields(tmp_path):
|
|||||||
_seed_minimal(db)
|
_seed_minimal(db)
|
||||||
with open_db(db) as conn:
|
with open_db(db) as conn:
|
||||||
# Call without scene_id/chat_clock_at — should default to None.
|
# Call without scene_id/chat_clock_at — should default to None.
|
||||||
eid, mid = record_turn_memory(
|
# T90.3: migrated from legacy ``record_turn_memory``.
|
||||||
|
result = record_turn_memory_for_present(
|
||||||
conn,
|
conn,
|
||||||
chat_id="chat_bot_a",
|
chat_id="chat_bot_a",
|
||||||
host_bot_id="bot_a",
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id=None,
|
||||||
narrative_text="A simple memory.",
|
narrative_text="A simple memory.",
|
||||||
)
|
)
|
||||||
|
eid, mid = result["bot_a"]
|
||||||
assert eid > 0
|
assert eid > 0
|
||||||
assert mid is not None and mid > 0
|
assert mid is not None and mid > 0
|
||||||
|
|
||||||
@@ -444,3 +452,193 @@ def test_record_for_present_dict_keys_match(tmp_path):
|
|||||||
narrative_text="Both bots witness this.",
|
narrative_text="Both bots witness this.",
|
||||||
)
|
)
|
||||||
assert set(result_with_guest.keys()) == {"bot_a", "bot_b"}
|
assert set(result_with_guest.keys()) == {"bot_a", "bot_b"}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T84: unified record_turn_memory_for_present API with you_present kwarg.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_turn_memory_you_present_false_writes_meanwhile_witness_mask(tmp_path):
|
||||||
|
"""When ``you_present=False`` the witness mask should be
|
||||||
|
``[you=0, host=1, guest=1]`` for both bots — the meanwhile shape."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_two_bots(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
result = record_turn_memory_for_present(
|
||||||
|
conn,
|
||||||
|
chat_id="chat_ab",
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id="bot_b",
|
||||||
|
narrative_text="BotA and BotB confer privately.",
|
||||||
|
scene_id=None,
|
||||||
|
chat_clock_at="2026-04-26T20:00:00+00:00",
|
||||||
|
you_present=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert set(result.keys()) == {"bot_a", "bot_b"}
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT owner_id, witness_you, witness_host, witness_guest "
|
||||||
|
"FROM memories ORDER BY owner_id"
|
||||||
|
).fetchall()
|
||||||
|
assert len(rows) == 2
|
||||||
|
for _owner, w_you, w_host, w_guest in rows:
|
||||||
|
assert w_you == 0
|
||||||
|
assert w_host == 1
|
||||||
|
assert w_guest == 1
|
||||||
|
|
||||||
|
# Two memory_written events were appended.
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM event_log WHERE kind = 'memory_written'"
|
||||||
|
)
|
||||||
|
assert cur.fetchone()[0] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_turn_memory_you_present_true_default_writes_normal_witness_mask(tmp_path):
|
||||||
|
"""Default ``you_present=True`` preserves Phase 2 behaviour:
|
||||||
|
``witness_you=1`` for the host POV row."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_minimal(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# No explicit you_present arg — should default to True.
|
||||||
|
result = record_turn_memory_for_present(
|
||||||
|
conn,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id=None,
|
||||||
|
narrative_text="BotA hums to herself.",
|
||||||
|
)
|
||||||
|
assert set(result.keys()) == {"bot_a"}
|
||||||
|
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT witness_you, witness_host, witness_guest "
|
||||||
|
"FROM memories WHERE owner_id = 'bot_a'"
|
||||||
|
).fetchone()
|
||||||
|
assert row is not None
|
||||||
|
w_you, w_host, w_guest = row
|
||||||
|
assert w_you == 1
|
||||||
|
assert w_host == 1
|
||||||
|
assert w_guest == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_turn_memory_you_present_false_requires_guest(tmp_path):
|
||||||
|
"""Calling with ``you_present=False`` and no ``guest_bot_id`` is a
|
||||||
|
programming error — meanwhile scenes always have both bots."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_minimal(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
with pytest.raises(ValueError, match="you_present=False requires guest_bot_id"):
|
||||||
|
record_turn_memory_for_present(
|
||||||
|
conn,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id=None,
|
||||||
|
narrative_text="invalid",
|
||||||
|
you_present=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T97: embedding-worker enqueue hook.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_turn_memory_enqueues_embedding_job(tmp_path):
|
||||||
|
"""When ``app.state.embedding_worker`` is wired, every per-witness
|
||||||
|
write enqueues an :class:`EmbeddingJob` carrying the freshly-projected
|
||||||
|
memory id and the narrative text. Two-bot turn -> two jobs."""
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
from chat.services.embedding_worker import EmbeddingJob
|
||||||
|
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_two_bots(db)
|
||||||
|
|
||||||
|
captured: list[EmbeddingJob] = []
|
||||||
|
|
||||||
|
class _StubWorker:
|
||||||
|
def enqueue(self, job: EmbeddingJob) -> None:
|
||||||
|
captured.append(job)
|
||||||
|
|
||||||
|
fake_app = SimpleNamespace(
|
||||||
|
state=SimpleNamespace(embedding_worker=_StubWorker())
|
||||||
|
)
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
result = record_turn_memory_for_present(
|
||||||
|
conn,
|
||||||
|
chat_id="chat_ab",
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id="bot_b",
|
||||||
|
narrative_text="Both bots witness this beat.",
|
||||||
|
app=fake_app,
|
||||||
|
)
|
||||||
|
|
||||||
|
# One job per witness — host first, then guest (matches result dict
|
||||||
|
# insertion order in record_turn_memory_for_present).
|
||||||
|
assert len(captured) == 2
|
||||||
|
expected_ids = {result["bot_a"][1], result["bot_b"][1]}
|
||||||
|
assert {job.memory_id for job in captured} == expected_ids
|
||||||
|
for job in captured:
|
||||||
|
assert job.text == "Both bots witness this beat."
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T109: memories.event_id deep-link column populated by the projector.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_memory_written_populates_event_id(tmp_path):
|
||||||
|
"""Schema 0014 added ``memories.event_id`` referencing ``event_log.id``.
|
||||||
|
|
||||||
|
The ``memory_written`` projector handler must populate the column with
|
||||||
|
the projecting event's id so T111 can deep-link cross-chat search hits
|
||||||
|
back to the originating turn.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_minimal(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
result = record_turn_memory_for_present(
|
||||||
|
conn,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id=None,
|
||||||
|
narrative_text="BotA shrugs.",
|
||||||
|
)
|
||||||
|
eid, mid = result["bot_a"]
|
||||||
|
assert eid > 0 and mid is not None
|
||||||
|
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT event_id FROM memories WHERE id = ?", (mid,)
|
||||||
|
).fetchone()
|
||||||
|
assert row is not None
|
||||||
|
assert row[0] == eid
|
||||||
|
|
||||||
|
|
||||||
|
def test_memory_event_id_column_is_nullable_for_backfill(tmp_path):
|
||||||
|
"""Backward compat: the ``event_id`` column is nullable so historical
|
||||||
|
memory rows projected before 0014 ran (or rows synthesised by tests
|
||||||
|
that bypass the projector) don't break the schema. A direct INSERT
|
||||||
|
omitting the column must succeed and read back NULL."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_minimal(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
conn.execute(
|
||||||
|
"INSERT INTO memories ("
|
||||||
|
"owner_id, chat_id, pov_summary, "
|
||||||
|
"witness_you, witness_host, witness_guest"
|
||||||
|
") VALUES (?, ?, ?, ?, ?, ?)",
|
||||||
|
("bot_a", "chat_bot_a", "legacy row", 1, 1, 0),
|
||||||
|
)
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT event_id FROM memories WHERE pov_summary = 'legacy row'"
|
||||||
|
).fetchone()
|
||||||
|
assert row is not None
|
||||||
|
assert row[0] is None
|
||||||
|
|||||||
@@ -1418,3 +1418,528 @@ def test_consumed_digest_does_not_render_again(tmp_path):
|
|||||||
body2 = msgs2[0].content
|
body2 = msgs2[0].content
|
||||||
assert "Meanwhile while you were away:" not in body2
|
assert "Meanwhile while you were away:" not in body2
|
||||||
assert digest_text not in body2
|
assert digest_text not in body2
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T80: scene_summarize polish bundle.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_scene_close_re_run_does_not_double_suffix(tmp_path):
|
||||||
|
"""T80.1: re-running ``apply_scene_close_summary`` on the same scene
|
||||||
|
must NOT stack a second "Key quotes:" suffix on each pov_summary. The
|
||||||
|
builder strips any existing suffix from candidate text before
|
||||||
|
composing the new one, and the per-POV write replaces (not appends
|
||||||
|
to) the existing suffix.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a heavy talk with you.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Things shifted.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
no_threads = json.dumps({"candidates": []})
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_single_bot_scene_no_memory(conn)
|
||||||
|
# Significance >= 2 triggers the Key quotes suffix path.
|
||||||
|
_seed_memory(conn, pov_summary="Maya quote one", significance=3)
|
||||||
|
_seed_memory(conn, pov_summary="Maya quote two", significance=2)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
# First close.
|
||||||
|
client = MockLLMClient(canned=[canned, no_threads])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT pov_summary FROM memories WHERE scene_id = 1"
|
||||||
|
).fetchall()
|
||||||
|
assert rows
|
||||||
|
for (pov,) in rows:
|
||||||
|
assert pov.count("Key quotes:") == 1
|
||||||
|
|
||||||
|
# Second close on the same scene with fresh canned responses.
|
||||||
|
client2 = MockLLMClient(canned=[canned, no_threads])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client2,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
rows2 = conn.execute(
|
||||||
|
"SELECT pov_summary FROM memories WHERE scene_id = 1"
|
||||||
|
).fetchall()
|
||||||
|
assert rows2
|
||||||
|
for (pov,) in rows2:
|
||||||
|
# Still exactly ONE "Key quotes:" suffix — no recursive bloat.
|
||||||
|
assert pov.count("Key quotes:") == 1
|
||||||
|
# And no nested-quote artifacts (the suffix wasn't sourced
|
||||||
|
# from a row whose text already contained the suffix).
|
||||||
|
inner_count = pov.count("Key quotes:")
|
||||||
|
assert inner_count == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_thread_detection_uses_scene_scoped_transcript(
|
||||||
|
tmp_path, monkeypatch
|
||||||
|
):
|
||||||
|
"""T80.2: when a chat has multiple closed scenes, the second scene's
|
||||||
|
close must hand ``detect_threads`` ONLY the second scene's turns —
|
||||||
|
not the chat-wide last-50, which would bleed in the first scene's
|
||||||
|
transcript and risk mis-closing threads."""
|
||||||
|
from chat.services import thread_detection as td_mod
|
||||||
|
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a quick chat.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Steady.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
captured_transcripts: list[list[dict]] = []
|
||||||
|
|
||||||
|
async def capturing_detect_threads(client, **kwargs):
|
||||||
|
captured_transcripts.append(list(kwargs["scene_transcript"]))
|
||||||
|
return td_mod.ThreadDetectionResult()
|
||||||
|
|
||||||
|
monkeypatch.setattr(td_mod, "detect_threads", capturing_detect_threads)
|
||||||
|
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Seed scene 1 + 3 turns + close.
|
||||||
|
_seed_single_bot_scene(conn)
|
||||||
|
# Add two extra distinct turns inside scene 1 so the transcript
|
||||||
|
# has clearly-scene-1 markers we can assert on.
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "SCENE_ONE_USER_TURN",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "SCENE_ONE_BOT_TURN",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": 2,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
# Close scene 1.
|
||||||
|
client = MockLLMClient(canned=[canned])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Open scene 2 with distinct dialogue. Use append_and_apply so
|
||||||
|
# the new events project incrementally without re-running the
|
||||||
|
# already-applied seed events.
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="scene_opened",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"container_id": 1,
|
||||||
|
"started_at": "2026-04-26T21:00:00+00:00",
|
||||||
|
"participants": ["you", "bot_a"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"scene_id": 2,
|
||||||
|
"pov_summary": "Original (scene 2)",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"significance": 1,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "SCENE_TWO_USER_TURN",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "SCENE_TWO_BOT_TURN",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": 3,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Close scene 2.
|
||||||
|
client2 = MockLLMClient(canned=[canned])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client2,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=2,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
# The second close's transcript holds only scene-2 markers.
|
||||||
|
assert len(captured_transcripts) == 2
|
||||||
|
scene_two_transcript = captured_transcripts[1]
|
||||||
|
joined = " ".join(t.get("text", "") for t in scene_two_transcript)
|
||||||
|
assert "SCENE_TWO" in joined
|
||||||
|
assert "SCENE_ONE" not in joined
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_detect_threads_failure_is_logged(tmp_path, monkeypatch, caplog):
|
||||||
|
"""T80.3: when ``detect_threads`` raises, the broad except must log
|
||||||
|
the failure at DEBUG so a programmer-error flap surfaces in local
|
||||||
|
logs even though the close pipeline keeps moving."""
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from chat.services import thread_detection as td_mod
|
||||||
|
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a quick chat.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Steady.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def boom(client, **kwargs):
|
||||||
|
raise RuntimeError("test-detect-threads-boom")
|
||||||
|
|
||||||
|
monkeypatch.setattr(td_mod, "detect_threads", boom)
|
||||||
|
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_single_bot_scene(conn)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
caplog.set_level(logging.DEBUG, logger="chat.services.scene_summarize")
|
||||||
|
client = MockLLMClient(canned=[canned])
|
||||||
|
# Close should NOT raise even though detect_threads did.
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Log carries the error message.
|
||||||
|
assert any(
|
||||||
|
"detect_threads failed" in rec.message
|
||||||
|
and "test-detect-threads-boom" in rec.message
|
||||||
|
for rec in caplog.records
|
||||||
|
), [r.message for r in caplog.records]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_thread_closed_uses_chat_clock_time(tmp_path, monkeypatch):
|
||||||
|
"""T80.4: emitted ``thread_closed`` events stamp ``closed_at`` with
|
||||||
|
the chat-clock time (chat["time"]), not the host's wall clock. The
|
||||||
|
rest of the close pipeline already does this; threads must agree
|
||||||
|
so timeline reconstruction stays consistent."""
|
||||||
|
from chat.services import thread_detection as td_mod
|
||||||
|
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a quick chat.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Steady.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def fake_detect_threads(client, **kwargs):
|
||||||
|
return td_mod.ThreadDetectionResult(
|
||||||
|
candidates=[
|
||||||
|
td_mod.ThreadCandidate(
|
||||||
|
action="close",
|
||||||
|
existing_thread_id="thr_x",
|
||||||
|
summary="resolved",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(td_mod, "detect_threads", fake_detect_threads)
|
||||||
|
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_single_bot_scene(conn)
|
||||||
|
# Pre-seed an open thread so the "close" candidate has something
|
||||||
|
# real to close, and pin the chat clock to a known value.
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
import chat.state.threads # noqa: F401
|
||||||
|
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="thread_opened",
|
||||||
|
payload={
|
||||||
|
"thread_id": "thr_x",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"title": "Lingering question",
|
||||||
|
"summary": "What did Maya hide?",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
# UPDATE chat_state AFTER project so the re-projection doesn't
|
||||||
|
# overwrite the pinned clock value.
|
||||||
|
chat_clock = "2026-04-26T10:00:00+00:00"
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE chat_state SET time = ? WHERE chat_id = ?",
|
||||||
|
(chat_clock, "chat_bot_a"),
|
||||||
|
)
|
||||||
|
|
||||||
|
client = MockLLMClient(canned=[canned])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log WHERE kind = 'thread_closed'"
|
||||||
|
).fetchall()
|
||||||
|
assert len(rows) == 1
|
||||||
|
payload = json.loads(rows[0][0])
|
||||||
|
assert payload["thread_id"] == "thr_x"
|
||||||
|
assert payload["closed_at"] == chat_clock
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T80.5: T58 coverage gaps (truncation, thread update/close emissions).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_key_quote_truncation_at_200_chars(tmp_path):
|
||||||
|
"""T80.5: when a memory's pov_summary exceeds 200 chars, the
|
||||||
|
Key-quote bullet truncates the source text to exactly 200 chars
|
||||||
|
(no ellipsis — a hard slice, per the existing T58 implementation)."""
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a heavy talk.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Things shifted.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
no_threads = json.dumps({"candidates": []})
|
||||||
|
long_text = "X" * 500 # 500 X's; expected slice is 200 X's.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_single_bot_scene_no_memory(conn)
|
||||||
|
_seed_memory(conn, pov_summary=long_text, significance=2)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
client = MockLLMClient(canned=[canned, no_threads])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
new_pov = conn.execute(
|
||||||
|
"SELECT pov_summary FROM memories WHERE scene_id = 1"
|
||||||
|
).fetchone()[0]
|
||||||
|
assert "Key quotes:" in new_pov
|
||||||
|
# The bullet should contain exactly 200 X's, not 500.
|
||||||
|
# Format from _build_key_quotes_suffix: ``- "<text>"``.
|
||||||
|
bullet_marker = '- "'
|
||||||
|
idx = new_pov.index(bullet_marker)
|
||||||
|
# Count consecutive X's after the bullet marker.
|
||||||
|
x_run = 0
|
||||||
|
for ch in new_pov[idx + len(bullet_marker):]:
|
||||||
|
if ch == "X":
|
||||||
|
x_run += 1
|
||||||
|
else:
|
||||||
|
break
|
||||||
|
assert x_run == 200, (
|
||||||
|
f"expected 200-char truncation, got {x_run}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_thread_detection_update_candidate_emits_thread_updated(
|
||||||
|
tmp_path, monkeypatch
|
||||||
|
):
|
||||||
|
"""T80.5: a detect_threads ``update`` candidate produces a
|
||||||
|
``thread_updated`` event with the candidate's summary and a
|
||||||
|
last_referenced_scene_id pointing at the closed scene."""
|
||||||
|
from chat.services import thread_detection as td_mod
|
||||||
|
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a quick chat.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Steady.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def fake_detect_threads(client, **kwargs):
|
||||||
|
return td_mod.ThreadDetectionResult(
|
||||||
|
candidates=[
|
||||||
|
td_mod.ThreadCandidate(
|
||||||
|
action="update",
|
||||||
|
existing_thread_id="thr_x",
|
||||||
|
summary="updated summary",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(td_mod, "detect_threads", fake_detect_threads)
|
||||||
|
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_single_bot_scene(conn)
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
import chat.state.threads # noqa: F401
|
||||||
|
|
||||||
|
# Pre-seed the open thread so the update has a row to target.
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="thread_opened",
|
||||||
|
payload={
|
||||||
|
"thread_id": "thr_x",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"title": "Lingering question",
|
||||||
|
"summary": "old summary",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
client = MockLLMClient(canned=[canned])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log WHERE kind = 'thread_updated'"
|
||||||
|
).fetchall()
|
||||||
|
assert len(rows) == 1
|
||||||
|
payload = json.loads(rows[0][0])
|
||||||
|
assert payload["thread_id"] == "thr_x"
|
||||||
|
assert payload["summary"] == "updated summary"
|
||||||
|
assert payload["last_referenced_scene_id"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_thread_detection_close_candidate_emits_thread_closed(
|
||||||
|
tmp_path, monkeypatch
|
||||||
|
):
|
||||||
|
"""T80.5: a detect_threads ``close`` candidate produces a
|
||||||
|
``thread_closed`` event for the existing thread."""
|
||||||
|
from chat.services import thread_detection as td_mod
|
||||||
|
|
||||||
|
canned = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA had a quick chat.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "Steady.",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
async def fake_detect_threads(client, **kwargs):
|
||||||
|
return td_mod.ThreadDetectionResult(
|
||||||
|
candidates=[
|
||||||
|
td_mod.ThreadCandidate(
|
||||||
|
action="close",
|
||||||
|
existing_thread_id="thr_x",
|
||||||
|
summary="resolved",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(td_mod, "detect_threads", fake_detect_threads)
|
||||||
|
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
_seed_single_bot_scene(conn)
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
import chat.state.threads # noqa: F401
|
||||||
|
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="thread_opened",
|
||||||
|
payload={
|
||||||
|
"thread_id": "thr_x",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"title": "Lingering question",
|
||||||
|
"summary": "open",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
client = MockLLMClient(canned=[canned])
|
||||||
|
await apply_scene_close_summary(
|
||||||
|
conn,
|
||||||
|
client,
|
||||||
|
classifier_model="x",
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
scene_id=1,
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log WHERE kind = 'thread_closed'"
|
||||||
|
).fetchall()
|
||||||
|
assert len(rows) == 1
|
||||||
|
payload = json.loads(rows[0][0])
|
||||||
|
assert payload["thread_id"] == "thr_x"
|
||||||
|
# closed_at field is present (T80.4 verifies its value).
|
||||||
|
assert "closed_at" in payload
|
||||||
|
|||||||
@@ -39,11 +39,11 @@ Cross-feature notes discovered while writing these tests:
|
|||||||
swallowed. Tests that don't care about thread coverage can omit the
|
swallowed. Tests that don't care about thread coverage can omit the
|
||||||
slot; test 2 includes a valid thread response to exercise the path.
|
slot; test 2 includes a valid thread response to exercise the path.
|
||||||
- ``consume_pending_meanwhile_digests`` is defined in chat.services.prompt
|
- ``consume_pending_meanwhile_digests`` is defined in chat.services.prompt
|
||||||
but is NOT currently wired into the post_turn flow. The digest stays
|
and is wired into the END of post_turn (after scene-close detection)
|
||||||
pending across turns until the helper is called explicitly. Test 4
|
by T82.1. Test 4 still drives the helper directly because it asserts
|
||||||
reflects this: it asserts the digest renders pre-consumption AND
|
the helper's contract in isolation (no post_turn round-trip in scope);
|
||||||
post-consumption (driven via the helper directly), and that the
|
the explicit call doubles as defensive coverage and is idempotent — a
|
||||||
meanwhile_digest_consumed event lands in the event_log.
|
second call on already-consumed digests is a no-op.
|
||||||
- The host-only ``apply_scene_close_summary`` canned queue layout is
|
- The host-only ``apply_scene_close_summary`` canned queue layout is
|
||||||
``[host_pov, thread_detection]`` (2 slots) when a single bot is present
|
``[host_pov, thread_detection]`` (2 slots) when a single bot is present
|
||||||
and there are dialogue rows, with thread_detection being optional /
|
and there are dialogue rows, with thread_detection being optional /
|
||||||
@@ -769,10 +769,11 @@ def test_meanwhile_close_digest_surfaces_then_consumed(
|
|||||||
— the digest is gone, and a meanwhile_digest_consumed event landed.
|
— the digest is gone, and a meanwhile_digest_consumed event landed.
|
||||||
|
|
||||||
Cross-feature finding: ``consume_pending_meanwhile_digests`` is
|
Cross-feature finding: ``consume_pending_meanwhile_digests`` is
|
||||||
defined in chat.services.prompt but is NOT wired into the post_turn
|
defined in chat.services.prompt and wired into post_turn by T82.1
|
||||||
flow. The digest stays pending across turns until callers invoke
|
(after scene-close detection). This test exercises the helper
|
||||||
the helper. Test exercises the helper directly so the consumption
|
directly so the consumption contract is pinned in isolation from
|
||||||
contract is pinned independent of any future post_turn integration.
|
the post_turn round-trip; T82.1's wiring is covered by a dedicated
|
||||||
|
test in tests/test_turn_flow.py.
|
||||||
|
|
||||||
Canned queue for the meanwhile turn:
|
Canned queue for the meanwhile turn:
|
||||||
1. parse_turn
|
1. parse_turn
|
||||||
|
|||||||
@@ -0,0 +1,891 @@
|
|||||||
|
"""Phase 4 cross-feature integration tests (T97 follow-up + T101).
|
||||||
|
|
||||||
|
Cross-feature flows for the Phase 4 retrieval + branching + drawer
|
||||||
|
features. Each test drives multiple Phase 4 surfaces end-to-end and
|
||||||
|
asserts both event_log and projected-state outcomes.
|
||||||
|
|
||||||
|
Test inventory:
|
||||||
|
|
||||||
|
* ``test_post_turn_embeddings_indexed_via_worker_hook`` (T97.5) —
|
||||||
|
pins the production turn route's ``app=request.app`` plumbing so
|
||||||
|
the embedding worker actually receives jobs.
|
||||||
|
|
||||||
|
T101 additions (the "Phase 4 cross-feature integration" suite):
|
||||||
|
|
||||||
|
1. ``test_vector_retrieval_feedback_loop`` — write a memory, drain
|
||||||
|
the embedding worker, assert the vector path retrieves it.
|
||||||
|
2. ``test_branch_diverge_main_intact`` — create a branch from a
|
||||||
|
mid-log turn, switch, append more events, switch back and assert
|
||||||
|
the original log past the branch point is still present (Phase 4
|
||||||
|
branching is metadata-only — no read-side filter yet).
|
||||||
|
3. ``test_surgical_delete_truncates_log_and_writes_snapshot`` —
|
||||||
|
compute impact, confirm via the drawer route, assert the log was
|
||||||
|
truncated and a pre-rewind snapshot landed on disk.
|
||||||
|
4. ``test_hide_then_unhide_round_trip_through_read_recent_dialogue``
|
||||||
|
— flip ``hidden`` via the drawer route both directions and assert
|
||||||
|
``read_recent_dialogue`` honours the flag in real time.
|
||||||
|
5. ``test_cross_chat_search_surfaces_memories_in_three_chats`` —
|
||||||
|
write memories in 3 chats, hit ``/search?q=...`` and assert all
|
||||||
|
three appear.
|
||||||
|
|
||||||
|
The T97.5 test monkeypatches ``app.state.embedding_worker.enqueue`` to
|
||||||
|
record jobs (rather than draining the worker) because the bug it pins
|
||||||
|
is "did the call site pass ``app`` at all". T101 test 1 takes the
|
||||||
|
opposite tack: it drives the worker for real to verify the entire
|
||||||
|
write -> index -> retrieve loop.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from chat.app import app
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.eventlog.log import append_and_apply, append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
from chat.llm.mock import MockLLMClient
|
||||||
|
|
||||||
|
|
||||||
|
def _zero_state() -> str:
|
||||||
|
return json.dumps(
|
||||||
|
{"affinity_delta": 0, "trust_delta": 0, "knowledge_facts": []}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _override_llm(canned: list[str]) -> MockLLMClient:
|
||||||
|
from chat.web.kickoff import get_llm_client
|
||||||
|
|
||||||
|
mock = MockLLMClient(canned=list(canned))
|
||||||
|
app.dependency_overrides[get_llm_client] = lambda: mock
|
||||||
|
return mock
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def app_state_setup(tmp_path, monkeypatch):
|
||||||
|
cfg = tmp_path / "config.toml"
|
||||||
|
cfg.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(cfg))
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(db))
|
||||||
|
with TestClient(app) as c:
|
||||||
|
# The background worker is disabled so the canned-response queue
|
||||||
|
# is consumed only by the request path. The embedding worker
|
||||||
|
# stays "started" but its loop won't observe the captured
|
||||||
|
# enqueues — we replace ``enqueue`` on the worker instance below.
|
||||||
|
app.state.background_worker.enabled = False
|
||||||
|
yield c
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def _seed(db_path: Path) -> None:
|
||||||
|
"""Mirror of ``tests/test_turn_flow.py::_seed`` — single bot + chat
|
||||||
|
+ edge + activities so the prompt assembler has something to render.
|
||||||
|
"""
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": "bot_a",
|
||||||
|
"name": "BotA",
|
||||||
|
"persona": "thoughtful, observant",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "...",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_bot_a",
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="edge_update",
|
||||||
|
payload={
|
||||||
|
"source_id": "bot_a",
|
||||||
|
"target_id": "you",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"knowledge_facts": ["coworker"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for entity_id, verb in [("you", "talking"), ("bot_a", "listening")]:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="activity_change",
|
||||||
|
payload={
|
||||||
|
"entity_id": entity_id,
|
||||||
|
"posture": "sitting",
|
||||||
|
"action": {
|
||||||
|
"verb": verb,
|
||||||
|
"interruptible": True,
|
||||||
|
"required_attention": "low",
|
||||||
|
"expected_duration": "ongoing",
|
||||||
|
},
|
||||||
|
"attention": "",
|
||||||
|
"holding": [],
|
||||||
|
"status": {},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_turn_embeddings_indexed_via_worker_hook(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""POST a turn; the route must pass ``app=request.app`` into
|
||||||
|
``record_turn_memory_for_present`` so the per-witness write enqueues
|
||||||
|
an :class:`EmbeddingJob` on ``app.state.embedding_worker``.
|
||||||
|
|
||||||
|
Without the T97.5 wiring this test fails: the call site previously
|
||||||
|
omitted ``app=`` and the helper's ``app is None`` branch silently
|
||||||
|
skipped every enqueue. We monkeypatch ``enqueue`` on the live
|
||||||
|
embedding worker (rather than draining the queue mid-request) so the
|
||||||
|
assertion does not depend on asyncio scheduling inside the
|
||||||
|
TestClient — the bug is in the wiring, and the wiring is what we
|
||||||
|
pin. The drain path is covered separately in
|
||||||
|
:mod:`tests.test_embedding_worker`.
|
||||||
|
"""
|
||||||
|
_seed(tmp_path / "test.db")
|
||||||
|
|
||||||
|
canned_parse = json.dumps(
|
||||||
|
{"segments": [{"kind": "dialogue", "text": "hello"}]}
|
||||||
|
)
|
||||||
|
_override_llm(
|
||||||
|
[canned_parse, "Hi there.", _zero_state(), _zero_state()]
|
||||||
|
)
|
||||||
|
|
||||||
|
captured: list = []
|
||||||
|
worker = app.state.embedding_worker
|
||||||
|
original_enqueue = worker.enqueue
|
||||||
|
worker.enqueue = captured.append # type: ignore[assignment]
|
||||||
|
try:
|
||||||
|
response = app_state_setup.post(
|
||||||
|
"/chats/chat_bot_a/turns", data={"prose": "hello"}
|
||||||
|
)
|
||||||
|
assert response.status_code == 204
|
||||||
|
finally:
|
||||||
|
worker.enqueue = original_enqueue # type: ignore[assignment]
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
|
||||||
|
# Single-bot turn -> one ``memory_written`` -> one EmbeddingJob.
|
||||||
|
# The job's ``memory_id`` should match the freshly-projected memory
|
||||||
|
# row, and its ``text`` should carry the assistant's narrative text.
|
||||||
|
assert len(captured) == 1
|
||||||
|
job = captured[0]
|
||||||
|
assert job.text == "Hi there."
|
||||||
|
|
||||||
|
with open_db(tmp_path / "test.db") as conn:
|
||||||
|
memory_ids = [
|
||||||
|
r[0]
|
||||||
|
for r in conn.execute(
|
||||||
|
"SELECT id FROM memories WHERE owner_id = ?",
|
||||||
|
("bot_a",),
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
assert job.memory_id in memory_ids
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# T101 — Phase 4 cross-feature integration suite.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
#
|
||||||
|
# Helpers + the five required scenarios. Each test drives multiple Phase 4
|
||||||
|
# features so a regression in any one of them fails an integration check.
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_minimal_chat(db_path: Path, chat_id: str = "chat_bot_a") -> None:
|
||||||
|
"""Seed bot_a, you, a chat, edges, and activities — same shape as
|
||||||
|
``tests/test_phase3_integration.py::_seed_single_bot_chat`` but
|
||||||
|
parameterised on chat_id so the cross-chat search test can stamp
|
||||||
|
several chats in the same database without renaming bots.
|
||||||
|
|
||||||
|
Uses ``append_and_apply`` rather than ``append_event`` + a final
|
||||||
|
``project`` so successive calls (e.g. one per chat in the
|
||||||
|
cross-chat-search test) don't try to re-project the cumulative
|
||||||
|
log and trip the ``chats.id`` UNIQUE constraint on the prior
|
||||||
|
chat's row.
|
||||||
|
"""
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
existing_bot = conn.execute(
|
||||||
|
"SELECT 1 FROM bots WHERE id = 'bot_a'"
|
||||||
|
).fetchone()
|
||||||
|
if existing_bot is None:
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": "bot_a",
|
||||||
|
"name": "BotA",
|
||||||
|
"persona": "thoughtful",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "...",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="you_authored",
|
||||||
|
payload={
|
||||||
|
"name": "Me",
|
||||||
|
"pronouns": "they/them",
|
||||||
|
"persona": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": chat_id,
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="edge_update",
|
||||||
|
payload={
|
||||||
|
"source_id": "bot_a",
|
||||||
|
"target_id": "you",
|
||||||
|
"chat_id": chat_id,
|
||||||
|
"knowledge_facts": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Activities are unique per (entity_id) — only seed them on the
|
||||||
|
# first call (when the bot row is also fresh).
|
||||||
|
if existing_bot is None:
|
||||||
|
for entity_id, verb in [
|
||||||
|
("you", "talking"),
|
||||||
|
("bot_a", "listening"),
|
||||||
|
]:
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="activity_change",
|
||||||
|
payload={
|
||||||
|
"entity_id": entity_id,
|
||||||
|
"posture": "sitting",
|
||||||
|
"action": {
|
||||||
|
"verb": verb,
|
||||||
|
"interruptible": True,
|
||||||
|
"required_attention": "low",
|
||||||
|
"expected_duration": "ongoing",
|
||||||
|
},
|
||||||
|
"attention": "",
|
||||||
|
"holding": [],
|
||||||
|
"status": {},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. Vector retrieval feedback loop.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_vector_retrieval_feedback_loop(tmp_path):
|
||||||
|
"""End-to-end: write a memory through
|
||||||
|
:func:`record_turn_memory_for_present` so an :class:`EmbeddingJob`
|
||||||
|
lands on a worker, drain the worker, then call
|
||||||
|
:func:`vector_search` with the SAME pseudo-embedding function and
|
||||||
|
assert the just-written memory is the top hit.
|
||||||
|
|
||||||
|
Why this test does NOT use the TestClient fixture: the live
|
||||||
|
``app.state.embedding_worker`` is created inside the FastAPI
|
||||||
|
lifespan's event loop. ``await``-ing on it from pytest-asyncio's
|
||||||
|
loop trips ``"got Future attached to a different loop"``. We
|
||||||
|
instead spin up a fresh :class:`EmbeddingWorker` in the test
|
||||||
|
loop, exactly mirroring ``tests/test_embedding_worker.py``'s
|
||||||
|
pattern. The T97.5 test above pins the wiring between the live
|
||||||
|
HTTP route and the live app worker; this test pins the
|
||||||
|
write -> index -> retrieve loop with no transport in scope.
|
||||||
|
|
||||||
|
Cross-feature gaps this test catches:
|
||||||
|
* Memory write enqueues to the worker but the worker never
|
||||||
|
drains (e.g. ``_run`` deadlock or sentinel mishandled).
|
||||||
|
* Worker uses a different embedding function than
|
||||||
|
``vector_search`` at query time, producing different vectors
|
||||||
|
and breaking cosine retrieval.
|
||||||
|
* ``embeddings`` projector handler is not registered (e.g.
|
||||||
|
import ordering bug) so the event fires but the table stays
|
||||||
|
empty.
|
||||||
|
"""
|
||||||
|
from types import SimpleNamespace
|
||||||
|
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.services.embedding_worker import EmbeddingWorker
|
||||||
|
from chat.services.embeddings import generate_embedding
|
||||||
|
from chat.services.memory_write import record_turn_memory_for_present
|
||||||
|
from chat.services.vector_search import vector_search
|
||||||
|
|
||||||
|
# Trigger projector handler registration. ``record_turn_memory_for_present``
|
||||||
|
# imports memory_write which imports the worker module, but the
|
||||||
|
# projector handlers live in ``chat.state.*`` modules and are
|
||||||
|
# registered as a side effect of import.
|
||||||
|
import chat.state.embeddings # noqa: F401
|
||||||
|
import chat.state.entities # noqa: F401
|
||||||
|
import chat.state.memory # noqa: F401
|
||||||
|
import chat.state.world # noqa: F401
|
||||||
|
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
_seed_minimal_chat(db)
|
||||||
|
|
||||||
|
# Spin up our own worker in the test event loop. ``client=None``
|
||||||
|
# is fine for the pseudo-embedding path — the local hash function
|
||||||
|
# does not require an LLM client.
|
||||||
|
worker = EmbeddingWorker(
|
||||||
|
conn_factory=lambda: open_db(db),
|
||||||
|
client=None,
|
||||||
|
)
|
||||||
|
await worker.start()
|
||||||
|
|
||||||
|
# Stub ``app`` — only ``app.state.embedding_worker`` is read by
|
||||||
|
# ``_write_one_memory``. SimpleNamespace gives us a stand-in that
|
||||||
|
# exposes ``state.embedding_worker`` without the full FastAPI app.
|
||||||
|
fake_app = SimpleNamespace(state=SimpleNamespace(embedding_worker=worker))
|
||||||
|
|
||||||
|
distinctive_text = "Maya watched the gondola lights drift across the lagoon."
|
||||||
|
with open_db(db) as conn:
|
||||||
|
record_turn_memory_for_present(
|
||||||
|
conn,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
host_bot_id="bot_a",
|
||||||
|
guest_bot_id=None,
|
||||||
|
narrative_text=distinctive_text,
|
||||||
|
app=fake_app,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Drain the worker via the sentinel. After this returns the
|
||||||
|
# ``embedding_indexed`` event has been projected.
|
||||||
|
await worker.stop()
|
||||||
|
|
||||||
|
# Generate a query embedding using the same function the worker
|
||||||
|
# used. The pseudo-embedding is deterministic so a query equal to
|
||||||
|
# the indexed text produces the identical vector and a cosine
|
||||||
|
# similarity of 1.0.
|
||||||
|
query_result = await generate_embedding(client=None, text=distinctive_text)
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
emb_count = conn.execute(
|
||||||
|
"SELECT COUNT(*) FROM embeddings"
|
||||||
|
).fetchone()[0]
|
||||||
|
assert emb_count == 1, (
|
||||||
|
"embedding worker did not project an embedding_indexed event"
|
||||||
|
)
|
||||||
|
|
||||||
|
hits = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="host", # bot_a is host, witness_host=1 by default
|
||||||
|
query_vector=query_result.vector,
|
||||||
|
k=4,
|
||||||
|
)
|
||||||
|
assert len(hits) == 1
|
||||||
|
top = hits[0]
|
||||||
|
assert top["pov_summary"] == distinctive_text
|
||||||
|
# Self-match: cosine of identical vectors is 1.0.
|
||||||
|
assert top["score"] == pytest.approx(1.0, abs=1e-9)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. Branch + diverge: main's post-branch tail stays intact (Phase 4
|
||||||
|
# branches are metadata-only).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_branch_diverge_main_intact(app_state_setup, tmp_path):
|
||||||
|
"""Append turns 1-12 on main, branch from turn 10's event_id, switch
|
||||||
|
to the new branch, append 3 more "play" turns, switch back to main,
|
||||||
|
assert the original turn 11+ events are untouched.
|
||||||
|
|
||||||
|
Phase 4's branches table is metadata-only — the read-side filter
|
||||||
|
isn't wired yet, so all events live in one log regardless of which
|
||||||
|
branch is "active". This test pins that contract: switching does
|
||||||
|
not mutate or hide existing events on either branch.
|
||||||
|
|
||||||
|
Canned LLM queue: none. ``user_turn`` / ``assistant_turn`` are
|
||||||
|
transcript-only kinds with no projector handler that needs an
|
||||||
|
LLM call, and ``branch_created`` / ``branch_switched`` are pure
|
||||||
|
state events. We use ``append_and_apply`` directly rather than
|
||||||
|
driving the HTTP turn route, which would require a 6-slot canned
|
||||||
|
queue per turn (parse + narrative + 2 state-updates + scene-close
|
||||||
|
+ memory) for 15 turns total = 90 slots of plumbing irrelevant to
|
||||||
|
the branch contract.
|
||||||
|
"""
|
||||||
|
from chat.services.branching import branch_from_event, switch_active_branch
|
||||||
|
from chat.state.branches import active_branch
|
||||||
|
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_minimal_chat(db)
|
||||||
|
|
||||||
|
# Append 12 user_turn / assistant_turn pairs on main. We collect
|
||||||
|
# the assistant_turn id at index 10 (1-based: "turn 10") so the
|
||||||
|
# branch fork point is unambiguous.
|
||||||
|
main_turn_ids: list[int] = []
|
||||||
|
with open_db(db) as conn:
|
||||||
|
for i in range(1, 13):
|
||||||
|
user_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": f"main turn {i}",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
asst_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": f"main reply {i}",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
main_turn_ids.append(asst_id)
|
||||||
|
turn_10_id = main_turn_ids[9]
|
||||||
|
|
||||||
|
# Snapshot the post-turn-10 main tail (turns 11, 12 + their
|
||||||
|
# user_turn predecessors) so we can byte-compare after the
|
||||||
|
# round-trip.
|
||||||
|
main_tail_before = conn.execute(
|
||||||
|
"SELECT id, kind, payload_json, hidden, superseded_by "
|
||||||
|
"FROM event_log WHERE id > ? ORDER BY id",
|
||||||
|
(turn_10_id,),
|
||||||
|
).fetchall()
|
||||||
|
assert len(main_tail_before) == 4 # 2 user + 2 assistant past turn 10
|
||||||
|
|
||||||
|
# Branch from turn 10. Phase 4's helper validates the origin
|
||||||
|
# event id exists and emits ``branch_created``.
|
||||||
|
branch_from_event(
|
||||||
|
conn,
|
||||||
|
name="experiment",
|
||||||
|
origin_event_id=turn_10_id,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
)
|
||||||
|
switch_active_branch(conn, name="experiment")
|
||||||
|
active = active_branch(conn)
|
||||||
|
assert active is not None and active["name"] == "experiment"
|
||||||
|
|
||||||
|
# Play 3 turns on the experiment branch.
|
||||||
|
for i in range(1, 4):
|
||||||
|
user_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": f"experiment turn {i}",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": f"experiment reply {i}",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Switch back to main.
|
||||||
|
switch_active_branch(conn, name="main")
|
||||||
|
active2 = active_branch(conn)
|
||||||
|
assert active2 is not None and active2["name"] == "main"
|
||||||
|
|
||||||
|
# Main's original tail past turn 10 is byte-identical: the
|
||||||
|
# branching events (branch_created, branch_switched x2) and the
|
||||||
|
# 3 experiment turns sit AFTER the original tail in event_log
|
||||||
|
# order, never overwriting it.
|
||||||
|
main_tail_after = conn.execute(
|
||||||
|
"SELECT id, kind, payload_json, hidden, superseded_by "
|
||||||
|
"FROM event_log "
|
||||||
|
"WHERE id > ? AND id <= ? ORDER BY id",
|
||||||
|
(turn_10_id, main_turn_ids[-1]),
|
||||||
|
).fetchall()
|
||||||
|
assert main_tail_after == main_tail_before
|
||||||
|
|
||||||
|
# The 6 experiment events (3 user + 3 assistant) all live in
|
||||||
|
# the same log past the original main tail. Verify their
|
||||||
|
# prose payloads to disambiguate from main's content.
|
||||||
|
diverged = conn.execute(
|
||||||
|
"SELECT kind, json_extract(payload_json, '$.prose'), "
|
||||||
|
" json_extract(payload_json, '$.text') "
|
||||||
|
"FROM event_log WHERE id > ? "
|
||||||
|
" AND kind IN ('user_turn', 'assistant_turn') ORDER BY id",
|
||||||
|
(main_turn_ids[-1],),
|
||||||
|
).fetchall()
|
||||||
|
assert len(diverged) == 6
|
||||||
|
prose_or_text = [(row[1] or row[2]) for row in diverged]
|
||||||
|
# Sequence: user1, asst1, user2, asst2, user3, asst3.
|
||||||
|
assert "experiment turn 1" in prose_or_text
|
||||||
|
assert "experiment reply 1" in prose_or_text
|
||||||
|
assert "experiment turn 3" in prose_or_text
|
||||||
|
assert "experiment reply 3" in prose_or_text
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. Surgical delete: impact preview -> confirm -> log truncated +
|
||||||
|
# pre-rewind snapshot saved.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_surgical_delete_truncates_log_and_writes_snapshot(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""Compute the delete-impact for a turn (read-only preview), then
|
||||||
|
confirm via the POST drawer route. Assert:
|
||||||
|
|
||||||
|
* The preview returns 200 + cascade markup.
|
||||||
|
* The event_log is physically truncated past ``target_id - 1``.
|
||||||
|
* A snapshot file lands under ``<data_dir>/snapshots/rewind/``.
|
||||||
|
* The pre-rewind snapshot's ``last_event_id`` matches the high
|
||||||
|
water mark BEFORE the truncate (so recovery can replay back to
|
||||||
|
pre-delete state).
|
||||||
|
|
||||||
|
Snapshot location: T97.5's ``data_dir`` derives from the db's
|
||||||
|
parent directory when ``CHAT_DATA_DIR`` is unset. The fixture
|
||||||
|
sets ``CHAT_DB_PATH = tmp_path / "test.db"`` so the snapshot
|
||||||
|
parent is ``tmp_path / "snapshots" / "rewind"``.
|
||||||
|
|
||||||
|
No canned LLM queue — the preview is pure SQL and the rewind path
|
||||||
|
is also pure SQL (delete + reproject). The drawer routes don't
|
||||||
|
invoke the LLM.
|
||||||
|
"""
|
||||||
|
import json as _json
|
||||||
|
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_minimal_chat(db)
|
||||||
|
|
||||||
|
# Append a small fixed turn sequence we can predict the cascade for.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
first_user = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "first message",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "first reply",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": first_user,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
target_user = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "this turn will be deleted",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
target_asst = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "and so will this reply",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": target_user,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# One trailing event past the target so we can verify the
|
||||||
|
# cascade catches >1 event.
|
||||||
|
trailing = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "trailing context",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
max_id_before = conn.execute(
|
||||||
|
"SELECT MAX(id) FROM event_log"
|
||||||
|
).fetchone()[0]
|
||||||
|
|
||||||
|
# ---- Preview: GET delete-preview returns 200 + the cascade list. ----
|
||||||
|
preview = app_state_setup.get(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/delete-preview/{target_user}"
|
||||||
|
)
|
||||||
|
assert preview.status_code == 200
|
||||||
|
body = preview.text
|
||||||
|
assert "delete-impact-modal" in body
|
||||||
|
assert f"Delete event {target_user}?" in body
|
||||||
|
assert "user_turn" in body
|
||||||
|
assert "assistant_turn" in body
|
||||||
|
# Confirm form points at the delete route.
|
||||||
|
assert f"/drawer/turn/delete/{target_user}" in body
|
||||||
|
|
||||||
|
# ---- Confirm: POST delete drops user, assistant, AND trailing. ----
|
||||||
|
confirm = app_state_setup.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/delete/{target_user}"
|
||||||
|
)
|
||||||
|
assert confirm.status_code == 200
|
||||||
|
|
||||||
|
# ---- Event log truncated past target_user - 1. ----
|
||||||
|
with open_db(db) as conn:
|
||||||
|
max_id_after = conn.execute(
|
||||||
|
"SELECT MAX(id) FROM event_log"
|
||||||
|
).fetchone()[0]
|
||||||
|
# delete_turn passes ``after_event_id = target_user - 1`` so
|
||||||
|
# everything from target_user forward is gone.
|
||||||
|
assert max_id_after == target_user - 1
|
||||||
|
for ev_id in (target_user, target_asst, trailing):
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT 1 FROM event_log WHERE id = ?", (ev_id,)
|
||||||
|
).fetchone()
|
||||||
|
assert row is None, f"event {ev_id} should have been deleted"
|
||||||
|
|
||||||
|
# ---- Pre-rewind snapshot landed on disk. ----
|
||||||
|
snapshot_dir = tmp_path / "snapshots" / "rewind"
|
||||||
|
assert snapshot_dir.exists(), (
|
||||||
|
f"snapshot dir not created: {snapshot_dir}"
|
||||||
|
)
|
||||||
|
snapshots = sorted(snapshot_dir.glob("*.json"))
|
||||||
|
assert len(snapshots) >= 1, (
|
||||||
|
f"no rewind snapshot written under {snapshot_dir}"
|
||||||
|
)
|
||||||
|
# Most-recent snapshot's last_event_id == pre-truncate high water
|
||||||
|
# mark, so a "restore" path could fully reverse the delete.
|
||||||
|
latest_snapshot = snapshots[-1]
|
||||||
|
snap_data = _json.loads(latest_snapshot.read_text())
|
||||||
|
assert snap_data["last_event_id"] == max_id_before
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. Hide + retrieval: drawer hide drops a turn from read_recent_dialogue,
|
||||||
|
# unhide restores it.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_hide_then_unhide_round_trip_through_read_recent_dialogue(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""Drive a hide -> read -> unhide -> read cycle through the drawer
|
||||||
|
HTTP route and assert ``read_recent_dialogue`` flips visibility
|
||||||
|
each step. T98.3 wires the route; T55 / turn_common owns the
|
||||||
|
``hidden = 0`` filter.
|
||||||
|
|
||||||
|
Cross-feature: the drawer HTTP handler emits a ``manual_edit``
|
||||||
|
event with branch ``turn_hidden``, the manual_edit projector
|
||||||
|
flips ``event_log.hidden``, and the prompt-window reader filters
|
||||||
|
on that column. Three layers — any one breaking would fail this
|
||||||
|
test.
|
||||||
|
|
||||||
|
No canned LLM queue — hide/unhide are pure SQL routes.
|
||||||
|
"""
|
||||||
|
from chat.services.turn_common import read_recent_dialogue
|
||||||
|
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_minimal_chat(db)
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
user_a = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "first user line",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
asst_a = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "first reply",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_a,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
user_b = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "second user line",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
asst_b = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "second reply",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": user_b,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Baseline: all 4 turns visible.
|
||||||
|
baseline = read_recent_dialogue(conn, "chat_bot_a", limit=10)
|
||||||
|
baseline_ids = {t["event_id"] for t in baseline}
|
||||||
|
assert {user_a, asst_a, user_b, asst_b} <= baseline_ids
|
||||||
|
|
||||||
|
# ---- Hide user_b via the drawer route. ----
|
||||||
|
hide_resp = app_state_setup.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/hide/{user_b}",
|
||||||
|
data={"hidden": "1"},
|
||||||
|
)
|
||||||
|
assert hide_resp.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# event_log.hidden flipped.
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT hidden FROM event_log WHERE id = ?", (user_b,)
|
||||||
|
).fetchone()
|
||||||
|
assert int(row[0]) == 1
|
||||||
|
|
||||||
|
# read_recent_dialogue drops user_b but keeps the others.
|
||||||
|
after_hide = read_recent_dialogue(conn, "chat_bot_a", limit=10)
|
||||||
|
after_hide_ids = {t["event_id"] for t in after_hide}
|
||||||
|
assert user_b not in after_hide_ids
|
||||||
|
# The other 3 turns still surface.
|
||||||
|
assert {user_a, asst_a, asst_b} <= after_hide_ids
|
||||||
|
|
||||||
|
# ---- Unhide via the SAME route with hidden=0. ----
|
||||||
|
unhide_resp = app_state_setup.post(
|
||||||
|
f"/chats/chat_bot_a/drawer/turn/hide/{user_b}",
|
||||||
|
data={"hidden": "0"},
|
||||||
|
)
|
||||||
|
assert unhide_resp.status_code == 200
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
row = conn.execute(
|
||||||
|
"SELECT hidden FROM event_log WHERE id = ?", (user_b,)
|
||||||
|
).fetchone()
|
||||||
|
assert int(row[0]) == 0
|
||||||
|
|
||||||
|
# read_recent_dialogue restores user_b.
|
||||||
|
after_unhide = read_recent_dialogue(conn, "chat_bot_a", limit=10)
|
||||||
|
after_unhide_ids = {t["event_id"] for t in after_unhide}
|
||||||
|
assert {user_a, asst_a, user_b, asst_b} <= after_unhide_ids
|
||||||
|
|
||||||
|
# Two manual_edit events landed (one per toggle), each with the
|
||||||
|
# turn_hidden branch tag.
|
||||||
|
edits = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'manual_edit' "
|
||||||
|
" AND json_extract(payload_json, '$.target_kind') = 'turn_hidden' "
|
||||||
|
"ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
assert len(edits) == 2
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 5. Cross-chat search: memories across 3 chats all surface from /search.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_cross_chat_search_surfaces_memories_in_three_chats(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""Seed 3 chats each owned by bot_a (so the bot row exists for the
|
||||||
|
search route's display-name hydration), write a distinctive
|
||||||
|
memory in each, then GET ``/search?q=<distinctive>`` and assert
|
||||||
|
every chat appears as a result row.
|
||||||
|
|
||||||
|
Cross-feature: T93's :func:`search_all_memories` (no per-owner
|
||||||
|
filter) + T100's HTML route (display-name hydration via
|
||||||
|
``get_bot``/``get_chat``). The route's empty-query short-circuit
|
||||||
|
is incidentally exercised by the request setup but isn't the
|
||||||
|
focus.
|
||||||
|
|
||||||
|
No canned LLM queue — memory_written events are projected directly
|
||||||
|
via ``append_and_apply`` and the search route is pure SQL +
|
||||||
|
template rendering.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
# Three chats, all hosted by bot_a so bot_a is the owner of all
|
||||||
|
# three memories. _seed_minimal_chat skips the bot/you bootstrap
|
||||||
|
# after the first call so the cumulative seed is consistent.
|
||||||
|
chat_ids = ["chat_bot_a", "chat_bot_a_2", "chat_bot_a_3"]
|
||||||
|
for chat_id in chat_ids:
|
||||||
|
_seed_minimal_chat(db, chat_id=chat_id)
|
||||||
|
|
||||||
|
# Distinctive token — "wisteria" appears nowhere else in the seed.
|
||||||
|
distinctive = "wisteria"
|
||||||
|
with open_db(db) as conn:
|
||||||
|
for idx, chat_id in enumerate(chat_ids):
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": chat_id,
|
||||||
|
"pov_summary": (
|
||||||
|
f"the {distinctive} bloomed by the gate (chat {idx})"
|
||||||
|
),
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": 1,
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---- GET /search?q=wisteria -> all 3 chats appear as result rows. ----
|
||||||
|
response = app_state_setup.get(f"/search?q={distinctive}")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
|
||||||
|
# Each chat_id appears in a result link href, e.g.
|
||||||
|
# ``href="/chats/chat_bot_a"``. The template renders one
|
||||||
|
# ``<a class="search-result-link" href="/chats/{chat_id}">`` per
|
||||||
|
# row, so a substring match per chat is sufficient.
|
||||||
|
for chat_id in chat_ids:
|
||||||
|
assert f'href="/chats/{chat_id}"' in body, (
|
||||||
|
f"chat {chat_id} missing from /search results: {body!r}"
|
||||||
|
)
|
||||||
|
# The owner display name (BotA) renders for each row — verify >= 3
|
||||||
|
# occurrences so we know all 3 result rows hydrated, not just 1.
|
||||||
|
assert body.count("BotA") >= 3
|
||||||
|
|
||||||
|
# ---- Sanity: distractor query yields no results. ----
|
||||||
|
distractor_response = app_state_setup.get(
|
||||||
|
"/search?q=nonexistentterm12345"
|
||||||
|
)
|
||||||
|
assert distractor_response.status_code == 200
|
||||||
|
distractor_body = distractor_response.text
|
||||||
|
# The "no matches" empty-state copy fires.
|
||||||
|
assert "No matches" in distractor_body
|
||||||
|
for chat_id in chat_ids:
|
||||||
|
assert f'href="/chats/{chat_id}"' not in distractor_body
|
||||||
@@ -21,7 +21,7 @@ import chat.state.world # noqa: F401
|
|||||||
import chat.state.events # noqa: F401
|
import chat.state.events # noqa: F401
|
||||||
import chat.state.threads # noqa: F401
|
import chat.state.threads # noqa: F401
|
||||||
from chat.llm.client import Message
|
from chat.llm.client import Message
|
||||||
from chat.services.prompt import assemble_narrative_prompt
|
from chat.services.prompt import _witness_role_for, assemble_narrative_prompt
|
||||||
|
|
||||||
|
|
||||||
def _seed_basic(conn) -> None:
|
def _seed_basic(conn) -> None:
|
||||||
@@ -852,3 +852,10 @@ def test_assemble_with_open_thread_renders_block(tmp_path):
|
|||||||
body = msgs[0].content
|
body = msgs[0].content
|
||||||
assert "Open threads:" in body
|
assert "Open threads:" in body
|
||||||
assert "Maya's job hunt" in body
|
assert "Maya's job hunt" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_witness_role_for_none_host_returns_host():
|
||||||
|
assert _witness_role_for("bot_a", None) == "host"
|
||||||
|
# Sanity check: existing semantics preserved.
|
||||||
|
assert _witness_role_for("bot_a", "bot_a") == "host"
|
||||||
|
assert _witness_role_for("bot_a", "bot_b") == "guest"
|
||||||
|
|||||||
@@ -662,3 +662,363 @@ def test_regenerate_drops_interjection_when_classifier_returns_false(
|
|||||||
new_primary_payload = json.loads(cur[0][0])
|
new_primary_payload = json.loads(cur[0][0])
|
||||||
assert new_primary_payload["text"] == "New primary text."
|
assert new_primary_payload["text"] == "New primary text."
|
||||||
assert "interjection_of" not in new_primary_payload
|
assert "interjection_of" not in new_primary_payload
|
||||||
|
|
||||||
|
|
||||||
|
def test_regenerate_with_prior_lifecycle_logs_warning(tmp_path, monkeypatch, caplog):
|
||||||
|
"""T83.4: when the superseded assistant_turn already produced
|
||||||
|
lifecycle transitions (event_started / event_completed /
|
||||||
|
event_cancelled), regenerate emits a WARNING naming the un-rolled-
|
||||||
|
back transitions. Phase 3.5 documents the gap; the actual rollback
|
||||||
|
is Phase 4 work.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from chat.config import Settings
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_and_apply
|
||||||
|
from chat.services.regenerate import regenerate_assistant_turn
|
||||||
|
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
cfg = tmp_path / "config.toml"
|
||||||
|
cfg.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(cfg))
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(db_path))
|
||||||
|
apply_migrations(db_path)
|
||||||
|
|
||||||
|
_ut_id, at_id = _seed_with_one_turn(db_path)
|
||||||
|
|
||||||
|
# After the assistant_turn lands, simulate that the turn flow
|
||||||
|
# produced an event_completed transition. ``append_and_apply`` is
|
||||||
|
# the standard path so the events projection updates.
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="event_planned",
|
||||||
|
payload={
|
||||||
|
"event_id": "evt_x",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"kind": "story_event",
|
||||||
|
"props": {},
|
||||||
|
"planned_for": "2026-04-30T18:00:00+00:00",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="event_started",
|
||||||
|
payload={
|
||||||
|
"event_id": "evt_x",
|
||||||
|
"started_at": "2026-04-30T19:00:00+00:00",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
completed_id = append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="event_completed",
|
||||||
|
payload={
|
||||||
|
"event_id": "evt_x",
|
||||||
|
"completed_at": "2026-04-30T19:30:00+00:00",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert completed_id is not None
|
||||||
|
|
||||||
|
state_canned = json.dumps(
|
||||||
|
{"affinity_delta": 0, "trust_delta": 0, "knowledge_facts": []}
|
||||||
|
)
|
||||||
|
mock_client = MockLLMClient(
|
||||||
|
canned=["Refreshed reply.", state_canned, state_canned]
|
||||||
|
)
|
||||||
|
settings = Settings(featherless_api_key="test")
|
||||||
|
|
||||||
|
caplog.set_level(logging.WARNING, logger="chat.services.regenerate")
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
asyncio.run(
|
||||||
|
regenerate_assistant_turn(
|
||||||
|
conn,
|
||||||
|
mock_client,
|
||||||
|
settings=settings,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
original_assistant_event_id=at_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# The warning records the count and at least one of the affected
|
||||||
|
# event_log ids (event_started + event_completed = at minimum 2).
|
||||||
|
warnings = [
|
||||||
|
r for r in caplog.records if r.levelname == "WARNING"
|
||||||
|
]
|
||||||
|
matching = [w for w in warnings if "lifecycle transition" in w.getMessage()]
|
||||||
|
assert matching, (
|
||||||
|
"expected a WARNING about un-rolled-back lifecycle transitions; "
|
||||||
|
f"got: {[w.getMessage() for w in warnings]}"
|
||||||
|
)
|
||||||
|
msg = matching[0].getMessage()
|
||||||
|
# Reference the original superseded turn's id and the event_completed
|
||||||
|
# row's id.
|
||||||
|
assert str(at_id) in msg
|
||||||
|
assert str(completed_id) in msg
|
||||||
|
# T90.2: wording was tightened from "from superseded turn" to
|
||||||
|
# "at-or-after turn <id>" — when regenerating an OLDER turn, the
|
||||||
|
# listed transitions may include legitimate intervening-turn ones
|
||||||
|
# that stand on their own. The new phrasing avoids implying the
|
||||||
|
# warning's target turn directly authored every listed transition.
|
||||||
|
assert "at-or-after turn" in msg
|
||||||
|
assert "from superseded turn" not in msg
|
||||||
|
|
||||||
|
|
||||||
|
def test_regenerate_sibling_lookup_scoped_to_chat(tmp_path, monkeypatch):
|
||||||
|
"""T83.3: regenerate's sibling-interjection lookup is scoped to the
|
||||||
|
chat being regenerated.
|
||||||
|
|
||||||
|
Setup: TWO chats, each with a primary + interjection turn group whose
|
||||||
|
rows happen to share the same ``user_turn_id`` value (the projector
|
||||||
|
assigns event_log ids monotonically across the whole database, so
|
||||||
|
when each chat is seeded back-to-back the chat A primary lands on a
|
||||||
|
different ``user_turn_id`` than chat B's — but in older versions the
|
||||||
|
sibling query had no chat predicate, so it could in principle latch
|
||||||
|
onto a row from a different chat if ids collided in some unusual
|
||||||
|
flow). We construct the seeding so chat B's interjection has the
|
||||||
|
SAME ``interjection_of`` value as the chat A primary's speaker_id —
|
||||||
|
pre-T83.3 the global query could have picked it up.
|
||||||
|
|
||||||
|
Assert: regenerating the chat A primary leaves chat B's rows
|
||||||
|
untouched (no supersede), and the regenerated chat A turn group's
|
||||||
|
interjection (the only one regenerate should regenerate) has its
|
||||||
|
``regenerated_from`` pointing at the chat A original interjection,
|
||||||
|
not chat B's.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from chat.config import Settings
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.services import regenerate as regenerate_module
|
||||||
|
from chat.services.interjection import InterjectionDecision
|
||||||
|
from chat.services.regenerate import regenerate_assistant_turn
|
||||||
|
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
cfg = tmp_path / "config.toml"
|
||||||
|
cfg.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(cfg))
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(db_path))
|
||||||
|
apply_migrations(db_path)
|
||||||
|
|
||||||
|
# Seed chat A's interjection group.
|
||||||
|
a_ut_id, a_primary_id, a_interjection_id = _seed_with_interjection_group(
|
||||||
|
db_path
|
||||||
|
)
|
||||||
|
|
||||||
|
# Seed chat B with the same shape but a different chat_id and bot
|
||||||
|
# ids, then add an interjection group whose ``interjection_of``
|
||||||
|
# points at "bot_a" so a global (unscoped) query could collide.
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
for bot_id, name in (("bot_c", "BotC"), ("bot_d", "BotD")):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": bot_id,
|
||||||
|
"name": name,
|
||||||
|
"persona": "",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_other",
|
||||||
|
"host_bot_id": "bot_c",
|
||||||
|
"guest_bot_id": "bot_d",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
b_ut_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_other",
|
||||||
|
"prose": "different chat",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
b_primary_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_other",
|
||||||
|
"speaker_id": "bot_c",
|
||||||
|
"text": "Other primary.",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": b_ut_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# The chat B interjection's ``interjection_of`` references
|
||||||
|
# "bot_a" — the chat A primary's speaker. Pre-T83.3 the global
|
||||||
|
# sibling query could mis-match this row.
|
||||||
|
b_interjection_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_other",
|
||||||
|
"speaker_id": "bot_d",
|
||||||
|
"text": "Cross-chat noise.",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": b_ut_id,
|
||||||
|
"interjection_of": "bot_a",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Stub the interjection classifier to return True so the regenerate
|
||||||
|
# actively walks the sibling-discovery path.
|
||||||
|
async def _stub_should_interject(*_args, **_kwargs):
|
||||||
|
return InterjectionDecision(should_interject=True, reason="fired")
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
regenerate_module, "detect_interjection", _stub_should_interject
|
||||||
|
)
|
||||||
|
|
||||||
|
state_canned = json.dumps(
|
||||||
|
{"affinity_delta": 0, "trust_delta": 0, "knowledge_facts": []}
|
||||||
|
)
|
||||||
|
canned: list[str] = (
|
||||||
|
["New chat A primary."]
|
||||||
|
+ [state_canned] * 6
|
||||||
|
+ ["New chat A interjection."]
|
||||||
|
+ [state_canned] * 6
|
||||||
|
)
|
||||||
|
mock_client = MockLLMClient(canned=list(canned))
|
||||||
|
settings = Settings(featherless_api_key="test")
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
new_text = asyncio.run(
|
||||||
|
regenerate_assistant_turn(
|
||||||
|
conn,
|
||||||
|
mock_client,
|
||||||
|
settings=settings,
|
||||||
|
chat_id="chat_multi",
|
||||||
|
original_assistant_event_id=a_primary_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert new_text == "New chat A primary."
|
||||||
|
|
||||||
|
# Chat B rows are untouched — neither superseded nor referenced.
|
||||||
|
b_primary_super = conn.execute(
|
||||||
|
"SELECT superseded_by FROM event_log WHERE id = ?",
|
||||||
|
(b_primary_id,),
|
||||||
|
).fetchone()[0]
|
||||||
|
b_interjection_super = conn.execute(
|
||||||
|
"SELECT superseded_by FROM event_log WHERE id = ?",
|
||||||
|
(b_interjection_id,),
|
||||||
|
).fetchone()[0]
|
||||||
|
assert b_primary_super is None
|
||||||
|
assert b_interjection_super is None
|
||||||
|
|
||||||
|
# Chat A's regenerated interjection has its ``regenerated_from``
|
||||||
|
# pointing at chat A's original interjection — NOT chat B's.
|
||||||
|
cur = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'assistant_turn' "
|
||||||
|
" AND id NOT IN (?, ?, ?, ?) "
|
||||||
|
" AND superseded_by IS NULL",
|
||||||
|
(a_primary_id, a_interjection_id, b_primary_id, b_interjection_id),
|
||||||
|
).fetchall()
|
||||||
|
# Two new rows: regenerated primary + regenerated interjection.
|
||||||
|
assert len(cur) == 2
|
||||||
|
payloads = [json.loads(row[0]) for row in cur]
|
||||||
|
# Find the regenerated interjection (carries interjection_of).
|
||||||
|
new_interject_payloads = [
|
||||||
|
p for p in payloads if p.get("interjection_of")
|
||||||
|
]
|
||||||
|
assert len(new_interject_payloads) == 1
|
||||||
|
assert new_interject_payloads[0]["regenerated_from"] == a_interjection_id
|
||||||
|
# Pin chat scope on every new row.
|
||||||
|
for p in payloads:
|
||||||
|
assert p["chat_id"] == "chat_multi"
|
||||||
|
|
||||||
|
|
||||||
|
def test_regenerate_registers_task_in_in_flight_tasks(tmp_path, monkeypatch):
|
||||||
|
"""T83.1: regenerate's streaming Task is registered in the chat-keyed
|
||||||
|
``_in_flight_tasks`` dict so the /turns/cancel route can cancel a
|
||||||
|
mid-regenerate stream. Mirrors the meanwhile registration pattern
|
||||||
|
pinned by tests/test_meanwhile_turn_flow.py.
|
||||||
|
|
||||||
|
Snapshot pattern: a custom MockLLMClient subclass captures the
|
||||||
|
presence of the chat_id in ``_in_flight_tasks`` at the first stream
|
||||||
|
yield (when the regenerate coroutine is awaiting our generator and
|
||||||
|
the task is alive). Post-flight, the entry must be cleaned up so the
|
||||||
|
next regenerate / turn registers a fresh task.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
from typing import AsyncIterator, Sequence
|
||||||
|
|
||||||
|
from chat.config import Settings
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.llm.client import Message
|
||||||
|
from chat.services.regenerate import regenerate_assistant_turn
|
||||||
|
from chat.web.turns import _in_flight_tasks
|
||||||
|
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
cfg = tmp_path / "config.toml"
|
||||||
|
cfg.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(cfg))
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(db_path))
|
||||||
|
apply_migrations(db_path)
|
||||||
|
|
||||||
|
_ut_id, at_id = _seed_with_one_turn(db_path)
|
||||||
|
|
||||||
|
in_flight_snapshot: dict = {}
|
||||||
|
|
||||||
|
class _SnapshotMock(MockLLMClient):
|
||||||
|
async def stream(
|
||||||
|
self, messages: Sequence[Message], *, model: str, **params
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
text = self._canned.pop(0)
|
||||||
|
for i, ch in enumerate(text):
|
||||||
|
if i == 0:
|
||||||
|
in_flight_snapshot["present"] = (
|
||||||
|
"chat_bot_a" in _in_flight_tasks
|
||||||
|
)
|
||||||
|
in_flight_snapshot["task"] = _in_flight_tasks.get(
|
||||||
|
"chat_bot_a"
|
||||||
|
)
|
||||||
|
yield ch
|
||||||
|
|
||||||
|
state_canned = json.dumps(
|
||||||
|
{"affinity_delta": 0, "trust_delta": 0, "knowledge_facts": []}
|
||||||
|
)
|
||||||
|
mock_client = _SnapshotMock(
|
||||||
|
canned=["Refreshed reply.", state_canned, state_canned]
|
||||||
|
)
|
||||||
|
|
||||||
|
settings = Settings(featherless_api_key="test")
|
||||||
|
|
||||||
|
# Pre-condition: registry empty for this chat.
|
||||||
|
assert "chat_bot_a" not in _in_flight_tasks
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
new_text = asyncio.run(
|
||||||
|
regenerate_assistant_turn(
|
||||||
|
conn,
|
||||||
|
mock_client,
|
||||||
|
settings=settings,
|
||||||
|
chat_id="chat_bot_a",
|
||||||
|
original_assistant_event_id=at_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert new_text == "Refreshed reply."
|
||||||
|
|
||||||
|
# Mid-flight: the streaming task was present in the registry, and
|
||||||
|
# the captured value was an asyncio.Task.
|
||||||
|
assert in_flight_snapshot.get("present") is True, (
|
||||||
|
"_in_flight_tasks was empty at first yield — regenerate stream "
|
||||||
|
"isn't registering its task"
|
||||||
|
)
|
||||||
|
assert isinstance(in_flight_snapshot.get("task"), asyncio.Task)
|
||||||
|
# Post-flight: the entry has been cleaned up.
|
||||||
|
assert "chat_bot_a" not in _in_flight_tasks
|
||||||
|
|||||||
@@ -85,3 +85,26 @@ def test_render_prose_mixed_full_message():
|
|||||||
assert '<em class="action">looks up</em>' in out
|
assert '<em class="action">looks up</em>' in out
|
||||||
# The apostrophe in ``she's`` is HTML-escaped to ``'``.
|
# The apostrophe in ``she's`` is HTML-escaped to ``'``.
|
||||||
assert '<span class="ooc">((she's tired))</span>' in out
|
assert '<span class="ooc">((she's tired))</span>' in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_turn_html_stamps_event_id_when_provided():
|
||||||
|
"""T86 follow-up: when ``event_id`` is supplied the wrapper DIV
|
||||||
|
carries ``id="turn-<event_id>"`` so the chat-page
|
||||||
|
``turn_html_replace`` SSE handler can locate the prior turn DOM
|
||||||
|
node by id and swap it in-place. Without the id the handler's
|
||||||
|
``getElementById('turn-' + supersedes_id)`` lookup misses and
|
||||||
|
the regenerated turn appends instead of replaces.
|
||||||
|
"""
|
||||||
|
out = render_turn_html("BotA", "Hello.", role="bot", event_id=42)
|
||||||
|
assert 'id="turn-42"' in out
|
||||||
|
# The id must sit on the wrapper DIV, not somewhere nested inside.
|
||||||
|
assert out.startswith('<div id="turn-42" class="turn turn-bot">')
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_turn_html_omits_id_when_event_id_missing():
|
||||||
|
"""Legacy callers (no ``event_id`` passed) get a clean DIV with no
|
||||||
|
id attribute — preserves the pre-T86 fragment shape.
|
||||||
|
"""
|
||||||
|
out = render_turn_html("BotA", "Hello.", role="bot")
|
||||||
|
assert "id=" not in out
|
||||||
|
assert out.startswith('<div class="turn turn-bot">')
|
||||||
|
|||||||
@@ -0,0 +1,163 @@
|
|||||||
|
"""T100 (Phase 4): cross-chat search UX (top-bar + results page).
|
||||||
|
|
||||||
|
Verifies the FastAPI ``/search`` route that wraps T93's
|
||||||
|
``search_all_memories`` service:
|
||||||
|
|
||||||
|
* ``/search?q=...`` returns 200 + an HTML page that lists matches drawn
|
||||||
|
from MULTIPLE chats (not just the current one) and links each result
|
||||||
|
back to ``/chats/{chat_id}``.
|
||||||
|
* ``/search`` with no query renders the page in its empty state with a
|
||||||
|
"enter a query" placeholder and no result rows (avoids hitting the
|
||||||
|
FTS index with an invalid empty MATCH).
|
||||||
|
* Result links navigate to the originating chat so users can pick up
|
||||||
|
the thread where the memory came from.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from chat.app import app
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
import chat.state.memory # noqa: F401 (registers memory_written handler)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client(tmp_path, monkeypatch):
|
||||||
|
config_path = tmp_path / "config.toml"
|
||||||
|
config_path.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(config_path))
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(tmp_path / "test.db"))
|
||||||
|
with TestClient(app) as c:
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_two_chats_with_memories(db_path: Path) -> None:
|
||||||
|
"""Seed: a ``you_entity``, two bots, two chats, and one ``rabbit``
|
||||||
|
memory per chat. Two-chat seeding lets the cross-chat assertion
|
||||||
|
actually distinguish "both chats appear" from "only the current
|
||||||
|
one does"."""
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="you_authored",
|
||||||
|
payload={"name": "Me", "pronouns": "", "persona": ""},
|
||||||
|
)
|
||||||
|
for bot_id, chat_id in (("bot_a", "chat_a"), ("bot_b", "chat_b")):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": bot_id,
|
||||||
|
"name": bot_id.upper(),
|
||||||
|
"persona": "thoughtful",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "friend",
|
||||||
|
"kickoff_prose": "kickoff",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": chat_id,
|
||||||
|
"host_bot_id": bot_id,
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload={
|
||||||
|
"owner_id": bot_id,
|
||||||
|
"chat_id": chat_id,
|
||||||
|
"pov_summary": f"the rabbit darted across {chat_id}",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": 1,
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_returns_results_from_multiple_chats(client, tmp_path):
|
||||||
|
"""A single ``/search?q=rabbit`` must surface matches from BOTH
|
||||||
|
chats — the whole point of the cross-chat search box is that it
|
||||||
|
isn't owner-scoped."""
|
||||||
|
_seed_two_chats_with_memories(tmp_path / "test.db")
|
||||||
|
resp = client.get("/search?q=rabbit")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
body = resp.text
|
||||||
|
# Both chats' memory snippets must appear in the rendered page.
|
||||||
|
assert "chat_a" in body
|
||||||
|
assert "chat_b" in body
|
||||||
|
assert "rabbit" in body.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_query_renders_placeholder_not_results(client, tmp_path):
|
||||||
|
"""``/search`` with no query renders the page in its empty state.
|
||||||
|
|
||||||
|
The placeholder copy is a contract with the user — they should see
|
||||||
|
"enter a query" rather than an empty result list that looks like a
|
||||||
|
no-match. Also: the FTS short-circuit means there are no result
|
||||||
|
rows to leak into the body."""
|
||||||
|
_seed_two_chats_with_memories(tmp_path / "test.db")
|
||||||
|
resp = client.get("/search")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
body = resp.text.lower()
|
||||||
|
assert "enter a query" in body
|
||||||
|
# Seeded "rabbit" memories must NOT appear: empty query => no results.
|
||||||
|
assert "the rabbit darted" not in resp.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_result_links_navigate_to_chat(client, tmp_path):
|
||||||
|
"""Each result links back to its originating chat so the user can
|
||||||
|
reopen the thread where the memory was first witnessed."""
|
||||||
|
_seed_two_chats_with_memories(tmp_path / "test.db")
|
||||||
|
resp = client.get("/search?q=rabbit")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
# The link target is chat-level (memories don't carry an event_id
|
||||||
|
# column today, so we don't deep-link to a specific turn).
|
||||||
|
assert 'href="/chats/chat_a"' in resp.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_search_results_use_batched_lookups(client, tmp_path):
|
||||||
|
"""T106: hydration must not fan out to per-row ``get_bot``/
|
||||||
|
``get_chat``/``get_scene`` calls.
|
||||||
|
|
||||||
|
The previous implementation called each helper once per result row
|
||||||
|
(worst case 50 rows x 3 helpers = 150 individual queries). The
|
||||||
|
batched implementation collects distinct ids and issues at most one
|
||||||
|
query per entity kind via ``WHERE id IN (...)``, so the per-row
|
||||||
|
helpers should not be invoked at all when there are matches.
|
||||||
|
|
||||||
|
We seed two chats (so both ``get_bot`` and ``get_chat`` would have
|
||||||
|
been hit pre-T106) and assert each helper sees zero per-row calls.
|
||||||
|
"""
|
||||||
|
_seed_two_chats_with_memories(tmp_path / "test.db")
|
||||||
|
with (
|
||||||
|
patch("chat.web.search.get_bot") as mock_get_bot,
|
||||||
|
patch("chat.web.search.get_chat") as mock_get_chat,
|
||||||
|
patch("chat.web.search.get_scene") as mock_get_scene,
|
||||||
|
):
|
||||||
|
resp = client.get("/search?q=rabbit")
|
||||||
|
assert resp.status_code == 200
|
||||||
|
# Batched IN-list queries replace the per-row helpers entirely.
|
||||||
|
assert mock_get_bot.call_count == 0
|
||||||
|
assert mock_get_chat.call_count == 0
|
||||||
|
assert mock_get_scene.call_count == 0
|
||||||
@@ -98,6 +98,49 @@ class _RaisingMock:
|
|||||||
yield # pragma: no cover - make this a generator
|
yield # pragma: no cover - make this a generator
|
||||||
|
|
||||||
|
|
||||||
|
class _RecordingMock:
|
||||||
|
"""Mock LLMClient that records the kwargs passed to ``generate``.
|
||||||
|
|
||||||
|
Used to assert that callers plumb through optional parameters like
|
||||||
|
``timeout_s`` instead of swallowing them. Returns a fixed string so
|
||||||
|
the surrounding fallback path is not exercised.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.captured_kwargs: dict | None = None
|
||||||
|
|
||||||
|
async def generate(
|
||||||
|
self, messages: Sequence[Message], *, model: str, **params
|
||||||
|
) -> str:
|
||||||
|
self.captured_kwargs = dict(params)
|
||||||
|
return "ok"
|
||||||
|
|
||||||
|
async def stream(
|
||||||
|
self, messages: Sequence[Message], *, model: str, **params
|
||||||
|
) -> AsyncIterator[str]:
|
||||||
|
raise RuntimeError("not used")
|
||||||
|
yield # pragma: no cover - make this a generator
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_narrate_skip_passes_timeout_through():
|
||||||
|
mock = _RecordingMock()
|
||||||
|
await narrate_skip(
|
||||||
|
mock,
|
||||||
|
narrative_model="x",
|
||||||
|
skip_kind="jump",
|
||||||
|
speaker_bot=_SPEAKER,
|
||||||
|
you_name="Me",
|
||||||
|
current_time="late evening",
|
||||||
|
new_time="next morning",
|
||||||
|
current_activity="winding down for the night",
|
||||||
|
landing_state_hint="having coffee in the kitchen",
|
||||||
|
timeout_s=12.5,
|
||||||
|
)
|
||||||
|
assert mock.captured_kwargs is not None
|
||||||
|
assert mock.captured_kwargs.get("timeout_s") == 12.5
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_narrate_falls_back_on_generation_failure():
|
async def test_narrate_falls_back_on_generation_failure():
|
||||||
new_time = "next morning"
|
new_time = "next morning"
|
||||||
|
|||||||
@@ -0,0 +1,204 @@
|
|||||||
|
"""Tests for Task 99 — snapshot UX (manual trigger + list + restore + preview).
|
||||||
|
|
||||||
|
Phase 4 surfaces the existing snapshot infrastructure (Phase 1 T20 / T31)
|
||||||
|
through HTML routes so the user can:
|
||||||
|
|
||||||
|
* see what snapshots exist,
|
||||||
|
* take one on demand,
|
||||||
|
* restore one with a hard confirm,
|
||||||
|
* peek at metadata before restoring.
|
||||||
|
|
||||||
|
The underlying service API lives in ``chat/services/snapshot.py`` and is
|
||||||
|
already exercised by ``test_snapshot.py``; here we only verify the web
|
||||||
|
surface wires the existing functions correctly.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
|
||||||
|
from chat.app import app
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
|
||||||
|
|
||||||
|
def _bot_payload(bot_id: str, name: str) -> dict:
|
||||||
|
return {
|
||||||
|
"id": bot_id,
|
||||||
|
"name": name,
|
||||||
|
"persona": "fancy",
|
||||||
|
"voice_samples": ["sample"],
|
||||||
|
"traits": ["shy"],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "coworker",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def client(tmp_path, monkeypatch):
|
||||||
|
"""A TestClient whose db + data_dir live under ``tmp_path``.
|
||||||
|
|
||||||
|
``load_settings`` derives ``data_dir`` from ``CHAT_DB_PATH``'s parent
|
||||||
|
when ``CHAT_DATA_DIR`` is unset (see ``chat/config.py``), so this also
|
||||||
|
isolates the ``data/snapshots/`` tree to ``tmp_path``.
|
||||||
|
"""
|
||||||
|
config_path = tmp_path / "config.toml"
|
||||||
|
config_path.write_text('featherless_api_key = "test"\n')
|
||||||
|
monkeypatch.setenv("CHAT_CONFIG_PATH", str(config_path))
|
||||||
|
monkeypatch.setenv("CHAT_DB_PATH", str(tmp_path / "test.db"))
|
||||||
|
|
||||||
|
with TestClient(app) as c:
|
||||||
|
c.tmp_path = tmp_path # type: ignore[attr-defined]
|
||||||
|
yield c
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_bot(db_path: Path, bot_id: str = "bot_a", name: str = "BotA") -> None:
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(conn, kind="bot_authored", payload=_bot_payload(bot_id, name))
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
|
||||||
|
def _take_snapshot_via_service(
|
||||||
|
db_path: Path, data_dir: Path, kind: str = "periodic"
|
||||||
|
) -> Path:
|
||||||
|
from chat.services.snapshot import take_snapshot
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
return take_snapshot(conn, data_dir=data_dir, kind=kind)
|
||||||
|
|
||||||
|
|
||||||
|
def test_list_snapshots_renders_page(client, tmp_path):
|
||||||
|
_seed_bot(tmp_path / "test.db", "bot_a", "BotA")
|
||||||
|
# Take two snapshots through the service so the listing has rows.
|
||||||
|
p1 = _take_snapshot_via_service(tmp_path / "test.db", tmp_path, kind="periodic")
|
||||||
|
p2 = _take_snapshot_via_service(tmp_path / "test.db", tmp_path, kind="rewind")
|
||||||
|
|
||||||
|
response = client.get("/snapshots")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
# Both filenames should appear in the listing.
|
||||||
|
assert p1.stem in body
|
||||||
|
assert p2.stem in body
|
||||||
|
# Both kinds should be visible.
|
||||||
|
assert "periodic" in body
|
||||||
|
assert "rewind" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_take_snapshot_creates_new(client, tmp_path):
|
||||||
|
_seed_bot(tmp_path / "test.db", "bot_a", "BotA")
|
||||||
|
snapshot_dir = tmp_path / "snapshots" / "periodic"
|
||||||
|
|
||||||
|
before = (
|
||||||
|
len(list(snapshot_dir.glob("*.json"))) if snapshot_dir.exists() else 0
|
||||||
|
)
|
||||||
|
response = client.post("/snapshots/take", follow_redirects=False)
|
||||||
|
assert response.status_code == 303
|
||||||
|
assert response.headers["location"] == "/snapshots"
|
||||||
|
|
||||||
|
after = len(list(snapshot_dir.glob("*.json")))
|
||||||
|
assert after == before + 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_snapshot_with_correct_confirm(client, tmp_path):
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
_seed_bot(db_path, "bot_a", "BotA")
|
||||||
|
snapshot_path = _take_snapshot_via_service(
|
||||||
|
db_path, tmp_path, kind="periodic"
|
||||||
|
)
|
||||||
|
snapshot_id = snapshot_path.stem # filename without extension
|
||||||
|
|
||||||
|
# Mutate the DB after the snapshot was taken — restoring should erase
|
||||||
|
# the new bot.
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn, kind="bot_authored", payload=_bot_payload("bot_b", "BotB")
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
bots_before = conn.execute(
|
||||||
|
"SELECT id FROM bots ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
assert {r[0] for r in bots_before} == {"bot_a", "bot_b"}
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/snapshots/restore/{snapshot_id}",
|
||||||
|
data={"confirm_id": snapshot_id, "kind": "periodic"},
|
||||||
|
follow_redirects=False,
|
||||||
|
)
|
||||||
|
assert response.status_code == 303
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
bots_after = conn.execute(
|
||||||
|
"SELECT id FROM bots ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
# The post-snapshot bot should be gone.
|
||||||
|
assert {r[0] for r in bots_after} == {"bot_a"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_snapshot_wrong_confirm_400(client, tmp_path):
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
_seed_bot(db_path, "bot_a", "BotA")
|
||||||
|
snapshot_path = _take_snapshot_via_service(
|
||||||
|
db_path, tmp_path, kind="periodic"
|
||||||
|
)
|
||||||
|
snapshot_id = snapshot_path.stem
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/snapshots/restore/{snapshot_id}",
|
||||||
|
data={"confirm_id": "not_the_right_id", "kind": "periodic"},
|
||||||
|
follow_redirects=False,
|
||||||
|
)
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_restore_without_kind_returns_400(client, tmp_path):
|
||||||
|
"""T105: Missing or empty ``kind`` must be rejected with 400.
|
||||||
|
|
||||||
|
Previously ``kind`` defaulted to ``"periodic"``, which silently 404'd
|
||||||
|
when the caller meant a rewind snapshot. Tighten the contract so the
|
||||||
|
client must always pass an explicit, valid ``kind``.
|
||||||
|
"""
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
_seed_bot(db_path, "bot_a", "BotA")
|
||||||
|
snapshot_path = _take_snapshot_via_service(
|
||||||
|
db_path, tmp_path, kind="periodic"
|
||||||
|
)
|
||||||
|
snapshot_id = snapshot_path.stem
|
||||||
|
|
||||||
|
response = client.post(
|
||||||
|
f"/snapshots/restore/{snapshot_id}",
|
||||||
|
data={"confirm_id": snapshot_id}, # no `kind`
|
||||||
|
follow_redirects=False,
|
||||||
|
)
|
||||||
|
assert response.status_code == 400
|
||||||
|
|
||||||
|
|
||||||
|
def test_preview_renders_metadata(client, tmp_path):
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
_seed_bot(db_path, "bot_a", "BotA")
|
||||||
|
snapshot_path = _take_snapshot_via_service(
|
||||||
|
db_path, tmp_path, kind="periodic"
|
||||||
|
)
|
||||||
|
snapshot_id = snapshot_path.stem
|
||||||
|
|
||||||
|
# Append more events post-snapshot so the delta is non-zero.
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn, kind="bot_authored", payload=_bot_payload("bot_b", "BotB")
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
response = client.get(
|
||||||
|
f"/snapshots/{snapshot_id}/preview", params={"kind": "periodic"}
|
||||||
|
)
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
assert snapshot_id in body
|
||||||
|
# Snapshot's last_event_id and current event_log size should appear.
|
||||||
|
dump = json.loads(snapshot_path.read_text())
|
||||||
|
assert str(dump["last_event_id"]) in body
|
||||||
@@ -174,3 +174,74 @@ def test_chat_html_includes_stop_streaming_script(client, tmp_path):
|
|||||||
assert "stop-streaming" in body or "isStreaming" in body
|
assert "stop-streaming" in body or "isStreaming" in body
|
||||||
# Cancel route reference must be wired so the Stop button can call it.
|
# Cancel route reference must be wired so the Stop button can call it.
|
||||||
assert "/turns/cancel" in body
|
assert "/turns/cancel" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_chat_html_has_turn_html_replace_listener(client, tmp_path):
|
||||||
|
"""T86: the chat shell wires a JS handler for the ``turn_html_replace``
|
||||||
|
SSE event so regenerate-driven swaps land in connected tabs without a
|
||||||
|
page refresh.
|
||||||
|
|
||||||
|
This is a presence / string-check test: it verifies the handler is
|
||||||
|
embedded in the rendered template but does NOT drive a real browser
|
||||||
|
(no headless runner is wired into this test environment). The end-to-
|
||||||
|
end behaviour — receiving the event over SSE and replacing the prior
|
||||||
|
turn's DOM node — is therefore not exercised here; a manual smoke
|
||||||
|
check or future browser-driven test would close that gap.
|
||||||
|
"""
|
||||||
|
_seed_chat(tmp_path / "test.db")
|
||||||
|
response = client.get("/chats/chat_bot_a")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
# The handler must be wired against the SSE event name the backend
|
||||||
|
# publishes (chat.services.regenerate -> "turn_html_replace").
|
||||||
|
assert "turn_html_replace" in body
|
||||||
|
# Confirm the handler reads the JSON payload's ``supersedes_id`` so
|
||||||
|
# it can locate the prior turn node. The exact lookup mechanism may
|
||||||
|
# vary, but the field name is part of the contract with the backend.
|
||||||
|
assert "supersedes_id" in body
|
||||||
|
|
||||||
|
|
||||||
|
def test_rendered_turn_html_includes_event_id(client, tmp_path):
|
||||||
|
"""T86 follow-up: the chat-detail Jinja loop stamps
|
||||||
|
``id="turn-<event_id>"`` on every rendered turn DIV. Without this id
|
||||||
|
the ``turn_html_replace`` SSE handler's ``getElementById`` lookup
|
||||||
|
misses, falls through to ``insertAdjacentHTML('beforeend', …)``, and
|
||||||
|
the regenerated turn appears APPENDED instead of swapped in-place
|
||||||
|
(rendering the primary handler path dead code — exactly the gap the
|
||||||
|
T86 reviewer flagged).
|
||||||
|
|
||||||
|
Seed a user_turn + assistant_turn, GET the chat page, and assert the
|
||||||
|
response body carries both turns' event ids on the wrapper DIVs.
|
||||||
|
"""
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
_seed_chat(db_path)
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
ut_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"prose": "hello bot",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
at_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "Hi there.",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": ut_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
response = client.get("/chats/chat_bot_a")
|
||||||
|
assert response.status_code == 200
|
||||||
|
body = response.text
|
||||||
|
# Both seeded turns must carry ``id="turn-<event_id>"`` so the SSE
|
||||||
|
# in-place swap can find them.
|
||||||
|
assert f'id="turn-{ut_id}"' in body
|
||||||
|
assert f'id="turn-{at_id}"' in body
|
||||||
|
|||||||
@@ -0,0 +1,297 @@
|
|||||||
|
"""Shared turn helpers (T83.2).
|
||||||
|
|
||||||
|
``chat.services.turn_common`` extracts two snippets that were duplicated
|
||||||
|
between ``chat.web.turns`` and ``chat.services.regenerate``: the recent
|
||||||
|
user-side / assistant_turn read, and the directed-pair edge gather for
|
||||||
|
the multi-pair state-update pass. These tests pin the helpers' behavior
|
||||||
|
independently of either call site.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
from chat.services.turn_common import gather_prior_edges, read_recent_dialogue
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_basic_chat(db_path):
|
||||||
|
"""Seed bot + chat + a couple of edges + one round of user/assistant
|
||||||
|
turns. Returns ``(user_turn_id, assistant_turn_id)``.
|
||||||
|
"""
|
||||||
|
apply_migrations(db_path)
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": "bot_a",
|
||||||
|
"name": "BotA",
|
||||||
|
"persona": "thoughtful",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_a",
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="edge_update",
|
||||||
|
payload={
|
||||||
|
"source_id": "bot_a",
|
||||||
|
"target_id": "you",
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"affinity_delta": 7,
|
||||||
|
"trust_delta": 3,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="edge_update",
|
||||||
|
payload={
|
||||||
|
"source_id": "you",
|
||||||
|
"target_id": "bot_a",
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"affinity_delta": 2,
|
||||||
|
"trust_delta": 1,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
ut_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"prose": "hello",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
at_id = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "Original.",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": ut_id,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
return ut_id, at_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_recent_dialogue_returns_chronological_pairs(tmp_path):
|
||||||
|
"""``read_recent_dialogue`` returns oldest-first ``{speaker, text}``
|
||||||
|
entries scoped to the requested chat. Speaker is "you" for user-side
|
||||||
|
rows and the assistant_turn's ``speaker_id`` for bot rows.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_basic_chat(db)
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = read_recent_dialogue(conn, "chat_a", limit=10)
|
||||||
|
|
||||||
|
# Each entry now carries the source ``event_log.id`` as ``event_id``
|
||||||
|
# (T86 follow-up) so the chat-detail Jinja loop can stamp
|
||||||
|
# ``id="turn-<n>"`` on each rendered turn DIV — needed by the
|
||||||
|
# ``turn_html_replace`` SSE handler for in-place regenerate swaps.
|
||||||
|
speakers = [(e["speaker"], e["text"]) for e in out]
|
||||||
|
assert speakers == [
|
||||||
|
("you", "hello"),
|
||||||
|
("bot_a", "Original."),
|
||||||
|
]
|
||||||
|
assert all("event_id" in e and isinstance(e["event_id"], int) for e in out)
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_recent_dialogue_filters_superseded_and_other_chats(tmp_path):
|
||||||
|
"""Superseded rows drop out (regenerate-aware). Rows scoped to a
|
||||||
|
different chat are also filtered. ``exclude_event_id`` excludes a
|
||||||
|
specific row even when it isn't superseded yet (regenerate uses this
|
||||||
|
to drop the original assistant_turn before the supersede UPDATE
|
||||||
|
lands).
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
ut_id, at_id = _seed_basic_chat(db)
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Append a second user/assistant pair.
|
||||||
|
ut_id2 = append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"prose": "how are you",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
at_id2 = append_event(
|
||||||
|
conn,
|
||||||
|
kind="assistant_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"speaker_id": "bot_a",
|
||||||
|
"text": "Second.",
|
||||||
|
"truncated": False,
|
||||||
|
"user_turn_id": ut_id2,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# And a row scoped to a different chat — must NOT appear.
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "other_chat",
|
||||||
|
"prose": "should be filtered",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Mark the first assistant_turn as superseded — must drop out.
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE event_log SET superseded_by = ? WHERE id = ?",
|
||||||
|
(at_id2, at_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
out = read_recent_dialogue(conn, "chat_a", limit=10)
|
||||||
|
# First (superseded) assistant turn dropped; "other_chat" rows
|
||||||
|
# filtered; first user_turn still present.
|
||||||
|
speakers = [(e["speaker"], e["text"]) for e in out]
|
||||||
|
assert speakers == [
|
||||||
|
("you", "hello"),
|
||||||
|
("you", "how are you"),
|
||||||
|
("bot_a", "Second."),
|
||||||
|
]
|
||||||
|
|
||||||
|
# exclude_event_id drops at_id2 even though it's not superseded.
|
||||||
|
out2 = read_recent_dialogue(
|
||||||
|
conn, "chat_a", limit=10, exclude_event_id=at_id2
|
||||||
|
)
|
||||||
|
speakers2 = [(e["speaker"], e["text"]) for e in out2]
|
||||||
|
assert ("bot_a", "Second.") not in speakers2
|
||||||
|
assert ("you", "how are you") in speakers2
|
||||||
|
|
||||||
|
# Ensure ut_id is still part of the dataset (sanity for the seed).
|
||||||
|
assert ut_id is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_recent_dialogue_limit_respects_chat_scope(tmp_path):
|
||||||
|
"""T90.1: ``read_recent_dialogue`` must push the chat_id filter into
|
||||||
|
SQL so that ``LIMIT N`` returns N rows scoped to the requested chat —
|
||||||
|
not N globally-recent rows that may then be filtered down to fewer in
|
||||||
|
Python.
|
||||||
|
|
||||||
|
Setup: two chats with 60 turns each, interleaved. With the old
|
||||||
|
post-fetch filter, ``LIMIT 50`` would pull 50 globally-recent rows
|
||||||
|
(most or all from chat_b — the most recent inserts) and then drop
|
||||||
|
chat_b ones via the Python check, yielding far fewer than 50 chat_a
|
||||||
|
rows. After the SQL pushdown, ``LIMIT 50`` should return exactly 50
|
||||||
|
chat_a rows.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
for chat_id, host_bot in (("chat_a", "bot_a"), ("chat_b", "bot_b")):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": host_bot,
|
||||||
|
"name": host_bot,
|
||||||
|
"persona": "...",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": chat_id,
|
||||||
|
"host_bot_id": host_bot,
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Interleave 60 user_turn rows in each chat — chat_b's go in last
|
||||||
|
# so they dominate the global tail.
|
||||||
|
for i in range(60):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_a",
|
||||||
|
"prose": f"a-{i}",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for i in range(60):
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="user_turn",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_b",
|
||||||
|
"prose": f"b-{i}",
|
||||||
|
"segments": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
out = read_recent_dialogue(conn, "chat_a", limit=50)
|
||||||
|
|
||||||
|
# All returned rows should belong to chat_a (texts a-* only).
|
||||||
|
assert len(out) == 50
|
||||||
|
for entry in out:
|
||||||
|
assert entry["text"].startswith("a-"), (
|
||||||
|
f"foreign chat row leaked: {entry!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_gather_prior_edges_fills_missing_with_default(tmp_path):
|
||||||
|
"""``gather_prior_edges`` returns one entry per directed pair across
|
||||||
|
``present_ids``. Missing rows fall back to the schema default
|
||||||
|
50/50 baseline; existing rows carry their stored values.
|
||||||
|
"""
|
||||||
|
db = tmp_path / "test.db"
|
||||||
|
_seed_basic_chat(db)
|
||||||
|
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out = gather_prior_edges(conn, ["bot_a", "you"])
|
||||||
|
|
||||||
|
# 2 entities -> 2 directed pairs (a->b and b->a, no self-pairs).
|
||||||
|
assert set(out.keys()) == {("bot_a", "you"), ("you", "bot_a")}
|
||||||
|
bot_to_you = out[("bot_a", "you")]
|
||||||
|
you_to_bot = out[("you", "bot_a")]
|
||||||
|
# Both edges seeded with deltas — they must reflect the projected
|
||||||
|
# affinity/trust (not the default 50/50).
|
||||||
|
assert bot_to_you["affinity"] == 57 # 50 + 7
|
||||||
|
assert bot_to_you["trust"] == 53 # 50 + 3
|
||||||
|
assert you_to_bot["affinity"] == 52
|
||||||
|
assert you_to_bot["trust"] == 51
|
||||||
|
|
||||||
|
# A pair with no row yet falls back to 50/50.
|
||||||
|
with open_db(db) as conn:
|
||||||
|
out_with_missing = gather_prior_edges(
|
||||||
|
conn, ["bot_a", "you", "ghost_bot"]
|
||||||
|
)
|
||||||
|
# 3 entities -> 6 directed pairs.
|
||||||
|
assert len(out_with_missing) == 6
|
||||||
|
fallback = out_with_missing[("bot_a", "ghost_bot")]
|
||||||
|
assert fallback["affinity"] == 50
|
||||||
|
assert fallback["trust"] == 50
|
||||||
|
assert fallback["summary"] == ""
|
||||||
@@ -734,6 +734,19 @@ def test_cancelled_turn_still_closes_scene_when_user_prose_signals_close(
|
|||||||
that as an exception, so we drive the request inside ``with
|
that as an exception, so we drive the request inside ``with
|
||||||
pytest.raises``. Despite the exception, the scene_closed event
|
pytest.raises``. Despite the exception, the scene_closed event
|
||||||
must land in the event_log.
|
must land in the event_log.
|
||||||
|
|
||||||
|
T108 NOTE — this test does NOT actually exercise the cancel path.
|
||||||
|
``_CancelOnStreamMock.stream`` writes ``raise asyncio.CancelledError``
|
||||||
|
but ``asyncio`` is not imported at module scope, so the first
|
||||||
|
iteration raises ``NameError`` (caught by ``except Exception:`` in
|
||||||
|
post_turn, which sets ``primary_truncated=True`` but leaves
|
||||||
|
``cancelled=False``). The function therefore returns 204 normally,
|
||||||
|
the dependency-managed connection commits, and ``scene_closed``
|
||||||
|
lands. Importing asyncio so the real CancelledError fires reveals
|
||||||
|
a transactional bug: ``post_turn``'s end-of-function re-raise
|
||||||
|
causes ``open_db``'s dependency teardown to skip ``conn.commit()``,
|
||||||
|
rolling back ALL post-cancel writes (user_turn, assistant_turn,
|
||||||
|
edge_updates, scene_closed). Deferred for triage — see T108 report.
|
||||||
"""
|
"""
|
||||||
from typing import AsyncIterator, Sequence
|
from typing import AsyncIterator, Sequence
|
||||||
|
|
||||||
@@ -828,12 +841,33 @@ def test_cancelled_turn_still_closes_scene_when_user_prose_signals_close(
|
|||||||
"SELECT payload_json FROM event_log "
|
"SELECT payload_json FROM event_log "
|
||||||
"WHERE kind = 'assistant_turn' ORDER BY id"
|
"WHERE kind = 'assistant_turn' ORDER BY id"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
|
# T108: pin the ordering — user_turn must commit before
|
||||||
|
# scene_closed (close detection runs on prose that is already
|
||||||
|
# in the event_log) and any assistant_turn the cancel produced
|
||||||
|
# must come last (truncated record written after both).
|
||||||
|
ordered = conn.execute(
|
||||||
|
"SELECT id, kind FROM event_log "
|
||||||
|
"WHERE kind IN ('user_turn', 'scene_closed', 'assistant_turn') "
|
||||||
|
"ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
|
||||||
# Scene close lands despite the cancel.
|
# Scene close lands despite the cancel.
|
||||||
assert scene_close_count == 1
|
assert scene_close_count == 1
|
||||||
# The cancelled assistant_turn was still recorded (truncated=True).
|
# The cancelled assistant_turn was still recorded (truncated=True).
|
||||||
assert len(assistant_payload) == 1
|
assert len(assistant_payload) == 1
|
||||||
assert json.loads(assistant_payload[0][0])["truncated"] is True
|
assert json.loads(assistant_payload[0][0])["truncated"] is True
|
||||||
|
# T108 ordering pin: user_turn lands first, the truncated
|
||||||
|
# assistant_turn (if any) is committed BEFORE the scene_close
|
||||||
|
# decision fires, and scene_closed lands last. Close detection
|
||||||
|
# relies on user prose being committed to the event_log BEFORE
|
||||||
|
# the close decision runs — and the cancelled assistant beat is
|
||||||
|
# recorded as a partial before close-detection too.
|
||||||
|
kinds_in_order = [row[1] for row in ordered]
|
||||||
|
user_idx = kinds_in_order.index("user_turn")
|
||||||
|
close_idx = kinds_in_order.index("scene_closed")
|
||||||
|
assert user_idx < close_idx
|
||||||
|
if "assistant_turn" in kinds_in_order:
|
||||||
|
assert user_idx < kinds_in_order.index("assistant_turn") < close_idx
|
||||||
|
|
||||||
|
|
||||||
def test_interjection_enqueues_significance_job(app_state_setup, tmp_path):
|
def test_interjection_enqueues_significance_job(app_state_setup, tmp_path):
|
||||||
@@ -1317,3 +1351,247 @@ def test_skip_command_does_not_run_narrative_classifier(
|
|||||||
"assemble_narrative_prompt was called on the skip path; the "
|
"assemble_narrative_prompt was called on the skip path; the "
|
||||||
"natural-language skip dispatch must bypass narrative assembly."
|
"natural-language skip dispatch must bypass narrative assembly."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Phase 3.5 (T82.1) — post_turn consumes pending meanwhile digests.
|
||||||
|
#
|
||||||
|
# The helper ``consume_pending_meanwhile_digests`` lives in
|
||||||
|
# chat.services.prompt and is now wired into the END of post_turn (after
|
||||||
|
# scene-close detection, before the response broadcast). This pins the
|
||||||
|
# wiring so future refactors don't accidentally drop the call and leave
|
||||||
|
# digests pending forever.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_post_turn_consumes_pending_meanwhile_digests(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""Seed a pending meanwhile digest via ``meanwhile_digest_created``,
|
||||||
|
POST a regular you-turn through post_turn, and assert:
|
||||||
|
|
||||||
|
1. A ``meanwhile_digest_consumed`` event lands in the event_log.
|
||||||
|
2. ``list_pending_meanwhile_digests`` returns empty after the turn.
|
||||||
|
|
||||||
|
The post_turn flow surfaces the digest in the prompt (T65) and then
|
||||||
|
consumes it (T82.1) so the next turn starts clean.
|
||||||
|
"""
|
||||||
|
_seed(tmp_path / "test.db")
|
||||||
|
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
# Seed a pending digest directly via the projection event. The scene_id
|
||||||
|
# field doesn't need to reference an existing meanwhile scene for the
|
||||||
|
# digest table — the FK is on the digest payload only.
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_and_apply(
|
||||||
|
conn,
|
||||||
|
kind="meanwhile_digest_created",
|
||||||
|
payload={
|
||||||
|
"scene_id": 99,
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"summary": "While you were away, the bots talked.",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Confirm the digest is pending before the turn lands.
|
||||||
|
from chat.state.meanwhile import list_pending_meanwhile_digests
|
||||||
|
|
||||||
|
assert len(list_pending_meanwhile_digests(conn, "chat_bot_a")) == 1
|
||||||
|
|
||||||
|
canned_parse = json.dumps(
|
||||||
|
{"segments": [{"kind": "dialogue", "text": "hello"}]}
|
||||||
|
)
|
||||||
|
# Standard 4-slot queue: parse + narrative + 2 state-updates. No
|
||||||
|
# active scene so scene-close detection short-circuits without an LLM
|
||||||
|
# call (consistent with the no-guest regression test).
|
||||||
|
mock = _override_llm(
|
||||||
|
[canned_parse, "Hi there.", _zero_state(), _zero_state()]
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
response = app_state_setup.post(
|
||||||
|
"/chats/chat_bot_a/turns", data={"prose": "hello"}
|
||||||
|
)
|
||||||
|
assert response.status_code == 204
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
# All canned slots drained — no extra classifier calls fired.
|
||||||
|
assert mock._canned == []
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
# A meanwhile_digest_consumed event landed for the seeded digest.
|
||||||
|
consumed_rows = conn.execute(
|
||||||
|
"SELECT payload_json FROM event_log "
|
||||||
|
"WHERE kind = 'meanwhile_digest_consumed' ORDER BY id"
|
||||||
|
).fetchall()
|
||||||
|
assert len(consumed_rows) == 1
|
||||||
|
|
||||||
|
# The pending list is empty after consumption.
|
||||||
|
from chat.state.meanwhile import list_pending_meanwhile_digests
|
||||||
|
|
||||||
|
assert list_pending_meanwhile_digests(conn, "chat_bot_a") == []
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Phase 3.5 (T82.2) — natural-language skip runs scene close detection.
|
||||||
|
#
|
||||||
|
# A user typing "fade out, skip an hour" should close the scene FIRST
|
||||||
|
# (so the close summary captures the closing scene's final beat) and
|
||||||
|
# THEN run the elision skip. Without this wiring, the skip dispatch
|
||||||
|
# branch bypasses scene close entirely.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_natural_language_skip_with_close_signal_closes_scene(
|
||||||
|
app_state_setup, tmp_path
|
||||||
|
):
|
||||||
|
"""Prose that hard-signals a close ("fade out, skip to morning") and
|
||||||
|
parses as ``intent=skip_elision`` must:
|
||||||
|
|
||||||
|
1. Land a ``scene_closed`` event before any skip event.
|
||||||
|
2. Run ``apply_scene_close_summary`` for the closing scene.
|
||||||
|
3. Land a ``time_skip_elision`` event AFTER the scene_closed.
|
||||||
|
|
||||||
|
Order matters — the scene_closed id must be lower than the
|
||||||
|
time_skip_elision id in the event_log.
|
||||||
|
|
||||||
|
Canned queue (single-bot, scene seeded, NO prior dialogue rows):
|
||||||
|
1. parse_turn -> intent=skip_elision
|
||||||
|
2. detect_scene_close -> should_close=True
|
||||||
|
3. apply_scene_close_summary host POV
|
||||||
|
4. narrate_skip narration
|
||||||
|
|
||||||
|
detect_threads (T58.2 fires on every close) short-circuits when the
|
||||||
|
scene-scoped transcript is empty — in this test no user/assistant
|
||||||
|
turns landed in scene 1 before the close, so no thread-detection
|
||||||
|
slot is needed.
|
||||||
|
"""
|
||||||
|
# Seed an open scene so detect_scene_close has something to act on.
|
||||||
|
db_path = tmp_path / "test.db"
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="bot_authored",
|
||||||
|
payload={
|
||||||
|
"id": "bot_a",
|
||||||
|
"name": "BotA",
|
||||||
|
"persona": "thoughtful, observant",
|
||||||
|
"voice_samples": [],
|
||||||
|
"traits": [],
|
||||||
|
"backstory": "",
|
||||||
|
"initial_relationship_to_you": "",
|
||||||
|
"kickoff_prose": "...",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="chat_created",
|
||||||
|
payload={
|
||||||
|
"id": "chat_bot_a",
|
||||||
|
"host_bot_id": "bot_a",
|
||||||
|
"initial_time": "2026-04-26T20:00:00+00:00",
|
||||||
|
"narrative_anchor": "Day 1",
|
||||||
|
"weather": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="container_created",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"name": "office",
|
||||||
|
"type": "workplace",
|
||||||
|
"properties": {},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="scene_opened",
|
||||||
|
payload={
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"container_id": 1,
|
||||||
|
"started_at": "2026-04-26T20:00:00+00:00",
|
||||||
|
"participants": ["you", "bot_a"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="edge_update",
|
||||||
|
payload={
|
||||||
|
"source_id": "bot_a",
|
||||||
|
"target_id": "you",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"knowledge_facts": ["coworker"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
for entity_id, verb in [("you", "talking"), ("bot_a", "listening")]:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="activity_change",
|
||||||
|
payload={
|
||||||
|
"entity_id": entity_id,
|
||||||
|
"posture": "sitting",
|
||||||
|
"action": {
|
||||||
|
"verb": verb,
|
||||||
|
"interruptible": True,
|
||||||
|
"required_attention": "low",
|
||||||
|
"expected_duration": "ongoing",
|
||||||
|
},
|
||||||
|
"attention": "",
|
||||||
|
"holding": [],
|
||||||
|
"status": {},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
canned_parse = json.dumps(
|
||||||
|
{
|
||||||
|
"segments": [
|
||||||
|
{"kind": "narration", "text": "fade out, skip to morning"}
|
||||||
|
],
|
||||||
|
"intent": "skip_elision",
|
||||||
|
"landing_state_hint": "morning at home",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
canned_close = json.dumps(
|
||||||
|
{"should_close": True, "reason": "fade out signaled"}
|
||||||
|
)
|
||||||
|
canned_pov = json.dumps(
|
||||||
|
{
|
||||||
|
"summary": "BotA noticed the day winding down.",
|
||||||
|
"knowledge_facts": [],
|
||||||
|
"relationship_summary": "warmer",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
canned_narration = "The night fades and morning arrives."
|
||||||
|
mock = _override_llm(
|
||||||
|
[
|
||||||
|
canned_parse,
|
||||||
|
canned_close,
|
||||||
|
canned_pov,
|
||||||
|
canned_narration,
|
||||||
|
]
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
response = app_state_setup.post(
|
||||||
|
"/chats/chat_bot_a/turns",
|
||||||
|
data={"prose": "fade out, skip to morning"},
|
||||||
|
)
|
||||||
|
assert response.status_code == 204
|
||||||
|
finally:
|
||||||
|
app.dependency_overrides.clear()
|
||||||
|
# All 4 canned slots drained — close + skip both ran end-to-end.
|
||||||
|
assert mock._canned == []
|
||||||
|
|
||||||
|
with open_db(db_path) as conn:
|
||||||
|
# scene_closed and time_skip_elision both landed.
|
||||||
|
scene_close_rows = conn.execute(
|
||||||
|
"SELECT id FROM event_log WHERE kind = 'scene_closed'"
|
||||||
|
).fetchall()
|
||||||
|
skip_rows = conn.execute(
|
||||||
|
"SELECT id FROM event_log WHERE kind = 'time_skip_elision'"
|
||||||
|
).fetchall()
|
||||||
|
assert len(scene_close_rows) == 1, "scene_closed must land"
|
||||||
|
assert len(skip_rows) == 1, "time_skip_elision must land"
|
||||||
|
# Order: scene close first, then skip.
|
||||||
|
assert scene_close_rows[0][0] < skip_rows[0][0], (
|
||||||
|
"scene_closed must precede time_skip_elision in the event_log"
|
||||||
|
)
|
||||||
|
|||||||
@@ -0,0 +1,242 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from chat.db.connection import open_db
|
||||||
|
from chat.db.migrate import apply_migrations
|
||||||
|
from chat.eventlog.log import append_event
|
||||||
|
from chat.eventlog.projector import project
|
||||||
|
import chat.state.memory # registers memory_written handler
|
||||||
|
import chat.state.embeddings # registers embedding handlers
|
||||||
|
from chat.services.vector_search import vector_search
|
||||||
|
|
||||||
|
|
||||||
|
def _base_memory(**overrides):
|
||||||
|
payload = {
|
||||||
|
"owner_id": "bot_a",
|
||||||
|
"chat_id": "chat_bot_a",
|
||||||
|
"scene_id": 1,
|
||||||
|
"pov_summary": "She laughed at his joke about owls.",
|
||||||
|
"witness_you": 1,
|
||||||
|
"witness_host": 1,
|
||||||
|
"witness_guest": 0,
|
||||||
|
"chat_clock_at": "2026-04-26T10:00:00",
|
||||||
|
"source": "direct",
|
||||||
|
"reliability": 1.0,
|
||||||
|
"significance": 1,
|
||||||
|
"pinned": 0,
|
||||||
|
"auto_pinned": 0,
|
||||||
|
}
|
||||||
|
payload.update(overrides)
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _one_hot(dim: int, idx: int) -> list[float]:
|
||||||
|
"""Return a one-hot vector of length ``dim`` with 1.0 at ``idx``."""
|
||||||
|
v = [0.0] * dim
|
||||||
|
v[idx] = 1.0
|
||||||
|
return v
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_memory_with_embedding(
|
||||||
|
conn,
|
||||||
|
*,
|
||||||
|
owner_id: str,
|
||||||
|
pov_summary: str,
|
||||||
|
vector: list[float],
|
||||||
|
significance: int = 1,
|
||||||
|
witness_you: int = 1,
|
||||||
|
witness_host: int = 1,
|
||||||
|
witness_guest: int = 0,
|
||||||
|
model: str = "test-model",
|
||||||
|
) -> int:
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload=_base_memory(
|
||||||
|
owner_id=owner_id,
|
||||||
|
pov_summary=pov_summary,
|
||||||
|
significance=significance,
|
||||||
|
witness_you=witness_you,
|
||||||
|
witness_host=witness_host,
|
||||||
|
witness_guest=witness_guest,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
memory_id = conn.execute(
|
||||||
|
"SELECT id FROM memories WHERE pov_summary = ?", (pov_summary,)
|
||||||
|
).fetchone()[0]
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="embedding_indexed",
|
||||||
|
payload={
|
||||||
|
"memory_id": memory_id,
|
||||||
|
"vector": vector,
|
||||||
|
"model": model,
|
||||||
|
"dim": len(vector),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
return memory_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_vector_search_returns_nearest_neighbors(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
dim = 8
|
||||||
|
ids = []
|
||||||
|
for i in range(5):
|
||||||
|
mid = _seed_memory_with_embedding(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
pov_summary=f"Memory {i}.",
|
||||||
|
vector=_one_hot(dim, i),
|
||||||
|
)
|
||||||
|
ids.append(mid)
|
||||||
|
|
||||||
|
# Query close to memory index 3 (one-hot at position 3, plus tiny noise).
|
||||||
|
query = _one_hot(dim, 3)
|
||||||
|
query[2] = 0.01
|
||||||
|
|
||||||
|
results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="you",
|
||||||
|
query_vector=query,
|
||||||
|
k=3,
|
||||||
|
)
|
||||||
|
assert len(results) == 3
|
||||||
|
# Top-1 must be memory at index 3.
|
||||||
|
assert results[0]["memory_id"] == ids[3]
|
||||||
|
assert results[0]["pov_summary"] == "Memory 3."
|
||||||
|
# Score for the near-perfect match should be very close to 1.0.
|
||||||
|
assert results[0]["score"] > 0.99
|
||||||
|
# Results sorted by score DESC.
|
||||||
|
scores = [r["score"] for r in results]
|
||||||
|
assert scores == sorted(scores, reverse=True)
|
||||||
|
# Second place should be memory index 2 (the small noise component).
|
||||||
|
assert results[1]["memory_id"] == ids[2]
|
||||||
|
|
||||||
|
|
||||||
|
def test_vector_search_respects_witness_filter(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
dim = 4
|
||||||
|
# Memory visible to you=1, host=1, guest=0.
|
||||||
|
_seed_memory_with_embedding(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
pov_summary="Restricted.",
|
||||||
|
vector=_one_hot(dim, 0),
|
||||||
|
witness_you=1,
|
||||||
|
witness_host=1,
|
||||||
|
witness_guest=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Guest sees nothing.
|
||||||
|
guest_results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="guest",
|
||||||
|
query_vector=_one_hot(dim, 0),
|
||||||
|
k=4,
|
||||||
|
)
|
||||||
|
assert guest_results == []
|
||||||
|
|
||||||
|
# Host sees the memory.
|
||||||
|
host_results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="host",
|
||||||
|
query_vector=_one_hot(dim, 0),
|
||||||
|
k=4,
|
||||||
|
)
|
||||||
|
assert len(host_results) == 1
|
||||||
|
assert host_results[0]["pov_summary"] == "Restricted."
|
||||||
|
|
||||||
|
# You also see it.
|
||||||
|
you_results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="you",
|
||||||
|
query_vector=_one_hot(dim, 0),
|
||||||
|
k=4,
|
||||||
|
)
|
||||||
|
assert len(you_results) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_vector_search_respects_owner_filter(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
dim = 4
|
||||||
|
_seed_memory_with_embedding(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
pov_summary="Owner A memory.",
|
||||||
|
vector=_one_hot(dim, 0),
|
||||||
|
)
|
||||||
|
_seed_memory_with_embedding(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_b",
|
||||||
|
pov_summary="Owner B memory.",
|
||||||
|
vector=_one_hot(dim, 0),
|
||||||
|
)
|
||||||
|
|
||||||
|
a_results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="you",
|
||||||
|
query_vector=_one_hot(dim, 0),
|
||||||
|
k=10,
|
||||||
|
)
|
||||||
|
assert len(a_results) == 1
|
||||||
|
assert a_results[0]["pov_summary"] == "Owner A memory."
|
||||||
|
|
||||||
|
b_results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_b",
|
||||||
|
witness_role="you",
|
||||||
|
query_vector=_one_hot(dim, 0),
|
||||||
|
k=10,
|
||||||
|
)
|
||||||
|
assert len(b_results) == 1
|
||||||
|
assert b_results[0]["pov_summary"] == "Owner B memory."
|
||||||
|
|
||||||
|
|
||||||
|
def test_vector_search_invalid_witness_role_raises(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
with pytest.raises(ValueError, match="witness_role"):
|
||||||
|
vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="invalid",
|
||||||
|
query_vector=[1.0, 0.0, 0.0],
|
||||||
|
k=4,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_vector_search_empty_when_no_embeddings_indexed(tmp_path):
|
||||||
|
db = tmp_path / "t.db"
|
||||||
|
apply_migrations(db)
|
||||||
|
with open_db(db) as conn:
|
||||||
|
# Seed a memory but don't index an embedding for it.
|
||||||
|
append_event(
|
||||||
|
conn,
|
||||||
|
kind="memory_written",
|
||||||
|
payload=_base_memory(owner_id="bot_a", pov_summary="No embedding here."),
|
||||||
|
)
|
||||||
|
project(conn)
|
||||||
|
|
||||||
|
results = vector_search(
|
||||||
|
conn,
|
||||||
|
owner_id="bot_a",
|
||||||
|
witness_role="you",
|
||||||
|
query_vector=[1.0, 0.0, 0.0, 0.0],
|
||||||
|
k=4,
|
||||||
|
)
|
||||||
|
assert results == []
|
||||||
+2
-2
@@ -324,11 +324,11 @@ def test_get_scene_returns_none_for_missing(tmp_path):
|
|||||||
assert active_scene(conn, "chat_missing") is None
|
assert active_scene(conn, "chat_missing") is None
|
||||||
|
|
||||||
|
|
||||||
def test_schema_version_after_migration_is_11(tmp_path):
|
def test_schema_version_after_migration_is_14(tmp_path):
|
||||||
db = tmp_path / "t.db"
|
db = tmp_path / "t.db"
|
||||||
apply_migrations(db)
|
apply_migrations(db)
|
||||||
with open_db(db) as conn:
|
with open_db(db) as conn:
|
||||||
row = conn.execute(
|
row = conn.execute(
|
||||||
"SELECT value FROM meta WHERE key = 'schema_version'"
|
"SELECT value FROM meta WHERE key = 'schema_version'"
|
||||||
).fetchone()
|
).fetchone()
|
||||||
assert int(row[0]) == 11
|
assert int(row[0]) == 14
|
||||||
|
|||||||
Reference in New Issue
Block a user