docs: add Phase 4 implementation plan (vector retrieval + branching + polish)
15 tasks across 8 waves landing the Phase 4 deliverables per requirements doc §13 + §14: - Vector retrieval via sqlite-vec (new external dependency) - Branching UI (event log forks) - Drawer-edit on every field (significance review, hide-from-view, surgical delete with cascade preview, branching affordances) - Backup tooling (snapshot UX surface) - Cross-chat search Plus the 3 Phase 3.6 carry-over fixes (T90 bundle). Wave structure: - W1 (parallel 3-way): schema foundation + carry-overs - W2 (parallel 3-way): embedding/search services - W3 (parallel 2-way): branching + delete services - W4 (single): combined retrieval ranking - W5 (single): memory write hook + backfill - W6 (single): drawer Phase 4 bundle (5 sub-features) - W7 (parallel 2-way): snapshot UX + cross-chat search UX - W8 (parallel 2-way): integration tests + docs External dependency: sqlite-vec must be installed BEFORE Wave 1. Embedding model choice (384-dim default) pinned in T91 before dispatch since the migration hardcodes the dimension. Schema baseline: 11 -> 13 (adds 0012_embeddings.sql + 0013_branches.sql). Task ids T88-T102 to avoid collision with prior phases.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,832 @@
|
||||
# Roleplay Engine — Phase 4 Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use `superpowers-extended-cc:executing-plans` to implement this plan task-by-task. Use the parallel-dispatch pattern documented under "Parallel-Execution Strategy" for parallel waves.
|
||||
|
||||
**Goal:** Land Phase 4 polish per requirements doc §13 + §14: vector retrieval, branching UI, drawer-edit on every field, backup tooling, significance review UI, surgical delete with cascade preview, hide-from-view soft delete, plus cross-chat search and the small Phase 3.6 carry-over fixes.
|
||||
|
||||
**Architecture:** Builds on Phase 3.5's stable base. Two new tables (`embeddings`, `branches`) and one external dependency (sqlite-vec extension). Embedding generation runs as a deferred async job — NOT inline with turns — so the play loop stays fast even when the embedding endpoint is slow. Branching is data-model-only at first (events + selectors); UI grafts on top. Surgical delete + cascade preview reuses the existing rewind-and-supersede plumbing. Cross-chat search piggybacks on the existing FTS5 + (now) vector retrieval.
|
||||
|
||||
**Tech Stack:**
|
||||
|
||||
- **NEW dependency: `sqlite-vec`** (or `sqlite-vss` — Phase 4 picks; recommended `sqlite-vec` for simpler load semantics and active maintenance). Add to `pyproject.toml`.
|
||||
- **Embedding model selection** is part of T91 spec. Recommended default: a small model on Featherless (e.g., `BAAI/bge-small-en-v1.5` if available) or a local CPU-friendly model via `sentence-transformers`. Document choice in CLAUDE.md.
|
||||
- Same as Phase 3 otherwise (Python 3.11+, FastAPI, HTMX, SQLite).
|
||||
|
||||
**Source-of-truth references:**
|
||||
|
||||
- Phase 4 scope: requirements doc §13 "Phase 4 — polish" + §14 "Open / Deferred Decisions".
|
||||
- Behavioral details: §6 (prompt assembly + retrieval), §10 (rewind / regenerate / reset), §11 (compression + significance), §12 (snapshots).
|
||||
- Conventions: [`CLAUDE.md`](../../CLAUDE.md) §"Behavioral defaults" + §"Phase 3 status" + §"Phase 3.5 status".
|
||||
- Phase 3.5 cleanup plan (style, file-bundling pattern): [2026-04-26-v3.5-phase3.5-cleanup.md](2026-04-26-v3.5-phase3.5-cleanup.md).
|
||||
|
||||
---
|
||||
|
||||
## Pre-flight
|
||||
|
||||
**Branch:** create `phase-4` from the latest `main` after Phase 3.5 has merged (it has — main is at `1b66a28`):
|
||||
|
||||
```bash
|
||||
git checkout main && git pull && git checkout -b phase-4
|
||||
```
|
||||
|
||||
**Schema baseline:** Phase 3.5 leaves the DB at version 11. Phase 4 adds two migrations: `0012_embeddings.sql` and `0013_branches.sql`. Final schema version: 13.
|
||||
|
||||
**External dependency setup (BEFORE T88 dispatch):**
|
||||
|
||||
The controlling agent should add `sqlite-vec` to `pyproject.toml` and run `pip install -e .` (or equivalent) so all worktrees pick up the new dependency. Confirm `sqlite_vec` imports cleanly:
|
||||
|
||||
```bash
|
||||
python -c "import sqlite_vec; print(sqlite_vec.__version__)"
|
||||
```
|
||||
|
||||
If `sqlite_vec` isn't on PyPI when this plan executes, fall back to `sqlite-vss` and adapt T88/T92 accordingly. Both expose vector-search SQL via a loadable extension.
|
||||
|
||||
**Pinned non-negotiables (carried forward):**
|
||||
|
||||
- State changes go through the event log. Use `append_and_apply(conn, kind, payload)` for the live path; `apply_event` only after a fresh `append_event` returning the new id.
|
||||
- Witness filter every memory read at SQL level (hard `WHERE` constraint; never a soft signal).
|
||||
- Per-POV scene summaries — never write omniscient narration.
|
||||
- TDD: every task starts with a failing test (or a regression test pinning existing contract before refactor).
|
||||
- One commit per task minimum. Tasks that bundle multiple sub-features SHOULD split commits internally.
|
||||
|
||||
**Verification before claiming done:** Use `superpowers-extended-cc:verification-before-completion` — run the test command, paste actual output. Don't assume green.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3.6 carry-overs folded in
|
||||
|
||||
Three small items from Phase 3.6 backlog are bundled into Phase 4's Wave 1 trivial-fixes task (T90):
|
||||
|
||||
1. `read_recent_dialogue` chat-id pushdown into SQL (T80 review nit)
|
||||
2. Lifecycle warning wording in regenerate (T83.4 — "at-or-after turn X" tightening)
|
||||
3. Legacy single-bot `record_turn_memory` consolidation (T84 review nit)
|
||||
|
||||
Three items remain DEFERRED beyond Phase 4 (Phase 4.5 if needed):
|
||||
|
||||
- Scene-close-on-cancel UX revisit (no action unless real play surfaces a regression).
|
||||
- Cross-feature canned-queue brittleness (structured fixture builder for tests — not blocking).
|
||||
- Full lifecycle-rollback in regenerate (warning log already shipped in T83.4; proper rollback needs schema-level back-references, deferred indefinitely).
|
||||
|
||||
---
|
||||
|
||||
## Parallel-Execution Strategy
|
||||
|
||||
Same pattern as Phase 3.5. Eight waves: parallel within each wave (file-disjoint), serial across waves.
|
||||
|
||||
### How to dispatch a wave in parallel
|
||||
|
||||
Use the **Agent tool with `isolation: "worktree"`** so each subagent gets its own git worktree. (If the controlling session's working directory is **not** the chat repo, create worktrees manually with `git worktree add .worktrees/<wave>-<task> -b <wave>/<task> phase-4` from inside the chat repo.)
|
||||
|
||||
Dispatch all tasks in a wave in a single message:
|
||||
|
||||
```
|
||||
Agent({ description: "Wave 1 — T88 embeddings table", prompt: "...", isolation: "worktree" })
|
||||
Agent({ description: "Wave 1 — T89 branches table", ... })
|
||||
Agent({ description: "Wave 1 — T90 phase 3.6 carry-overs", ... })
|
||||
```
|
||||
|
||||
### After a wave completes
|
||||
|
||||
1. Each subagent returns its worktree path and commit SHA(s).
|
||||
2. **Run a spec + code-quality reviewer subagent on each completed task.** Combined review acceptable for trivial tasks (T90 carry-overs); separate spec + quality reviewers for vector-retrieval tasks (T91, T92, T96, T97) since the integration surface is wider.
|
||||
3. **Merge the wave into `phase-4`** in any order (file-disjointness guarantees no conflict). Use `--no-ff`.
|
||||
4. **Run the full test suite** on the merged `phase-4`. If red, the wave's mutual-independence assumption was violated — bisect, fix, re-merge.
|
||||
5. **Push `phase-4`** to gitea.
|
||||
6. Optionally clean up worktrees.
|
||||
|
||||
### Conflict prevention checklist
|
||||
|
||||
For each parallel wave, verify the **Files** sections of all tasks have **no overlapping paths**. Hot files in this plan: `chat/web/drawer.py` + `chat/templates/_drawer.html` (T98 only — bundled), `chat/state/memory.py` (T96 only), `chat/services/memory_write.py` (T90 + T97 — sequential), `chat/web/turns.py` (T98 only via delete affordance — sequential after T96).
|
||||
|
||||
### Why each wave is parallel-safe
|
||||
|
||||
| Wave | Tasks | Hot files touched | Disjoint? |
|
||||
|------|-------|-------------------|-----------|
|
||||
| 1 | T88, T89, T90 | new migrations + new state modules; T90 touches `turn_common.py` + `regenerate.py` + `memory_write.py` (additive only) | ✅ |
|
||||
| 2 | T91, T92, T93 | new service modules (embeddings, vector_search, cross_chat_search) | ✅ |
|
||||
| 3 | T94, T95 | new service modules (branching, delete_impact) | ✅ |
|
||||
| 4 | T96 | `chat/state/memory.py` (combined retrieval ranking) | (single task) |
|
||||
| 5 | T97 | `chat/services/memory_write.py` + new backfill script | (single task) |
|
||||
| 6 | T98 | `chat/web/drawer.py` + `chat/templates/_drawer.html` (drawer Phase 4 bundle) | (single task) |
|
||||
| 7 | T99, T100 | new files: `chat/web/snapshots.py` + `chat/templates/snapshots.html` (T99); `chat/web/search.py` + `chat/templates/search.html` + small chat.html top-bar addition (T100) | ✅ (disjoint) |
|
||||
| 8 | T101, T102 | new test file (T101); CLAUDE.md + design doc (T102) | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## Task overview
|
||||
|
||||
```
|
||||
Wave 1 ─┬─ T88: embeddings table + projector handlers
|
||||
├─ T89: branches table + projector handlers
|
||||
└─ T90: Phase 3.6 carry-overs trio (chat-id SQL pushdown + lifecycle wording + legacy-fn consolidation)
|
||||
|
||||
Wave 2 ─┬─ T91: embedding generation service (Featherless or local)
|
||||
├─ T92: vector search service via sqlite-vec
|
||||
└─ T93: cross-chat search service (FTS over all owners)
|
||||
|
||||
Wave 3 ─┬─ T94: branch_from_event service (event-log fork, branch metadata)
|
||||
└─ T95: delete-impact computation service (cascade preview)
|
||||
|
||||
Wave 4 ─── T96: combined FTS + vector retrieval ranking in search_memories
|
||||
|
||||
Wave 5 ─── T97: memory_write enqueues embedding job + backfill script for existing memories
|
||||
|
||||
Wave 6 ─── T98: drawer Phase 4 bundle — branching UI + significance review + hide-from-view + surgical delete + remaining v1 edits
|
||||
|
||||
Wave 7 ─┬─ T99: snapshot UX (manual trigger, retention display, restore-from-snapshot UI)
|
||||
└─ T100: cross-chat search UX (top-bar input + search results page)
|
||||
|
||||
Wave 8 ─┬─ T101: cross-feature integration tests (vector × branching × delete × snapshot × search)
|
||||
└─ T102: Phase 4 documentation update
|
||||
```
|
||||
|
||||
Critical path: 8 sequential merge points. Total tasks: 15. Parallelism: Waves 1, 2, 3, 7, 8 dispatch concurrently (3-way and 2-way). Waves 4, 5, 6 are single-task by hot-file constraint.
|
||||
|
||||
---
|
||||
|
||||
## Wave 1 — Schema foundation + Phase 3.6 carry-overs (parallel)
|
||||
|
||||
### Task 88: Embeddings table + projector handlers
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/db/migrations/0012_embeddings.sql`
|
||||
- Create: `chat/state/embeddings.py`
|
||||
- Create: `tests/test_embeddings_state.py`
|
||||
- Modify: `pyproject.toml` (add `sqlite-vec` dependency — controlling agent should pre-install before dispatch; the worktree commits the dependency declaration)
|
||||
|
||||
**Spec:**
|
||||
|
||||
Adds the `embeddings` table that stores per-memory embedding vectors for vector retrieval. Uses `sqlite-vec` virtual-table syntax for cosine-similarity search. Schema:
|
||||
|
||||
```sql
|
||||
-- Load sqlite-vec extension at connection time (handled in chat/db/connection.py).
|
||||
-- Embeddings are stored as blobs in a vec0 virtual table for fast similarity search.
|
||||
|
||||
CREATE VIRTUAL TABLE embeddings USING vec0(
|
||||
memory_id INTEGER PRIMARY KEY,
|
||||
embedding FLOAT[384] -- 384-dim default; adjust per chosen model
|
||||
);
|
||||
|
||||
-- Sidecar table for non-vector metadata (model used, dim, indexed_at).
|
||||
CREATE TABLE embeddings_meta (
|
||||
memory_id INTEGER PRIMARY KEY,
|
||||
model TEXT NOT NULL,
|
||||
dim INTEGER NOT NULL,
|
||||
indexed_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
FOREIGN KEY (memory_id) REFERENCES memories(id)
|
||||
);
|
||||
```
|
||||
|
||||
(If `sqlite-vss` is chosen instead, replace `vec0` with `vss0` and adapt the dim declaration. Both have similar Python loading semantics.)
|
||||
|
||||
**`chat/state/embeddings.py`:**
|
||||
|
||||
- `@on("embedding_indexed")` payload `{memory_id, model, dim, vector: list[float]}`. Inserts into both `embeddings` and `embeddings_meta`. Idempotent via `INSERT OR REPLACE` (re-indexing a memory replaces the prior vector).
|
||||
- `@on("embedding_deindexed")` payload `{memory_id}`. Deletes from both tables. Used when a memory is purged via reset/cascade.
|
||||
- Reader `get_embedding_meta(conn, memory_id) -> dict | None` returns the meta row.
|
||||
|
||||
The `chat/db/connection.py` `open_db` helper needs to load the sqlite-vec extension on each connection. Add:
|
||||
|
||||
```python
|
||||
import sqlite_vec
|
||||
# Inside open_db, after connection is opened:
|
||||
conn.enable_load_extension(True)
|
||||
sqlite_vec.load(conn)
|
||||
conn.enable_load_extension(False)
|
||||
```
|
||||
|
||||
This is a small modification to `connection.py`. Include it in T88's diff.
|
||||
|
||||
**Tests:** 3 minimum.
|
||||
|
||||
1. `test_embedding_indexed_inserts_row`: append `bot_authored`, `chat_created`, `memory_written` (creates a memory), then `embedding_indexed` with `vector=[0.1] * 384`. Project. Assert `embeddings_meta` row exists for that memory_id with the right model.
|
||||
2. `test_embedding_deindexed_removes_row`: same setup; index then de-index; assert row is gone.
|
||||
3. `test_vector_similarity_search_returns_nearest`: index two memories with distinct vectors; query for nearest neighbor of one vector; assert correct memory_id returned. Uses `sqlite-vec`'s `MATCH '...'` syntax (verify against actual sqlite-vec docs; adapt if needed).
|
||||
|
||||
If running tests requires sqlite-vec to be loaded, the test fixture may need to skip / xfail when the extension isn't installed. Use `pytest.importorskip("sqlite_vec")` at the top of the test file.
|
||||
|
||||
**Commit:** `feat: embeddings table + projector handlers via sqlite-vec (T88)`.
|
||||
|
||||
**Notes:**
|
||||
|
||||
- Schema version after migration alone: 12. T89 adds 0013, taking final to 13. The schema_version assertion in `tests/test_world.py` updates to 13 in the wave-merge step.
|
||||
- The `connection.py` change is small but cross-cutting — affects every `open_db` call. Verify the existing 343 tests still pass after the change.
|
||||
|
||||
---
|
||||
|
||||
### Task 89: Branches table + projector handlers
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/db/migrations/0013_branches.sql`
|
||||
- Create: `chat/state/branches.py`
|
||||
- Create: `tests/test_branches_state.py`
|
||||
|
||||
**Spec:**
|
||||
|
||||
Adds the `branches` table that records named alternate event-log forks. A branch is metadata: a name, an `origin_event_id` (the event we forked from), and a `head_event_id` (the latest event in this branch). The event log itself is unchanged — the branch table just **labels** linear ranges of event ids.
|
||||
|
||||
```sql
|
||||
CREATE TABLE branches (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
origin_event_id INTEGER NOT NULL,
|
||||
head_event_id INTEGER NOT NULL,
|
||||
chat_id TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
is_active INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
|
||||
-- Exactly one row may have is_active = 1 at any time.
|
||||
CREATE UNIQUE INDEX branches_active_idx ON branches(is_active) WHERE is_active = 1;
|
||||
```
|
||||
|
||||
The "main" branch is implicit and bootstrapped by the migration: `INSERT INTO branches (name, origin_event_id, head_event_id, is_active) VALUES ('main', 0, 0, 1);`. Subsequent branches reference an `origin_event_id` (the event that the branch forked from).
|
||||
|
||||
`chat/state/branches.py`:
|
||||
|
||||
- `@on("branch_created")` payload `{name, origin_event_id, chat_id?, head_event_id}`. Inserts a new row with `is_active=0`. Idempotent re-insertion via `INSERT OR IGNORE`.
|
||||
- `@on("branch_switched")` payload `{name}`. Sets `is_active=1` on the named branch and `is_active=0` on all others. Atomic via a single UPDATE.
|
||||
- `@on("branch_head_updated")` payload `{name, head_event_id}`. Updates `head_event_id` on the named branch. Used by the orchestrator when new events extend the branch.
|
||||
- Readers: `get_branch(conn, name)`, `list_branches(conn, chat_id=None)`, `active_branch(conn)`.
|
||||
|
||||
**Tests:** 3 minimum.
|
||||
|
||||
1. `test_branch_created_inserts_row`: append `branch_created` with name="experiment", origin_event_id=42; project; assert `get_branch(conn, "experiment")` returns the row.
|
||||
2. `test_branch_switched_atomic`: seed two branches; switch from one to the other; assert exactly one is active.
|
||||
3. `test_main_branch_bootstrapped_by_migration`: open a fresh DB, apply migrations; assert `active_branch(conn)["name"] == "main"`.
|
||||
|
||||
**Commit:** `feat: branches table + projector handlers (T89)`.
|
||||
|
||||
**Notes:**
|
||||
|
||||
- Schema version after this migration alone: 13. Combined with T88: 13 (since T88 was 12, T89 stacks). Wave-merge bumps `tests/test_world.py` schema_version assertion to 13.
|
||||
- This task does NOT yet teach the orchestrator to consult `is_active` — the existing event_log queries assume a single timeline. T98 (drawer branching UI) will enable user-driven switches, but the actual "follow only the active branch" filter on event reads is a follow-up (Phase 4.5 nit; document in T102 docs sweep).
|
||||
|
||||
---
|
||||
|
||||
### Task 90: Phase 3.6 carry-overs trio
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `chat/services/turn_common.py` (push chat_id filter into SQL)
|
||||
- Modify: `chat/services/regenerate.py` (lifecycle warning wording tightening)
|
||||
- Modify: `chat/services/memory_write.py` (consolidate legacy `record_turn_memory` into the unified API or delete it)
|
||||
- Modify: `tests/test_turn_common.py`, `tests/test_regenerate.py`, `tests/test_memory_write.py`
|
||||
|
||||
**Spec:** Three small Phase 3.6 carry-over fixes bundled because each is 1-line + 1-test.
|
||||
|
||||
#### 90.1 — `read_recent_dialogue` chat-id SQL pushdown
|
||||
|
||||
Per T80 review nit. Currently `read_recent_dialogue` filters chat_id post-fetch in Python. Push into SQL for tighter LIMIT semantics:
|
||||
|
||||
```sql
|
||||
SELECT id, kind, payload_json
|
||||
FROM event_log
|
||||
WHERE kind IN ('user_turn', 'user_turn_edit', 'assistant_turn')
|
||||
AND superseded_by IS NULL
|
||||
AND hidden = 0
|
||||
AND json_extract(payload_json, '$.chat_id') = ?
|
||||
ORDER BY id DESC
|
||||
LIMIT ?
|
||||
```
|
||||
|
||||
Then the post-fetch loop becomes a simple reverse + slice — no chat_id check needed.
|
||||
|
||||
**Test added:** `test_read_recent_dialogue_limit_respects_chat_scope` — seed two chats with 60 turns each; query chat_a with `limit=50`; assert returned rows are exactly 50 chat_a rows (not 50 cross-chat rows that filter down to <50 after Python).
|
||||
|
||||
**Commit:** `perf: read_recent_dialogue pushes chat-id filter into SQL (T90.1)`.
|
||||
|
||||
#### 90.2 — Lifecycle warning wording tightening
|
||||
|
||||
Per T83.4 review nit. Current warning lists "lifecycle transitions from superseded turn are NOT being rolled back". When user regenerates an OLDER turn (T29 supports this), the warning lists intervening-turn transitions that legitimately stand. Tighten wording to "lifecycle transitions at-or-after turn X" so operators reading logs aren't misled.
|
||||
|
||||
Change is one log message string. Test asserts the new wording appears.
|
||||
|
||||
**Commit:** `chore: clarify regenerate lifecycle warning wording (T90.2)`.
|
||||
|
||||
#### 90.3 — Legacy `record_turn_memory` consolidation
|
||||
|
||||
Per T84 review nit. The original Phase 1 single-bot `record_turn_memory` function still exists alongside the unified `record_turn_memory_for_present`. Either:
|
||||
|
||||
- (a) Remove the legacy function entirely; update any remaining callers to use the unified API.
|
||||
- (b) Convert it to a thin wrapper for backward compat.
|
||||
|
||||
Pick (a) if there are zero remaining callers; (b) if any callers exist. Read the codebase to confirm. The mock-data seed scripts may still use the legacy fn.
|
||||
|
||||
**Commit:** `refactor: consolidate legacy record_turn_memory into unified API (T90.3)`.
|
||||
|
||||
**TDD process for T90:**
|
||||
|
||||
1. Read all 3 affected files + their tests.
|
||||
2. Implement 90.1 with test; commit.
|
||||
3. Implement 90.2 with test; commit.
|
||||
4. Implement 90.3 with test; commit.
|
||||
5. Run full suite — should be 343 + 3 = 346 (or +2 if 90.3 had no behavioral change).
|
||||
|
||||
---
|
||||
|
||||
## Wave 2 — Embedding & search services (parallel)
|
||||
|
||||
Three new service modules. Fully file-disjoint.
|
||||
|
||||
### Task 91: Embedding generation service
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/services/embeddings.py`
|
||||
- Create: `tests/test_embeddings.py`
|
||||
|
||||
**Spec:** Wraps the embedding API call. Signature:
|
||||
|
||||
```python
|
||||
class EmbeddingResult(BaseModel):
|
||||
vector: list[float]
|
||||
model: str
|
||||
dim: int
|
||||
|
||||
async def generate_embedding(
|
||||
client: LLMClient, # or a separate embedding-specific client
|
||||
*,
|
||||
text: str,
|
||||
model: str,
|
||||
timeout_s: float = 30.0,
|
||||
) -> EmbeddingResult:
|
||||
"""Generate an embedding vector for the given text. Falls back to a
|
||||
zero-vector with model='fallback' on failure (so callers get a deterministic
|
||||
sentinel they can detect and skip indexing)."""
|
||||
```
|
||||
|
||||
**Implementation:** call the embedding endpoint (Featherless OpenAI-compatible `/v1/embeddings`, or a local `sentence-transformers` model). Add a new method `client.embed(text, model)` to `LLMClient` Protocol (and to `MockLLMClient` and `FeatherlessClient`).
|
||||
|
||||
**Embedding model choice:**
|
||||
|
||||
Default to a small CPU-friendly model accessible through the existing Featherless setup:
|
||||
|
||||
- If Featherless has `BAAI/bge-small-en-v1.5` or similar 384-dim model: use that.
|
||||
- If not: fall back to local `sentence-transformers/all-MiniLM-L6-v2` (384-dim, runs CPU). Add `sentence-transformers` to `pyproject.toml`.
|
||||
- Document choice in CLAUDE.md (T102 docs sweep).
|
||||
|
||||
The 384 dim is hardcoded in T88's migration. If a different model with different dim is chosen, update T88's schema accordingly BEFORE T88 dispatches.
|
||||
|
||||
**Tests:** 3 minimum.
|
||||
|
||||
1. `test_generate_embedding_returns_vector_of_correct_dim`: mock embedding response with a 384-element vector; assert returned `vector` length is 384.
|
||||
2. `test_generate_embedding_returns_correct_model_metadata`: assert `result.model` matches the input.
|
||||
3. `test_generate_embedding_falls_back_on_failure`: mock the client to raise; assert the result is a 384-element zero vector with `model="fallback"`.
|
||||
|
||||
**Commit:** `feat: embedding generation service (T91)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 92: Vector search service via sqlite-vec
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/services/vector_search.py`
|
||||
- Create: `tests/test_vector_search.py`
|
||||
|
||||
**Spec:** Wraps sqlite-vec's `MATCH` syntax for cosine-similarity search over the `embeddings` virtual table. Witness-filter aware (joins through `memories` table for the witness check).
|
||||
|
||||
```python
|
||||
def vector_search(
|
||||
conn,
|
||||
*,
|
||||
owner_id: str,
|
||||
witness_role: str, # "you" | "host" | "guest"
|
||||
query_vector: list[float],
|
||||
k: int = 4,
|
||||
) -> list[dict]:
|
||||
"""Return top-K memories by cosine similarity to query_vector,
|
||||
witness-filtered for the requesting bot's POV. Returns same row
|
||||
shape as state.memory.search_memories for combined-ranking
|
||||
compatibility."""
|
||||
```
|
||||
|
||||
SQL pattern (sqlite-vec):
|
||||
|
||||
```sql
|
||||
SELECT m.id, m.text, m.pov_summary, m.significance, e.distance
|
||||
FROM embeddings e
|
||||
JOIN memories m ON m.id = e.memory_id
|
||||
WHERE e.embedding MATCH ?
|
||||
AND k = ?
|
||||
AND m.owner_id = ?
|
||||
AND m.witness_<role> = 1
|
||||
ORDER BY e.distance ASC
|
||||
LIMIT ?
|
||||
```
|
||||
|
||||
(Adapt to actual sqlite-vec syntax — use `vec0` MATCH semantics. The `witness_<role>` interpolation needs the same allowlist guard pattern as Phase 2.5 T72.3.)
|
||||
|
||||
**Tests:** 3 minimum.
|
||||
|
||||
1. `test_vector_search_returns_nearest_neighbors`: index 5 memories with synthetic vectors; query for nearest 3; assert correct order.
|
||||
2. `test_vector_search_respects_witness_filter`: index a memory with witness `[1, 1, 0]`; query with `witness_role="guest"`; assert empty result.
|
||||
3. `test_vector_search_respects_owner_filter`: index memories for two owners; assert query for owner_a doesn't return owner_b's memories.
|
||||
|
||||
**Commit:** `feat: vector search service via sqlite-vec (T92)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 93: Cross-chat search service
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/services/cross_chat_search.py`
|
||||
- Create: `tests/test_cross_chat_search.py`
|
||||
|
||||
**Spec:** FTS5-based search across ALL chats and all owners (admin-style search; no witness filter). For "where did I last see this person mention X?" queries.
|
||||
|
||||
```python
|
||||
def search_all_memories(
|
||||
conn,
|
||||
*,
|
||||
query: str,
|
||||
k: int = 20,
|
||||
) -> list[dict]:
|
||||
"""Search FTS across all owners and chats. Returns rows with
|
||||
{memory_id, owner_id, chat_id, text, pov_summary, scene_id,
|
||||
significance, ts}. Sorted by FTS rank."""
|
||||
```
|
||||
|
||||
This is intentionally NOT witness-filtered — it's a power-user search surface. The UI (T100) prompts the user to acknowledge they're seeing memories across POVs.
|
||||
|
||||
**Tests:** 3 minimum.
|
||||
|
||||
1. `test_search_all_memories_returns_matches_across_owners`: seed 2 owners with overlapping keyword; search; assert both owner's matches appear.
|
||||
2. `test_search_all_memories_orders_by_fts_rank`: seed memories with varying FTS-match strength; assert order.
|
||||
3. `test_search_all_memories_respects_k_limit`.
|
||||
|
||||
**Commit:** `feat: cross-chat search service (FTS5 over all owners) (T93)`.
|
||||
|
||||
---
|
||||
|
||||
## Wave 3 — Branching + delete services (parallel)
|
||||
|
||||
Two new service modules. Fully file-disjoint.
|
||||
|
||||
### Task 94: branch_from_event service
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/services/branching.py`
|
||||
- Create: `tests/test_branching.py`
|
||||
|
||||
**Spec:**
|
||||
|
||||
```python
|
||||
def branch_from_event(
|
||||
conn,
|
||||
*,
|
||||
name: str,
|
||||
origin_event_id: int,
|
||||
chat_id: str | None = None,
|
||||
) -> int:
|
||||
"""Create a new named branch forking from origin_event_id.
|
||||
Emits a branch_created event. Returns the new branch's row id.
|
||||
Raises ValueError if name already exists."""
|
||||
|
||||
def switch_active_branch(conn, *, name: str) -> None:
|
||||
"""Make the named branch active. Emits branch_switched. Subsequent
|
||||
event reads should consult is_active to filter."""
|
||||
|
||||
def list_branches_with_metadata(conn, chat_id: str | None = None) -> list[dict]:
|
||||
"""List branches with: name, origin_event_id, head_event_id, is_active,
|
||||
event_count (number of events between origin and head, inclusive),
|
||||
created_at."""
|
||||
```
|
||||
|
||||
Tests cover: basic create, duplicate-name raises, switch updates `is_active` exclusively, list returns metadata.
|
||||
|
||||
**Commit:** `feat: branching service (T94)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 95: Delete-impact computation service
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/services/delete_impact.py`
|
||||
- Create: `tests/test_delete_impact.py`
|
||||
|
||||
**Spec:** Computes the cascade impact of deleting a single event_log row (or a turn group: user_turn + assistant_turn + interjection if any). Returns a structured `ImpactReport` for the UI to render.
|
||||
|
||||
```python
|
||||
class DeletedItem(BaseModel):
|
||||
kind: str # "memory" | "edge_update" | "scene_close" | etc.
|
||||
description: str # human-readable
|
||||
target_id: int | str | None
|
||||
|
||||
class ImpactReport(BaseModel):
|
||||
target_event_id: int
|
||||
cascading: list[DeletedItem]
|
||||
notes: list[str] # warnings, e.g. "this turn opened scene_X which has 3 subsequent turns"
|
||||
|
||||
def compute_delete_impact(conn, *, target_event_id: int) -> ImpactReport:
|
||||
"""Walk the event log forward from target_event_id and identify
|
||||
everything that depends on this event: child memory_written events,
|
||||
edge_update events with this turn as source, scene_closed events
|
||||
triggered by this turn, etc. Also identify subsequent turns that
|
||||
REFERENCE this event (regenerated_from chains, etc.).
|
||||
|
||||
Does NOT mutate the database. Pure computation for preview."""
|
||||
```
|
||||
|
||||
The actual delete (truncate + supersede) is the existing rewind path from Phase 1 T31. T95 just builds the preview.
|
||||
|
||||
**Tests:** 4 minimum.
|
||||
|
||||
1. `test_impact_for_simple_turn_lists_memory_and_edges`: seed a chat with a turn that wrote 1 memory + 2 edge_updates. Compute impact. Assert the 3 items appear in `cascading`.
|
||||
2. `test_impact_for_scene_opening_turn_warns_about_subsequent_turns`: seed a turn that opened a scene + 5 subsequent turns. Assert `notes` mentions the dependency.
|
||||
3. `test_impact_for_regenerated_turn_lists_supersede_chain`: seed a turn that's been regenerated (has `superseded_by`). Compute impact for the original. Assert the chain appears.
|
||||
4. `test_impact_does_not_mutate_database`: snapshot event_log before + after; assert byte-identical.
|
||||
|
||||
**Commit:** `feat: delete-impact computation service (T95)`.
|
||||
|
||||
---
|
||||
|
||||
## Wave 4 — Combined retrieval ranking (single)
|
||||
|
||||
### Task 96: Combined FTS + vector retrieval ranking
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `chat/state/memory.py` — extend `search_memories` to optionally include vector hits
|
||||
- Modify: `tests/test_memory_search.py` — add 4 tests
|
||||
|
||||
**Spec:**
|
||||
|
||||
`search_memories` currently does FTS5 + Python-side significance/recency re-rank. Phase 4 adds:
|
||||
|
||||
- An optional `query_vector: list[float] | None = None` kwarg.
|
||||
- When `query_vector` is provided, run `vector_search` (T92) for top-K-vector candidates.
|
||||
- Merge with FTS top-K candidates via reciprocal-rank fusion (RRF) or a simpler sum-of-ranks scheme — implementer's choice. Document the merge formula.
|
||||
- Final result is top-K from the fused set, with the existing significance + recency boosts applied as a final pass.
|
||||
|
||||
When `query_vector` is None: existing behavior unchanged. Phase 1/2/3 callers that don't pass `query_vector` see no change.
|
||||
|
||||
**Implementation note:** the embedding for the query (the speaker's recent context) must be generated by the caller (Wave 5 T97 wires the prompt-assembly pipeline to call `generate_embedding` on the dialogue tail). T96 only handles the search side — assumes the vector is pre-computed.
|
||||
|
||||
**Tests:** 4 added.
|
||||
|
||||
1. `test_search_memories_without_query_vector_uses_fts_only`: regression — call without `query_vector`; assert the existing FTS+rerank behavior.
|
||||
2. `test_search_memories_with_query_vector_includes_vector_hits`: index 5 memories where 1 is FTS-only-matching, 1 is vector-only-matching, 3 are unrelated. Pass both `query=...` and `query_vector=...`. Assert both the FTS hit and the vector hit appear in results.
|
||||
3. `test_search_memories_fusion_significance_bias_still_applies`: confirm the existing significance bias rerank still works on top of fused results.
|
||||
4. `test_search_memories_fusion_handles_empty_vector_results`: pass a vector for a memory that has no embeddings indexed; assert FTS-only results still come back.
|
||||
|
||||
**Commit:** `feat: combined FTS + vector retrieval ranking (T96)`.
|
||||
|
||||
---
|
||||
|
||||
## Wave 5 — Memory write hook + backfill (single)
|
||||
|
||||
### Task 97: Embedding generation hook + backfill script
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `chat/services/memory_write.py` — after each `memory_written` event, enqueue a background embedding job
|
||||
- Create: `chat/services/embedding_worker.py` — async worker that consumes the queue and emits `embedding_indexed` events
|
||||
- Create: `scripts/backfill_embeddings.py` — one-time script that walks all existing memories and embeds them
|
||||
- Modify: `chat/app.py` — wire the embedding worker into the lifespan startup
|
||||
- Modify: `tests/test_memory_write.py` — add 2 tests for the enqueue hook
|
||||
- Create: `tests/test_embedding_worker.py` — 3 tests for the worker drain logic
|
||||
|
||||
**Spec:**
|
||||
|
||||
After each successful `memory_written` event, enqueue an embedding job. The worker dequeues and:
|
||||
|
||||
1. Reads the memory text (via `get_memory(conn, memory_id)`).
|
||||
2. Calls `generate_embedding(client, text=memory.text, model=settings.embedding_model)`.
|
||||
3. Appends `embedding_indexed` event with the result. (Skip if `result.model == "fallback"` — leave the memory un-indexed; will retry later via backfill.)
|
||||
|
||||
The worker pattern mirrors Phase 1's `chat/services/significance.py` SignificanceWorker. Reuse its queue + lifecycle pattern.
|
||||
|
||||
**Backfill script:**
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/backfill_embeddings.py [--limit N] [--dry-run]
|
||||
```
|
||||
|
||||
Walks all memories where no `embeddings_meta` row exists. For each, generates an embedding and emits `embedding_indexed`. Useful for the initial migration after Phase 4 lands AND for periodic re-runs if an embedding model changes.
|
||||
|
||||
**Tests:**
|
||||
|
||||
`tests/test_memory_write.py`:
|
||||
1. `test_record_turn_memory_enqueues_embedding_job`: monkeypatch the worker's enqueue method; record_turn_memory_for_present; assert the worker received a job per memory.
|
||||
|
||||
`tests/test_embedding_worker.py`:
|
||||
1. `test_worker_drains_jobs_and_emits_indexed_events`: enqueue 3 jobs with mock embeddings; run worker; assert 3 `embedding_indexed` events landed.
|
||||
2. `test_worker_skips_fallback_results`: mock the embedding service to return a fallback result; assert NO `embedding_indexed` event landed for that job.
|
||||
3. `test_worker_handles_concurrent_jobs_serially`: pin the Featherless 2-conn cap behavior (worker calls embed sequentially under the existing semaphore).
|
||||
|
||||
**Commit (split):**
|
||||
|
||||
- `feat: embedding worker drains queue and emits embedding_indexed events (T97.1)`
|
||||
- `feat: memory_write enqueues embedding job after each memory_written (T97.2)`
|
||||
- `feat: backfill_embeddings script for existing memories (T97.3)`
|
||||
|
||||
**Verification gates:**
|
||||
|
||||
- All Phase 1/2/3/3.5 memory tests still pass (regression critical).
|
||||
- New tests pass.
|
||||
- Manual smoke: run `scripts/backfill_embeddings.py --dry-run` against a seeded DB and verify expected count.
|
||||
|
||||
---
|
||||
|
||||
## Wave 6 — Drawer Phase 4 bundle (single task)
|
||||
|
||||
### Task 98: Drawer Phase 4 features
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `chat/web/drawer.py` (add many new POST routes and GET extensions)
|
||||
- Modify: `chat/templates/_drawer.html` (add 5 new sections)
|
||||
- Create: `tests/test_drawer_phase4.py`
|
||||
|
||||
**Spec:** Drawer affordances for 5 Phase 4 features. Single task by hot-file constraint; split into 5 commits internally.
|
||||
|
||||
#### 98.1 — Branching UI
|
||||
|
||||
GET drawer extension: `list_branches_with_metadata(conn)` → render in a "Branches" section (active branch highlighted + count of events).
|
||||
|
||||
POST routes:
|
||||
- `/drawer/branch/create` — form `{name, origin_event_id}` → `branch_from_event` service.
|
||||
- `/drawer/branch/switch` — form `{name}` → `switch_active_branch`.
|
||||
- `/drawer/branch/from-turn/{event_id}` — convenience: branch from a specific turn (used by per-turn UI affordance).
|
||||
|
||||
#### 98.2 — Significance review panel
|
||||
|
||||
GET extension: significance distribution per chat (`SELECT significance, COUNT(*) GROUP BY significance`) → render histogram.
|
||||
|
||||
POST route:
|
||||
- `/drawer/memory/significance/{memory_id}` — form `{new_value}` (already supported via T22 `manual_edit` `target_kind=memory_significance`); just add the UI form.
|
||||
|
||||
Bulk re-rate is a Phase 4.5 polish — not in scope here. Just per-memory edit + distribution display.
|
||||
|
||||
#### 98.3 — Hide-from-view toggle
|
||||
|
||||
POST route:
|
||||
- `/drawer/turn/hide/{event_id}` — form `{hidden: bool}` → emits a `manual_edit` with `target_kind="turn_hidden"`.
|
||||
|
||||
NEW `manual_edit` projector branch for `turn_hidden`: sets `event_log.hidden = ?` for the target event. Reuses the existing `hidden` column.
|
||||
|
||||
UI affordance: per-turn checkbox in the chat surface or drawer (per-turn list with hide toggle).
|
||||
|
||||
#### 98.4 — Surgical delete with cascade preview
|
||||
|
||||
GET extension:
|
||||
- `/drawer/turn/delete-preview/{event_id}` → returns the `ImpactReport` (T95) rendered as a modal.
|
||||
|
||||
POST route:
|
||||
- `/drawer/turn/delete/{event_id}` — invokes the rewind-and-truncate path (Phase 1 T31's `rewind_to_turn`) restricted to the target turn group.
|
||||
|
||||
Important: this reuses the existing pre-rewind snapshot path so the action is undoable.
|
||||
|
||||
#### 98.5 — Remaining v1 edits
|
||||
|
||||
Audit: are any v1 fields STILL not editable from the drawer? Phase 2.5 T72.1 added edge_trust/edge_summary/memory_pov_summary/edge_knowledge_facts. T72.3 added witness flags. Anything left?
|
||||
|
||||
Likely candidates: scene `narrative_anchor`, scene `weather`, container `properties` JSON. Add edit forms for any that surface during the audit. If none, this sub-fix is a no-op.
|
||||
|
||||
**Tests:** 8+ in `tests/test_drawer_phase4.py` (one per sub-feature × happy path; plus 1 for the cascade-preview rendering).
|
||||
|
||||
**Commits (5):**
|
||||
|
||||
- `feat: drawer branching UI (T98.1)`
|
||||
- `feat: drawer significance review panel (T98.2)`
|
||||
- `feat: drawer hide-from-view toggle + manual_edit turn_hidden branch (T98.3)`
|
||||
- `feat: drawer surgical delete with cascade preview (T98.4)`
|
||||
- `feat: drawer remaining v1 field edits (T98.5)` (or "no-op audit" if nothing left)
|
||||
|
||||
---
|
||||
|
||||
## Wave 7 — Snapshot + cross-chat search UX (parallel)
|
||||
|
||||
### Task 99: Snapshot UX
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/web/snapshots.py` (new route module)
|
||||
- Create: `chat/templates/snapshots.html` (snapshot list page)
|
||||
- Modify: `chat/templates/layout.html` (add "Snapshots" nav link)
|
||||
- Create: `tests/test_snapshot_ux.py`
|
||||
|
||||
**Spec:** Surface the existing snapshot infrastructure (Phase 1 T20 wrote snapshots; Phase 4 makes them visible).
|
||||
|
||||
GET `/snapshots` — list all snapshots (periodic + pre-rewind) with metadata: kind, created_at, event_log_size, file_size_bytes.
|
||||
|
||||
POST `/snapshots/take` — manually trigger a snapshot now.
|
||||
|
||||
POST `/snapshots/restore/{snapshot_id}` — restore from snapshot (with hard confirmation).
|
||||
|
||||
GET `/snapshots/{snapshot_id}/preview` — show what's in the snapshot vs. current state.
|
||||
|
||||
**Tests:** 4 minimum (list, take, restore, preview).
|
||||
|
||||
**Commit:** `feat: snapshot UX (manual trigger, list, restore) (T99)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 100: Cross-chat search UX
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `chat/web/search.py` (new route module)
|
||||
- Create: `chat/templates/search.html` (search results page)
|
||||
- Modify: `chat/templates/layout.html` (add top-bar search input)
|
||||
- Create: `tests/test_search_ux.py`
|
||||
|
||||
**Spec:** Top-bar search box submits to `/search?q=...`. Results page shows up to 50 matches across all chats and all owners (uses T93's `search_all_memories`). Each result shows: chat name, owner bot name, scene context, memory text excerpt with FTS highlight, "Open chat at this turn" link.
|
||||
|
||||
**Tests:** 3 minimum.
|
||||
1. Search returns results from multiple chats.
|
||||
2. Empty query returns empty result set.
|
||||
3. Result links navigate to the right chat anchor.
|
||||
|
||||
**Commit:** `feat: cross-chat search UX (top-bar input + results page) (T100)`.
|
||||
|
||||
---
|
||||
|
||||
## Wave 8 — Polish (parallel)
|
||||
|
||||
### Task 101: Cross-feature integration tests
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `tests/test_phase4_integration.py`
|
||||
|
||||
**Spec:** End-to-end multi-feature flows. 5 tests minimum.
|
||||
|
||||
1. **Vector retrieval feedback loop**: write a memory → embedding worker indexes it → search retrieves it via vector path.
|
||||
2. **Branch + diverge**: create branch B from turn 10 → switch to B → play 3 new turns → switch back to main → assert main's turn 11+ are still intact.
|
||||
3. **Surgical delete**: compute impact for a turn → confirm → assert event log truncated correctly + pre-rewind snapshot saved.
|
||||
4. **Hide + retrieval**: hide a turn → assert it doesn't appear in `read_recent_dialogue` (existing `hidden = 0` filter) → unhide → assert it reappears.
|
||||
5. **Cross-chat search**: write memories in 3 chats → search for keyword present in all 3 → assert all 3 appear in results.
|
||||
|
||||
**Commit:** `test: phase 4 cross-feature integration coverage (T101)`.
|
||||
|
||||
---
|
||||
|
||||
### Task 102: Phase 4 documentation update
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `CLAUDE.md` (add "Phase 4 status" section; update behavioral defaults; add "Phase 4.5 / 5 backlog" with carry-overs)
|
||||
- Modify: `docs/plans/2026-04-26-v1-requirements-design.md` (annotate §13 Phase 4 as **Status: shipped 2026-04-27**)
|
||||
|
||||
**Spec:**
|
||||
|
||||
Mirror the Phase 3 / 3.5 status sections. Document:
|
||||
|
||||
- **Vector retrieval**: sqlite-vec virtual table, embedding worker async pipeline, combined FTS + vector ranking via RRF.
|
||||
- **Branching**: forks the event log; UI in drawer; `is_active` flag plus orchestrator filter (caveat — see backlog if filter not yet wired into all readers).
|
||||
- **Drawer-edit on every field**: branching, significance review, hide-from-view, surgical delete with preview, plus any audit findings.
|
||||
- **Backup tooling**: snapshots panel surfaces existing infra.
|
||||
- **Significance review UI**: distribution + per-memory edit.
|
||||
- **Surgical delete + cascade preview**: piggybacks on rewind path; impact report from T95.
|
||||
- **Hide-from-view soft delete**: `manual_edit` `turn_hidden` branch.
|
||||
- **Cross-chat search**: top-bar + results page over T93's service.
|
||||
|
||||
**Phase 4.5 / 5 backlog candidates** (reflect any discovered during execution):
|
||||
|
||||
- Branching read-side filter — if T89's `is_active` isn't yet consulted by every event reader, this is the work to do.
|
||||
- Bulk significance re-rate (per T98.2 deferral).
|
||||
- Snapshot retention policy UI controls (per Phase 1 T19 deferred).
|
||||
- Auto-pin override UI (per Phase 2 design).
|
||||
- Embedding model swap migration tooling (when changing embedding model, need to re-embed everything).
|
||||
- Vector index optimization (HNSW vs flat — Phase 5 if needed).
|
||||
- Carry-overs that remained deferred from Phase 3.6: scene-close-on-cancel UX revisit, canned-queue brittleness fixture builder, full lifecycle rollback in regenerate.
|
||||
|
||||
**Commit:** `docs: phase 4 status, behavioral defaults, deferred items (T102)`.
|
||||
|
||||
---
|
||||
|
||||
## Wrap-up
|
||||
|
||||
After Wave 8 lands:
|
||||
|
||||
1. **Run full suite** on `phase-4`: should be ~390+ tests passing (343 from Phase 3.5 + ~50 new).
|
||||
2. **Manual smoke** (recommended before opening the PR):
|
||||
- Run `scripts/backfill_embeddings.py` against a seeded DB to verify vector indexing works.
|
||||
- Search for a phrase that's substring-distinct but semantically similar to a memory; verify vector path returns it (FTS would miss).
|
||||
- Create a branch from an old turn; switch; play a few turns; switch back.
|
||||
- Trigger surgical delete on a turn; verify the impact preview matches what actually gets removed.
|
||||
- Hide a turn; verify it disappears from the chat surface; unhide.
|
||||
- Use top-bar search to find a phrase; verify cross-chat results appear.
|
||||
- Click the "Snapshots" nav link; trigger a manual snapshot; verify it appears.
|
||||
3. **Push `phase-4`** to gitea.
|
||||
4. **Open PR** `phase-4 → main`.
|
||||
|
||||
---
|
||||
|
||||
## Notes for the controller running this plan
|
||||
|
||||
- **External dependency**: `sqlite-vec` (or `sqlite-vss`) MUST be added to `pyproject.toml` and installed BEFORE Wave 1 dispatches. The migration in T88 expects the extension to be loadable.
|
||||
- **Embedding model choice**: pin in T91 spec before dispatch. The 384 dim is hardcoded in T88's migration; if a different dim is used, update T88 first.
|
||||
- **After each parallel wave**, run a code-review subagent. Combined spec+quality acceptable for trivial tasks (T90 carry-overs); separate spec + quality reviewers for vector-retrieval and integration tasks (T91, T96, T97, T98, T101) — surface area is larger.
|
||||
- **Don't dispatch Wave 5 until Wave 4 merged green.** T97 (memory_write enqueue) calls into the embedding-aware worker; the worker uses T91's `generate_embedding`. Both must be merged into `phase-4` first.
|
||||
- **Don't dispatch Wave 6 until Wave 5 merged green.** T98 (drawer) wires UI affordances over services from earlier waves.
|
||||
- **Token-spend rough estimate**: Phase 4 should be ~70-80% the size of Phase 3 (similar scope, larger per-task because vector + branching are non-trivial). Per-task spend similar to Phase 3's larger tasks (T59, T64).
|
||||
- **DO NOT break existing v1/v2/v3/v3.5 surface contracts.** Every test file that was green at the start of Phase 4 must stay green at the end. The cross-feature integration tests from Phase 3 (`tests/test_phase3_integration.py`) are particularly load-bearing.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-04-27-v4-phase4-implementation.md",
|
||||
"tasks": [
|
||||
{"id": 88, "subject": "T88: embeddings table + projector handlers (sqlite-vec)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||
{"id": 89, "subject": "T89: branches table + projector handlers", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||
{"id": 90, "subject": "T90: phase 3.6 carry-overs (chat-id pushdown + lifecycle wording + legacy fn consolidation)", "status": "pending", "wave": 1, "parallelGroup": "wave-1"},
|
||||
{"id": 91, "subject": "T91: embedding generation service", "status": "pending", "wave": 2, "parallelGroup": "wave-2", "blockedBy": [88]},
|
||||
{"id": 92, "subject": "T92: vector search service via sqlite-vec", "status": "pending", "wave": 2, "parallelGroup": "wave-2", "blockedBy": [88]},
|
||||
{"id": 93, "subject": "T93: cross-chat search service (FTS5 over all owners)", "status": "pending", "wave": 2, "parallelGroup": "wave-2"},
|
||||
{"id": 94, "subject": "T94: branch_from_event service", "status": "pending", "wave": 3, "parallelGroup": "wave-3", "blockedBy": [89]},
|
||||
{"id": 95, "subject": "T95: delete-impact computation service", "status": "pending", "wave": 3, "parallelGroup": "wave-3"},
|
||||
{"id": 96, "subject": "T96: combined FTS + vector retrieval ranking in search_memories", "status": "pending", "wave": 4, "parallelGroup": null, "blockedBy": [91, 92]},
|
||||
{"id": 97, "subject": "T97: memory_write enqueues embedding job + backfill script", "status": "pending", "wave": 5, "parallelGroup": null, "blockedBy": [91, 96]},
|
||||
{"id": 98, "subject": "T98: drawer Phase 4 bundle (branching + sig review + hide + surgical delete + remaining edits)", "status": "pending", "wave": 6, "parallelGroup": null, "blockedBy": [94, 95, 97]},
|
||||
{"id": 99, "subject": "T99: snapshot UX (manual trigger + list + restore + preview)", "status": "pending", "wave": 7, "parallelGroup": "wave-7"},
|
||||
{"id": 100, "subject": "T100: cross-chat search UX (top-bar + results page)", "status": "pending", "wave": 7, "parallelGroup": "wave-7", "blockedBy": [93]},
|
||||
{"id": 101, "subject": "T101: cross-feature integration tests (vector × branching × delete × snapshot × search)", "status": "pending", "wave": 8, "parallelGroup": "wave-8", "blockedBy": [98, 99, 100]},
|
||||
{"id": 102, "subject": "T102: Phase 4 documentation update", "status": "pending", "wave": 8, "parallelGroup": "wave-8", "blockedBy": [98, 99, 100]}
|
||||
],
|
||||
"lastUpdated": "2026-04-27T00:00:00Z",
|
||||
"notes": "15 tasks across 8 waves. Adds vector retrieval (sqlite-vec), branching UI, drawer-edit on every field, backup tooling, significance review UI, surgical delete with cascade preview, hide-from-view, and cross-chat search. Phase 3.6 carry-overs (3 small fixes) bundled into T90. External dependency: sqlite-vec must be installed BEFORE Wave 1 dispatch. Embedding model choice (default: 384-dim small model) pinned in T91 spec before dispatch — schema 0012 hardcodes 384 dim. Two new schema migrations (0012 embeddings, 0013 branches), final schema version 13. Uses task ids T88-T102."
|
||||
}
|
||||
Reference in New Issue
Block a user