Step 1 of 4
Build the lexical layer
Exact matches are the floor of trustworthy recall.
Implement lexical search over memory entries. Requirements: tokenize entries and queries (lowercase, split identifiers on common separators so auth_handler matches 'auth handler'); score with BM25 (use SQLite FTS5 or a small library - do not hand-roll unless trivial); support quoted exact-phrase matching for error strings; return top-K with matched terms highlighted per result. Test with queries that MUST work lexically: an exact file name, an exact error message substring, and a specific command flag - all should hit their entries at rank 1.
Expected after this step
BM25 lexical search passing the exact-match tests.
Should not happen
- ✕Vector-only recall that cannot find an exact error string
- ✕Lexical-only recall that misses every paraphrase
- ✕Unbounded result lists stuffing recall noise into context
- ✕Opaque retrieval nobody can debug when the wrong memory surfaces
Verify before continuing
Do not move on until every check is true. The complete button stays locked until then.
Do not continue if…
- !Vector-only recall that cannot find an exact error string
- !Lexical-only recall that misses every paraphrase
- !Unbounded result lists stuffing recall noise into context
- !Opaque retrieval nobody can debug when the wrong memory surfaces
If the AI messes this up
Use this when the AI fakes progress or breaks the feature. It forces a real fix.
Exact file names miss. Check tokenization of paths and identifiers - index both the raw token (auth_handler.py) and its split parts (auth, handler, py).