Vector vs Markdown Memory: Make the Decision
Choose your memory backend on evidence: when plain files win, when vectors earn their complexity, and how to test it on your own data.
The route
3 steps to Done
- 01
Profile the workload honestly
Scale and query style decide this - measure both.
Preview prompt + verify gate ▾ Hide ▴
Profile my memory workload. Quantify: current entry count and average length; growth per week (from session frequency); query styles from real usage - what fraction are exact lookups (names, errors, commands) vs paraphrases vs broad topical asks (sample 20 real queries if logs exist, else write 20 realistic ones); latency needs; and inspectability requirements (do humans read/edit memory? does it live in git?). Research context: practitioner systems report plain files serving well into the hundreds of entries when queries skew exact; vector value concentrates in paraphrase-heavy, large-store recall. Deliver the profile with the 20-query sample labeled by style.
- ✓ Counts and growth measured, not guessed
- ✓ 20 queries labeled exact/paraphrase/topical
- ✓ Inspectability needs stated
- 02
Benchmark both on your data
An afternoon of testing beats a month of opinions.
Preview prompt + verify gate ▾ Hide ▴
Run the benchmark. Setup A (files): memories in markdown, recall via lexical search (grep/BM25 as available). Setup B (vectors): same memories embedded, cosine top-K recall. Load the SAME memory set into both. Run the 20 labeled queries against each, judging the top-3 results correct/incorrect against known answers. Report per style: exact-lookup accuracy, paraphrase accuracy, topical usefulness, plus setup time and moving parts count for each backend. Also test the hybrid intuition: note queries where the two setups' correct answers do not overlap - a high count is the argument for hybrid.
- ✓ Same data both sides
- ✓ Accuracy judged against known answers
- ✓ Non-overlap cases counted for the hybrid case
- 03
Decide and set the tripwires
Commit on the numbers; leave a door for the future.
Preview prompt + verify gate ▾ Hide ▴
Make the decision. Score files, vectors, and hybrid on: recall quality per my query mix (weighted by the actual style ratios), operational cost (dependencies, embedding upkeep, backup story), debuggability (can I see why recall returned X?), inspectability (human read/edit, git-diffable), and migration cost later. Write the decision paragraph citing the benchmark numbers. Then set revisit triggers with values: entry count exceeding N, paraphrase-miss rate above M% in recall logs, or recall latency above T. Add the trigger checks to the memory health metrics if those exist. Whatever wins, ensure a human-readable export path exists so memory is never opaque.
- ✓ Scores weighted by the real query mix
- ✓ Decision cites the benchmark
- ✓ Triggers have numeric thresholds wired to metrics where possible
Research-backed
Sources behind this flow
Tier 3 · memory
semantic-memory (RecursiveIntell)
Local-first hybrid semantic search backed by authoritative SQLite state plus a vector sidecar: facts, chunked documents, conversation messages, and episodes searched via BM25 (FTS5) and vector retrieval fused with Reciprocal Rank Fusion, with explainable search.
Tier 3 · memory
Agent Memory Systems Survey
A panoramic survey (2026.03) of 30+ agent memory projects across 15 dimensions - standalone memory frameworks plus agents' built-in memory - including LongMemEval scores. Notably documents the trend of agentic/observational memory overtaking pure vector memory.