LangMem vs Plain RAG: accuracy, cost and latency
A memory system against a no-memory baseline: does the memory layer earn its cost? Independent results on the same 30 questions of LongMemEval-S (cleaned, 2025-09), with the same reader model and the official judge. Pilot sample, so accuracy is shown with 95% intervals.
Verdict
Generated from the pilot data, n=30 per systemIs a memory layer worth it here?
Not in this pilot: the Plain RAG baseline is more accurate than LangMem (LangMem 33.3% vs Plain RAG 86.7%; the 95% intervals do not overlap), and LangMem costs 11× more per 1,000 questions. The reader sees a median of 23k characters of context per question with LangMem and 127k with Plain RAG.
- AccuracyPlain RAG is more accurate: 86.7% (95% CI 70–95%) vs 33.3% (95% CI 19–51%) for LangMem. The 95% intervals do not overlap, even at n=30.
- CostPlain RAG costs 11× less per 1,000 questions: $5.45 vs $61.6 for LangMem.
- LatencyLangMem answers faster: median 3.5s vs 4.8s for Plain RAG per question (p90 5.3s vs 7.4s).
- IngestionPlain RAG ingests a chat history faster: 1s vs 8.5 min for LangMem (median per question).
Key numbers side by side
Bold marks the better value. Accuracy rows are not bolded when the 95% intervals overlap.
| Metric | LangMem | Plain RAG |
|---|---|---|
| Accuracy (official judge) | 33.3% | 86.7% |
| 95% interval | 19–51% | 70–95% |
| Panel accuracy | 26.7% | 86.7% |
| Cost per 1,000 questions | $61.6 | $5.45 |
| of which memory side | $61.0 | $2.08 |
| of which answering | $0.61 | $3.37 |
| Latency p50 | 3.5s | 4.8s |
| Latency p90 | 5.3s | 7.4s |
| Ingestion per history (median) | 8.5 min | 1s |
| Context given to reader (median chars) | 23k | 127k |
LangMem and Plain RAG among all tested systems
LangMem and Plain RAG are highlighted; other systems are muted for context, baselines lightest. Each name links to the system's page.
Accuracy
Higher is betterShare of 30 questions answered correctly (official judge). Whiskers: 95% interval.
- Cognee90.0%74–97%
- Hindsight90.0%74–97%
- Plain RAG86.7%70–95%
- Mem083.3%66–93%
- Full context83.3%66–93%
- LangMem33.3%19–51%
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Cost per 1,000 questions
Lower is betterUSD billed (prices as of 2026-10-08). Solid: memory side (ingestion and retrieval). Light: answering.
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Latency per question
Lower is betterRetrieval plus answer, in seconds. Solid: median (p50). Light: up to p90.
- LangMem3.5sp90 5.3s
- Cognee4.0sp90 8.1s
- Mem04.3sp90 7.4s
- Full context4.8sp90 8.0s
- Plain RAG4.8sp90 7.4s
- Hindsight5.4sp90 8.1s
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Ingestion time
Lower is betterMedian time to load one question's chat history into the system.
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Accuracy by question type
Difference is LangMem minus Plain RAG, in percentage points. Each type has only a few pilot questions (n), so one question can move a row by 13 points or more.
| Question type | n | LangMem | Plain RAG | Difference |
|---|---|---|---|---|
| Single-session (user) | 4 | 75% | 100% | −25 pts in favour of Plain RAG |
| Single-session (assistant) | 3 | 0% | 100% | −100 pts in favour of Plain RAG |
| Preferences | 2 | 50% | 100% | −50 pts in favour of Plain RAG |
| Multi-session | 8 | 13% | 75% | −62 pts in favour of Plain RAG |
| Knowledge update | 4 | 50% | 100% | −50 pts in favour of Plain RAG |
| Temporal reasoning | 8 | 38% | 88% | −50 pts in favour of Plain RAG |
| Abstention | 1 | 0% | 0% | 0 |
What each one is
LangMem
A memory manager that asks an LLM to extract, consolidate and update memories in a LangGraph store.
How we ran it
create_memory_store_manager with default instructions, one invoke() per session, LangGraph InMemoryStore with OpenAI embeddings, store.search() with the default limit (10).
Plain RAG
No memory system. Each past session is embedded once; the 10 most similar sessions are given to the reader in date order.
How we ran it
text-embedding-3-small, cosine similarity, top 10 sessions.
What the vendors report
LangMem
| Score | Variant | Reader | Judge | Note | Source |
|---|---|---|---|---|---|
| 33.3% | S (cleaned, 2025-09), 30-question pilot | openai/gpt-6-luna | gpt-4o-2024-08-06, official prompt | MemVerdict measurement, LangMem 0.0.30 | This page |
| None found | - | - | - | No self-reported LongMemEval score found. | - |
Vendor-reported scores and ours are measured differently, so a gap does not by itself mean either number is wrong. Our pilot uses openai/gpt-6-luna as the reader that writes each answer, gpt-4o-2024-08-06 with the official LongMemEval judge prompt, LongMemEval-S (cleaned, 2025-09) and 30 questions, with each system set up as described above. Scores move with the reader model, the judge model and its prompt, the dataset version, the number of questions, and whether a hosted platform or the open-source package is tested.
Frequently asked questions
Is LangMem better than Plain RAG?
Plain RAG is more accurate: 86.7% (95% CI 70–95%) vs 33.3% (95% CI 19–51%) for LangMem. The 95% intervals do not overlap, even at n=30. With the 3-judge panel majority instead of the official judge: LangMem 26.7%, Plain RAG 86.7%. The largest gap by question type is Multi-session: 1 of 8 vs 6 of 8 correct, in Plain RAG's favour; each question type has only 1 to 8 questions in this pilot. These are pilot numbers (30 questions each); the full 500-question run will narrow the intervals.
Which is cheaper, LangMem or Plain RAG?
Plain RAG costs 11× less per 1,000 questions: $5.45 vs $61.6 for LangMem. LangMem: $61.6 per 1,000 questions, of which $61.0 (99%) is memory-side (ingestion and retrieval) and $0.61 is answering. Plain RAG: $5.45 per 1,000 questions, of which $2.08 (38%) is memory-side (ingestion and retrieval) and $3.37 is answering. Prices as of 2026-10-08.
Which is faster, LangMem or Plain RAG?
LangMem answers faster: median 3.5s vs 4.8s for Plain RAG per question (p90 5.3s vs 7.4s). Plain RAG ingests a chat history faster: 1s vs 8.5 min for LangMem (median per question). Latency is measured per question as retrieval plus answering; ingestion is the time to load one question's chat history.
How were LangMem and Plain RAG tested?
Both were run on the same 30 questions of LongMemEval-S (cleaned, 2025-09) as every other system, by the same harness. For each question, the system ingests that question's chat history, then retrieves context that one reader model (openai/gpt-6-luna) uses to answer with the official LongMemEval prompt. Answers are graded by the official judge (gpt-4o-2024-08-06) and cross-checked by 3 other judges; all judges agreed on 95% of graded answers. Costs are what the providers billed (prices as of 2026-10-08). LangMem: create_memory_store_manager with default instructions, one invoke() per session, LangGraph InMemoryStore with OpenAI embeddings, store.search() with the default limit (10). Plain RAG: text-embedding-3-small, cosine similarity, top 10 sessions.
Which should I use, LangMem or Plain RAG?
Not in this pilot: the Plain RAG baseline is more accurate than LangMem (LangMem 33.3% vs Plain RAG 86.7%; the 95% intervals do not overlap), and LangMem costs 11× more per 1,000 questions. The reader sees a median of 23k characters of context per question with LangMem and 127k with Plain RAG. If the Plain RAG baseline is as accurate on your own data, it is the simpler option to run; test both on a sample of your real conversations before committing. These are pilot numbers (30 questions each); the full 500-question run will narrow the intervals.
More comparisons
Other comparisons with LangMem
- Cognee vs LangMemCognee more accurate; LangMem 2.3× cheaper
- Full context vs LangMemFull context more accurate and 4.7× cheaper
- Graphiti vs LangMemGraphiti: pilot result withdrawn, rerun in progress
- Hindsight vs LangMemHindsight more accurate; LangMem 2.8× cheaper
- LangMem vs Mem0Mem0 more accurate; LangMem 1.5× cheaper
Other comparisons with Plain RAG
- Cognee vs Plain RAGStatistically tied on accuracy at n=30; Plain RAG 26× cheaper
- Full context vs Plain RAGStatistically tied on accuracy at n=30; Plain RAG 2.4× cheaper
- Graphiti vs Plain RAGGraphiti: pilot result withdrawn, rerun in progress
- Hindsight vs Plain RAGStatistically tied on accuracy at n=30; Plain RAG 31× cheaper
- Mem0 vs Plain RAGStatistically tied on accuracy at n=30; Plain RAG 17× cheaper