MemVerdict

Pilot results: 30 of 500 questions per system. Gaps of a few points are within noise; 95% intervals are shown. The full run is in progress.

Memory system

LangMem: LongMemEval results

A memory manager that asks an LLM to extract, consolidate and update memories in a LangGraph store.

Vendor
LangChain
Version tested
0.0.30
License
MIT
Repository
github.com/langchain-ai/langmem

Headline numbers

Pilot: 30 questions of LongMemEval-S (cleaned, 2025-09). With this few questions, gaps of several points between systems are within noise, so read accuracy together with its 95% interval.

Accuracy (official judge)
33.3%
95% CI 19–51% · 10/30 correct
Rank by accuracy
#6 of 6
Among all scored systems, baselines included. #4 of 4 memory systems.
Panel accuracy (3-judge majority)
26.7%
Cross-check with gpt-6-sol, claude-sonnet-5.5, gemini-3.1-pro-preview.
Cost per 1,000 questions
$61.6
Memory side $61.0 · answering $0.61 · prices as of 2026-10-08
Latency per question
3.5s p50
p90 5.3s · retrieval plus answer
Ingestion and context
8.5 min
Median time to ingest one chat history; the reader sees a median 23k characters of context.

How LangMem compares

All scored systems on the same questions, reader and judge. LangMem is highlighted. Baselines are shown in a lighter grey.

Accuracy

Higher is better

Share of 30 questions answered correctly (official judge). Whiskers: 95% interval.

  1. Cognee
    90.0%74–97%
  2. Hindsight
    90.0%74–97%
  3. Plain RAG
    86.7%70–95%
  4. Mem0
    83.3%66–93%
  5. Full context
    83.3%66–93%
  6. LangMem
    33.3%19–51%

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Cost per 1,000 questions

Lower is better

USD billed (prices as of 2026-10-08). Solid: memory side (ingestion and retrieval). Light: answering.

  1. Plain RAG
    $5.45
  2. Full context
    $13.1
  3. LangMem
    $61.6
  4. Mem0
    $93.0
  5. Cognee
    $141
  6. Hindsight
    $172

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Latency per question

Lower is better

Retrieval plus answer, in seconds. Solid: median (p50). Light: up to p90.

  1. LangMem
    3.5sp90 5.3s
  2. Cognee
    4.0sp90 8.1s
  3. Mem0
    4.3sp90 7.4s
  4. Full context
    4.8sp90 8.0s
  5. Plain RAG
    4.8sp90 7.4s
  6. Hindsight
    5.4sp90 8.1s

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Ingestion time

Lower is better

Median time to load one question's chat history into the system.

  1. Full context
    none
  2. Plain RAG
    1s
  3. Cognee
    2.8 min
  4. LangMem
    8.5 min
  5. Hindsight
    8.8 min
  6. Mem0
    15.2 min

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Accuracy by question type

LongMemEval groups questions by the memory ability they test. n is the number of pilot questions in each group. Highest: Single-session (user) (n=4) at 75%. Lowest: Abstention (n=1), Single-session (assistant) (n=3) at 0%. Category samples are small, so treat these as hints.

  • Single-session (user) n=475% (3/4)
  • Single-session (assistant) n=30% (0/3)
  • Preferences n=250% (1/2)
  • Multi-session n=813% (1/8)
  • Knowledge update n=450% (2/4)
  • Temporal reasoning n=838% (3/8)
  • Abstention n=10% (0/1)

How we ran it

create_memory_store_manager with default instructions, one invoke() per session, LangGraph InMemoryStore with OpenAI embeddings, store.search() with the default limit (10).

Every system gets each question's chat history, then returns context that the same reader model (openai/gpt-6-luna) uses to answer with the official LongMemEval prompt. Answers are graded by the official judge (gpt-4o-2024-08-06) and cross-checked by 3 other judges. Full details are on the methodology page.

What the vendor reports

LongMemEval scores reported for LangMem
ScoreVariantReaderJudgeNoteSource
33.3%S (cleaned, 2025-09), 30-question pilotopenai/gpt-6-lunagpt-4o-2024-08-06, official promptMemVerdict measurement, LangMem 0.0.30This page
None found---No self-reported LongMemEval score found.-

Vendor-reported scores and ours are measured differently, so a gap does not by itself mean either number is wrong. Our pilot uses openai/gpt-6-luna as the reader that writes each answer, gpt-4o-2024-08-06 with the official LongMemEval judge prompt, LongMemEval-S (cleaned, 2025-09) and 30 questions, with LangMem 0.0.30 set up as described above. Scores move with the reader model, the judge model and its prompt, the dataset version, the number of questions, and whether a hosted platform or the open-source package is tested.

Frequently asked questions

How accurate is LangMem on LongMemEval?

LangMem answered 10 of 30 questions correctly (33.3%, 95% CI 19–51%) under the official judge. Rank 6 of 6 scored systems including baselines; 4 of 4 memory systems. The 3-judge panel majority gives 26.7%. This is a 30-question pilot of LongMemEval-S (cleaned, 2025-09); the full 500-question run is in progress.

How much does LangMem cost to run?

LangMem: $61.6 per 1,000 questions, of which $61.0 (99%) is memory-side (ingestion and retrieval) and $0.61 is answering. Memory-side cost covers ingesting each question's chat history and retrieving from it, as billed by the providers (prices as of 2026-10-08).

How fast is LangMem?

Median latency is 3.5s per question (p90 5.3s), measured as retrieval plus answering. Ingesting one question's chat history takes 8.5 min (median).

Has LangChain published a LongMemEval score for LangMem?

No self-reported LongMemEval score found. The score on this page is our own independent measurement of LangMem 0.0.30.

Does LangMem beat the simple baselines?

Plain RAG is more accurate: 86.7% (95% CI 70–95%) vs 33.3% (95% CI 19–51%) for LangMem. The 95% intervals do not overlap, even at n=30. Full context is more accurate: 83.3% (95% CI 66–93%) vs 33.3% (95% CI 19–51%) for LangMem. The 95% intervals do not overlap, even at n=30.

Compare LangMem with