MemVerdict

Pilot results: 30 of 500 questions per system. Gaps of a few points are within noise; 95% intervals are shown. The full run is in progress.

Memory system

Hindsight: LongMemEval results

Extracts facts into a memory bank on Postgres, consolidates them into observations in the background, and recalls with hybrid search plus a local reranker.

Vendor
Vectorize
Version tested
0.10.3
License
MIT
Repository
github.com/vectorize-io/hindsight

Headline numbers

Pilot: 30 questions of LongMemEval-S (cleaned, 2025-09). With this few questions, gaps of several points between systems are within noise, so read accuracy together with its 95% interval.

Accuracy (official judge)
90.0%
95% CI 74–97% · 27/30 correct
Rank by accuracy
#1 of 6
Among all scored systems, baselines included. Tied with Cognee. #1 of 4 memory systems.
Panel accuracy (3-judge majority)
86.7%
Cross-check with gpt-6-sol, claude-sonnet-5.5, gemini-3.1-pro-preview.
Cost per 1,000 questions
$172
Memory side $171 · answering $0.71 · prices as of 2026-10-08
Latency per question
5.4s p50
p90 8.1s · retrieval plus answer
Ingestion and context
8.8 min
Median time to ingest one chat history; the reader sees a median 18k characters of context.

How Hindsight compares

All scored systems on the same questions, reader and judge. Hindsight is highlighted. Baselines are shown in a lighter grey.

Accuracy

Higher is better

Share of 30 questions answered correctly (official judge). Whiskers: 95% interval.

  1. Cognee
    90.0%74–97%
  2. Hindsight
    90.0%74–97%
  3. Plain RAG
    86.7%70–95%
  4. Mem0
    83.3%66–93%
  5. Full context
    83.3%66–93%
  6. LangMem
    33.3%19–51%

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Cost per 1,000 questions

Lower is better

USD billed (prices as of 2026-10-08). Solid: memory side (ingestion and retrieval). Light: answering.

  1. Plain RAG
    $5.45
  2. Full context
    $13.1
  3. LangMem
    $61.6
  4. Mem0
    $93.0
  5. Cognee
    $141
  6. Hindsight
    $172

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Latency per question

Lower is better

Retrieval plus answer, in seconds. Solid: median (p50). Light: up to p90.

  1. LangMem
    3.5sp90 5.3s
  2. Cognee
    4.0sp90 8.1s
  3. Mem0
    4.3sp90 7.4s
  4. Full context
    4.8sp90 8.0s
  5. Plain RAG
    4.8sp90 7.4s
  6. Hindsight
    5.4sp90 8.1s

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Ingestion time

Lower is better

Median time to load one question's chat history into the system.

  1. Full context
    none
  2. Plain RAG
    1s
  3. Cognee
    2.8 min
  4. LangMem
    8.5 min
  5. Hindsight
    8.8 min
  6. Mem0
    15.2 min

Not shown: Graphiti (pilot result withdrawn, rerun in progress).

Accuracy by question type

LongMemEval groups questions by the memory ability they test. n is the number of pilot questions in each group. Highest: Abstention (n=1), Knowledge update (n=4), Preferences (n=2), Single-session (user) (n=4), Temporal reasoning (n=8) at 100%. Lowest: Single-session (assistant) (n=3) at 33%. Category samples are small, so treat these as hints.

  • Single-session (user) n=4100% (4/4)
  • Single-session (assistant) n=333% (1/3)
  • Preferences n=2100% (2/2)
  • Multi-session n=888% (7/8)
  • Knowledge update n=4100% (4/4)
  • Temporal reasoning n=8100% (8/8)
  • Abstention n=1100% (1/1)

How we ran it

Embedded daemon (pg0 Postgres), one bank per question. Sessions retained with their real timestamp; recall with the question date and default budget. We wait for background consolidation to finish before asking.

Every system gets each question's chat history, then returns context that the same reader model (openai/gpt-6-luna) uses to answer with the official LongMemEval prompt. Answers are graded by the official judge (gpt-4o-2024-08-06) and cross-checked by 3 other judges. Full details are on the methodology page.

What the vendor reports

LongMemEval scores reported for Hindsight
ScoreVariantReaderJudgeNoteSource
90.0%S (cleaned, 2025-09), 30-question pilotopenai/gpt-6-lunagpt-4o-2024-08-06, official promptMemVerdict measurement, Hindsight 0.10.3This page
91.4%SGemini 3 ProGPT-OSS-120B-arxiv.org/abs/2512.12818
94.6%S (cleaned)Gemini 3.1 ProGemini 2.5 Flash-Lite-benchmarks.hindsight.vectorize.io

Vendor-reported scores and ours are measured differently, so a gap does not by itself mean either number is wrong. Our pilot uses openai/gpt-6-luna as the reader that writes each answer, gpt-4o-2024-08-06 with the official LongMemEval judge prompt, LongMemEval-S (cleaned, 2025-09) and 30 questions, with Hindsight 0.10.3 set up as described above. Scores move with the reader model, the judge model and its prompt, the dataset version, the number of questions, and whether a hosted platform or the open-source package is tested.

Frequently asked questions

How accurate is Hindsight on LongMemEval?

Hindsight answered 27 of 30 questions correctly (90.0%, 95% CI 74–97%) under the official judge. Rank 1 of 6 scored systems including baselines (tied); 1 of 4 memory systems (tied). The 3-judge panel majority gives 86.7%. This is a 30-question pilot of LongMemEval-S (cleaned, 2025-09); the full 500-question run is in progress.

How much does Hindsight cost to run?

Hindsight: $172 per 1,000 questions, of which $171 (over 99%) is memory-side (ingestion and retrieval) and $0.71 is answering. Memory-side cost covers ingesting each question's chat history and retrieving from it, as billed by the providers (prices as of 2026-10-08).

How fast is Hindsight?

Median latency is 5.4s per question (p90 8.1s), measured as retrieval plus answering. Ingesting one question's chat history takes 8.8 min (median).

Why is this Hindsight score different from the one Vectorize reports?

Vectorize reports 91.4% (LongMemEval-S; reader Gemini 3 Pro; judge GPT-OSS-120B) and 94.6% (LongMemEval-S (cleaned); reader Gemini 3.1 Pro; judge Gemini 2.5 Flash-Lite). We measured 90.0% (95% CI 74–97%). Vendor-reported scores and ours are measured differently, so a gap does not by itself mean either number is wrong. Our pilot uses openai/gpt-6-luna as the reader that writes each answer, gpt-4o-2024-08-06 with the official LongMemEval judge prompt, LongMemEval-S (cleaned, 2025-09) and 30 questions, with Hindsight 0.10.3 set up as described above. Scores move with the reader model, the judge model and its prompt, the dataset version, the number of questions, and whether a hosted platform or the open-source package is tested.

Does Hindsight beat the simple baselines?

Statistically tied on accuracy at n=30: Hindsight 90.0% (95% CI 74–97%) vs Plain RAG 86.7% (95% CI 70–95%). The 95% intervals overlap, so the 3.3-point gap could be noise. Statistically tied on accuracy at n=30: Hindsight 90.0% (95% CI 74–97%) vs Full context 83.3% (95% CI 66–93%). The 95% intervals overlap, so the 6.7-point gap could be noise.

Compare Hindsight with