Full context vs Hindsight: accuracy, cost and latency
A memory system against a no-memory baseline: does the memory layer earn its cost? Independent results on the same 30 questions of LongMemEval-S (cleaned, 2025-09), with the same reader model and the official judge. Pilot sample, so accuracy is shown with 95% intervals.
Full context
Baseline, not a memory system- Vendor
- Baseline
- Version
- -
- License
- -
- Result
- 83.3% (95% CI 66–93%)
Hindsight
Memory system- Vendor
- Vectorize
- Version
- 0.10.3
- License
- MIT
- Result
- 90.0% (95% CI 74–97%)
Verdict
Generated from the pilot data, n=30 per systemIs a memory layer worth it here?
Not demonstrated in this pilot: Hindsight and the Full context baseline are statistically tied on accuracy at n=30 (Hindsight 90.0% vs Full context 83.3%; the 95% intervals overlap), and Hindsight costs 13× more per 1,000 questions. The reader sees a median of 18k characters of context per question with Hindsight and 500k with Full context.
- AccuracyStatistically tied on accuracy at n=30: Hindsight 90.0% (95% CI 74–97%) vs Full context 83.3% (95% CI 66–93%). The 95% intervals overlap, so the 6.7-point gap could be noise.
- CostFull context costs 13× less per 1,000 questions: $13.1 vs $172 for Hindsight.
- LatencySimilar latency: median 4.8s for Full context vs 5.4s for Hindsight per question (p90 8.0s vs 8.1s).
- IngestionThe Full context baseline needs no ingestion step; Hindsight takes 8.8 min to ingest one chat history (median).
Key numbers side by side
Bold marks the better value. Accuracy rows are not bolded when the 95% intervals overlap.
| Metric | Full context | Hindsight |
|---|---|---|
| Accuracy (official judge) | 83.3% | 90.0% |
| 95% interval | 66–93% | 74–97% |
| Panel accuracy | 86.7% | 86.7% |
| Cost per 1,000 questions | $13.1 | $172 |
| of which memory side | $0.00 | $171 |
| of which answering | $13.1 | $0.71 |
| Latency p50 | 4.8s | 5.4s |
| Latency p90 | 8.0s | 8.1s |
| Ingestion per history (median) | none | 8.8 min |
| Context given to reader (median chars) | 500k | 18k |
Full context and Hindsight among all tested systems
Full context and Hindsight are highlighted; other systems are muted for context, baselines lightest. Each name links to the system's page.
Accuracy
Higher is betterShare of 30 questions answered correctly (official judge). Whiskers: 95% interval.
- Cognee90.0%74–97%
- Hindsight90.0%74–97%
- Plain RAG86.7%70–95%
- Mem083.3%66–93%
- Full context83.3%66–93%
- LangMem33.3%19–51%
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Cost per 1,000 questions
Lower is betterUSD billed (prices as of 2026-10-08). Solid: memory side (ingestion and retrieval). Light: answering.
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Latency per question
Lower is betterRetrieval plus answer, in seconds. Solid: median (p50). Light: up to p90.
- LangMem3.5sp90 5.3s
- Cognee4.0sp90 8.1s
- Mem04.3sp90 7.4s
- Full context4.8sp90 8.0s
- Plain RAG4.8sp90 7.4s
- Hindsight5.4sp90 8.1s
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Ingestion time
Lower is betterMedian time to load one question's chat history into the system.
Not shown: Graphiti (pilot result withdrawn, rerun in progress).
Accuracy by question type
Difference is Full context minus Hindsight, in percentage points. Each type has only a few pilot questions (n), so one question can move a row by 13 points or more.
| Question type | n | Full context | Hindsight | Difference |
|---|---|---|---|---|
| Single-session (user) | 4 | 100% | 100% | 0 |
| Single-session (assistant) | 3 | 100% | 33% | +67 pts in favour of Full context |
| Preferences | 2 | 100% | 100% | 0 |
| Multi-session | 8 | 75% | 88% | −13 pts in favour of Hindsight |
| Knowledge update | 4 | 100% | 100% | 0 |
| Temporal reasoning | 8 | 75% | 100% | −25 pts in favour of Hindsight |
| Abstention | 1 | 0% | 100% | −100 pts in favour of Hindsight |
What each one is
Full context
No memory system. The entire chat history (about 100k tokens) is placed in the reader's prompt.
How we ran it
Official LongMemEval long-context setting with the official answer prompt.
Hindsight
Extracts facts into a memory bank on Postgres, consolidates them into observations in the background, and recalls with hybrid search plus a local reranker.
How we ran it
Embedded daemon (pg0 Postgres), one bank per question. Sessions retained with their real timestamp; recall with the question date and default budget. We wait for background consolidation to finish before asking.
What the vendors report
Hindsight
| Score | Variant | Reader | Judge | Note | Source |
|---|---|---|---|---|---|
| 90.0% | S (cleaned, 2025-09), 30-question pilot | openai/gpt-6-luna | gpt-4o-2024-08-06, official prompt | MemVerdict measurement, Hindsight 0.10.3 | This page |
| 91.4% | S | Gemini 3 Pro | GPT-OSS-120B | - | arxiv.org/abs/2512.12818 |
| 94.6% | S (cleaned) | Gemini 3.1 Pro | Gemini 2.5 Flash-Lite | - | benchmarks.hindsight.vectorize.io |
Vendor-reported scores and ours are measured differently, so a gap does not by itself mean either number is wrong. Our pilot uses openai/gpt-6-luna as the reader that writes each answer, gpt-4o-2024-08-06 with the official LongMemEval judge prompt, LongMemEval-S (cleaned, 2025-09) and 30 questions, with each system set up as described above. Scores move with the reader model, the judge model and its prompt, the dataset version, the number of questions, and whether a hosted platform or the open-source package is tested.
Frequently asked questions
Is Full context better than Hindsight?
Statistically tied on accuracy at n=30: Hindsight 90.0% (95% CI 74–97%) vs Full context 83.3% (95% CI 66–93%). The 95% intervals overlap, so the 6.7-point gap could be noise. With the 3-judge panel majority instead of the official judge: Full context 86.7%, Hindsight 86.7%. The largest gap by question type is Single-session (assistant): 3 of 3 vs 1 of 3 correct, in Full context's favour; each question type has only 1 to 8 questions in this pilot. These are pilot numbers (30 questions each); the full 500-question run will narrow the intervals.
Which is cheaper, Full context or Hindsight?
Full context costs 13× less per 1,000 questions: $13.1 vs $172 for Hindsight. Full context: $13.1 per 1,000 questions, all of it answering (no memory-side cost). Hindsight: $172 per 1,000 questions, of which $171 (over 99%) is memory-side (ingestion and retrieval) and $0.71 is answering. Prices as of 2026-10-08.
Which is faster, Full context or Hindsight?
Similar latency: median 4.8s for Full context vs 5.4s for Hindsight per question (p90 8.0s vs 8.1s). The Full context baseline needs no ingestion step; Hindsight takes 8.8 min to ingest one chat history (median). Latency is measured per question as retrieval plus answering; ingestion is the time to load one question's chat history.
How were Full context and Hindsight tested?
Both were run on the same 30 questions of LongMemEval-S (cleaned, 2025-09) as every other system, by the same harness. For each question, the system ingests that question's chat history, then retrieves context that one reader model (openai/gpt-6-luna) uses to answer with the official LongMemEval prompt. Answers are graded by the official judge (gpt-4o-2024-08-06) and cross-checked by 3 other judges; all judges agreed on 95% of graded answers. Costs are what the providers billed (prices as of 2026-10-08). Full context: Official LongMemEval long-context setting with the official answer prompt. Hindsight: Embedded daemon (pg0 Postgres), one bank per question. Sessions retained with their real timestamp; recall with the question date and default budget. We wait for background consolidation to finish before asking.
Which should I use, Full context or Hindsight?
Not demonstrated in this pilot: Hindsight and the Full context baseline are statistically tied on accuracy at n=30 (Hindsight 90.0% vs Full context 83.3%; the 95% intervals overlap), and Hindsight costs 13× more per 1,000 questions. The reader sees a median of 18k characters of context per question with Hindsight and 500k with Full context. If the Full context baseline is as accurate on your own data, it is the simpler option to run; test both on a sample of your real conversations before committing. These are pilot numbers (30 questions each); the full 500-question run will narrow the intervals.
More comparisons
Other comparisons with Full context
- Cognee vs Full contextStatistically tied on accuracy at n=30; Full context 11× cheaper
- Full context vs GraphitiGraphiti: pilot result withdrawn, rerun in progress
- Full context vs LangMemFull context more accurate and 4.7× cheaper
- Full context vs Mem0Tied on accuracy (83.3% each); Full context 7.1× cheaper
- Full context vs Plain RAGStatistically tied on accuracy at n=30; Plain RAG 2.4× cheaper
Other comparisons with Hindsight
- Cognee vs HindsightTied on accuracy (90.0% each); Cognee 1.2× cheaper
- Graphiti vs HindsightGraphiti: pilot result withdrawn, rerun in progress
- Hindsight vs LangMemHindsight more accurate; LangMem 2.8× cheaper
- Hindsight vs Mem0Statistically tied on accuracy at n=30; Mem0 1.8× cheaper
- Hindsight vs Plain RAGStatistically tied on accuracy at n=30; Plain RAG 31× cheaper