fleet-memory/benchmarks/longmemeval

LongMemEval Benchmark

Purpose: Evaluate long-term interactive memory capabilities across five core abilities.

Overview

LongMemEval is a comprehensive benchmark that tests chat assistants on realistic long-term memory scenarios. The benchmark evaluates five core memory abilities:

1. Information Extraction - Retrieving specific facts from conversation history
2. Multi-Session Reasoning - Connecting information across multiple conversations
3. Temporal Reasoning - Understanding time-based relationships and changes
4. Knowledge Updates - Handling conflicting or updated information
5. Abstention - Recognizing when information is insufficient to answer

Dataset

• Source: LongMemEval (ICLR 2025)
• File: longmemeval_s_cleaned.json
• Size: 500 question-answer pairs
• Context: ~40 sessions per instance (~115k tokens)
• Format: Multi-turn conversations with timestamped sessions

Dataset Structure

Each instance contains:

• question_id: Unique identifier
• question_type: Category (single-session, multi-session, temporal, knowledge-update, abstention)
• question: Query text
• answer: Expected answer
• question_date: Query timestamp
• haystack_sessions: List of conversation sessions with turns
• answer_session_ids: Evidence session identifiers

Quick Start

Prerequisites

1. Python 3.11+ with dependencies installed (uv sync)
2. PostgreSQL database configured
3. OpenAI API key set in environment

export OPENAI_API_KEY="your-api-key"

Download Dataset

cd benchmarks/longmemeval
curl -L "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json" -o longmemeval_s_cleaned.json

Run Benchmark

Full evaluation (500 questions):

uv run python run_benchmark.py

Quick test (first 5 instances):

uv run python run_benchmark.py --max-instances 5

Custom settings:

uv run python run_benchmark.py \
  --max-instances 10 \
  --thinking-budget 100 \
  --top-k 20 \
  --output my_results.json

View Results

Results are saved to benchmark_results.json and include:

• Per-question scores and predictions
• Performance breakdown by question type
• Retrieved memory units for debugging
• Evaluation explanations

Methodology

1. Ingestion Phase

For each instance:

1. Parse all conversation sessions with timestamps
2. Process each turn (user and assistant messages)
3. Store in memory system with:
   • Coreference resolution (pronouns → entities)
   • Entity extraction and disambiguation
   • Temporal, semantic, and entity link creation

2. Retrieval Phase

For each question:

1. Generate query embedding
2. Find entry points (top-3 similar memories)
3. Spread activation through memory graph
4. Apply recency and frequency weighting
5. Return top-k most relevant memory units

3. Answer Generation

1. Format retrieved memories as context
2. Generate answer using GPT-4o-mini
3. Enforce answering only from provided memories
4. Handle abstention cases appropriately

4. Evaluation

1. Compare predicted answer to gold answer
2. Use GPT-4o as judge for semantic equivalence
3. Binary scoring (1 = correct, 0 = incorrect)
4. Aggregate by question type

Parameters

Parameter	Default	Description
--max-instances	500	Number of instances to evaluate
--max-questions	None	Limit questions per instance (for testing)
--thinking-budget	100	Exploration budget for spreading activation
--top-k	20	Number of memory units to retrieve
--output	benchmark_results.json	Output file path

Expected Performance

Based on published results:

System	Accuracy
Human	~95%
Zep	75.2%
Letta (GPT-4o-mini)	74.0%
Mem0 Graph	68.5%
Target	65-75%

Performance by Question Type

Expected breakdown:

• Single-session: 70-80% (easiest - information in one session)
• Multi-session: 60-70% (requires connecting across sessions)
• Temporal reasoning: 60-70% (requires time-based reasoning)
• Knowledge updates: 50-65% (hardest - handling conflicting info)
• Abstention: 65-75% (recognizing insufficient information)

Computational Cost

Per instance (~40 sessions, ~200 turns):

• Ingestion: ~200 embedding generations (local model, fast)
• Query: 1 embedding generation + graph search
• Answer: 1 GPT-4o-mini call (~200 tokens)
• Evaluation: 1 GPT-4o call (~150 tokens)

Full benchmark (500 instances):

• Runtime: 2-4 hours (depends on API rate limits)
• Cost: ~$50-80 (OpenAI API for answer generation + evaluation)
• Embeddings: Free (local model)

Example Output

LongMemEval Benchmark Evaluation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Performance
┏━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Metric         ┃ Value ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ Total          │ 500   │
│ Correct        │ 345   │
│ Incorrect      │ 155   │
│ Accuracy       │ 69.0% │
└────────────────┴───────┘

Performance by Question Type
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃ Question Type      ┃ Total ┃ Correct ┃ Accuracy ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│ single-session     │ 150   │ 115     │ 76.7%    │
│ multi-session      │ 120   │ 78      │ 65.0%    │
│ temporal-reasoning │ 100   │ 65      │ 65.0%    │
│ knowledge-update   │ 80    │ 48      │ 60.0%    │
│ abstention         │ 50    │ 39      │ 78.0%    │
└────────────────────┴───────┴─────────┴──────────┘

Troubleshooting

Dataset not found

cd benchmarks/longmemeval
curl -L "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json" -o longmemeval_s_cleaned.json

OpenAI API key error

export OPENAI_API_KEY="your-api-key"

Memory ingestion slow

• This is expected for first-time entity resolution
• Subsequent queries are fast (graph search)
• Consider using --max-instances for quick testing

Low accuracy

• Try increasing --thinking-budget (default: 100)
• Try increasing --top-k (default: 20)
• Check retrieved memories in results JSON for debugging

Architecture Integration

This benchmark tests the full memory system architecture:

1. ✅ Coreference Resolution: Makes memories self-contained
2. ✅ Entity Extraction: Identifies people, organizations, places
3. ✅ Entity Disambiguation: Links mentions across sessions
4. ✅ Temporal Links: Connects memories by time proximity
5. ✅ Semantic Links: Connects memories by meaning
6. ✅ Entity Links: Connects memories by shared entities
7. ✅ Spreading Activation: Graph-aware retrieval
8. ✅ Recency/Frequency Weighting: Importance signals

Citation

If you use this benchmark, please cite:

@inproceedings{wu2025longmemeval,
  title={LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory},
  author={Wu, Di and Wang, Hongwei and Liu, Wenhao and Wang, Jiaheng and Li, Zihan and Huang, Yiqin and Patel, Zelin and Liu, Yiheng and Meng, Bo and Pan, Sinong and others},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}

Related Benchmarks

• LoComo: Multi-session conversational QA
• MemGPT Tasks: Long-context question answering
• Custom Temporal Reasoning: Time-based memory retrieval