Nicolò Boschi 6d7223b14e initial commit

2025-10-30 12:53:12 +01:00

7.9 KiB

Raw Blame History

LongMemEval Benchmark

Purpose: Evaluate long-term interactive memory capabilities across five core abilities.

Overview

LongMemEval is a comprehensive benchmark that tests chat assistants on realistic long-term memory scenarios. The benchmark evaluates five core memory abilities:

Information Extraction - Retrieving specific facts from conversation history
Multi-Session Reasoning - Connecting information across multiple conversations
Temporal Reasoning - Understanding time-based relationships and changes
Knowledge Updates - Handling conflicting or updated information
Abstention - Recognizing when information is insufficient to answer

Dataset

Source: LongMemEval (ICLR 2025)
File: longmemeval_s_cleaned.json
Size: 500 question-answer pairs
Context: ~40 sessions per instance (~115k tokens)
Format: Multi-turn conversations with timestamped sessions

Dataset Structure

Each instance contains:

question_id: Unique identifier
question_type: Category (single-session, multi-session, temporal, knowledge-update, abstention)
question: Query text
answer: Expected answer
question_date: Query timestamp
haystack_sessions: List of conversation sessions with turns
answer_session_ids: Evidence session identifiers

Quick Start

Prerequisites

Python 3.11+ with dependencies installed (uv sync)
PostgreSQL database configured
OpenAI API key set in environment

export OPENAI_API_KEY="your-api-key"

Download Dataset

cd benchmarks/longmemeval
curl -L "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json" -o longmemeval_s_cleaned.json

Run Benchmark

Full evaluation (500 questions):

uv run python run_benchmark.py

Quick test (first 5 instances):

uv run python run_benchmark.py --max-instances 5

Custom settings:

uv run python run_benchmark.py \
  --max-instances 10 \
  --thinking-budget 100 \
  --top-k 20 \
  --output my_results.json

View Results

Results are saved to benchmark_results.json and include:

Per-question scores and predictions
Performance breakdown by question type
Retrieved memory units for debugging
Evaluation explanations

Methodology

1. Ingestion Phase

For each instance:

Parse all conversation sessions with timestamps
Process each turn (user and assistant messages)
Store in memory system with:
- Coreference resolution (pronouns → entities)
- Entity extraction and disambiguation
- Temporal, semantic, and entity link creation

2. Retrieval Phase

For each question:

Generate query embedding
Find entry points (top-3 similar memories)
Spread activation through memory graph
Apply recency and frequency weighting
Return top-k most relevant memory units

3. Answer Generation

Format retrieved memories as context
Generate answer using GPT-4o-mini
Enforce answering only from provided memories
Handle abstention cases appropriately

4. Evaluation

Compare predicted answer to gold answer
Use GPT-4o as judge for semantic equivalence
Binary scoring (1 = correct, 0 = incorrect)
Aggregate by question type

Parameters

Parameter	Default	Description
`--max-instances`	500	Number of instances to evaluate
`--max-questions`	None	Limit questions per instance (for testing)
`--thinking-budget`	100	Exploration budget for spreading activation
`--top-k`	20	Number of memory units to retrieve
`--output`	`benchmark_results.json`	Output file path

Expected Performance

Based on published results:

System	Accuracy
Human	~95%
Zep	75.2%
Letta (GPT-4o-mini)	74.0%
Mem0 Graph	68.5%
Target	65-75%

Performance by Question Type

Expected breakdown:

Single-session: 70-80% (easiest - information in one session)
Multi-session: 60-70% (requires connecting across sessions)
Temporal reasoning: 60-70% (requires time-based reasoning)
Knowledge updates: 50-65% (hardest - handling conflicting info)
Abstention: 65-75% (recognizing insufficient information)

Computational Cost

Per instance (~40 sessions, ~200 turns):

Ingestion: ~200 embedding generations (local model, fast)
Query: 1 embedding generation + graph search
Answer: 1 GPT-4o-mini call (~200 tokens)
Evaluation: 1 GPT-4o call (~150 tokens)

Full benchmark (500 instances):

Runtime: 2-4 hours (depends on API rate limits)
Cost: ~$50-80 (OpenAI API for answer generation + evaluation)
Embeddings: Free (local model)

Example Output

LongMemEval Benchmark Evaluation
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Overall Performance
┏━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Metric         ┃ Value ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ Total          │ 500   │
│ Correct        │ 345   │
│ Incorrect      │ 155   │
│ Accuracy       │ 69.0% │
└────────────────┴───────┘

Performance by Question Type
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃ Question Type      ┃ Total ┃ Correct ┃ Accuracy ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│ single-session     │ 150   │ 115     │ 76.7%    │
│ multi-session      │ 120   │ 78      │ 65.0%    │
│ temporal-reasoning │ 100   │ 65      │ 65.0%    │
│ knowledge-update   │ 80    │ 48      │ 60.0%    │
│ abstention         │ 50    │ 39      │ 78.0%    │
└────────────────────┴───────┴─────────┴──────────┘

Troubleshooting

Dataset not found

cd benchmarks/longmemeval
curl -L "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json" -o longmemeval_s_cleaned.json

OpenAI API key error

export OPENAI_API_KEY="your-api-key"

Memory ingestion slow

This is expected for first-time entity resolution
Subsequent queries are fast (graph search)
Consider using --max-instances for quick testing

Low accuracy

Try increasing --thinking-budget (default: 100)
Try increasing --top-k (default: 20)
Check retrieved memories in results JSON for debugging

Architecture Integration

This benchmark tests the full memory system architecture:

✅ Coreference Resolution: Makes memories self-contained
✅ Entity Extraction: Identifies people, organizations, places
✅ Entity Disambiguation: Links mentions across sessions
✅ Temporal Links: Connects memories by time proximity
✅ Semantic Links: Connects memories by meaning
✅ Entity Links: Connects memories by shared entities
✅ Spreading Activation: Graph-aware retrieval
✅ Recency/Frequency Weighting: Importance signals

Citation

If you use this benchmark, please cite:

@inproceedings{wu2025longmemeval,
  title={LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory},
  author={Wu, Di and Wang, Hongwei and Liu, Wenhao and Wang, Jiaheng and Li, Zihan and Huang, Yiqin and Patel, Zelin and Liu, Yiheng and Meng, Bo and Pan, Sinong and others},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}

LoComo: Multi-session conversational QA
MemGPT Tasks: Long-context question answering
Custom Temporal Reasoning: Time-based memory retrieval

7.9 KiB Raw Blame History