147 lines
4.3 KiB
Markdown
147 lines
4.3 KiB
Markdown
# Benchmarks
|
||
|
||
This directory contains benchmark evaluations for the Entity-Aware Memory System.
|
||
|
||
## LoComo Benchmark
|
||
|
||
**Location**: `locomo/`
|
||
|
||
**Purpose**: Evaluate long-term conversational memory through Question Answering on multi-session conversations.
|
||
|
||
### Quick Start
|
||
|
||
1. **Run full benchmark** (10 conversations, ~2000 questions):
|
||
```bash
|
||
cd locomo
|
||
uv run python run_benchmark.py
|
||
```
|
||
|
||
2. **Run quick test** (1 conversation, 10 questions):
|
||
```bash
|
||
cd locomo
|
||
uv run python run_benchmark.py --max-conversations 1 --max-questions 10
|
||
```
|
||
|
||
3. **View results**:
|
||
- Detailed report: `locomo/RESULTS.md`
|
||
- Raw data: `locomo/benchmark_results.json`
|
||
|
||
### Dataset
|
||
|
||
- **Source**: [Snap Research LoComo](https://github.com/snap-research/locomo)
|
||
- **File**: `locomo10.json` (10 conversations)
|
||
- **Size**: Each conversation has ~300 turns over ~35 sessions spanning several months
|
||
- **Tasks**: Question Answering with 3 reasoning types (single-hop, temporal, multi-hop)
|
||
|
||
### Methodology
|
||
|
||
1. **Ingest** each conversation turn-by-turn with timestamps
|
||
2. **Apply** coreference resolution and entity extraction
|
||
3. **Create** temporal, semantic, and entity links
|
||
4. **Answer** questions using spreading activation search
|
||
5. **Evaluate** using LLM-as-judge (GPT-4o-mini)
|
||
|
||
### Expected Performance
|
||
|
||
Based on published results:
|
||
- **Human**: ~95%
|
||
- **Letta (GPT-4o-mini)**: 74.0%
|
||
- **Mem0 Graph**: 68.5%
|
||
- **Our target**: 65-75% (competitive with state-of-the-art)
|
||
|
||
### Computational Cost
|
||
|
||
**Per conversation** (~300 turns):
|
||
- ~300 embedding API calls (ingestion)
|
||
- ~200 embedding API calls (queries)
|
||
- ~200 LLM API calls (answer generation)
|
||
- ~200 LLM API calls (judgment)
|
||
|
||
**Estimated runtime**: 2-5 minutes per conversation (API-dependent)
|
||
|
||
**Estimated cost**: $0.50-1.00 per conversation (OpenAI pricing)
|
||
|
||
## LongMemEval Benchmark
|
||
|
||
**Location**: `longmemeval/`
|
||
|
||
**Purpose**: Evaluate five core long-term interactive memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention.
|
||
|
||
### Quick Start
|
||
|
||
1. **Download dataset**:
|
||
```bash
|
||
cd longmemeval
|
||
curl -L "https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json" -o longmemeval_s_cleaned.json
|
||
```
|
||
|
||
2. **Run full benchmark** (500 questions):
|
||
```bash
|
||
cd longmemeval
|
||
uv run python run_benchmark.py
|
||
```
|
||
|
||
3. **Run quick test** (5 instances):
|
||
```bash
|
||
cd longmemeval
|
||
uv run python run_benchmark.py --max-instances 5
|
||
```
|
||
|
||
4. **View results**:
|
||
- Raw data: `longmemeval/benchmark_results.json`
|
||
|
||
### Dataset
|
||
|
||
- **Source**: [LongMemEval (ICLR 2025)](https://github.com/xiaowu0162/LongMemEval)
|
||
- **File**: `longmemeval_s_cleaned.json` (500 instances)
|
||
- **Size**: ~40 sessions per instance (~115k tokens)
|
||
- **Tasks**: 5 memory abilities across different question types
|
||
|
||
### Methodology
|
||
|
||
1. **Ingest** multi-session conversations with timestamps
|
||
2. **Apply** coreference resolution and entity extraction
|
||
3. **Create** temporal, semantic, and entity links
|
||
4. **Retrieve** relevant memories using spreading activation
|
||
5. **Generate** answers using GPT-4o-mini
|
||
6. **Evaluate** using GPT-4o as judge
|
||
|
||
### Expected Performance
|
||
|
||
Based on published results:
|
||
- **Human**: ~95%
|
||
- **Zep**: 75.2%
|
||
- **Letta (GPT-4o-mini)**: 74.0%
|
||
- **Mem0 Graph**: 68.5%
|
||
- **Our target**: 65-75% (competitive with state-of-the-art)
|
||
|
||
### Computational Cost
|
||
|
||
**Full benchmark** (500 instances):
|
||
- Embeddings: Free (local model)
|
||
- Answer generation: 500 × GPT-4o-mini calls
|
||
- Evaluation: 500 × GPT-4o calls
|
||
- **Estimated runtime**: 2-4 hours
|
||
- **Estimated cost**: $50-80 (OpenAI API)
|
||
|
||
## Future Benchmarks
|
||
|
||
- **MemGPT Tasks**: Long-context question answering
|
||
- **Custom Temporal Reasoning**: Time-based memory retrieval
|
||
- **Entity-Centric Queries**: Testing entity link effectiveness
|
||
|
||
## Adding New Benchmarks
|
||
|
||
1. Create a new directory: `benchmarks/{benchmark_name}/`
|
||
2. Add dataset: `benchmarks/{benchmark_name}/data/`
|
||
3. Implement adapter: `benchmarks/{benchmark_name}/run_benchmark.py`
|
||
4. Document results: `benchmarks/{benchmark_name}/RESULTS.md`
|
||
|
||
## Results Summary
|
||
|
||
| Benchmark | Metric | Our System | Best Published | Status |
|
||
|-----------|--------|------------|----------------|--------|
|
||
| LoComo QA | Accuracy | {TBD}% | 74.0% (Letta) | In Progress |
|
||
| LongMemEval | Accuracy | {TBD}% | 75.2% (Zep) | Ready to Run |
|
||
|
||
*Last updated: 2025-10-30*
|