97 lines
3.4 KiB
Markdown
97 lines
3.4 KiB
Markdown
# Consolidation Performance Benchmark
|
|
|
|
## Overview
|
|
|
|
This benchmark measures consolidation throughput (operations per second) and identifies bottlenecks in the consolidation pipeline.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Run with default settings (100 memories)
|
|
./scripts/benchmarks/run-consolidation.sh
|
|
|
|
# Run with custom number of memories
|
|
NUM_MEMORIES=50 ./scripts/benchmarks/run-consolidation.sh
|
|
|
|
# Run with different model
|
|
export HINDSIGHT_API_CONSOLIDATION_LLM_MODEL=llama-3.1-70b-versatile
|
|
NUM_MEMORIES=100 ./scripts/benchmarks/run-consolidation.sh
|
|
```
|
|
|
|
## What It Measures
|
|
|
|
The benchmark:
|
|
1. Creates N test memories with diverse content (similar facts, contradictions, different entities)
|
|
2. Runs consolidation and measures time spent in each component:
|
|
- **Recall**: Finding related observations
|
|
- **LLM**: Deciding on consolidation actions
|
|
- **Embedding**: Generating embeddings for new/updated observations
|
|
- **DB Write**: Writing to database
|
|
3. Reports throughput (op/sec) and detailed timing breakdown
|
|
|
|
## Interpreting Results
|
|
|
|
### Metrics
|
|
- **Throughput (op/sec)**: Memories processed per second
|
|
- **Timing Breakdown**: % of time spent in each component
|
|
- **Observations Created/Updated**: Quality indicator
|
|
|
|
### Baseline Performance (groq/openai/gpt-oss-120b)
|
|
- **~0.7-1.0 op/sec** (1-1.4 seconds per memory)
|
|
- **LLM: 80-87%** of time (main bottleneck)
|
|
- **Recall: 10-17%** of time (secondary bottleneck)
|
|
|
|
## Results
|
|
|
|
See:
|
|
- `ANALYSIS.md` - Detailed bottleneck analysis
|
|
- `RESULTS.md` - Performance results and recommendations
|
|
- `benchmarks/results/` - Raw benchmark data (JSON)
|
|
|
|
## Optimizations
|
|
|
|
### Implemented
|
|
✅ Batch database queries (fixed N+1 problem)
|
|
✅ Reduced recall token budget (5000 → 2000)
|
|
✅ Limited observation results (top 15)
|
|
|
|
### Recommended
|
|
🔧 Use faster LLM model for consolidation
|
|
🔧 Enable prompt caching (if available)
|
|
🔧 Optimize prompt verbosity
|
|
|
|
See `RESULTS.md` for detailed recommendations.
|
|
|
|
## Configuration
|
|
|
|
Environment variables:
|
|
- `NUM_MEMORIES`: Number of memories to create (default: 100)
|
|
- `HINDSIGHT_API_CONSOLIDATION_LLM_MODEL`: Model for consolidation
|
|
- `HINDSIGHT_API_CONSOLIDATION_LLM_PROVIDER`: Provider for consolidation
|
|
- `HINDSIGHT_API_DATABASE_URL`: Database URL
|
|
- `HINDSIGHT_LOG_LEVEL`: Logging level (INFO for detailed logs)
|
|
|
|
## Example Output
|
|
|
|
```
|
|
Consolidation Benchmark Results
|
|
┌────────────────────────────────┬─────────────┐
|
|
│ Metric │ Value │
|
|
├────────────────────────────────┼─────────────┤
|
|
│ Total Time │ 60.28s │
|
|
│ Memories Processed │ 43 │
|
|
│ Throughput │ 0.71 op/sec │
|
|
│ Avg Time/Memory │ 1.402s │
|
|
│ │ │
|
|
│ Observations Created │ 4 │
|
|
│ Observations Updated │ 38 │
|
|
│ Observations Merged │ 0 │
|
|
│ Skipped (No Durable Knowledge) │ 1 │
|
|
└────────────────────────────────┴─────────────┘
|
|
|
|
Timing breakdown:
|
|
recall=6.295s (10.4%)
|
|
llm=52.144s (86.5%) ← BOTTLENECK
|
|
embedding=1.717s (2.8%)
|
|
db_write=0.075s (0.1%)
|
|
```
|