fleet-memory/hindsight-dev/benchmarks

Hindsight Benchmarks

This directory contains benchmark suites for evaluating Hindsight's memory capabilities.

Prerequisites

1. Set up your environment variables in .env at the project root:

   cp .env.example .env
   # Edit .env with your API keys
   

2. Make sure you have uv installed.

Available Benchmarks

LoComo

Tests conversational memory with multi-turn dialogues.

# Run from project root
./scripts/benchmarks/run-locomo.sh

# With options
./scripts/benchmarks/run-locomo.sh --max-conversations 10
./scripts/benchmarks/run-locomo.sh --skip-ingestion  # Reuse existing data
./scripts/benchmarks/run-locomo.sh --use-think       # Use think API
./scripts/benchmarks/run-locomo.sh --conversation conv-26  # Single conversation

Options:

• --max-conversations N - Limit number of conversations
• --max-questions N - Limit questions per conversation
• --skip-ingestion - Skip data ingestion, use existing
• --use-think - Use think API instead of search + LLM
• --conversation NAME - Run specific conversation only
• --api-url URL - Custom API URL (default: local memory)
• --only-failed - Retry only failed questions
• --only-invalid - Retry only invalid questions

LongMemEval

Tests long-term memory across different categories.

# Run from project root
./scripts/benchmarks/run-longmemeval.sh

# With options
./scripts/benchmarks/run-longmemeval.sh --max-instances 50
./scripts/benchmarks/run-longmemeval.sh --category single-session-user
./scripts/benchmarks/run-longmemeval.sh --parallel 4  # Faster evaluation

Options:

• --max-instances N - Limit total questions
• --max-instances-per-category N - Limit per category
• --skip-ingestion - Skip data ingestion
• --category NAME - Filter by category:
  • single-session-user
  • multi-session
  • single-session-preference
  • temporal-reasoning
  • knowledge-update
  • single-session-assistant
• --parallel N - Parallel instances (default: 1)
• --only-failed - Retry failed questions
• --fill - Resume interrupted runs

Consolidation Performance

Tests consolidation throughput and identifies bottlenecks.

./scripts/benchmarks/run-consolidation.sh

# With custom memory count
NUM_MEMORIES=200 ./scripts/benchmarks/run-consolidation.sh

Retain Performance

Measures retain operation performance (throughput and token usage).

Prerequisites: API server must be running (./scripts/dev/start-api.sh)

# Basic usage
./scripts/benchmarks/run-retain-perf.sh \
    --document hindsight-dev/benchmarks/perf/test_data/sample_document.txt

# Save results to JSON
./scripts/benchmarks/run-retain-perf.sh \
    --document ./my_document.txt \
    --bank-id my-test-bank \
    --output results/retain_perf.json

Options:

• --document PATH - Document file to retain (required)
• --bank-id ID - Bank ID to use (default: perf-test)
• --context TEXT - Optional context
• --api-url URL - API URL (default: http://localhost:8000)
• --timeout SECONDS - Request timeout (default: 300)
• --output PATH - Save results to JSON file

See perf/README.md for detailed documentation.

Visualizer

View benchmark results in a web UI:

./scripts/benchmarks/start-visualizer.sh
# Opens at http://localhost:8001

Results

Results are saved in JSON format in each benchmark's results/ directory.