holetron-lab/fleet-memory
1
0
Fork 0
Code Releases Packages Activity Actions
fleet-memory/hindsight-dev/benchmarks
History
Nicolò Boschi ad2cf72aab
perf: add GIN index on source_memory_ids for observation lookup (#485) 
* perf: add GIN index on source_memory_ids for observation lookup

Addresses a 927x performance regression (45ms → 0.049ms) reported by a
user with ~77k observations. The array overlap operator (&&) on
source_memory_ids was doing a full sequential scan over all observations,
causing recall timeouts (57-64s) and slow user recall (18-27s avg).

The partial GIN index reduces consolidation recall from timeout to ~15s
and user recall to ~6s.

* fix: use pre-bounded memory_links for observation graph expansion

Replace raw unit_entities join in _expand_observations() with the same
memory_links entity graph used by non-observation fact types. The previous
approach joined unit_entities twice (seeds→entities→connected_sources),
which explodes at scale (30-70s at 100k observations). The LIMIT 500
workaround was non-deterministic and dropped valid results.

Using memory_links (pre-bounded to MAX_LINKS_PER_ENTITY=50 at retain time)
is algorithmically identical to the non-observation entity expansion and
keeps graph retrieval at ~2s p50 even at 100k observations.

Also fix migration down_revision (z1u2v3w4x5y6 → d2e3f4a5b6c7) and add
observation generation + fact-type filtering to the recall perf benchmark.
2026-03-05 10:15:40 +01:00
..
common feat: batch observations consolidation (#430) 2026-02-24 15:19:39 +01:00
consolidation feat: consolidation performance benchmark and optimization (#227) 2026-01-29 11:24:15 +01:00
locomo fix: catch ValueError instead of bare except in date parsing (#438) 2026-02-25 10:33:56 +01:00
longmemeval chore: cleanup benchmarks runner with old flags (#212) 2026-01-28 13:22:48 +01:00
perf perf: add GIN index on source_memory_ids for observation lookup (#485) 2026-03-05 10:15:40 +01:00
visualizer feat: add reflect mode to LoComo benchmark and improve reflect agent (#428) 2026-02-24 09:48:23 +01:00
.DS_Store rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
__init__.py rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
README.md fix: improve async batch retain with large payloads (#366) 2026-02-16 12:51:42 +01:00

README.md

Hindsight Benchmarks

This directory contains benchmark suites for evaluating Hindsight's memory capabilities.

Prerequisites

  1. Set up your environment variables in .env at the project root:

    cp .env.example .env
# Edit .env with your API keys

  2. Make sure you have uv installed.

Available Benchmarks

LoComo

Tests conversational memory with multi-turn dialogues.

# Run from project root
./scripts/benchmarks/run-locomo.sh

# With options
./scripts/benchmarks/run-locomo.sh --max-conversations 10
./scripts/benchmarks/run-locomo.sh --skip-ingestion  # Reuse existing data
./scripts/benchmarks/run-locomo.sh --use-think       # Use think API
./scripts/benchmarks/run-locomo.sh --conversation conv-26  # Single conversation

Options:

  • --max-conversations N - Limit number of conversations
  • --max-questions N - Limit questions per conversation
  • --skip-ingestion - Skip data ingestion, use existing
  • --use-think - Use think API instead of search + LLM
  • --conversation NAME - Run specific conversation only
  • --api-url URL - Custom API URL (default: local memory)
  • --only-failed - Retry only failed questions
  • --only-invalid - Retry only invalid questions

LongMemEval

Tests long-term memory across different categories.

# Run from project root
./scripts/benchmarks/run-longmemeval.sh

# With options
./scripts/benchmarks/run-longmemeval.sh --max-instances 50
./scripts/benchmarks/run-longmemeval.sh --category single-session-user
./scripts/benchmarks/run-longmemeval.sh --parallel 4  # Faster evaluation

Options:

  • --max-instances N - Limit total questions
  • --max-instances-per-category N - Limit per category
  • --skip-ingestion - Skip data ingestion
  • --category NAME - Filter by category:
    • single-session-user
    • multi-session
    • single-session-preference
    • temporal-reasoning
    • knowledge-update
    • single-session-assistant
  • --parallel N - Parallel instances (default: 1)
  • --only-failed - Retry failed questions
  • --fill - Resume interrupted runs

Consolidation Performance

Tests consolidation throughput and identifies bottlenecks.

./scripts/benchmarks/run-consolidation.sh

# With custom memory count
NUM_MEMORIES=200 ./scripts/benchmarks/run-consolidation.sh

Retain Performance

Measures retain operation performance (throughput and token usage).

Prerequisites: API server must be running (./scripts/dev/start-api.sh)

# Basic usage
./scripts/benchmarks/run-retain-perf.sh \
    --document hindsight-dev/benchmarks/perf/test_data/sample_document.txt

# Save results to JSON
./scripts/benchmarks/run-retain-perf.sh \
    --document ./my_document.txt \
    --bank-id my-test-bank \
    --output results/retain_perf.json

Options:

  • --document PATH - Document file to retain (required)
  • --bank-id ID - Bank ID to use (default: perf-test)
  • --context TEXT - Optional context
  • --api-url URL - API URL (default: http://localhost:8000)
  • --timeout SECONDS - Request timeout (default: 300)
  • --output PATH - Save results to JSON file

See perf/README.md for detailed documentation.

Visualizer

View benchmark results in a web UI:

./scripts/benchmarks/start-visualizer.sh
# Opens at http://localhost:8001

Results

Results are saved in JSON format in each benchmark's results/ directory.