History

Nicolò Boschi 1c9a7a0d5e chore: cleanup benchmarks runner with old flags (#212 ) * chore: cleanup benchmarks runner with old flags * fix tests * fix: observations rely on source_memory_ids, no link copying Observations no longer copy any memory_links from their source facts. Instead, retrieval uses source_memory_ids to traverse: - Entity connections: observation → source_memory_ids → unit_entities - Semantic similarity: observations have their own embeddings - Temporal proximity: observations have their own temporal fields This avoids data duplication and fixes bidirectionality issues with entity links being copied to observations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: update consolidation test for source_memory_ids behavior Updated test_consolidation_creates_memory_links to test_consolidation_uses_source_memory_ids to reflect the new behavior where observations use source_memory_ids instead of memory_links for traversal. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>		2026-01-28 13:22:48 +01:00
..
common	chore: cleanup benchmarks runner with old flags (#212 )	2026-01-28 13:22:48 +01:00
locomo	chore: cleanup benchmarks runner with old flags (#212 )	2026-01-28 13:22:48 +01:00
longmemeval	chore: cleanup benchmarks runner with old flags (#212 )	2026-01-28 13:22:48 +01:00
visualizer	feat: extensions (#54 )	2025-12-22 11:05:23 +01:00
.DS_Store	rename to hindsight (#2 )	2025-11-25 19:28:26 +01:00
__init__.py	rename to hindsight (#2 )	2025-11-25 19:28:26 +01:00
README.md	add run benchmarks instructions	2025-12-16 17:24:45 +01:00

README.md

Hindsight Benchmarks

This directory contains benchmark suites for evaluating Hindsight's memory capabilities.

Prerequisites

Set up your environment variables in .env at the project root:
```
cp .env.example .env
# Edit .env with your API keys
```
Make sure you have uv installed.

Available Benchmarks

LoComo

Tests conversational memory with multi-turn dialogues.

# Run from project root
./scripts/benchmarks/run-locomo.sh

# With options
./scripts/benchmarks/run-locomo.sh --max-conversations 10
./scripts/benchmarks/run-locomo.sh --skip-ingestion  # Reuse existing data
./scripts/benchmarks/run-locomo.sh --use-think       # Use think API
./scripts/benchmarks/run-locomo.sh --conversation conv-26  # Single conversation

Options:

--max-conversations N - Limit number of conversations
--max-questions N - Limit questions per conversation
--skip-ingestion - Skip data ingestion, use existing
--use-think - Use think API instead of search + LLM
--conversation NAME - Run specific conversation only
--api-url URL - Custom API URL (default: local memory)
--only-failed - Retry only failed questions
--only-invalid - Retry only invalid questions

LongMemEval

Tests long-term memory across different categories.

# Run from project root
./scripts/benchmarks/run-longmemeval.sh

# With options
./scripts/benchmarks/run-longmemeval.sh --max-instances 50
./scripts/benchmarks/run-longmemeval.sh --category single-session-user
./scripts/benchmarks/run-longmemeval.sh --parallel 4  # Faster evaluation

Options:

--max-instances N - Limit total questions
--max-instances-per-category N - Limit per category
--skip-ingestion - Skip data ingestion
--category NAME - Filter by category:
- single-session-user
- multi-session
- single-session-preference
- temporal-reasoning
- knowledge-update
- single-session-assistant
--parallel N - Parallel instances (default: 1)
--only-failed - Retry failed questions
--fill - Resume interrupted runs

Visualizer

View benchmark results in a web UI:

./scripts/benchmarks/start-visualizer.sh
# Opens at http://localhost:8001

Results

Results are saved in JSON format in each benchmark's results/ directory.