# Performance

Hindsight is designed for high-performance semantic memory operations at scale. This page covers performance characteristics, optimization strategies, and best practices.

## Overview

Hindsight's performance is optimized across three key operations:

- **Retain (Ingestion)**: Batch processing with async operations for large-scale memory storage
- **Recall (Search)**: Sub-second semantic search with configurable thinking budgets
- **Reflect (Reasoning)**: Personality-aware answer generation with controllable compute

## Design Philosophy: Optimized for Fast Reads

Hindsight is **architected from the ground up to prioritize read performance over write performance**. This design decision reflects the typical usage pattern of memory systems: memories are written once but read many times.

The system makes deliberate trade-offs to ensure **sub-second recall operations**:

- **Pre-computed embeddings**: All memory embeddings are generated and indexed during retention
- **Optimized vector search**: HNSW indexes enable fast approximate nearest neighbor search
- **Fact extraction at write time**: Complex LLM-based fact extraction happens during retention, not retrieval
- **Structured memory graphs**: Relationships and temporal information are resolved upfront

This means **Recall (search) operations are blazingly fast** because all the heavy lifting has already been done.

### Performance Comparison

| Operation | Typical Latency | Primary Bottleneck | Optimization Strategy |
|-----------|----------------|-------------------|----------------------|
| **Recall** | 100-600ms | Vector search, graph traversal | ✅ Already optimized |
| **Reflect** | 800-3000ms | LLM generation + search | Reduce search budget, use faster LLM |
| **Retain** | 500ms-2000ms per batch | **LLM fact extraction** | Use high-throughput LLM provider |

Hindsight is designed to ensure your **application's read path (recall/reflect) is always fast**, even if it means spending more time upfront during writes. This is the right trade-off for memory systems where:

- Memories are retained in background processes or during low-traffic periods
- Memories are queried frequently in user-facing, latency-sensitive contexts
- The ratio of reads to writes is high (typically 10:1 or higher)

---

## Retain Performance

**Retain (write) operations are inherently slower** because they involve LLM-based fact extraction, entity recognition, temporal reasoning, relationship mapping, and embedding generation. **The LLM is the primary bottleneck for write latency.**

### Hindsight Doesn't Need a Smart Model

The fact extraction process is structured and well-defined, so smaller, faster models work extremely well. Our recommended model is `gpt-oss-20b` (available via Groq and other providers).

To maximize retention throughput:

1. **Use high-throughput LLM providers**: Choose providers with high requests-per-minute (RPM) limits and low latency
   - ✅ **Fast**: [Groq](https://groq.com) with `gpt-oss-20b` or other openai-oss models, self-hosted models on GPU clusters (vLLM, TGI)
   - ⚠️ **Slower**: Standard cloud LLM providers with rate limits

2. **Batch your operations**: Group related content into batch requests. The only limit is the HTTP payload size — Hindsight automatically splits large batches into smaller, optimized chunks under the hood, so you don't have to worry about it.

3. **Use async mode for large datasets**: Queue operations in the background

4. **Parallel processing**: For very large datasets, use multiple concurrent retention requests with different `document_id` values

### Throughput

Typical ingestion performance:

| Mode | Items/second | Use Case |
|------|--------------|----------|
| Synchronous | ~50-100 | Real-time updates, small batches |
| Async (batched) | ~500-1000 | Bulk imports, background processing |
| Parallel async | ~2000-5000 | Large-scale data migration |

**Factors affecting throughput:**
- Document size and complexity
- LLM provider rate limits (for fact extraction)
- Database write performance
- Available CPU/memory resources

---

## Recall Performance

### Budget

The `budget` parameter controls the search depth and quality. Choose based on query complexity — comprehensive questions that need thorough analysis benefit from higher budgets:

| Budget | Latency | Memory Activation | Use Case |
|--------|---------|-------------------|----------|
| `low` | 100-300ms | ~10-50 facts | Quick lookups, real-time chat |
| `mid` | 300-600ms | ~50-200 facts | Standard queries, balanced performance |
| `high` | 500-1500ms | ~200-500 facts | Comprehensive questions, thorough analysis |

### Search Optimization

1. **Appropriate budgets**: Use lower budgets for simple queries, higher for comprehensive reasoning
2. **Limit result tokens**: Set `max_tokens` to control response size (default: 4096)
3. **Include entities/chunks**: Use `include_entities` and `include_chunks` to retrieve additional context when needed — each has its own token budget

### Database Performance

Hindsight uses PostgreSQL with pgvector for efficient vector search:

- **Index type**: HNSW for approximate nearest neighbor search
- **Typical query time**: 10-50ms for vector search on 100K+ facts
- **Scalability**: Tested with millions of facts per bank

## Reflect Performance

### Performance Characteristics

| Component | Latency | Description |
|-----------|---------|-------------|
| Memory search | 300-1000ms | Based on budget (low/mid/high) |
| LLM generation | 500-2000ms | Depends on provider and response length |
| **Total** | **800-3000ms** | Typical end-to-end latency |

### Optimization Strategies

1. **Budget selection**: Use lower budgets when context is sufficient
2. **Context provision**: Provide relevant `context` to reduce search requirements
3. **Streaming responses**: Use streaming APIs (when available) for faster time-to-first-token
4. **Caching**: Cache frequent queries at the application level

## Best Practices

### Operations
- **Use appropriate budgets**: Don't over-provision for simple queries; use higher budgets for comprehensive reasoning
- **Batch retain operations**: Group related content together for better efficiency
- **Cache frequent queries**: Cache at the application level for repeated queries
- **Profile with trace**: Use the `trace` parameter to identify slow operations

### Scaling
- **Horizontal scaling**: Deploy multiple API instances behind a load balancer with shared PostgreSQL
- **Concurrency**: 100+ simultaneous requests supported; memory search scales with CPU cores
- **LLM rate limits**: Distribute load across multiple API keys/providers (typically 60-500 RPM per key)

### Cost Optimization
- **Use efficient models**: `gpt-oss-20b` via Groq for retain — Hindsight doesn't need frontier models
- **Control token budgets**: Limit `max_tokens` for recall, use lower budgets when possible
- **Optimize chunks**: Larger chunks (1000-2000 tokens) are more efficient than many small ones

### Monitoring
- **Prometheus metrics**: Available at `/metrics` — track latency percentiles, throughput, and error rates
- **Key metrics**: `hindsight_recall_duration_seconds`, `hindsight_reflect_duration_seconds`, `hindsight_retain_items_total`