* fix docker cp image build on ci

* fix docker

* fix docker again

2025-12-03 21:10:46 +01:00

14 KiB

Raw Blame History

Performance

Hindsight is designed for high-performance semantic memory operations at scale. This page covers performance characteristics, optimization strategies, and best practices.

Overview

Hindsight's performance is optimized across three key operations:

Retain (Ingestion): Batch processing with async operations for large-scale memory storage
Recall (Search): Sub-second semantic search with configurable thinking budgets
Reflect (Reasoning): Personality-aware answer generation with controllable compute

Design Philosophy: Optimized for Fast Reads

Hindsight is architected from the ground up to prioritize read performance over write performance. This design decision reflects the typical usage pattern of memory systems: memories are written once but read many times.

Read-Optimized Architecture

The system makes deliberate trade-offs to ensure sub-second recall operations:

Pre-computed embeddings: All memory embeddings are generated and indexed during retention
Optimized vector search: HNSW indexes enable fast approximate nearest neighbor search
Fact extraction at write time: Complex LLM-based fact extraction happens during retention, not retrieval
Structured memory graphs: Relationships and temporal information are resolved upfront

This means Recall (search) operations are blazingly fast because all the heavy lifting has already been done.

Write Performance: LLM-Bound Operations

The trade-off is that Retain (write) operations are inherently slower because they involve:

LLM-based fact extraction: Converting raw text into structured semantic facts
Entity recognition and resolution: Identifying and linking entities across memories
Temporal reasoning: Extracting and normalizing time references
Relationship mapping: Building the semantic graph structure
Embedding generation: Creating vector representations for search

The LLM is the primary bottleneck for write latency. Each piece of content requires one or more LLM calls for fact extraction, which typically takes 500ms-2000ms per batch depending on content complexity.

Achieving Fast Writes

To maximize retention throughput, we recommend:

Use high-throughput LLM providers: Choose providers with high requests-per-minute (RPM) limits
- ✅ Recommended: Groq (up to 30 RPM for Llama models), OpenAI GPT-4 Turbo/Mini
- ⚠️ Slower: Claude with lower rate limits, local models

Batch your operations: Group related content into batch requests to amortize overhead

# Good: Batch retention
client.retain_memories(bank_id="...", items=batch_of_100_items)

# Less efficient: Individual retention
for item in items:
    client.retain_memories(bank_id="...", items=[item])

Use async mode for large datasets: Queue operations in the background

client.retain_memories(bank_id="...", items=large_batch, async_=True)

Parallel processing: For very large datasets, use multiple concurrent retention requests with different document_id values

Performance Comparison

Operation	Typical Latency	Primary Bottleneck	Optimization Strategy
Recall	100-600ms	Vector search, graph traversal	✅ Already optimized
Reflect	800-3000ms	LLM generation + search	Reduce search budget, use faster LLM
Retain	500ms-2000ms per batch	LLM fact extraction	Use high-throughput LLM provider

The Bottom Line

Hindsight is designed to ensure your application's read path (recall/reflect) is always fast, even if it means spending more time upfront during writes. This is the right trade-off for memory systems where:

Memories are retained in background processes or during low-traffic periods
Memories are queried frequently in user-facing, latency-sensitive contexts
The ratio of reads to writes is high (typically 10:1 or higher)

If your use case requires extremely fast writes, focus on LLM provider selection and batching strategies rather than database or infrastructure optimization.

Retain Performance

Batch Ingestion

Hindsight supports high-throughput batch ingestion for efficient memory storage:

from hindsight_client import HindsightClient

client = HindsightClient(base_url="http://localhost:8888")

# Batch retain for better performance
items = [
    {"content": "Memory 1", "context": "Context 1"},
    {"content": "Memory 2", "context": "Context 2"},
    # ... up to thousands of items
]

result = client.retain_memories(
    bank_id="my-bank",
    items=items,
    document_id="batch-doc-001"
)

Async Operations

For very large datasets, use async operations to avoid blocking:

# Queue for background processing
result = client.retain_memories(
    bank_id="my-bank",
    items=large_dataset,
    async_=True  # Process in background
)

print(f"Queued {result.items_count} items for processing")

# Check operation status
operations = client.list_operations(bank_id="my-bank")
for op in operations:
    print(f"Operation {op.id}: {op.status}")

Ingestion Throughput

Typical ingestion performance on standard hardware:

Mode	Items/second	Use Case
Synchronous	~50-100	Real-time updates, small batches
Async (batched)	~500-1000	Bulk imports, background processing
Parallel async	~2000-5000	Large-scale data migration

Factors affecting throughput:

Document size and complexity
LLM provider rate limits (for fact extraction)
Database write performance
Available CPU/memory resources

Optimization Tips

Batch related memories: Group related content into the same document for better context
Use async for large batches: Set async_=True for batches > 100 items
Optimize chunk sizes: Larger chunks (1000-2000 tokens) are more efficient than many small chunks
Parallel processing: Use multiple concurrent requests with different document_id values

Recall Performance

Search Latency

Hindsight provides sub-second semantic search with configurable performance/quality tradeoffs:

# Fast search (low budget)
result = client.recall_memories(
    bank_id="my-bank",
    query="What did we discuss about the project?",
    budget="low"  # ~100-200ms
)

# Balanced search (mid budget)
result = client.recall_memories(
    bank_id="my-bank",
    query="What did we discuss about the project?",
    budget="mid"  # ~300-500ms
)

# Thorough search (high budget)
result = client.recall_memories(
    bank_id="my-bank",
    query="What did we discuss about the project?",
    budget="high"  # ~500-1000ms
)

Thinking Budget

The budget parameter controls the search depth and quality:

Budget	Latency	Memory Activation	Use Case
`low`	100-300ms	~10-50 facts	Quick lookups, real-time chat
`mid`	300-600ms	~50-200 facts	Standard queries, balanced performance
`high`	500-1500ms	~200-500 facts	Complex questions, thorough analysis

Search Optimization

Appropriate budgets: Use lower budgets for simple queries, higher for complex reasoning
Limit result tokens: Set max_tokens to control response size (default: 4096)
Filter by fact type: Specify types to search only relevant fact categories
Temporal filtering: Use query_timestamp for time-aware search

Database Performance

Hindsight uses PostgreSQL with pgvector for efficient vector search:

Index type: HNSW for approximate nearest neighbor search
Typical query time: 10-50ms for vector search on 100K+ facts
Scalability: Tested with millions of facts per bank

Reflect Performance

Answer Generation

Reflect combines semantic search with personality-aware reasoning:

result = client.reflect(
    bank_id="my-bank",
    query="What should we prioritize next quarter?",
    budget="mid",  # Controls memory search depth
    context="We have limited resources"
)

print(result.text)  # Personality-aware answer

Performance Characteristics

Component	Latency	Description
Memory search	300-1000ms	Based on budget (low/mid/high)
LLM generation	500-2000ms	Depends on provider and response length
Total	800-3000ms	Typical end-to-end latency

Optimization Strategies

Budget selection: Use lower budgets when context is sufficient
Context provision: Provide relevant context to reduce search requirements
Streaming responses: Use streaming APIs (when available) for faster time-to-first-token
Caching: Cache frequent queries at the application level

Concurrent Operations

Parallelism

Hindsight supports high levels of concurrent operations:

import asyncio
from hindsight_client import AsyncHindsightClient

async def parallel_recall():
    client = AsyncHindsightClient(base_url="http://localhost:8888")

    # Execute multiple recalls in parallel
    tasks = [
        client.recall_memories("bank-1", query="query 1"),
        client.recall_memories("bank-2", query="query 2"),
        client.recall_memories("bank-3", query="query 3"),
    ]

    results = await asyncio.gather(*tasks)
    return results

Concurrency Limits

Default limits (configurable in server settings):

Database connections: Pool of 20 connections
LLM rate limits: Depends on provider (typically 60-500 RPM)
Memory search: No hard limit, scales with CPU cores
Concurrent requests: 100+ simultaneous requests supported

Scaling Strategies

Horizontal Scaling

Hindsight can be scaled horizontally for high-throughput scenarios:

Multiple API instances: Deploy multiple Hindsight servers behind a load balancer
Shared database: All instances connect to the same PostgreSQL database
LLM provider limits: Distribute load across multiple API keys/providers
Bank isolation: Distribute banks across different instances for better isolation

Database Scaling

For very large deployments:

Connection pooling: Use pgBouncer for connection management
Read replicas: Use PostgreSQL read replicas for read-heavy workloads
Partitioning: Partition large banks by time or topic
Vacuum and analyze: Regular maintenance for optimal query performance

Resource Requirements

Recommended specifications per 1M facts:

Resource	Minimum	Recommended
CPU	2 cores	4-8 cores
RAM	4GB	8-16GB
Database storage	10GB	20GB+ (with indexes)
Vector index RAM	2GB	4GB+

Benchmarks

LoComo Benchmark Results

Hindsight has been evaluated on the LoComo (Long Context Memory) benchmark:

Dataset: 10 conversations with multi-hop, temporal, and reasoning questions
Overall accuracy: ~65-75% (varies by category)
Average recall latency: 400-600ms (mid budget)
Average reflect latency: 1500-2500ms (end-to-end)

See the GitHub repository for detailed benchmark results.

Performance Metrics

Key performance indicators to monitor:

Latency percentiles: Track p50, p95, p99 for recall/reflect operations
Throughput: Requests per second for each operation type
Error rates: Failed requests, timeouts, LLM errors
Resource utilization: CPU, memory, database connection pool usage
LLM costs: Token usage and API costs per operation

Monitoring and Optimization

Enable Trace Information

Use the trace parameter to analyze performance:

result = client.recall_memories(
    bank_id="my-bank",
    query="test query",
    trace=True
)

if result.trace:
    print(f"Total time: {result.trace.get('total_time')}ms")
    print(f"Activations: {result.trace.get('activation_count')}")

Metrics Collection

Hindsight exposes Prometheus metrics for monitoring:

curl http://localhost:8888/metrics

Key metrics:

hindsight_recall_duration_seconds: Recall operation latency
hindsight_reflect_duration_seconds: Reflect operation latency
hindsight_retain_items_total: Number of items retained
hindsight_database_connections: Active database connections

Performance Tuning

Server configuration options (environment variables):

# Database connection pool
export DB_POOL_SIZE=20
export DB_MAX_OVERFLOW=10

# LLM configuration
export LLM_PROVIDER=openai
export LLM_MAX_RETRIES=3
export LLM_TIMEOUT=30

# Search configuration
export DEFAULT_THINKING_BUDGET=500
export MAX_SEARCH_RESULTS=100

Best Practices

Use appropriate budgets: Don't over-provision thinking budget for simple queries
Batch operations: Group related retains together for better efficiency
Monitor costs: Track LLM token usage and optimize prompts
Cache when possible: Cache frequently accessed queries at the application level
Clean old data: Regularly archive or delete unused memory banks
Profile queries: Use trace information to identify slow operations
Load test: Test your specific workload before production deployment

Cost Optimization

LLM Token Usage

Optimize costs by controlling token usage:

Chunk size: Larger chunks reduce overhead but increase individual LLM calls
Max tokens: Limit max_tokens to reduce response size
Fact extraction: Use efficient models (e.g., GPT-4 Mini) for retain operations
Budget management: Lower budgets reduce the number of facts processed

Typical Costs

Example costs using OpenAI GPT-4:

Operation	Tokens	Cost per request	Notes
Retain (1 item)	~1000-2000	$0.01-0.02	Fact extraction
Recall	~2000-8000	$0.02-0.08	Depends on budget
Reflect	~4000-12000	$0.04-0.12	Search + generation

Note: Costs vary significantly by model provider and configuration. Use cheaper models (GPT-4 Mini, Claude Haiku) for non-critical operations.

Future Improvements

Planned optimizations:

Adaptive budgeting: Automatically adjust thinking budget based on query complexity
Incremental updates: Update facts without full re-extraction
Query caching: Built-in cache for frequently accessed memories
Multi-modal support: Efficient processing of images and documents
Distributed search: Shard large banks across multiple databases

For specific performance issues or questions, please open an issue on GitHub.

14 KiB Raw Blame History