529 lines
13 KiB
Markdown
529 lines
13 KiB
Markdown
# Models
|
|
|
|
Hindsight uses several machine learning models for different tasks. This page explains what models are used, why they're chosen, and how to optimize their performance.
|
|
|
|
## Model Overview
|
|
|
|
Hindsight's processing pipeline uses four types of models:
|
|
|
|
| Model Type | Purpose | Default Model | Configurable |
|
|
|------------|---------|---------------|--------------|
|
|
| **Embedding** | Vector representations for semantic search | `all-MiniLM-L6-v2` | Yes |
|
|
| **Cross-Encoder** | Reranking search results | `ms-marco-MiniLM-L-6-v2` | Yes |
|
|
| **Temporal Parser** | Understanding time expressions | `t5-small` | Yes |
|
|
| **LLM** | Fact extraction, reasoning, generation | Provider-specific | Yes |
|
|
|
|
All local models (embedding, cross-encoder, temporal) are automatically downloaded from HuggingFace on first run and cached in `~/.cache/huggingface/`.
|
|
|
|
## Embedding Model
|
|
|
|
### Purpose
|
|
|
|
The embedding model converts text into dense vector representations (embeddings) for semantic similarity search.
|
|
|
|
**Used for**:
|
|
- Encoding memory units during retention
|
|
- Encoding search queries during recall
|
|
- Vector similarity calculations
|
|
|
|
### Default: all-MiniLM-L6-v2
|
|
|
|
```
|
|
Model: sentence-transformers/all-MiniLM-L6-v2
|
|
Dimensions: 384
|
|
Size: ~90MB
|
|
Performance: ~2000 texts/second on CPU
|
|
```
|
|
|
|
**Why this model?**
|
|
- **Fast**: Optimized for CPU inference
|
|
- **Small**: Only 384 dimensions, efficient storage
|
|
- **Accurate**: Strong performance on semantic similarity tasks
|
|
- **Well-balanced**: Good trade-off between speed and quality
|
|
|
|
### Performance Optimization
|
|
|
|
#### 1. Use GPU Acceleration
|
|
|
|
```bash
|
|
# Enable CUDA (NVIDIA GPUs)
|
|
export HINDSIGHT_API_EMBEDDING_DEVICE=cuda
|
|
|
|
# Enable MPS (Apple Silicon)
|
|
export HINDSIGHT_API_EMBEDDING_DEVICE=mps
|
|
|
|
# Verify GPU usage in logs
|
|
hindsight-api --log-level debug
|
|
# Should see: "Loading embedding model on device: cuda"
|
|
```
|
|
|
|
**Expected speedup**:
|
|
- CPU: ~2000 texts/second
|
|
- GPU (CUDA): ~10,000-20,000 texts/second
|
|
- Apple Silicon (MPS): ~5,000-10,000 texts/second
|
|
|
|
#### 2. Increase Batch Size
|
|
|
|
```bash
|
|
# Default batch size
|
|
export HINDSIGHT_API_EMBEDDING_BATCH_SIZE=32
|
|
|
|
# Larger batch size for better throughput (requires more memory)
|
|
export HINDSIGHT_API_EMBEDDING_BATCH_SIZE=128
|
|
|
|
# Smaller batch size for limited memory
|
|
export HINDSIGHT_API_EMBEDDING_BATCH_SIZE=16
|
|
```
|
|
|
|
**Guidelines**:
|
|
- **CPU**: 32-64 (diminishing returns beyond 64)
|
|
- **GPU**: 128-256 (can go higher with more VRAM)
|
|
- **Memory-constrained**: 8-16
|
|
|
|
#### 3. Alternative Embedding Models
|
|
|
|
For different use cases, you can use other embedding models:
|
|
|
|
**Higher Quality (Slower)**
|
|
|
|
```bash
|
|
# 768 dimensions, better accuracy, slower
|
|
export HINDSIGHT_API_EMBEDDING_MODEL=sentence-transformers/all-mpnet-base-v2
|
|
```
|
|
|
|
**Multilingual Support**
|
|
|
|
```bash
|
|
# Supports 50+ languages
|
|
export HINDSIGHT_API_EMBEDDING_MODEL=sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
|
|
```
|
|
|
|
**Larger Context Window**
|
|
|
|
```bash
|
|
# 512 token context (vs 256 for MiniLM)
|
|
export HINDSIGHT_API_EMBEDDING_MODEL=sentence-transformers/all-roberta-large-v1
|
|
```
|
|
|
|
### Memory Requirements
|
|
|
|
| Model | Dimensions | Model Size | Runtime RAM (CPU) | Runtime RAM (GPU) |
|
|
|-------|------------|------------|-------------------|-------------------|
|
|
| all-MiniLM-L6-v2 | 384 | 90MB | ~500MB | ~1GB |
|
|
| all-mpnet-base-v2 | 768 | 420MB | ~1GB | ~2GB |
|
|
| all-roberta-large-v1 | 1024 | 1.3GB | ~2GB | ~4GB |
|
|
|
|
## Cross-Encoder (Reranker)
|
|
|
|
### Purpose
|
|
|
|
The cross-encoder reranks initial search results to improve precision.
|
|
|
|
**How it works**:
|
|
1. Vector search returns top 50-100 candidates (fast but approximate)
|
|
2. Cross-encoder scores each candidate with the query (slower but accurate)
|
|
3. Results are reranked by cross-encoder score
|
|
|
|
### Default: ms-marco-MiniLM-L-6-v2
|
|
|
|
```
|
|
Model: cross-encoder/ms-marco-MiniLM-L-6-v2
|
|
Size: ~85MB
|
|
Performance: ~500 pairs/second on CPU
|
|
```
|
|
|
|
**Why this model?**
|
|
- **Accurate**: Trained on Microsoft MARCO dataset for passage ranking
|
|
- **Fast enough**: Can rerank 50 results in ~100ms on CPU
|
|
- **Small**: Efficient memory footprint
|
|
|
|
### Performance Optimization
|
|
|
|
#### 1. Control Reranking Scope
|
|
|
|
```bash
|
|
# Rerank top 50 results (default)
|
|
export HINDSIGHT_API_RERANK_TOP_K=50
|
|
|
|
# More thorough reranking (slower)
|
|
export HINDSIGHT_API_RERANK_TOP_K=100
|
|
|
|
# Faster reranking (less accurate)
|
|
export HINDSIGHT_API_RERANK_TOP_K=20
|
|
|
|
# Disable reranking entirely (fastest, less accurate)
|
|
export HINDSIGHT_API_RERANK_ENABLED=false
|
|
```
|
|
|
|
**Trade-offs**:
|
|
- More reranking = Better precision, higher latency
|
|
- Less reranking = Faster queries, lower precision
|
|
- No reranking = Fastest, relies only on vector similarity
|
|
|
|
#### 2. GPU Acceleration
|
|
|
|
Cross-encoders also benefit from GPU:
|
|
|
|
```bash
|
|
# Uses same device as embedding model
|
|
export HINDSIGHT_API_EMBEDDING_DEVICE=cuda
|
|
```
|
|
|
|
**Speedup**: ~5-10x faster on GPU vs CPU
|
|
|
|
### Alternative Reranker Models
|
|
|
|
**Higher Accuracy**
|
|
|
|
```bash
|
|
export HINDSIGHT_API_RERANK_MODEL=cross-encoder/ms-marco-MiniLM-L-12-v2
|
|
# Larger model, ~200MB, better accuracy
|
|
```
|
|
|
|
**Multilingual**
|
|
|
|
```bash
|
|
export HINDSIGHT_API_RERANK_MODEL=cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
|
|
# Supports multiple languages
|
|
```
|
|
|
|
## Temporal Parser
|
|
|
|
### Purpose
|
|
|
|
Parses natural language time expressions into structured dates.
|
|
|
|
**Examples**:
|
|
- "last spring" → 2024-03-20 to 2024-06-20
|
|
- "in June 2024" → 2024-06-01 to 2024-06-30
|
|
- "two weeks ago" → 2024-05-15 to 2024-05-15
|
|
|
|
### Default: t5-small
|
|
|
|
```
|
|
Model: google/t5-small
|
|
Size: ~240MB
|
|
Performance: ~100 expressions/second on CPU
|
|
```
|
|
|
|
**Why this model?**
|
|
- **Accurate**: Good performance on temporal expression parsing
|
|
- **Compact**: Small enough for CPU inference
|
|
- **Standard**: Well-established model for sequence-to-sequence tasks
|
|
|
|
### Performance Optimization
|
|
|
|
Temporal parsing is typically not a bottleneck, but you can:
|
|
|
|
1. **Use a larger model for better accuracy**:
|
|
```bash
|
|
export HINDSIGHT_API_TEMPORAL_MODEL=google/t5-base
|
|
# ~850MB, better at complex temporal expressions
|
|
```
|
|
|
|
2. **Use GPU** (shared with other models):
|
|
```bash
|
|
export HINDSIGHT_API_EMBEDDING_DEVICE=cuda
|
|
```
|
|
|
|
## LLM (Large Language Model)
|
|
|
|
### Purpose
|
|
|
|
The LLM is used for high-level reasoning tasks that require language understanding and generation.
|
|
|
|
**Used for**:
|
|
- **Fact extraction**: Converting text into structured facts (retention)
|
|
- **Entity resolution**: Identifying and linking entities (retention)
|
|
- **Opinion generation**: Creating personality-based opinions (reflection)
|
|
- **Answer synthesis**: Generating responses from memories (reflect)
|
|
|
|
### Default: Provider-Specific
|
|
|
|
Hindsight supports multiple LLM providers. The default depends on your configuration:
|
|
|
|
| Provider | Default Model | Best For |
|
|
|----------|---------------|----------|
|
|
| **Groq** | `llama-3.1-70b-versatile` | High throughput, fast inference |
|
|
| **OpenAI** | `gpt-4o` | Best quality, general-purpose |
|
|
| **Anthropic** | `claude-3-5-sonnet-20241022` | Long context, complex reasoning |
|
|
| **Ollama** | User-specified | Local deployment, privacy |
|
|
|
|
### Performance Optimization
|
|
|
|
**The LLM is the primary bottleneck for write operations (retention).** See [Performance](./performance.md) for detailed optimization strategies.
|
|
|
|
#### 1. Choose the Right Provider
|
|
|
|
For **high-throughput retention** (many memories/second):
|
|
|
|
```bash
|
|
# Groq - fastest inference
|
|
export HINDSIGHT_API_LLM_PROVIDER=groq
|
|
export HINDSIGHT_API_LLM_MODEL=llama-3.1-70b-versatile
|
|
export HINDSIGHT_API_LLM_API_KEY=gsk_xxxxxxxxxxxx
|
|
```
|
|
|
|
**Groq advantages**:
|
|
- 10-30x faster than OpenAI for similar models
|
|
- High rate limits (30+ RPM for free tier)
|
|
- Low latency (~500ms for retention)
|
|
|
|
For **best quality** (reasoning, complex fact extraction):
|
|
|
|
```bash
|
|
# OpenAI GPT-4
|
|
export HINDSIGHT_API_LLM_PROVIDER=openai
|
|
export HINDSIGHT_API_LLM_MODEL=gpt-4o
|
|
export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx
|
|
```
|
|
|
|
For **cost optimization**:
|
|
|
|
```bash
|
|
# OpenAI GPT-4 Mini - 60x cheaper than GPT-4
|
|
export HINDSIGHT_API_LLM_PROVIDER=openai
|
|
export HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
|
export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx
|
|
```
|
|
|
|
For **local/private deployment**:
|
|
|
|
```bash
|
|
# Ollama with local Llama 3.1
|
|
export HINDSIGHT_API_LLM_PROVIDER=ollama
|
|
export HINDSIGHT_API_LLM_BASE_URL=http://localhost:11434/v1
|
|
export HINDSIGHT_API_LLM_MODEL=llama3.1
|
|
```
|
|
|
|
#### 2. Optimize LLM Configuration
|
|
|
|
```bash
|
|
# Increase timeout for slower providers
|
|
export HINDSIGHT_API_LLM_TIMEOUT=60 # seconds
|
|
|
|
# Increase retries for reliability
|
|
export HINDSIGHT_API_LLM_MAX_RETRIES=5
|
|
|
|
# Enable request caching (if supported by provider)
|
|
export HINDSIGHT_API_LLM_CACHE_ENABLED=true
|
|
```
|
|
|
|
#### 3. Rate Limit Management
|
|
|
|
For providers with strict rate limits:
|
|
|
|
1. **Use async retention** to queue operations:
|
|
```python
|
|
client.retain_memories(bank_id="...", items=batch, async_=True)
|
|
```
|
|
|
|
2. **Distribute across multiple API keys**:
|
|
```bash
|
|
# Rotate between keys in application logic
|
|
export HINDSIGHT_API_LLM_API_KEY_1=sk-key1
|
|
export HINDSIGHT_API_LLM_API_KEY_2=sk-key2
|
|
```
|
|
|
|
3. **Use multiple providers** for different operations:
|
|
```bash
|
|
# Groq for retention (fast)
|
|
# OpenAI for reflection (quality)
|
|
```
|
|
|
|
### Model Comparison
|
|
|
|
| Provider | Model | Speed | Quality | Cost/1M tokens | Rate Limit |
|
|
|----------|-------|-------|---------|----------------|------------|
|
|
| Groq | llama-3.1-70b | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | Free tier | 30 RPM |
|
|
| OpenAI | gpt-4o-mini | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | $0.15 / $0.60 | 500 RPM |
|
|
| OpenAI | gpt-4o | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | $2.50 / $10.00 | 500 RPM |
|
|
| Anthropic | claude-3-5-sonnet | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | $3.00 / $15.00 | 50 RPM |
|
|
| Ollama | llama3.1 (local) | ⭐⭐ | ⭐⭐⭐ | Free | Unlimited |
|
|
|
|
## Resource Requirements
|
|
|
|
### Minimal Configuration (Development)
|
|
|
|
```
|
|
CPU: 2 cores
|
|
RAM: 4GB
|
|
Storage: 5GB (models + data)
|
|
```
|
|
|
|
Models loaded:
|
|
- Embedding model (~500MB RAM)
|
|
- Cross-encoder (~300MB RAM)
|
|
- Temporal parser (~500MB RAM)
|
|
- **Total**: ~1.5GB for models + 2GB for application
|
|
|
|
### Recommended Configuration (Production)
|
|
|
|
```
|
|
CPU: 4-8 cores
|
|
RAM: 8-16GB
|
|
GPU: Optional (NVIDIA with 4GB+ VRAM for 10x speedup)
|
|
Storage: 20GB+ (models + database)
|
|
```
|
|
|
|
Models loaded:
|
|
- Same models as minimal
|
|
- Additional RAM for connection pooling
|
|
- PostgreSQL in separate container/server
|
|
|
|
### High-Performance Configuration
|
|
|
|
```
|
|
CPU: 8-16 cores
|
|
RAM: 16-32GB
|
|
GPU: NVIDIA T4, V100, or A100 (8-40GB VRAM)
|
|
Storage: 50GB+ SSD
|
|
```
|
|
|
|
Benefits:
|
|
- GPU acceleration for embeddings: 10x faster
|
|
- More RAM for larger batch sizes
|
|
- More CPU cores for parallel processing
|
|
|
|
## Model Caching and Storage
|
|
|
|
### Cache Locations
|
|
|
|
```bash
|
|
# HuggingFace models
|
|
~/.cache/huggingface/
|
|
|
|
# Model-specific caches
|
|
~/.cache/torch/
|
|
|
|
# Clear caches
|
|
rm -rf ~/.cache/huggingface/
|
|
rm -rf ~/.cache/torch/
|
|
```
|
|
|
|
### Preloading Models
|
|
|
|
To avoid download delays in production:
|
|
|
|
```bash
|
|
# Pre-download all models
|
|
python -c "
|
|
from sentence_transformers import SentenceTransformer, CrossEncoder
|
|
from transformers import T5ForConditionalGeneration, T5Tokenizer
|
|
|
|
# Download embedding model
|
|
SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
|
|
|
|
# Download cross-encoder
|
|
CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
|
|
|
|
# Download temporal parser
|
|
T5ForConditionalGeneration.from_pretrained('google/t5-small')
|
|
T5Tokenizer.from_pretrained('google/t5-small')
|
|
"
|
|
```
|
|
|
|
Or build into Docker image:
|
|
|
|
```dockerfile
|
|
FROM python:3.11-slim
|
|
|
|
# Install dependencies
|
|
RUN pip install hindsight-all
|
|
|
|
# Pre-download models
|
|
RUN python -c "from sentence_transformers import SentenceTransformer; \
|
|
SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')"
|
|
|
|
# Rest of Dockerfile...
|
|
```
|
|
|
|
## Monitoring Model Performance
|
|
|
|
### Check Model Loading
|
|
|
|
```bash
|
|
# Enable debug logging
|
|
export HINDSIGHT_API_LOG_LEVEL=debug
|
|
hindsight-api
|
|
|
|
# Look for logs like:
|
|
# INFO: Loading embedding model: all-MiniLM-L6-v2 on device: cpu
|
|
# INFO: Loading cross-encoder: ms-marco-MiniLM-L-6-v2
|
|
# INFO: Loading temporal parser: t5-small
|
|
```
|
|
|
|
### Monitor Resource Usage
|
|
|
|
```python
|
|
# In your application logs
|
|
import psutil
|
|
|
|
# Memory usage
|
|
print(f"RAM: {psutil.virtual_memory().percent}%")
|
|
|
|
# CPU usage
|
|
print(f"CPU: {psutil.cpu_percent()}%")
|
|
|
|
# GPU usage (if available)
|
|
import torch
|
|
if torch.cuda.is_available():
|
|
print(f"GPU Memory: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Models Not Downloaded
|
|
|
|
```bash
|
|
# Check cache directory
|
|
ls -lh ~/.cache/huggingface/
|
|
|
|
# Manually download
|
|
python -c "from sentence_transformers import SentenceTransformer; \
|
|
SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')"
|
|
|
|
# Check network connectivity
|
|
curl https://huggingface.co/
|
|
```
|
|
|
|
### Out of Memory
|
|
|
|
```bash
|
|
# Reduce batch size
|
|
export HINDSIGHT_API_EMBEDDING_BATCH_SIZE=8
|
|
|
|
# Use smaller models
|
|
export HINDSIGHT_API_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2 # smallest
|
|
|
|
# Disable reranking
|
|
export HINDSIGHT_API_RERANK_ENABLED=false
|
|
```
|
|
|
|
### Slow Inference
|
|
|
|
```bash
|
|
# Enable GPU if available
|
|
export HINDSIGHT_API_EMBEDDING_DEVICE=cuda
|
|
|
|
# Check GPU availability
|
|
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
|
|
|
|
# Increase batch size (if you have RAM)
|
|
export HINDSIGHT_API_EMBEDDING_BATCH_SIZE=128
|
|
```
|
|
|
|
### LLM Rate Limits
|
|
|
|
```bash
|
|
# Use Groq for higher limits
|
|
export HINDSIGHT_API_LLM_PROVIDER=groq
|
|
|
|
# Use async retention to queue operations
|
|
# (in your application code)
|
|
client.retain_memories(..., async_=True)
|
|
```
|
|
|
|
---
|
|
|
|
For model-related questions or issues, please [open an issue](https://github.com/your-repo/hindsight/issues) on GitHub.
|