237 lines
7.9 KiB
Markdown
237 lines
7.9 KiB
Markdown
# Models
|
|
|
|
Hindsight uses several machine learning models for different tasks.
|
|
|
|
## Overview
|
|
|
|
| Model Type | Purpose | Default | Configurable |
|
|
|------------|---------|---------|--------------|
|
|
| **LLM** | Fact extraction, reasoning, generation | Provider-specific | Yes |
|
|
| **Embedding** | Vector representations for semantic search | `BAAI/bge-small-en-v1.5` | Yes |
|
|
| **Cross-Encoder** | Reranking search results | `cross-encoder/ms-marco-MiniLM-L-6-v2` | Yes |
|
|
|
|
All local models (embedding, cross-encoder) are automatically downloaded from HuggingFace on first run.
|
|
|
|
---
|
|
|
|
## LLM
|
|
|
|
Used for fact extraction, entity resolution, mental model consolidation, and answer synthesis.
|
|
|
|
**Supported providers:** OpenAI, Anthropic, Gemini, Groq, Ollama, LM Studio, and **any OpenAI-compatible API**
|
|
|
|
:::tip OpenAI-Compatible Providers
|
|
Hindsight works with any provider that exposes an OpenAI-compatible API (e.g., Azure OpenAI). Simply set `HINDSIGHT_API_LLM_PROVIDER=openai` and configure `HINDSIGHT_API_LLM_BASE_URL` to point to your provider's endpoint.
|
|
|
|
See [Configuration](./configuration#llm-provider) for setup examples.
|
|
:::
|
|
|
|
### Tested Models
|
|
|
|
The following models have been tested and verified to work correctly with Hindsight:
|
|
|
|
| Provider | Model |
|
|
|----------|-------|
|
|
| **OpenAI** | `gpt-5.2` |
|
|
| **OpenAI** | `gpt-5` |
|
|
| **OpenAI** | `gpt-5-mini` |
|
|
| **OpenAI** | `gpt-5-nano` |
|
|
| **OpenAI** | `gpt-4.1-mini` |
|
|
| **OpenAI** | `gpt-4.1-nano` |
|
|
| **OpenAI** | `gpt-4o-mini` |
|
|
| **Anthropic** | `claude-sonnet-4-20250514` |
|
|
| **Anthropic** | `claude-3-5-sonnet-20241022` |
|
|
| **Gemini** | `gemini-3-pro-preview` |
|
|
| **Gemini** | `gemini-2.5-flash` |
|
|
| **Gemini** | `gemini-2.5-flash-lite` |
|
|
| **Groq** | `openai/gpt-oss-120b` |
|
|
| **Groq** | `openai/gpt-oss-20b` |
|
|
|
|
### Using Other Models
|
|
|
|
Other LLM models not listed above may work with Hindsight, but they must support **at least 65,000 output tokens** to ensure reliable fact extraction. If you need support for a specific model that doesn't meet this requirement, please [open an issue](https://github.com/hindsight-ai/hindsight/issues) to request an exception.
|
|
|
|
### Configuration
|
|
|
|
```bash
|
|
# Groq (recommended)
|
|
export HINDSIGHT_API_LLM_PROVIDER=groq
|
|
export HINDSIGHT_API_LLM_API_KEY=gsk_xxxxxxxxxxxx
|
|
export HINDSIGHT_API_LLM_MODEL=openai/gpt-oss-20b
|
|
|
|
# OpenAI
|
|
export HINDSIGHT_API_LLM_PROVIDER=openai
|
|
export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx
|
|
export HINDSIGHT_API_LLM_MODEL=gpt-4o
|
|
|
|
# Gemini
|
|
export HINDSIGHT_API_LLM_PROVIDER=gemini
|
|
export HINDSIGHT_API_LLM_API_KEY=xxxxxxxxxxxx
|
|
export HINDSIGHT_API_LLM_MODEL=gemini-2.0-flash
|
|
|
|
# Anthropic
|
|
export HINDSIGHT_API_LLM_PROVIDER=anthropic
|
|
export HINDSIGHT_API_LLM_API_KEY=sk-ant-xxxxxxxxxxxx
|
|
export HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-20250514
|
|
|
|
# Ollama (local)
|
|
export HINDSIGHT_API_LLM_PROVIDER=ollama
|
|
export HINDSIGHT_API_LLM_BASE_URL=http://localhost:11434/v1
|
|
export HINDSIGHT_API_LLM_MODEL=llama3
|
|
|
|
# LM Studio (local)
|
|
export HINDSIGHT_API_LLM_PROVIDER=lmstudio
|
|
export HINDSIGHT_API_LLM_BASE_URL=http://localhost:1234/v1
|
|
export HINDSIGHT_API_LLM_MODEL=your-local-model
|
|
```
|
|
|
|
**Note:** The LLM is the primary bottleneck for retain operations. See [Performance](./performance) for optimization strategies.
|
|
|
|
---
|
|
|
|
## Embedding Model
|
|
|
|
Converts text into dense vector representations for semantic similarity search.
|
|
|
|
**Default:** `BAAI/bge-small-en-v1.5` (384 dimensions, ~130MB)
|
|
|
|
### Supported Providers
|
|
|
|
| Provider | Description | Best For |
|
|
|----------|-------------|----------|
|
|
| `local` | SentenceTransformers (default) | Development, low latency |
|
|
| `openai` | OpenAI embeddings API | Production, high quality |
|
|
| `cohere` | Cohere embeddings API | Production, multilingual |
|
|
| `tei` | HuggingFace Text Embeddings Inference | Production, self-hosted |
|
|
| `litellm` | LiteLLM proxy (unified gateway) | Multi-provider setups |
|
|
|
|
### Local Models
|
|
|
|
| Model | Dimensions | Use Case |
|
|
|-------|------------|----------|
|
|
| `BAAI/bge-small-en-v1.5` | 384 | Default, fast, good quality |
|
|
| `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` | 384 | Multilingual (50+ languages) |
|
|
|
|
### OpenAI Models
|
|
|
|
| Model | Dimensions | Use Case |
|
|
|-------|------------|----------|
|
|
| `text-embedding-3-small` | 1536 | Default OpenAI, cost-effective |
|
|
| `text-embedding-3-large` | 3072 | Higher quality, more expensive |
|
|
| `text-embedding-ada-002` | 1536 | Legacy model |
|
|
|
|
### Cohere Models
|
|
|
|
| Model | Dimensions | Use Case |
|
|
|-------|------------|----------|
|
|
| `embed-english-v3.0` | 1024 | English text |
|
|
| `embed-multilingual-v3.0` | 1024 | 100+ languages |
|
|
|
|
:::warning Embedding Dimensions
|
|
Hindsight automatically detects the embedding dimension at startup and adjusts the database schema. Once memories are stored, you cannot change dimensions without losing data.
|
|
:::
|
|
|
|
**Configuration Examples:**
|
|
|
|
```bash
|
|
# Local provider (default)
|
|
export HINDSIGHT_API_EMBEDDINGS_PROVIDER=local
|
|
export HINDSIGHT_API_EMBEDDINGS_LOCAL_MODEL=BAAI/bge-small-en-v1.5
|
|
|
|
# OpenAI
|
|
export HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai
|
|
export HINDSIGHT_API_EMBEDDINGS_OPENAI_API_KEY=sk-xxxxxxxxxxxx
|
|
export HINDSIGHT_API_EMBEDDINGS_OPENAI_MODEL=text-embedding-3-small
|
|
|
|
# Cohere
|
|
export HINDSIGHT_API_EMBEDDINGS_PROVIDER=cohere
|
|
export HINDSIGHT_API_COHERE_API_KEY=your-api-key
|
|
export HINDSIGHT_API_EMBEDDINGS_COHERE_MODEL=embed-english-v3.0
|
|
|
|
# TEI (self-hosted)
|
|
export HINDSIGHT_API_EMBEDDINGS_PROVIDER=tei
|
|
export HINDSIGHT_API_EMBEDDINGS_TEI_URL=http://localhost:8080
|
|
|
|
# LiteLLM proxy
|
|
export HINDSIGHT_API_EMBEDDINGS_PROVIDER=litellm
|
|
export HINDSIGHT_API_LITELLM_API_BASE=http://localhost:4000
|
|
export HINDSIGHT_API_EMBEDDINGS_LITELLM_MODEL=text-embedding-3-small
|
|
```
|
|
|
|
See [Configuration](./configuration#embeddings) for all options including Azure OpenAI and custom endpoints.
|
|
|
|
---
|
|
|
|
## Cross-Encoder (Reranker)
|
|
|
|
Reranks initial search results to improve precision.
|
|
|
|
**Default:** `cross-encoder/ms-marco-MiniLM-L-6-v2` (~85MB)
|
|
|
|
### Supported Providers
|
|
|
|
| Provider | Description | Best For |
|
|
|----------|-------------|----------|
|
|
| `local` | SentenceTransformers CrossEncoder (default) | Development, low latency |
|
|
| `cohere` | Cohere rerank API | Production, high quality |
|
|
| `tei` | HuggingFace Text Embeddings Inference | Production, self-hosted |
|
|
| `flashrank` | FlashRank (lightweight, fast) | Resource-constrained environments |
|
|
| `litellm` | LiteLLM proxy (unified gateway) | Multi-provider setups |
|
|
| `rrf` | RRF-only (no neural reranking) | Testing, minimal resources |
|
|
|
|
### Local Models
|
|
|
|
| Model | Use Case |
|
|
|-------|----------|
|
|
| `cross-encoder/ms-marco-MiniLM-L-6-v2` | Default, fast |
|
|
| `cross-encoder/ms-marco-MiniLM-L-12-v2` | Higher accuracy |
|
|
| `cross-encoder/mmarco-mMiniLMv2-L12-H384-v1` | Multilingual |
|
|
|
|
### Cohere Models
|
|
|
|
| Model | Use Case |
|
|
|-------|----------|
|
|
| `rerank-english-v3.0` | English text |
|
|
| `rerank-multilingual-v3.0` | 100+ languages |
|
|
|
|
### LiteLLM Supported Providers
|
|
|
|
LiteLLM supports multiple reranking providers via the `/rerank` endpoint:
|
|
|
|
| Provider | Model Example |
|
|
|----------|---------------|
|
|
| Cohere | `cohere/rerank-english-v3.0` |
|
|
| Together AI | `together_ai/...` |
|
|
| Voyage AI | `voyage/rerank-2` |
|
|
| Jina AI | `jina_ai/...` |
|
|
| AWS Bedrock | `bedrock/...` |
|
|
|
|
**Configuration Examples:**
|
|
|
|
```bash
|
|
# Local provider (default)
|
|
export HINDSIGHT_API_RERANKER_PROVIDER=local
|
|
export HINDSIGHT_API_RERANKER_LOCAL_MODEL=cross-encoder/ms-marco-MiniLM-L-6-v2
|
|
|
|
# Cohere
|
|
export HINDSIGHT_API_RERANKER_PROVIDER=cohere
|
|
export HINDSIGHT_API_COHERE_API_KEY=your-api-key
|
|
export HINDSIGHT_API_RERANKER_COHERE_MODEL=rerank-english-v3.0
|
|
|
|
# TEI (self-hosted)
|
|
export HINDSIGHT_API_RERANKER_PROVIDER=tei
|
|
export HINDSIGHT_API_RERANKER_TEI_URL=http://localhost:8081
|
|
|
|
# FlashRank (lightweight)
|
|
export HINDSIGHT_API_RERANKER_PROVIDER=flashrank
|
|
|
|
# LiteLLM proxy
|
|
export HINDSIGHT_API_RERANKER_PROVIDER=litellm
|
|
export HINDSIGHT_API_LITELLM_API_BASE=http://localhost:4000
|
|
export HINDSIGHT_API_RERANKER_LITELLM_MODEL=cohere/rerank-english-v3.0
|
|
|
|
# RRF-only (no neural reranking)
|
|
export HINDSIGHT_API_RERANKER_PROVIDER=rrf
|
|
```
|
|
|
|
See [Configuration](./configuration#reranker) for all options including Azure-hosted endpoints and batch settings.
|