holetron-lab/fleet-memory
1
0
Fork 0
Code Releases Packages Activity Actions

feat(litellm): async retain, reflect support, and API cleanup (#167)

Browse source 
* feat(litellm): async retain with sync option, fix client session cleanup

- Add sync parameter to retain() for blocking vs background operation
- Default to async retain (sync=False) for better performance
- Add get_pending_retain_errors() to check async failures
- Fix "Unclosed client session" warnings by properly closing clients
- Fix "Timeout context manager" asyncio errors by creating fresh clients
- Each API call now creates and closes its own client (aiohttp limitation)
- Add _get_client() and _close_client() helpers for consistent handling
- Update recall(), reflect(), _retain_sync() and _inject_memories()

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(litellm): add reflect support and require explicit hindsight_query

- Make hindsight_query required when inject_memories=True to enforce
  intentional memory queries (no automatic last-user-message fallback)
- Add reflect_context parameter for shaping LLM reasoning in reflect
- Add reflect_response_schema for structured JSON output from reflect
- Add _reflect_sync() and _reflect_async() methods in callbacks
- Update wrappers.py to support response_schema in reflect/areflect

This improves the developer experience by making memory injection
explicit and adds full reflect API support through the integration.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat(litellm): rename recall_budget to budget, add per-call reflect context

- Rename `recall_budget` parameter to `budget` for consistency with API
- Add `hindsight_reflect_context` kwarg for per-call reflect context override
- Fix reflect() to not pass None values for optional parameters

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* docs(litellm): update README for new API structure and features

- Document configure() vs set_defaults() separation
- Add hindsight_query requirement when inject_memories=True
- Document async retain (sync=False default) and get_pending_retain_errors()
- Add hindsight_reflect_context per-call override documentation
- Document budget parameter (renamed from recall_budget)
- Add reflect_context and reflect_response_schema options
- Update all code examples to use new API structure
- Add new functions to API Reference table

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* test(litellm): update tests for new configure/set_defaults API

- Update tests to use separate configure() and set_defaults() calls
- Fix test assertions to check config vs defaults appropriately
- Add tests for legacy parameter backwards compatibility
- Add new TestSetDefaults test class
- Fix _format_memories test call signature (settings, config order)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add set_bank_mission(), deprecate set_bank_background()

- Add mission parameter to hindsight_client.create_bank()
- Add set_bank_mission() function to hindsight_litellm
- Deprecate set_bank_background() with DeprecationWarning
- Update _create_or_update_bank() to support mission parameter
- Update README and docstrings to document the new API

The 'background' field has been deprecated in the Hindsight API in favor
of 'mission' which is used for mental model generation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Remove deprecated background parameter and legacy configure() parameters

- Remove set_bank_background() in favor of set_bank_mission()
- Remove background parameter from _create_or_update_bank()
- Remove background parameter from hindsight_client.create_bank()
- Remove legacy parameters from configure() (bank_id, document_id, budget, etc.)
- These have been replaced by the set_defaults() API
- Remove legacy test cases for deprecated parameters

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: update tests and docs to use mission instead of background

The create_bank() parameter was renamed from background to mission.
Update all tests and doc examples to use the new parameter name.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
This commit is contained in:
DK09876 2026-01-26 02:07:32 -07:00 committed by GitHub
parent 8e39cb7bc8
commit 1d4879a206
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
10 changed files with 1928 additions and 694 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

11
hindsight-clients/python/hindsight_client/hindsight_client.py
View file

@ -327,7 +327,14 @@ class Hindsight:
background: str | None = None,
disposition: dict[str, float] | None = None,
) -> BankProfileResponse:
"""Create or update a memory bank."""
"""Create or update a memory bank.
Args:
bank_id: Unique identifier for the bank
name: Human-readable display name
mission: Instructions guiding what Hindsight should learn and remember (for mental models)
disposition: Optional disposition traits (skepticism, literalism, empathy)
"""
from hindsight_client_api.models import create_bank_request, disposition_traits
disposition_obj = None
@ -336,7 +343,7 @@ class Hindsight:
request_obj = create_bank_request.CreateBankRequest(
name=name,
background=background,
mission=mission,
disposition=disposition_obj,
)

6
hindsight-clients/python/tests/test_main_operations.py
View file

@ -139,7 +139,7 @@ class TestReflect:
"""Setup: Store some test memories and bank background."""
client.create_bank(
bank_id=bank_id,
background="I am a helpful AI assistant interested in technology and science.",
mission="I am a helpful AI assistant interested in technology and science.",
)
client.retain_batch(
@ -261,7 +261,7 @@ class TestEndToEndWorkflow:
# 1. Create bank
client.create_bank(
bank_id=workflow_bank_id,
background="I am a software engineer who loves Python programming.",
mission="I am a software engineer who loves Python programming.",
)
# 2. Store memories
@ -666,7 +666,7 @@ class TestDeleteBank:
# Create bank with some data
client.create_bank(
bank_id=bank_id,
background="This bank will be deleted",
mission="This bank will be deleted",
)
client.retain(
bank_id=bank_id,

8
hindsight-docs/examples/api/memory-banks.py
View file

@ -23,7 +23,7 @@ client = Hindsight(base_url=HINDSIGHT_URL)
client.create_bank(
bank_id="my-bank",
name="Research Assistant",
background="I am a research assistant specializing in machine learning",
mission="I am a research assistant specializing in machine learning",
disposition={
"skepticism": 4,
"literalism": 3,
@ -33,15 +33,15 @@ client.create_bank(
# [/docs:create-bank]
# [docs:bank-background]
# [docs:bank-mission]
client.create_bank(
bank_id="financial-advisor",
name="Financial Advisor",
background="""I am a conservative financial advisor with 20 years of experience.
mission="""I am a conservative financial advisor with 20 years of experience.
I prioritize capital preservation over aggressive growth.
I have seen multiple market crashes and believe in diversification."""
)
# [/docs:bank-background]
# [/docs:bank-mission]
# =============================================================================

2
hindsight-docs/examples/api/reflect.py
View file

@ -54,7 +54,7 @@ response = client.reflect(
client.create_bank(
bank_id="cautious-advisor",
name="Cautious Advisor",
background="I am a risk-aware financial advisor",
mission="I am a risk-aware financial advisor",
disposition={
"skepticism": 5, # Very skeptical of claims
"literalism": 4, # Focuses on exact requirements

237
hindsight-integrations/litellm/README.md
View file

@ -5,13 +5,14 @@ Universal LLM memory integration via LiteLLM. Add persistent memory to any LLM a
## Features
- **Universal LLM Support** - Works with 100+ LLM providers via LiteLLM (OpenAI, Anthropic, Groq, Azure, AWS Bedrock, Google Vertex AI, and more)
- **Simple Integration** - Just configure, enable, and use `hindsight_litellm.completion()`
- **Simple Integration** - Just configure, set defaults, enable, and use `hindsight_litellm.completion()`
- **Automatic Memory Injection** - Relevant memories are injected into prompts before LLM calls
- **Automatic Conversation Storage** - Conversations are stored to Hindsight for future recall
- **Automatic Conversation Storage** - Conversations are stored to Hindsight for future recall (async by default for performance)
- **Two Memory Modes** - Choose between `reflect` (synthesized context) or `recall` (raw memory retrieval)
- **Direct Memory APIs** - Query, synthesize, and store memories manually
- **Native Client Wrappers** - Alternative wrappers for OpenAI and Anthropic SDKs
- **Debug Mode** - Inspect exactly what memories are being injected
- **Async Error Tracking** - Check for background operation failures with `get_pending_retain_errors()`
## Installation
@ -24,20 +25,31 @@ pip install hindsight-litellm
```python
import hindsight_litellm
# Configure and enable memory integration
# Step 1: Configure static settings
hindsight_litellm.configure(
hindsight_api_url="http://localhost:8888",
bank_id="my-agent",
verbose=True,
)
# Step 2: Set defaults (bank_id is required)
hindsight_litellm.set_defaults(
bank_id="my-agent",
use_reflect=True, # Use reflect for synthesized context
)
# Step 3: Enable memory integration
hindsight_litellm.enable()
# Use the convenience wrapper - memory is automatically injected and stored
# Step 4: Use with explicit hindsight_query (required when inject_memories=True)
response = hindsight_litellm.completion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What did we discuss about AI?"}]
messages=[{"role": "user", "content": "What did we discuss about AI?"}],
hindsight_query="What do I know about AI discussions?", # Required!
)
```
**Important:** When `inject_memories=True` (default), you must provide `hindsight_query` to specify what to search for in memory. This ensures intentional, focused memory queries.
## How It Works
Here's what happens under the hood when you call `completion()`:
@ -121,56 +133,89 @@ The memory injection and storage happen automatically - you just use `completion
## Configuration Options
The API is split into two functions for clarity:
### 1. `configure()` - Static Settings
Settings that typically don't change during a session:
```python
hindsight_litellm.configure(
# Required
hindsight_api_url="http://localhost:8888", # Hindsight API server URL
bank_id="my-agent", # Memory bank ID
api_key="your-api-key", # Optional API key for authentication
# Optional - Authentication
api_key="your-api-key", # API key for Hindsight authentication
# Optional - Memory behavior
store_conversations=True, # Store conversations after LLM calls
inject_memories=True, # Inject relevant memories into prompts
use_reflect=False, # Use reflect API (synthesized) vs recall (raw memories)
reflect_include_facts=False, # Include source facts with reflect responses
max_memories=None, # Maximum memories to inject (None = unlimited)
max_memory_tokens=4096, # Maximum tokens for memory context
recall_budget="mid", # Recall budget: "low", "mid", "high"
fact_types=["world", "agent"], # Filter fact types to inject
# Optional - Bank Configuration
bank_name="My Agent", # Human-readable display name for the memory bank
background="This agent...", # Instructions guiding what Hindsight should remember (see below)
sync_storage=False, # False = async storage (default, better performance)
# True = sync storage (blocks, raises errors immediately)
# Optional - Advanced
injection_mode="system_message", # or "prepend_user"
excluded_models=["gpt-3.5*"], # Exclude certain models
injection_mode="system_message", # How to inject: "system_message" or "prepend_user"
excluded_models=["gpt-3.5*"], # Exclude certain models from interception
verbose=True, # Enable verbose logging and debug info
)
```
### Bank Configuration: background and bank_name
### 2. `set_defaults()` - Per-Call Defaults
The `background` and `bank_name` parameters configure the memory bank itself. When provided, `configure()` will automatically create or update the bank with these settings.
- **bank_name**: A human-readable display name for the memory bank. Useful for identifying banks in the Hindsight UI or when managing multiple banks.
- **background**: Instructions that guide Hindsight on what information is important to extract and remember from conversations. This influences memory extraction during the `retain` operation and can affect how the bank's "disposition" (skepticism, literalism, empathy) is calibrated.
Default values for per-call settings. These can be overridden on individual calls using `hindsight_*` kwargs:
```python
# Example: Customer support routing agent
hindsight_litellm.configure(
hindsight_api_url="http://localhost:8888",
bank_id="support-router",
bank_name="Customer Support Router",
background="""This agent routes customer support requests to the appropriate team.
Remember which types of issues should go to which teams (billing, technical, sales).
Track customer preferences for communication channels and past issue resolutions.
Note any escalation patterns or VIP customers who need special handling.""",
hindsight_litellm.set_defaults(
# Required
bank_id="my-agent", # Memory bank ID
# Optional - Memory retrieval
budget="mid", # Budget level: "low", "mid", "high"
fact_types=["world", "opinion"], # Filter fact types to retrieve
max_memories=10, # Maximum memories to inject (None = unlimited)
max_memory_tokens=4096, # Maximum tokens for memory context
include_entities=True, # Include entity observations in recall
# Optional - Reflect mode
use_reflect=True, # Use reflect API (synthesized) vs recall (raw memories)
reflect_include_facts=False, # Include source facts in debug info
reflect_context="I am a delivery agent finding recipients.", # Context for reflect reasoning
reflect_response_schema={...}, # JSON Schema for structured reflect output
# Optional - Debugging
trace=False, # Enable trace info for debugging
document_id="conversation-1", # Document ID for grouping conversations
)
```
### 3. Per-Call Overrides
Override any default on individual calls using `hindsight_*` kwargs:
```python
response = hindsight_litellm.completion(
model="gpt-4o-mini",
messages=[...],
hindsight_query="Where is Alice located?", # REQUIRED when inject_memories=True
hindsight_reflect_context="Currently on floor 3", # Per-call reflect context override
# hindsight_bank_id="other-bank", # Override bank_id for this call
)
```
### Bank Configuration: mission
Use `set_bank_mission()` to configure what the memory bank should learn and remember (used for mental models):
```python
hindsight_litellm.set_bank_mission(
mission="""This agent routes customer support requests to the appropriate team.
Remember which types of issues should go to which teams (billing, technical, sales).
Track customer preferences for communication channels and past issue resolutions.""",
name="Customer Support Router", # Optional display name
)
```
### Memory Modes: Reflect vs Recall
- **Recall mode** (`use_reflect=False`, default): Retrieves raw memory facts and injects them as a numbered list. Best when you need precise, individual memories.
@ -178,18 +223,19 @@ hindsight_litellm.configure(
```python
# Recall mode - raw memories
hindsight_litellm.configure(
bank_id="my-agent",
use_reflect=False, # Default
)
hindsight_litellm.set_defaults(bank_id="my-agent", use_reflect=False)
# Injects: "1. [WORLD] User prefers Python\n2. [OPINION] User dislikes Java..."
# Reflect mode - synthesized context
hindsight_litellm.configure(
hindsight_litellm.set_defaults(bank_id="my-agent", use_reflect=True)
# Injects: "Based on previous conversations, the user is a Python developer who..."
# Reflect with context - shapes LLM reasoning (not retrieval)
hindsight_litellm.set_defaults(
bank_id="my-agent",
use_reflect=True,
reflect_context="I am a delivery agent looking for package recipients.",
)
# Injects: "Based on previous conversations, the user is a Python developer who..."
```
## Multi-Provider Support
@ -199,29 +245,29 @@ Works with any LiteLLM-supported provider:
```python
import hindsight_litellm
hindsight_litellm.configure(
hindsight_api_url="http://localhost:8888",
bank_id="my-agent",
)
hindsight_litellm.configure(hindsight_api_url="http://localhost:8888")
hindsight_litellm.set_defaults(bank_id="my-agent")
hindsight_litellm.enable()
messages = [{"role": "user", "content": "Hello!"}]
# OpenAI
hindsight_litellm.completion(model="gpt-4o", messages=[...])
hindsight_litellm.completion(model="gpt-4o", messages=messages, hindsight_query="greeting")
# Anthropic
hindsight_litellm.completion(model="claude-3-5-sonnet-20241022", messages=[...])
hindsight_litellm.completion(model="claude-3-5-sonnet-20241022", messages=messages, hindsight_query="greeting")
# Groq
hindsight_litellm.completion(model="groq/llama-3.1-70b-versatile", messages=[...])
hindsight_litellm.completion(model="groq/llama-3.1-70b-versatile", messages=messages, hindsight_query="greeting")
# Azure OpenAI
hindsight_litellm.completion(model="azure/gpt-4", messages=[...])
hindsight_litellm.completion(model="azure/gpt-4", messages=messages, hindsight_query="greeting")
# AWS Bedrock
hindsight_litellm.completion(model="bedrock/anthropic.claude-3", messages=[...])
hindsight_litellm.completion(model="bedrock/anthropic.claude-3", messages=messages, hindsight_query="greeting")
# Google Vertex AI
hindsight_litellm.completion(model="vertex_ai/gemini-pro", messages=[...])
hindsight_litellm.completion(model="vertex_ai/gemini-pro", messages=messages, hindsight_query="greeting")
```
## Direct Memory APIs
@ -229,9 +275,10 @@ hindsight_litellm.completion(model="vertex_ai/gemini-pro", messages=[...])
### Recall - Query raw memories
```python
from hindsight_litellm import configure, recall
from hindsight_litellm import configure, set_defaults, recall
configure(bank_id="my-agent", hindsight_api_url="http://localhost:8888")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="my-agent")
# Query memories
memories = recall("what projects am I working on?", budget="mid")
@ -246,9 +293,10 @@ for m in memories:
### Reflect - Get synthesized context
```python
from hindsight_litellm import configure, reflect
from hindsight_litellm import configure, set_defaults, reflect
configure(bank_id="my-agent", hindsight_api_url="http://localhost:8888")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="my-agent")
# Get synthesized memory context
result = reflect("what do you know about the user's preferences?")
@ -256,21 +304,42 @@ print(result.text)
# Output:
# "Based on our conversations, the user prefers Python for backend development..."
# With context to shape the response (doesn't affect retrieval)
result = reflect(
query="what do I know about Alice?",
context="I am a delivery agent looking for package recipients.",
)
```
### Retain - Store memories
```python
from hindsight_litellm import configure, retain
from hindsight_litellm import configure, set_defaults, retain, get_pending_retain_errors
configure(bank_id="my-agent", hindsight_api_url="http://localhost:8888")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="my-agent")
# Store a memory
# Async retain (default) - fast, non-blocking
# Returns immediately; actual storage happens in background
result = retain(
content="User mentioned they're working on a machine learning project",
context="Discussion about current projects",
)
print(f"Retained successfully: {result.success}, items: {result.items_count}")
# result.success is True immediately (actual errors collected separately)
# Sync retain - blocks until complete, raises errors immediately
result = retain(
content="Critical information that must be stored",
context="Important data",
sync=True, # Block until storage completes
)
# Check for async retain errors (call periodically)
errors = get_pending_retain_errors()
if errors:
for e in errors:
print(f"Background retain failed: {e}")
```
### Async APIs
@ -332,19 +401,16 @@ response = wrapped.messages.create(
When `verbose=True`, you can inspect exactly what memories are being injected:
```python
from hindsight_litellm import configure, enable, completion, get_last_injection_debug
from hindsight_litellm import configure, set_defaults, enable, completion, get_last_injection_debug
configure(
bank_id="my-agent",
hindsight_api_url="http://localhost:8888",
verbose=True,
use_reflect=True,
)
configure(hindsight_api_url="http://localhost:8888", verbose=True)
set_defaults(bank_id="my-agent", use_reflect=True)
enable()
response = completion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What's my favorite color?"}]
messages=[{"role": "user", "content": "What's my favorite color?"}],
hindsight_query="What is the user's favorite color?",
)
# Inspect what was injected
@ -365,7 +431,11 @@ from hindsight_litellm import hindsight_memory
import litellm
with hindsight_memory(bank_id="user-123"):
response = litellm.completion(model="gpt-4", messages=[...])
response = litellm.completion(
model="gpt-4",
messages=[{"role": "user", "content": "Hello!"}],
hindsight_query="greeting context",
)
# Memory integration automatically disabled after context
```
@ -387,7 +457,8 @@ cleanup()
| Function | Description |
|----------|-------------|
| `configure(...)` | Configure global Hindsight settings |
| `configure(...)` | Configure static Hindsight settings (API URL, auth, storage options) |
| `set_defaults(...)` | Set defaults for per-call settings (bank_id, budget, reflect options) |
| `enable()` | Enable memory integration with LiteLLM |
| `disable()` | Disable memory integration |
| `is_enabled()` | Check if memory integration is enabled |
@ -397,20 +468,30 @@ cleanup()
| Function | Description |
|----------|-------------|
| `get_config()` | Get current configuration |
| `is_configured()` | Check if Hindsight is configured |
| `reset_config()` | Reset configuration to defaults |
| `get_config()` | Get current static configuration |
| `get_defaults()` | Get current per-call defaults |
| `is_configured()` | Check if Hindsight is configured with a bank_id |
| `reset_config()` | Reset all configuration to defaults |
| `set_document_id(id)` | Convenience function to update document_id |
| `set_bank_mission(...)` | Set mission/instructions for a memory bank (for mental models) |
### Memory Functions
| Function | Description |
|----------|-------------|
| `recall(query, ...)` | Synchronously query raw memories |
| `arecall(query, ...)` | Asynchronously query raw memories |
| `reflect(query, ...)` | Synchronously get synthesized memory context |
| `areflect(query, ...)` | Asynchronously get synthesized memory context |
| `retain(content, ...)` | Synchronously store a memory |
| `aretain(content, ...)` | Asynchronously store a memory |
| `recall(query, ...)` | Query raw memories (sync) |
| `arecall(query, ...)` | Query raw memories (async) |
| `reflect(query, ...)` | Get synthesized memory context (sync) |
| `areflect(query, ...)` | Get synthesized memory context (async) |
| `retain(content, sync=False, ...)` | Store a memory (async by default, use `sync=True` to block) |
| `aretain(content, ...)` | Store a memory (async) |
### Error Tracking Functions
| Function | Description |
|----------|-------------|
| `get_pending_retain_errors()` | Get and clear errors from background retain operations |
| `get_pending_storage_errors()` | Get and clear errors from background conversation storage |
### Debug Functions

777
hindsight-integrations/litellm/hindsight_litellm/__init__.py
View file

File diff suppressed because it is too large Load diff

616
hindsight-integrations/litellm/hindsight_litellm/callbacks.py
View file

@ -19,7 +19,14 @@ import concurrent.futures
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.utils import ModelResponse
from .config import get_config, is_configured, HindsightConfig, MemoryInjectionMode
from .config import (
get_config,
get_defaults,
is_configured,
HindsightConfig,
HindsightDefaults,
MemoryInjectionMode,
)
# Use requests for sync HTTP calls to avoid async event loop issues
try:
@ -37,6 +44,16 @@ except ImportError:
logger = logging.getLogger(__name__)
class HindsightError(Exception):
"""Exception raised when a Hindsight operation fails.
This is raised when inject_memories=True and recall fails,
or when store_conversations=True and store fails.
"""
pass
# Thread pool for running async operations in background
_executor = concurrent.futures.ThreadPoolExecutor(max_workers=4, thread_name_prefix="hindsight-")
@ -76,6 +93,43 @@ class HindsightCallback(CustomLogger):
self._recent_hashes: Set[str] = set()
self._max_hash_cache = 1000
def _get_effective_settings(self, kwargs: Dict[str, Any]) -> HindsightDefaults:
"""Get effective per-call settings from kwargs with fallback to defaults.
Per-call kwargs (hindsight_*) override defaults. Supported kwargs:
- hindsight_bank_id: Override bank_id
- hindsight_document_id: Override document_id
- hindsight_budget: Override budget
- hindsight_fact_types: Override fact_types
- hindsight_max_memories: Override max_memories
- hindsight_max_memory_tokens: Override max_memory_tokens
- hindsight_use_reflect: Override use_reflect
- hindsight_reflect_include_facts: Override reflect_include_facts
- hindsight_context: Override reflect_context
- hindsight_response_schema: Override reflect_response_schema
- hindsight_include_entities: Override include_entities
- hindsight_trace: Override trace
Note: hindsight_query is handled separately in log_pre_api_call since it's
always per-call (no sensible default for dynamic queries).
"""
defaults = get_defaults() or HindsightDefaults()
return HindsightDefaults(
bank_id=kwargs.get("hindsight_bank_id", defaults.bank_id),
document_id=kwargs.get("hindsight_document_id", defaults.document_id),
budget=kwargs.get("hindsight_budget", defaults.budget),
fact_types=kwargs.get("hindsight_fact_types", defaults.fact_types),
max_memories=kwargs.get("hindsight_max_memories", defaults.max_memories),
max_memory_tokens=kwargs.get("hindsight_max_memory_tokens", defaults.max_memory_tokens),
use_reflect=kwargs.get("hindsight_use_reflect", defaults.use_reflect),
reflect_include_facts=kwargs.get("hindsight_reflect_include_facts", defaults.reflect_include_facts),
reflect_context=kwargs.get("hindsight_context", defaults.reflect_context),
reflect_response_schema=kwargs.get("hindsight_response_schema", defaults.reflect_response_schema),
include_entities=kwargs.get("hindsight_include_entities", defaults.include_entities),
trace=kwargs.get("hindsight_trace", defaults.trace),
)
def _get_http_session(self):
"""Get or create a requests Session (thread-safe)."""
if self._http_session is None:
@ -92,14 +146,18 @@ class HindsightCallback(CustomLogger):
)
return self._http_session
def _http_post(self, url: str, json_data: dict, config: HindsightConfig) -> Optional[dict]:
"""Make a synchronous HTTP POST request."""
try:
session = self._get_http_session()
headers = {"Content-Type": "application/json"}
if config.api_key:
headers["Authorization"] = f"Bearer {config.api_key}"
def _http_post(self, url: str, json_data: dict, config: HindsightConfig) -> dict:
"""Make a synchronous HTTP POST request.
Raises:
HindsightError: If the request fails for any reason.
"""
session = self._get_http_session()
headers = {"Content-Type": "application/json"}
if config.api_key:
headers["Authorization"] = f"Bearer {config.api_key}"
try:
if HAS_REQUESTS:
response = session.post(url, json=json_data, headers=headers, timeout=30)
response.raise_for_status()
@ -108,10 +166,14 @@ class HindsightCallback(CustomLogger):
response = session.post(url, json=json_data, headers=headers)
response.raise_for_status()
return response.json()
else:
raise HindsightError("No HTTP client available (install requests or httpx)")
except HindsightError:
raise
except Exception as e:
if config.verbose:
logger.warning(f"HTTP POST failed: {e}")
return None
logger.error(f"HTTP POST failed: {e}")
raise HindsightError(f"Hindsight API request failed: {e}") from e
def _should_skip_model(self, model: str, config: HindsightConfig) -> bool:
"""Check if this model should be excluded from interception."""
@ -138,6 +200,20 @@ class HindsightCallback(CustomLogger):
return " ".join(text_parts)
return None
def _messages_to_query(self, messages: List[Dict[str, Any]]) -> str:
"""Concatenate all message contents into a single query string."""
message_parts = []
for msg in messages:
content = msg.get("content", "")
if isinstance(content, str) and content:
message_parts.append(content)
elif isinstance(content, list):
# Handle structured content (e.g., vision messages)
for item in content:
if isinstance(item, dict) and item.get("type") == "text":
message_parts.append(item.get("text", ""))
return "\n".join(message_parts)
def _compute_conversation_hash(
self,
user_input: str,
@ -163,6 +239,7 @@ class HindsightCallback(CustomLogger):
def _format_memories(
self,
results: List[Any],
settings: HindsightDefaults,
config: HindsightConfig
) -> str:
"""Format memory recall results into a context string.
@ -174,7 +251,7 @@ class HindsightCallback(CustomLogger):
return ""
# Apply limit if set, otherwise use all results
results_to_use = results[:config.max_memories] if config.max_memories else results
results_to_use = results[:settings.max_memories] if settings.max_memories else results
memory_lines = []
for i, result in enumerate(results_to_use, 1):
# Handle both RecallResult objects and dicts
@ -248,194 +325,343 @@ class HindsightCallback(CustomLogger):
return updated_messages
def _get_bank_id(self, config: HindsightConfig) -> str:
"""Get the bank_id for API calls."""
return config.bank_id
def _recall_memories_sync(
self,
query: str,
settings: HindsightDefaults,
config: HindsightConfig
) -> List[Dict[str, Any]]:
"""Recall relevant memories from Hindsight (sync) using direct HTTP."""
"""Recall relevant memories from Hindsight (sync) using direct HTTP.
Raises:
HindsightError: If inject_memories=True and recall fails.
"""
bank_id = settings.bank_id
if not bank_id:
raise HindsightError(
"No bank_id configured. Call set_defaults(bank_id=...) "
"or pass hindsight_bank_id=... to the completion call."
)
url = f"{config.hindsight_api_url}/v1/default/banks/{bank_id}/memories/recall"
request_data = {
"query": query,
"budget": settings.budget or "mid",
"max_tokens": settings.max_memory_tokens or 4096,
}
if settings.fact_types:
request_data["types"] = settings.fact_types
# Add trace parameter for debugging
if settings.trace:
request_data["trace"] = True
# Add include options for entity observations
# include_entities=True -> include: {entities: {}}
# include_entities=False -> include: {entities: null}
if settings.include_entities:
request_data["include"] = {"entities": {}}
else:
request_data["include"] = {"entities": None}
try:
bank_id = self._get_bank_id(config)
url = f"{config.hindsight_api_url}/v1/default/banks/{bank_id}/memories/recall"
request_data = {
"query": query,
"budget": config.recall_budget or "mid",
"max_tokens": config.max_memory_tokens or 4096,
}
if config.fact_types:
request_data["types"] = config.fact_types
response = self._http_post(url, request_data, config)
if response and "results" in response:
return response["results"]
return []
except Exception as e:
except HindsightError as e:
if config.verbose:
logger.warning(f"Failed to recall memories: {e}")
return []
logger.error(f"Failed to recall memories: {e}")
raise HindsightError(f"Memory recall failed: {e}") from e
async def _recall_memories_async(
self,
query: str,
settings: HindsightDefaults,
config: HindsightConfig
) -> List[Any]:
"""Recall relevant memories from Hindsight (async).
Uses thread pool executor with sync HTTP to avoid event loop conflicts.
Raises:
HindsightError: If inject_memories=True and recall fails.
"""
try:
loop = asyncio.get_running_loop()
results = await loop.run_in_executor(
_executor,
self._recall_memories_sync,
query,
config
loop = asyncio.get_running_loop()
results = await loop.run_in_executor(
_executor,
lambda: self._recall_memories_sync(query, settings, config)
)
return results if isinstance(results, list) else []
def _reflect_sync(
self,
query: str,
settings: HindsightDefaults,
config: HindsightConfig
) -> Optional[str]:
"""Generate a reflection response from Hindsight (sync) using direct HTTP.
Returns:
The reflect response text, or None if no response.
Raises:
HindsightError: If inject_memories=True and reflect fails.
"""
bank_id = settings.bank_id
if not bank_id:
raise HindsightError(
"No bank_id configured. Call set_defaults(bank_id=...) "
"or pass hindsight_bank_id=... to the completion call."
)
return results if isinstance(results, list) else []
url = f"{config.hindsight_api_url}/v1/default/banks/{bank_id}/reflect"
except Exception as e:
request_data: Dict[str, Any] = {
"query": query,
"budget": settings.budget or "mid",
"max_tokens": settings.max_memory_tokens or 4096,
}
# Add context if provided (shapes reasoning but not retrieval)
if settings.reflect_context:
request_data["context"] = settings.reflect_context
# Add response_schema for structured output
if settings.reflect_response_schema:
request_data["response_schema"] = settings.reflect_response_schema
# Add include options for facts if requested
if settings.reflect_include_facts:
request_data["include"] = {"facts": {}}
try:
response = self._http_post(url, request_data, config)
if response:
# Handle structured output if schema was provided
if settings.reflect_response_schema and "structured_output" in response:
# Return structured output as JSON string for injection
import json
return json.dumps(response["structured_output"], indent=2)
# Otherwise return text response
return response.get("text", "")
return None
except HindsightError as e:
if config.verbose:
logger.warning(f"Failed to recall memories: {e}")
return []
logger.error(f"Failed to reflect: {e}")
raise HindsightError(f"Reflect failed: {e}") from e
async def _reflect_async(
self,
query: str,
settings: HindsightDefaults,
config: HindsightConfig
) -> Optional[str]:
"""Generate a reflection response from Hindsight (async).
Uses thread pool executor with sync HTTP to avoid event loop conflicts.
Returns:
The reflect response text, or None if no response.
Raises:
HindsightError: If inject_memories=True and reflect fails.
"""
loop = asyncio.get_running_loop()
result = await loop.run_in_executor(
_executor,
lambda: self._reflect_sync(query, settings, config)
)
return result
def _store_conversation_sync(
self,
messages: List[Dict[str, Any]],
response: ModelResponse,
model: str,
settings: HindsightDefaults,
config: HindsightConfig,
) -> None:
"""Store the conversation to Hindsight (sync) using direct HTTP.
By default, stores the full conversation history passed to the LLM.
Each message is stored as a separate item, all linked by document_id.
IMPORTANT: This intentionally sends the FULL conversation history each call,
not just the new messages. This is required because Hindsight's retain API
with document_id performs an UPSERT (replace), not an append.
Hindsight will process the document as a whole for memory extraction.
If we only sent deltas (new messages), Hindsight would only have the latest
fragment and lose all prior context. By sending the full conversation each
time, Hindsight always has the complete context to extract meaningful facts.
Example with delta-only (WRONG):
Call 1: "USER: deliver to Alex\\nASSISTANT_TOOL_CALLS: look_at_business"
Call 2: "TOOL_RESULT: TechStart Labs\\nASSISTANT_TOOL_CALLS: go_up" # Lost context!
Example with full conversation (CORRECT):
Call 1: "USER: deliver to Alex\\nASSISTANT_TOOL_CALLS: look_at_business"
Call 2: "USER: deliver to Alex\\nASSISTANT_TOOL_CALLS: look_at_business\\n
TOOL_RESULT: TechStart Labs\\nASSISTANT_TOOL_CALLS: go_up" # Full context!
Each upsert replaces the previous, so the final stored document contains
the complete conversation for Hindsight to process.
Raises:
HindsightError: If store_conversations=True and store fails.
"""
try:
# Extract assistant response from the LLM response
assistant_output = ""
if response.choices and len(response.choices) > 0:
choice = response.choices[0]
if hasattr(choice, "message") and choice.message:
assistant_output = choice.message.content or ""
bank_id = settings.bank_id
if not bank_id:
raise HindsightError(
"No bank_id configured. Call set_defaults(bank_id=...) "
"or pass hindsight_bank_id=... to the completion call."
)
if not assistant_output:
return
# Extract assistant response from the LLM response
assistant_output = ""
assistant_tool_calls = []
if response.choices and len(response.choices) > 0:
choice = response.choices[0]
if hasattr(choice, "message") and choice.message:
assistant_output = choice.message.content or ""
# Also capture tool calls
if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
for tc in choice.message.tool_calls:
if hasattr(tc, "function"):
assistant_tool_calls.append(f"{tc.function.name}({tc.function.arguments})")
# Build conversation items - each message becomes a separate item
# All linked by document_id for Hindsight to process together
items = []
for msg in messages:
role = msg.get("role", "").upper()
content = msg.get("content", "")
# Skip if no content AND no tool calls - nothing to store
if not assistant_output and not assistant_tool_calls:
return
# Skip system messages - they're instructions, not conversation
if role == "SYSTEM":
continue
# Build conversation items - each message becomes a separate item
# All linked by document_id for Hindsight to process together
items = []
for msg in messages:
role = msg.get("role", "").upper()
content = msg.get("content", "")
# Skip if this looks like our injected memory context
if isinstance(content, str) and content.startswith("# Relevant Memories"):
continue
# Skip system messages - they're instructions, not conversation
if role == "SYSTEM":
continue
# Handle structured content (e.g., vision messages)
if isinstance(content, list):
text_parts = []
for item in content:
if isinstance(item, dict) and item.get("type") == "text":
text_parts.append(item.get("text", ""))
content = " ".join(text_parts)
# Skip if this looks like our injected memory context
if isinstance(content, str) and content.startswith("# Relevant Memories"):
continue
# Handle tool messages (results from tool calls)
if role == "TOOL":
tool_call_id = msg.get("tool_call_id", "")
items.append(f"TOOL_RESULT: {content}")
continue
# Handle assistant messages with tool calls
tool_calls = msg.get("tool_calls", [])
if tool_calls:
tc_strs = []
for tc in tool_calls:
if hasattr(tc, "function"):
tc_strs.append(f"{tc.function.name}({tc.function.arguments})")
elif isinstance(tc, dict) and "function" in tc:
func = tc["function"]
tc_strs.append(f"{func.get('name', '')}({func.get('arguments', '')})")
if tc_strs:
items.append(f"ASSISTANT_TOOL_CALLS: {'; '.join(tc_strs)}")
if content:
# Map roles to clearer labels
label = "USER" if role == "USER" else "ASSISTANT"
items.append(f"{label}: {content}")
items.append(f"ASSISTANT: {content}")
continue
# Add the new assistant response
# Handle structured content (e.g., vision messages)
if isinstance(content, list):
text_parts = []
for item in content:
if isinstance(item, dict) and item.get("type") == "text":
text_parts.append(item.get("text", ""))
content = " ".join(text_parts)
if content:
# Map roles to clearer labels
label = "USER" if role == "USER" else "ASSISTANT"
items.append(f"{label}: {content}")
# Add the new assistant response (text or tool calls)
if assistant_output:
items.append(f"ASSISTANT: {assistant_output}")
if assistant_tool_calls:
items.append(f"ASSISTANT_TOOL_CALLS: {'; '.join(assistant_tool_calls)}")
if not items:
return
if not items:
return
# Use last user message for deduplication hash
user_input = self._extract_user_query(messages) or ""
# Use last user message for deduplication hash
user_input = self._extract_user_query(messages) or ""
# Deduplication check
conv_hash = self._compute_conversation_hash(user_input, assistant_output)
if self._is_duplicate(conv_hash):
if config.verbose:
logger.debug(f"Skipping duplicate conversation: {conv_hash}")
return
# Deduplication check - include tool calls if no text content
dedup_output = assistant_output or ";".join(assistant_tool_calls)
conv_hash = self._compute_conversation_hash(user_input, dedup_output)
if self._is_duplicate(conv_hash):
if config.verbose:
logger.debug(f"Skipping duplicate conversation: {conv_hash}")
return
# Build the full conversation as a single item for now
# (Future: could store each message as separate item in same document)
conversation_text = "\n\n".join(items)
# Build the full conversation as a single item for now
# (Future: could store each message as separate item in same document)
conversation_text = "\n\n".join(items)
# Build metadata
metadata = {
"source": "litellm",
"model": model,
}
# Build metadata
metadata = {
"source": "litellm",
"model": model,
}
# Add token usage if available
if hasattr(response, "usage") and response.usage:
if hasattr(response.usage, "total_tokens"):
metadata["tokens"] = str(response.usage.total_tokens)
# Add token usage if available
if hasattr(response, "usage") and response.usage:
if hasattr(response.usage, "total_tokens"):
metadata["tokens"] = str(response.usage.total_tokens)
bank_id = self._get_bank_id(config)
url = f"{config.hindsight_api_url}/v1/default/banks/{bank_id}/memories"
url = f"{config.hindsight_api_url}/v1/default/banks/{bank_id}/memories"
request_data = {
"items": [
{
"content": conversation_text,
"context": f"conversation:litellm:{model}",
"metadata": metadata,
"document_id": config.document_id, # Group by document
}
],
}
request_data = {
"items": [
{
"content": conversation_text,
"context": f"conversation:litellm:{model}",
"metadata": metadata,
"document_id": settings.document_id, # Group by document
}
],
}
try:
self._http_post(url, request_data, config)
if config.verbose:
logger.info(f"Stored conversation to Hindsight bank: {config.bank_id}")
except Exception as e:
logger.info(f"Stored conversation to Hindsight bank: {bank_id}")
except HindsightError as e:
if config.verbose:
logger.warning(f"Failed to store conversation: {e}")
logger.error(f"Failed to store conversation: {e}")
raise HindsightError(f"Memory storage failed: {e}") from e
async def _store_conversation_async(
self,
messages: List[Dict[str, Any]],
response: ModelResponse,
model: str,
settings: HindsightDefaults,
config: HindsightConfig,
) -> None:
"""Store the conversation to Hindsight (async).
Uses thread pool executor with sync HTTP to avoid event loop conflicts.
Raises:
HindsightError: If store_conversations=True and store fails.
"""
try:
loop = asyncio.get_running_loop()
await loop.run_in_executor(
_executor,
self._store_conversation_sync,
messages,
response,
model,
config
loop = asyncio.get_running_loop()
await loop.run_in_executor(
_executor,
lambda: self._store_conversation_sync(
messages, response, model, settings, config
)
except Exception as e:
if config.verbose:
logger.warning(f"Failed to store conversation: {e}")
)
# ========== LiteLLM CustomLogger Interface ==========
@ -449,28 +675,53 @@ class HindsightCallback(CustomLogger):
This is where we inject memories into the messages.
"""
if not is_configured():
config = get_config()
if not config or not config.inject_memories:
return
config = get_config()
if not config or not config.enabled or not config.inject_memories:
return
# Get effective settings (kwargs override defaults)
settings = self._get_effective_settings(kwargs)
if not settings.bank_id:
raise ValueError(
"No bank_id configured. Either call set_defaults(bank_id=...) "
"or pass hindsight_bank_id=... to the completion call."
)
if self._should_skip_model(model, config):
return
# Extract user query
user_query = self._extract_user_query(messages)
if not user_query:
return
# hindsight_query is required when inject_memories=True
custom_query = kwargs.get("hindsight_query")
if not custom_query:
raise ValueError(
"hindsight_query is required when inject_memories=True. "
"Pass hindsight_query='your query' to specify what to search for in memory. "
"Example: hindsight_query=recipient_name or hindsight_query='What do I know about Alice?'"
)
# Recall relevant memories
memories = self._recall_memories_sync(user_query, config)
if not memories:
return
user_query = custom_query
# Use reflect or recall based on settings
if settings.use_reflect:
# Use reflect API for disposition-aware reasoning
reflect_response = self._reflect_sync(user_query, settings, config)
if not reflect_response:
return
# Format reflect response as context
memory_context = (
"# Relevant Context from Memory\n"
f"{reflect_response}"
)
else:
# Use recall API for raw fact retrieval
memories = self._recall_memories_sync(user_query, settings, config)
if not memories:
return
# Format and inject memories
memory_context = self._format_memories(memories, settings, config)
# Format and inject memories
memory_context = self._format_memories(memories, config)
updated_messages = self._inject_memories_into_messages(
messages, memory_context, config
)
@ -480,7 +731,8 @@ class HindsightCallback(CustomLogger):
messages.extend(updated_messages)
if config.verbose:
logger.info(f"Injected {len(memories)} memories into prompt")
mode = "reflect" if settings.use_reflect else "recall"
logger.info(f"Injected memory context via {mode}")
async def async_log_pre_api_call(
self,
@ -492,28 +744,53 @@ class HindsightCallback(CustomLogger):
This is where we inject memories into the messages.
"""
if not is_configured():
config = get_config()
if not config or not config.inject_memories:
return
config = get_config()
if not config or not config.enabled or not config.inject_memories:
return
# Get effective settings (kwargs override defaults)
settings = self._get_effective_settings(kwargs)
if not settings.bank_id:
raise ValueError(
"No bank_id configured. Either call set_defaults(bank_id=...) "
"or pass hindsight_bank_id=... to the completion call."
)
if self._should_skip_model(model, config):
return
# Extract user query
user_query = self._extract_user_query(messages)
if not user_query:
return
# hindsight_query is required when inject_memories=True
custom_query = kwargs.get("hindsight_query")
if not custom_query:
raise ValueError(
"hindsight_query is required when inject_memories=True. "
"Pass hindsight_query='your query' to specify what to search for in memory. "
"Example: hindsight_query=recipient_name or hindsight_query='What do I know about Alice?'"
)
# Recall relevant memories
memories = await self._recall_memories_async(user_query, config)
if not memories:
return
user_query = custom_query
# Use reflect or recall based on settings
if settings.use_reflect:
# Use reflect API for disposition-aware reasoning
reflect_response = await self._reflect_async(user_query, settings, config)
if not reflect_response:
return
# Format reflect response as context
memory_context = (
"# Relevant Context from Memory\n"
f"{reflect_response}"
)
else:
# Use recall API for raw fact retrieval
memories = await self._recall_memories_async(user_query, settings, config)
if not memories:
return
# Format and inject memories
memory_context = self._format_memories(memories, settings, config)
# Format and inject memories
memory_context = self._format_memories(memories, config)
updated_messages = self._inject_memories_into_messages(
messages, memory_context, config
)
@ -523,7 +800,8 @@ class HindsightCallback(CustomLogger):
messages.extend(updated_messages)
if config.verbose:
logger.info(f"Injected {len(memories)} memories into prompt")
mode = "reflect" if settings.use_reflect else "recall"
logger.info(f"Injected memory context via {mode}")
def log_success_event(
self,
@ -536,11 +814,14 @@ class HindsightCallback(CustomLogger):
This is where we store the conversation.
"""
if not is_configured():
config = get_config()
if not config or not config.store_conversations:
return
config = get_config()
if not config or not config.enabled or not config.store_conversations:
# Get effective settings (kwargs override defaults)
settings = self._get_effective_settings(kwargs)
if not settings.bank_id:
# bank_id validation already done in log_pre_api_call
return
model = kwargs.get("model", "unknown")
@ -552,7 +833,7 @@ class HindsightCallback(CustomLogger):
return
# Store the conversation
self._store_conversation_sync(messages, response_obj, model, config)
self._store_conversation_sync(messages, response_obj, model, settings, config)
async def async_log_success_event(
self,
@ -565,11 +846,14 @@ class HindsightCallback(CustomLogger):
This is where we store the conversation.
"""
if not is_configured():
config = get_config()
if not config or not config.store_conversations:
return
config = get_config()
if not config or not config.enabled or not config.store_conversations:
# Get effective settings (kwargs override defaults)
settings = self._get_effective_settings(kwargs)
if not settings.bank_id:
# bank_id validation already done in async_log_pre_api_call
return
model = kwargs.get("model", "unknown")
@ -581,7 +865,7 @@ class HindsightCallback(CustomLogger):
return
# Store the conversation
await self._store_conversation_async(messages, response_obj, model, config)
await self._store_conversation_async(messages, response_obj, model, settings, config)
def log_failure_event(
self,

401
hindsight-integrations/litellm/hindsight_litellm/config.py
View file

@ -1,179 +1,258 @@
"""Global configuration for Hindsight-LiteLLM integration."""
"""Global configuration for Hindsight-LiteLLM integration.
from typing import Optional, List
This module provides a clean API for configuring Hindsight integration:
1. configure() - Static settings that rarely change during a session
- API URL, authentication, logging, injection mode, etc.
2. set_defaults() - Default values for per-call settings
- bank_id, document_id, budget, fact_types, etc.
- These are used when per-call kwargs are not provided
3. Per-call kwargs (hindsight_* prefix) - Override any default per-call
- hindsight_bank_id, hindsight_document_id, etc.
4. set_bank_mission() - Set the mission for a memory bank (for mental models)
"""
from typing import Optional, List, Any, Dict
from dataclasses import dataclass, field
from enum import Enum
class MemoryInjectionMode(str, Enum):
"""How memories should be injected into the prompt."""
SYSTEM_MESSAGE = "system_message" # Add as system message
PREPEND_USER = "prepend_user" # Prepend to user message
DISABLED = "disabled" # Don't inject memories
"""How memories should be injected into the prompt.
Use inject_memories=False if you don't want memory injection.
"""
SYSTEM_MESSAGE = "system_message" # Add to/create system message
PREPEND_USER = "prepend_user" # Prepend to last user message
@dataclass
class HindsightConfig:
"""Configuration for Hindsight integration with LiteLLM.
"""Static configuration for Hindsight integration with LiteLLM.
These settings typically don't change during a session.
Attributes:
hindsight_api_url: URL of the Hindsight API server
bank_id: Memory bank ID for memory operations (required). For multi-user
support, use different bank_ids per user (e.g., f"user-{user_id}")
api_key: Optional API key for Hindsight authentication
store_conversations: Whether to store conversations to Hindsight
inject_memories: Whether to inject relevant memories into prompts
injection_mode: How to inject memories (system_message or prepend_user)
max_memories: Maximum number of memories to inject
max_memory_tokens: Maximum tokens for injected memory context
recall_budget: Budget level for memory recall (low, mid, high)
fact_types: List of fact types to filter recall (world, agent, opinion, observation)
document_id: Optional document ID for grouping stored conversations
enabled: Master switch to enable/disable Hindsight integration
excluded_models: List of model patterns to exclude from interception
verbose: Enable verbose logging
bank_name: Optional display name for the memory bank
background: Optional background/instructions for memory extraction
use_reflect: Use reflect API instead of recall for memory injection (synthesizes answer)
sync_storage: If True, storage runs synchronously and raises errors immediately.
If False (default), storage runs in background thread for better performance.
"""
hindsight_api_url: str = "http://localhost:8888"
bank_id: Optional[str] = None
api_key: Optional[str] = None
store_conversations: bool = True
inject_memories: bool = True
injection_mode: MemoryInjectionMode = MemoryInjectionMode.SYSTEM_MESSAGE
max_memories: Optional[int] = None # None = no limit (use all results from API)
max_memory_tokens: int = 4096
recall_budget: str = "mid" # low, mid, high
fact_types: Optional[List[str]] = None # world, agent, opinion, observation
document_id: Optional[str] = None
enabled: bool = True
excluded_models: List[str] = field(default_factory=list)
verbose: bool = False
bank_name: Optional[str] = None # Display name for the memory bank
background: Optional[str] = None # Background/instructions for memory extraction
use_reflect: bool = False # Use reflect instead of recall for memory injection
reflect_include_facts: bool = False # Include facts used by reflect in debug info
sync_storage: bool = False
# Global configuration instance
@dataclass
class HindsightDefaults:
"""Default values for per-call settings.
These can be overridden on a per-call basis using hindsight_* kwargs.
Attributes:
bank_id: Memory bank ID for memory operations
document_id: Optional document ID for grouping stored conversations
budget: Budget level for memory recall (low, mid, high)
fact_types: List of fact types to filter recall (world, experience, opinion, observation)
max_memories: Maximum number of memories to inject (None = no limit)
max_memory_tokens: Maximum tokens for injected memory context
use_reflect: Use reflect API instead of recall for memory injection
reflect_include_facts: Include facts used by reflect in debug info
reflect_context: Additional context for reflect reasoning (does not affect retrieval)
reflect_response_schema: JSON Schema for structured reflect output
include_entities: Include entity observations in recall results
trace: Enable trace info for recall debugging
Note:
For custom queries, use the hindsight_query kwarg per-call instead of a default,
since queries typically need to be dynamic (e.g., include recipient name).
"""
bank_id: Optional[str] = None
document_id: Optional[str] = None
budget: str = "mid" # low, mid, high
fact_types: Optional[List[str]] = None # world, experience, opinion, observation
max_memories: Optional[int] = None # None = no limit
max_memory_tokens: int = 4096
use_reflect: bool = False
reflect_include_facts: bool = False
reflect_context: Optional[str] = None # Context for reflect reasoning
reflect_response_schema: Optional[Dict[str, Any]] = None # JSON Schema for structured output
include_entities: bool = True # Include entity observations by default
trace: bool = False # Enable trace info for debugging
# Global instances
_global_config: Optional[HindsightConfig] = None
_global_defaults: Optional[HindsightDefaults] = None
def configure(
hindsight_api_url: str = "http://localhost:8888",
bank_id: Optional[str] = None,
api_key: Optional[str] = None,
store_conversations: bool = True,
inject_memories: bool = True,
injection_mode: MemoryInjectionMode = MemoryInjectionMode.SYSTEM_MESSAGE,
max_memories: Optional[int] = None,
max_memory_tokens: int = 4096,
recall_budget: str = "mid",
fact_types: Optional[List[str]] = None,
document_id: Optional[str] = None,
enabled: bool = True,
excluded_models: Optional[List[str]] = None,
verbose: bool = False,
bank_name: Optional[str] = None,
background: Optional[str] = None,
use_reflect: bool = False,
reflect_include_facts: bool = False,
sync_storage: bool = False,
) -> HindsightConfig:
"""Configure global Hindsight integration settings for LiteLLM.
"""Configure static Hindsight integration settings for LiteLLM.
This function sets up the global configuration that will be used by the
LiteLLM callbacks to inject memories and store conversations.
This sets up settings that typically don't change during a session.
For per-call settings like bank_id, use set_defaults() or per-call kwargs.
Args:
hindsight_api_url: URL of the Hindsight API server
bank_id: Memory bank ID for memory operations (required). For multi-user
support, use different bank_ids per user (e.g., f"user-{user_id}")
api_key: Optional API key for Hindsight authentication
store_conversations: Whether to store conversations to Hindsight
inject_memories: Whether to inject relevant memories into prompts
injection_mode: How to inject memories into the prompt
max_memories: Maximum number of memories to inject
max_memory_tokens: Maximum tokens for injected memory context
recall_budget: Budget level for memory recall (low, mid, high)
fact_types: List of fact types to filter (world, agent, opinion, observation)
document_id: Optional document ID for grouping stored conversations
enabled: Master switch to enable/disable Hindsight integration
excluded_models: List of model patterns to exclude from interception
verbose: Enable verbose logging
bank_name: Optional display name for the memory bank
background: Optional background/instructions that help Hindsight understand
what information is important to extract and remember from conversations.
This is passed to create_bank() to configure the memory bank.
use_reflect: Use reflect API instead of recall for memory injection.
When True, Hindsight will synthesize a contextual answer based on
memories rather than returning raw memory facts.
reflect_include_facts: When use_reflect=True, include the facts that
were used to generate the reflect response in the debug info.
This is useful for debugging what memories the reflect API used.
sync_storage: If True, storage runs synchronously and raises errors immediately.
If False (default), storage runs in background for better performance.
Use get_pending_storage_errors() to check for async storage failures.
Returns:
The configured HindsightConfig instance
Example:
>>> from hindsight_litellm import configure, enable
>>> from hindsight_litellm import configure, set_defaults, enable
>>> configure(
... hindsight_api_url="http://localhost:8888",
... bank_id="user-123", # Per-user bank for multi-user support
... store_conversations=True,
... inject_memories=True,
... background="This agent routes customer requests to support channels. "
... "Remember which types of issues should go to which channels.",
... api_key="your-api-key",
... verbose=True,
... )
>>> enable() # Register callbacks with LiteLLM
>>> set_defaults(bank_id="user-123")
>>> enable() # Start memory integration
"""
global _global_config
_global_config = HindsightConfig(
hindsight_api_url=hindsight_api_url,
bank_id=bank_id,
api_key=api_key,
store_conversations=store_conversations,
inject_memories=inject_memories,
injection_mode=injection_mode,
max_memories=max_memories,
max_memory_tokens=max_memory_tokens,
recall_budget=recall_budget,
fact_types=fact_types,
document_id=document_id,
enabled=enabled,
excluded_models=excluded_models or [],
verbose=verbose,
bank_name=bank_name,
background=background,
use_reflect=use_reflect,
reflect_include_facts=reflect_include_facts,
sync_storage=sync_storage,
)
# If background or bank_name is provided, create/update the bank
if bank_id and (background or bank_name):
_create_or_update_bank(
hindsight_api_url=hindsight_api_url,
bank_id=bank_id,
name=bank_name,
background=background,
verbose=verbose,
)
return _global_config
def set_defaults(
bank_id: Optional[str] = None,
document_id: Optional[str] = None,
budget: Optional[str] = None,
fact_types: Optional[List[str]] = None,
max_memories: Optional[int] = None,
max_memory_tokens: Optional[int] = None,
use_reflect: Optional[bool] = None,
reflect_include_facts: Optional[bool] = None,
reflect_context: Optional[str] = None,
reflect_response_schema: Optional[Dict[str, Any]] = None,
include_entities: Optional[bool] = None,
trace: Optional[bool] = None,
) -> HindsightDefaults:
"""Set default values for per-call settings.
These defaults are used when per-call kwargs are not provided.
Any of these can be overridden on individual LLM calls using
hindsight_* kwargs (e.g., hindsight_bank_id="other-bank").
Args:
bank_id: Default memory bank ID for memory operations
document_id: Default document ID for grouping stored conversations
budget: Default budget level for memory recall (low, mid, high)
fact_types: Default fact types to filter (world, experience, opinion, observation)
max_memories: Default max number of memories to inject
max_memory_tokens: Default max tokens for memory context
use_reflect: Default whether to use reflect API instead of recall
reflect_include_facts: Default whether to include facts in reflect debug info
reflect_context: Default context for reflect reasoning (shapes LLM response, not retrieval)
reflect_response_schema: Default JSON Schema for structured reflect output
include_entities: Default whether to include entity observations in recall (default True)
trace: Default whether to enable trace info for debugging (default False)
Returns:
The configured HindsightDefaults instance
Note:
For custom memory queries, use hindsight_query per-call instead of a default,
since queries typically need to be dynamic (e.g., include recipient name).
Example:
>>> from hindsight_litellm import set_defaults
>>> set_defaults(
... bank_id="my-agent",
... budget="high",
... fact_types=["world", "opinion"],
... reflect_context="I am a delivery agent finding package recipients.",
... )
>>>
>>> # Override per-call with dynamic query:
>>> response = litellm.completion(
... model="gpt-4",
... messages=[...],
... hindsight_query=f"Where is {recipient_name} located?", # Dynamic query
... )
"""
global _global_defaults
# Get current defaults or create new
current = _global_defaults or HindsightDefaults()
# Update only provided values
_global_defaults = HindsightDefaults(
bank_id=bank_id if bank_id is not None else current.bank_id,
document_id=document_id if document_id is not None else current.document_id,
budget=budget if budget is not None else current.budget,
fact_types=fact_types if fact_types is not None else current.fact_types,
max_memories=max_memories if max_memories is not None else current.max_memories,
max_memory_tokens=max_memory_tokens if max_memory_tokens is not None else current.max_memory_tokens,
use_reflect=use_reflect if use_reflect is not None else current.use_reflect,
reflect_include_facts=reflect_include_facts if reflect_include_facts is not None else current.reflect_include_facts,
reflect_context=reflect_context if reflect_context is not None else current.reflect_context,
reflect_response_schema=reflect_response_schema if reflect_response_schema is not None else current.reflect_response_schema,
include_entities=include_entities if include_entities is not None else current.include_entities,
trace=trace if trace is not None else current.trace,
)
return _global_defaults
def _create_or_update_bank(
hindsight_api_url: str,
bank_id: str,
name: Optional[str] = None,
background: Optional[str] = None,
mission: Optional[str] = None,
verbose: bool = False,
) -> None:
"""Create or update a memory bank with the given configuration.
This is called automatically by configure() when background or bank_name is provided.
Args:
hindsight_api_url: URL of the Hindsight API server
bank_id: The bank ID to create/update
name: Optional display name for the bank
mission: Instructions guiding what Hindsight should learn and remember
verbose: Enable verbose logging
"""
try:
from hindsight_client import Hindsight
@ -182,18 +261,18 @@ def _create_or_update_bank(
client.create_bank(
bank_id=bank_id,
name=name,
background=background,
mission=mission,
)
if verbose:
import logging
logging.getLogger("hindsight_litellm").info(
f"Created/updated bank '{bank_id}' with background"
f"Created/updated bank '{bank_id}' with mission"
)
except ImportError:
if verbose:
import logging
logging.getLogger("hindsight_litellm").warning(
"hindsight_client not installed. Cannot create bank with background. "
"hindsight_client not installed. Cannot create bank. "
"Install with: pip install hindsight-client"
)
except Exception as e:
@ -205,7 +284,7 @@ def _create_or_update_bank(
def get_config() -> Optional[HindsightConfig]:
"""Get the current global configuration.
"""Get the current global static configuration.
Returns:
The current HindsightConfig instance, or None if not configured
@ -213,20 +292,136 @@ def get_config() -> Optional[HindsightConfig]:
return _global_config
def is_configured() -> bool:
"""Check if Hindsight has been configured.
def get_defaults() -> Optional[HindsightDefaults]:
"""Get the current global defaults for per-call settings.
Returns:
True if configure() has been called with a valid bank_id
The current HindsightDefaults instance, or None if not set
"""
return _global_defaults
def is_configured() -> bool:
"""Check if Hindsight has been configured with a valid bank_id.
Returns:
True if configure() has been called and a bank_id is set in defaults
"""
return (
_global_config is not None
and _global_config.enabled
and _global_config.bank_id is not None
and _global_defaults is not None
and _global_defaults.bank_id is not None
)
def reset_config() -> None:
"""Reset the global configuration to None."""
global _global_config
"""Reset all global configuration to None."""
global _global_config, _global_defaults
_global_config = None
_global_defaults = None
def set_document_id(document_id: str | None) -> None:
"""Set the document_id for grouping stored conversations.
This is a convenience function that updates just the document_id
in the defaults without requiring a full set_defaults() call.
When document_id is set, Hindsight uses upsert behavior:
- Same document_id = replace previous version
- Hindsight deduplicates facts automatically
Args:
document_id: Document ID for grouping conversations, or None to clear
Example:
>>> from hindsight_litellm import configure, set_defaults, enable, set_document_id
>>> configure(hindsight_api_url="http://localhost:8888")
>>> set_defaults(bank_id="my-agent")
>>> enable()
>>>
>>> # Start a new conversation
>>> set_document_id("conversation-123")
>>> response = litellm.completion(model="gpt-4", messages=[...])
>>>
>>> # Switch to another conversation
>>> set_document_id("conversation-456")
>>> response = litellm.completion(model="gpt-4", messages=[...])
"""
global _global_defaults
if _global_defaults is not None:
_global_defaults = HindsightDefaults(
bank_id=_global_defaults.bank_id,
document_id=document_id,
budget=_global_defaults.budget,
fact_types=_global_defaults.fact_types,
max_memories=_global_defaults.max_memories,
max_memory_tokens=_global_defaults.max_memory_tokens,
use_reflect=_global_defaults.use_reflect,
reflect_include_facts=_global_defaults.reflect_include_facts,
reflect_context=_global_defaults.reflect_context,
reflect_response_schema=_global_defaults.reflect_response_schema,
include_entities=_global_defaults.include_entities,
trace=_global_defaults.trace,
)
else:
# Create defaults with just document_id if none exist
_global_defaults = HindsightDefaults(document_id=document_id)
def set_bank_mission(
bank_id: Optional[str] = None,
mission: Optional[str] = None,
name: Optional[str] = None,
) -> None:
"""Set or update the mission for a memory bank.
The mission guides Hindsight on what information to learn and remember,
and is used for mental model generation. If the bank doesn't exist,
it will be auto-created.
Args:
bank_id: The bank ID to update. If not provided, uses the default bank_id.
mission: Instructions guiding what Hindsight should learn and remember.
name: Optional display name for the bank.
Raises:
ValueError: If no bank_id is provided and no default is set.
RuntimeError: If configure() hasn't been called.
Example:
>>> from hindsight_litellm import configure, set_defaults, set_bank_mission
>>> configure(hindsight_api_url="http://localhost:8888")
>>> set_defaults(bank_id="delivery-agent")
>>> set_bank_mission(
... mission="You are a delivery agent navigating a building. "
... "Remember employee locations, building layout, and optimal paths."
... )
"""
config = get_config()
if not config:
raise RuntimeError("Hindsight not configured. Call configure() first.")
# Determine which bank_id to use
effective_bank_id = bank_id
if effective_bank_id is None:
defaults = get_defaults()
if defaults:
effective_bank_id = defaults.bank_id
if not effective_bank_id:
raise ValueError(
"No bank_id provided and no default bank_id set. "
"Either pass bank_id or call set_defaults(bank_id=...) first."
)
# Use the Hindsight API to create/update the bank
_create_or_update_bank(
hindsight_api_url=config.hindsight_api_url,
bank_id=effective_bank_id,
name=name,
mission=mission,
verbose=config.verbose,
)

332
hindsight-integrations/litellm/hindsight_litellm/wrappers.py
View file

@ -8,15 +8,36 @@ integration with native client libraries.
"""
import logging
import threading
from typing import Any, Dict, List, Optional, Union
from dataclasses import dataclass
from .config import get_config, is_configured, HindsightConfig
from .config import get_config, get_defaults, is_configured, HindsightConfig
# Background thread support for async retain
_retain_errors: List[Exception] = []
_retain_errors_lock = threading.Lock()
logger = logging.getLogger(__name__)
def _get_client(api_url: str):
"""Create a fresh Hindsight client for the given URL.
Note: We create a fresh client each time because the hindsight_client
uses aiohttp internally, and reusing clients across different sync
calls causes asyncio context issues.
"""
from hindsight_client import Hindsight
return Hindsight(base_url=api_url, timeout=30.0)
def _close_client():
"""No-op for compatibility. Clients are now closed after each use."""
pass
@dataclass
class RecallResult:
"""A single memory recall result."""
@ -106,24 +127,25 @@ def recall(
>>> if memories.debug:
... print(f"Queried bank: {memories.debug.bank_id}")
"""
# Get config or use overrides
# Get config and defaults, or use overrides
config = get_config()
defaults = get_defaults()
api_url = hindsight_api_url or (config.hindsight_api_url if config else None)
target_bank_id = bank_id or (config.bank_id if config else None)
target_fact_types = fact_types or (config.fact_types if config else None)
target_budget = budget or (config.recall_budget if config else "mid")
target_max_tokens = max_tokens or (config.max_memory_tokens if config else 4096)
target_bank_id = bank_id or (defaults.bank_id if defaults else None)
target_fact_types = fact_types or (defaults.fact_types if defaults else None)
target_budget = budget or (defaults.budget if defaults else "mid")
target_max_tokens = max_tokens or (defaults.max_memory_tokens if defaults else 4096)
if not api_url or not target_bank_id:
raise RuntimeError(
"Hindsight not configured. Call configure() or provide bank_id and hindsight_api_url."
)
client = None
try:
from hindsight_client import Hindsight
client = Hindsight(base_url=api_url, timeout=30.0)
# Create fresh client for this operation
client = _get_client(api_url)
# Call recall API
results = client.recall(
@ -178,6 +200,12 @@ def recall(
if config and config.verbose:
logger.warning(f"Failed to recall memories: {e}")
raise
finally:
if client is not None:
try:
client.close()
except Exception:
pass
async def arecall(
@ -233,6 +261,7 @@ def reflect(
bank_id: Optional[str] = None,
budget: Optional[str] = None,
context: Optional[str] = None,
response_schema: Optional[dict] = None,
hindsight_api_url: Optional[str] = None,
) -> ReflectResult:
"""Generate a contextual answer based on memories.
@ -246,10 +275,11 @@ def reflect(
use different bank_ids per user (e.g., f"user-{user_id}")
budget: Budget level for reflection (low, mid, high)
context: Additional context to include in the reflection
response_schema: JSON Schema for structured output
hindsight_api_url: Override the configured API URL
Returns:
ReflectResult with synthesized answer text
ReflectResult with synthesized answer text (or structured_output if schema provided)
Raises:
RuntimeError: If Hindsight is not configured and no overrides provided
@ -264,28 +294,33 @@ def reflect(
Based on our conversations, you're working on a FastAPI project...
"""
config = get_config()
defaults = get_defaults()
api_url = hindsight_api_url or (config.hindsight_api_url if config else None)
target_bank_id = bank_id or (config.bank_id if config else None)
target_budget = budget or (config.recall_budget if config else "mid")
target_bank_id = bank_id or (defaults.bank_id if defaults else None)
target_budget = budget or (defaults.budget if defaults else "mid")
if not api_url or not target_bank_id:
raise RuntimeError(
"Hindsight not configured. Call configure() or provide bank_id and hindsight_api_url."
)
client = None
try:
from hindsight_client import Hindsight
client = Hindsight(base_url=api_url, timeout=30.0)
# Create fresh client for this operation
client = _get_client(api_url)
# Call reflect API
result = client.reflect(
bank_id=target_bank_id,
query=query,
budget=target_budget,
context=context,
)
reflect_kwargs = {
"bank_id": target_bank_id,
"query": query,
"budget": target_budget,
}
if context is not None:
reflect_kwargs["context"] = context
if response_schema is not None:
reflect_kwargs["response_schema"] = response_schema
result = client.reflect(**reflect_kwargs)
# Convert to ReflectResult
text = result.text if hasattr(result, 'text') else str(result)
@ -310,6 +345,12 @@ def reflect(
if config and config.verbose:
logger.warning(f"Failed to reflect: {e}")
raise
finally:
if client is not None:
try:
client.close()
except Exception:
pass
async def areflect(
@ -317,6 +358,7 @@ async def areflect(
bank_id: Optional[str] = None,
budget: Optional[str] = None,
context: Optional[str] = None,
response_schema: Optional[dict] = None,
hindsight_api_url: Optional[str] = None,
) -> ReflectResult:
"""Async version of reflect().
@ -332,6 +374,7 @@ async def areflect(
bank_id=bank_id,
budget=budget,
context=context,
response_schema=response_schema,
hindsight_api_url=hindsight_api_url,
)
)
@ -359,64 +402,20 @@ class RetainResult:
return self.success
def retain(
def _retain_sync(
content: str,
bank_id: Optional[str] = None,
context: Optional[str] = None,
document_id: Optional[str] = None,
metadata: Optional[Dict[str, str]] = None,
hindsight_api_url: Optional[str] = None,
api_url: str,
target_bank_id: str,
context: Optional[str],
target_document_id: Optional[str],
metadata: Optional[Dict[str, str]],
verbose: bool,
) -> RetainResult:
"""Store content to Hindsight memory.
This function allows you to manually store content to memory without
making an LLM call. Useful for storing feedback, user preferences,
or any other information you want the system to remember.
Args:
content: The text content to store
bank_id: Override the configured bank_id. For multi-user support,
use different bank_ids per user (e.g., f"user-{user_id}")
context: Context description for the memory (e.g., "customer_feedback")
document_id: Optional document ID for grouping related memories
metadata: Optional key-value metadata to attach to the memory
hindsight_api_url: Override the configured API URL
Returns:
RetainResult indicating success
Raises:
RuntimeError: If Hindsight is not configured and no overrides provided
Example:
>>> from hindsight_litellm import configure, retain
>>> configure(bank_id="my-agent", hindsight_api_url="http://localhost:8888")
>>>
>>> # Store feedback
>>> retain("User prefers dark mode", context="user_preference")
>>>
>>> # Store with metadata
>>> retain(
... "Customer reported billing issue resolved",
... context="support_ticket",
... metadata={"ticket_id": "12345", "status": "resolved"}
... )
"""
config = get_config()
api_url = hindsight_api_url or (config.hindsight_api_url if config else None)
target_bank_id = bank_id or (config.bank_id if config else None)
target_document_id = document_id or (config.document_id if config else None)
if not api_url or not target_bank_id:
raise RuntimeError(
"Hindsight not configured. Call configure() or provide bank_id and hindsight_api_url."
)
"""Internal synchronous retain implementation."""
client = None
try:
from hindsight_client import Hindsight
client = Hindsight(base_url=api_url, timeout=30.0)
# Create fresh client for this operation
client = _get_client(api_url)
# Call retain API
result = client.retain(
@ -433,7 +432,7 @@ def retain(
# Include debug info if verbose
debug_info = None
if config and config.verbose:
if verbose:
logger.info(f"Stored content to Hindsight bank: {target_bank_id}")
debug_info = RetainDebugInfo(
content=content,
@ -449,9 +448,158 @@ def retain(
except ImportError as e:
raise RuntimeError(f"hindsight-client not installed: {e}")
except Exception as e:
if config and config.verbose:
if verbose:
logger.warning(f"Failed to retain: {e}")
raise
finally:
if client is not None:
try:
client.close()
except Exception:
pass
def _retain_background(
content: str,
api_url: str,
target_bank_id: str,
context: Optional[str],
target_document_id: Optional[str],
metadata: Optional[Dict[str, str]],
verbose: bool,
) -> None:
"""Background thread worker for async retain."""
global _retain_errors
try:
_retain_sync(
content=content,
api_url=api_url,
target_bank_id=target_bank_id,
context=context,
target_document_id=target_document_id,
metadata=metadata,
verbose=verbose,
)
except Exception as e:
with _retain_errors_lock:
_retain_errors.append(e)
logger.warning(f"Background retain failed: {e}")
def get_pending_retain_errors() -> List[Exception]:
"""Get and clear any pending errors from background retain operations.
When using async retain (sync=False), errors are collected in the background.
Call this periodically to check for and handle any failures.
Returns:
List of exceptions from failed background retain operations.
The list is cleared after calling this function.
Example:
>>> errors = get_pending_retain_errors()
>>> if errors:
... for e in errors:
... print(f"Retain failed: {e}")
"""
global _retain_errors
with _retain_errors_lock:
errors = _retain_errors.copy()
_retain_errors.clear()
return errors
def retain(
content: str,
bank_id: Optional[str] = None,
context: Optional[str] = None,
document_id: Optional[str] = None,
metadata: Optional[Dict[str, str]] = None,
hindsight_api_url: Optional[str] = None,
sync: bool = False,
) -> RetainResult:
"""Store content to Hindsight memory.
This function allows you to manually store content to memory without
making an LLM call. Useful for storing feedback, user preferences,
or any other information you want the system to remember.
Args:
content: The text content to store
bank_id: Override the configured bank_id. For multi-user support,
use different bank_ids per user (e.g., f"user-{user_id}")
context: Context description for the memory (e.g., "customer_feedback")
document_id: Optional document ID for grouping related memories
metadata: Optional key-value metadata to attach to the memory
hindsight_api_url: Override the configured API URL
sync: If True, block until storage completes. If False (default),
run in background thread for better performance. Use
get_pending_retain_errors() to check for async failures.
Returns:
RetainResult indicating success. For async mode (sync=False),
always returns success=True immediately; actual errors are
collected via get_pending_retain_errors().
Raises:
RuntimeError: If Hindsight is not configured and no overrides provided
Exception: Only raised in sync mode if storage fails
Example:
>>> from hindsight_litellm import configure, retain
>>> configure(bank_id="my-agent", hindsight_api_url="http://localhost:8888")
>>>
>>> # Async retain (default) - fast, non-blocking
>>> retain("User prefers dark mode", context="user_preference")
>>>
>>> # Sync retain - blocks until complete
>>> retain("Critical data", sync=True)
>>>
>>> # Check for async errors
>>> errors = get_pending_retain_errors()
"""
config = get_config()
defaults = get_defaults()
api_url = hindsight_api_url or (config.hindsight_api_url if config else None)
target_bank_id = bank_id or (defaults.bank_id if defaults else None)
target_document_id = document_id or (defaults.document_id if defaults else None)
verbose = config.verbose if config else False
if not api_url or not target_bank_id:
raise RuntimeError(
"Hindsight not configured. Call configure() or provide bank_id and hindsight_api_url."
)
if sync:
# Synchronous mode - block and return result
return _retain_sync(
content=content,
api_url=api_url,
target_bank_id=target_bank_id,
context=context,
target_document_id=target_document_id,
metadata=metadata,
verbose=verbose,
)
else:
# Async mode - run in background thread
thread = threading.Thread(
target=_retain_background,
args=(
content,
api_url,
target_bank_id,
context,
target_document_id,
metadata,
verbose,
),
daemon=True,
)
thread.start()
# Return immediate success - actual errors collected via get_pending_retain_errors()
return RetainResult(success=True, items_count=0)
async def aretain(
@ -509,7 +657,7 @@ class HindsightOpenAI:
store_conversations: bool = True,
inject_memories: bool = True,
max_memories: Optional[int] = None,
recall_budget: str = "mid",
budget: str = "mid",
verbose: bool = False,
):
"""Initialize the wrapped OpenAI client.
@ -523,7 +671,7 @@ class HindsightOpenAI:
store_conversations: Whether to store conversations
inject_memories: Whether to inject relevant memories
max_memories: Maximum number of memories to inject (None = no limit)
recall_budget: Budget level for memory recall (low, mid, high)
budget: Budget level for memory recall (low, mid, high)
verbose: Enable verbose logging
"""
self._client = client
@ -533,7 +681,7 @@ class HindsightOpenAI:
self._store_conversations = store_conversations
self._inject_memories = inject_memories
self._max_memories = max_memories
self._recall_budget = recall_budget
self._budget = budget
self._verbose = verbose
self._hindsight_client = None
@ -560,7 +708,7 @@ class HindsightOpenAI:
results = client.recall(
bank_id=self._bank_id,
query=query,
budget=self._recall_budget,
budget=self._budget,
max_tokens=self._max_memories * 200 if self._max_memories else 4096,
)
@ -713,7 +861,7 @@ class HindsightAnthropic:
store_conversations: bool = True,
inject_memories: bool = True,
max_memories: Optional[int] = None,
recall_budget: str = "mid",
budget: str = "mid",
verbose: bool = False,
):
"""Initialize the wrapped Anthropic client.
@ -727,7 +875,7 @@ class HindsightAnthropic:
store_conversations: Whether to store conversations
inject_memories: Whether to inject relevant memories
max_memories: Maximum number of memories to inject (None = no limit)
recall_budget: Budget level for memory recall (low, mid, high)
budget: Budget level for memory recall (low, mid, high)
verbose: Enable verbose logging
"""
self._client = client
@ -737,7 +885,7 @@ class HindsightAnthropic:
self._store_conversations = store_conversations
self._inject_memories = inject_memories
self._max_memories = max_memories
self._recall_budget = recall_budget
self._budget = budget
self._verbose = verbose
self._hindsight_client = None
@ -764,7 +912,7 @@ class HindsightAnthropic:
results = client.recall(
bank_id=self._bank_id,
query=query,
budget=self._recall_budget,
budget=self._budget,
max_tokens=self._max_memories * 200 if self._max_memories else 4096,
)
@ -889,7 +1037,7 @@ def wrap_openai(
store_conversations: bool = True,
inject_memories: bool = True,
max_memories: Optional[int] = None,
recall_budget: str = "mid",
budget: str = "mid",
verbose: bool = False,
) -> HindsightOpenAI:
"""Wrap an OpenAI client with Hindsight memory integration.
@ -906,7 +1054,7 @@ def wrap_openai(
store_conversations: Whether to store conversations
inject_memories: Whether to inject relevant memories
max_memories: Maximum number of memories to inject (None = no limit)
recall_budget: Budget level for memory recall (low, mid, high)
budget: Budget level for memory recall (low, mid, high)
verbose: Enable verbose logging
Returns:
@ -935,7 +1083,7 @@ def wrap_openai(
store_conversations=store_conversations,
inject_memories=inject_memories,
max_memories=max_memories,
recall_budget=recall_budget,
budget=budget,
verbose=verbose,
)
@ -948,7 +1096,7 @@ def wrap_anthropic(
store_conversations: bool = True,
inject_memories: bool = True,
max_memories: Optional[int] = None,
recall_budget: str = "mid",
budget: str = "mid",
verbose: bool = False,
) -> HindsightAnthropic:
"""Wrap an Anthropic client with Hindsight memory integration.
@ -965,7 +1113,7 @@ def wrap_anthropic(
store_conversations: Whether to store conversations
inject_memories: Whether to inject relevant memories
max_memories: Maximum number of memories to inject (None = no limit)
recall_budget: Budget level for memory recall (low, mid, high)
budget: Budget level for memory recall (low, mid, high)
verbose: Enable verbose logging
Returns:
@ -995,6 +1143,6 @@ def wrap_anthropic(
store_conversations=store_conversations,
inject_memories=inject_memories,
max_memories=max_memories,
recall_budget=recall_budget,
budget=budget,
verbose=verbose,
)

232
hindsight-integrations/litellm/tests/test_integration.py
View file

@ -6,6 +6,8 @@ from typing import List, Dict, Any
from hindsight_litellm import (
configure,
set_defaults,
get_defaults,
enable,
disable,
is_enabled,
@ -14,9 +16,10 @@ from hindsight_litellm import (
is_configured,
reset_config,
HindsightConfig,
HindsightDefaults,
MemoryInjectionMode,
)
from hindsight_litellm.callbacks import HindsightCallback, get_callback, cleanup_callback
from hindsight_litellm.callbacks import HindsightCallback
class TestConfiguration:
@ -34,47 +37,53 @@ class TestConfiguration:
def test_configure_creates_config(self):
"""Test that configure creates a config object."""
config = configure(
bank_id="test-agent",
hindsight_api_url="http://localhost:8888",
)
# Set defaults separately (new API)
defaults = set_defaults(bank_id="test-agent")
assert config is not None
assert config.bank_id == "test-agent"
assert config.hindsight_api_url == "http://localhost:8888"
assert config.enabled is True
assert defaults.bank_id == "test-agent"
def test_configure_with_all_options(self):
"""Test configure with all options."""
config = configure(
hindsight_api_url="http://custom:9999",
bank_id="custom-agent",
api_key="secret-key",
store_conversations=False,
inject_memories=False,
injection_mode=MemoryInjectionMode.PREPEND_USER,
max_memories=5,
max_memory_tokens=1000,
recall_budget="high",
fact_types=["world", "opinion"],
document_id="doc-123",
enabled=True,
excluded_models=["gpt-3.5*"],
verbose=True,
sync_storage=True,
)
# Set defaults separately (new API)
defaults = set_defaults(
bank_id="custom-agent",
max_memories=5,
max_memory_tokens=1000,
budget="high",
fact_types=["world", "opinion"],
document_id="doc-123",
)
assert config.hindsight_api_url == "http://custom:9999"
assert config.bank_id == "custom-agent"
assert config.api_key == "secret-key"
assert config.store_conversations is False
assert config.inject_memories is False
assert config.injection_mode == MemoryInjectionMode.PREPEND_USER
assert config.max_memories == 5
assert config.max_memory_tokens == 1000
assert config.recall_budget == "high"
assert config.fact_types == ["world", "opinion"]
assert config.document_id == "doc-123"
assert config.excluded_models == ["gpt-3.5*"]
assert config.verbose is True
assert config.sync_storage is True
assert defaults.bank_id == "custom-agent"
assert defaults.max_memories == 5
assert defaults.max_memory_tokens == 1000
assert defaults.budget == "high"
assert defaults.fact_types == ["world", "opinion"]
assert defaults.document_id == "doc-123"
def test_is_configured_without_bank_id(self):
"""Test is_configured returns False without bank_id."""
@ -83,16 +92,19 @@ class TestConfiguration:
def test_is_configured_with_bank_id(self):
"""Test is_configured returns True with bank_id."""
configure(bank_id="test-agent")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="test-agent")
assert is_configured() is True
def test_reset_config(self):
"""Test reset_config clears the configuration."""
configure(bank_id="test-agent")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="test-agent")
assert is_configured() is True
reset_config()
assert get_config() is None
assert get_defaults() is None
assert is_configured() is False
@ -112,44 +124,42 @@ class TestEnableDisable:
with pytest.raises(RuntimeError, match="not configured"):
enable()
def test_enable_registers_callback(self):
"""Test enable registers callback with LiteLLM."""
import litellm
def test_enable_without_bank_id_raises(self):
"""Test enable raises error without bank_id."""
configure(hindsight_api_url="http://localhost:8888")
with pytest.raises(RuntimeError, match="bank_id not set"):
enable()
configure(bank_id="test-agent")
def test_enable_sets_enabled_flag(self):
"""Test enable sets the enabled flag."""
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="test-agent")
enable()
callback = get_callback()
assert callback in litellm.callbacks
assert is_enabled() is True
def test_disable_removes_callback(self):
"""Test disable removes callback from LiteLLM."""
import litellm
configure(bank_id="test-agent")
def test_disable_clears_enabled_flag(self):
"""Test disable clears the enabled flag."""
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="test-agent")
enable()
assert is_enabled() is True
disable()
callback = get_callback()
assert callback not in litellm.callbacks
assert is_enabled() is False
def test_enable_idempotent(self):
"""Test enable is idempotent (can be called multiple times)."""
import litellm
configure(bank_id="test-agent")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="test-agent")
# Enable multiple times
enable()
enable()
enable()
# Should only have one callback
callback = get_callback()
assert litellm.callbacks.count(callback) == 1
# Should still be enabled
assert is_enabled() is True
class TestCallback:
@ -221,14 +231,21 @@ class TestCallback:
def test_format_memories(self):
"""Test formatting memories into context string."""
callback = HindsightCallback()
config = HindsightConfig(bank_id="test", max_memories=10, verbose=False)
# Create config and defaults with new API
configure(hindsight_api_url="http://localhost:8888", verbose=False)
set_defaults(bank_id="test", max_memories=10)
config = get_config()
defaults = get_defaults()
memories = [
{"text": "User likes Python", "fact_type": "world", "weight": 0.95},
{"text": "User works at Google", "fact_type": "world", "weight": 0.8},
]
formatted = callback._format_memories(memories, config)
# Signature is: _format_memories(results, settings, config)
formatted = callback._format_memories(memories, defaults, config)
assert "Relevant Memories" in formatted
assert "User likes Python" in formatted
@ -238,23 +255,34 @@ class TestCallback:
def test_format_memories_with_verbose(self):
"""Test formatting memories with verbose mode shows weights."""
callback = HindsightCallback()
config = HindsightConfig(bank_id="test", max_memories=10, verbose=True)
# Create config and defaults with new API
configure(hindsight_api_url="http://localhost:8888", verbose=True)
set_defaults(bank_id="test", max_memories=10)
config = get_config()
defaults = get_defaults()
memories = [
{"text": "User likes Python", "fact_type": "world", "weight": 0.95},
]
formatted = callback._format_memories(memories, config)
# Signature is: _format_memories(results, settings, config)
formatted = callback._format_memories(memories, defaults, config)
assert "relevance: 0.95" in formatted
def test_inject_memories_as_system_message(self):
"""Test injecting memories as system message."""
callback = HindsightCallback()
config = HindsightConfig(
bank_id="test",
configure(
hindsight_api_url="http://localhost:8888",
injection_mode=MemoryInjectionMode.SYSTEM_MESSAGE,
)
set_defaults(bank_id="test")
config = get_config()
messages = [
{"role": "user", "content": "Hello"},
@ -271,10 +299,14 @@ class TestCallback:
def test_inject_memories_prepend_to_existing_system(self):
"""Test injecting memories appends to existing system message."""
callback = HindsightCallback()
config = HindsightConfig(
bank_id="test",
configure(
hindsight_api_url="http://localhost:8888",
injection_mode=MemoryInjectionMode.SYSTEM_MESSAGE,
)
set_defaults(bank_id="test")
config = get_config()
messages = [
{"role": "system", "content": "You are helpful."},
@ -292,10 +324,14 @@ class TestCallback:
def test_inject_memories_prepend_user_mode(self):
"""Test injecting memories in prepend_user mode."""
callback = HindsightCallback()
config = HindsightConfig(
bank_id="test",
configure(
hindsight_api_url="http://localhost:8888",
injection_mode=MemoryInjectionMode.PREPEND_USER,
)
set_defaults(bank_id="test")
config = get_config()
messages = [
{"role": "user", "content": "What's my name?"},
@ -312,10 +348,14 @@ class TestCallback:
def test_should_skip_model_exact_match(self):
"""Test model exclusion with exact match."""
callback = HindsightCallback()
config = HindsightConfig(
bank_id="test",
configure(
hindsight_api_url="http://localhost:8888",
excluded_models=["gpt-3.5-turbo"],
)
set_defaults(bank_id="test")
config = get_config()
assert callback._should_skip_model("gpt-3.5-turbo", config) is True
assert callback._should_skip_model("gpt-4", config) is False
@ -323,10 +363,14 @@ class TestCallback:
def test_should_skip_model_wildcard(self):
"""Test model exclusion with wildcard pattern."""
callback = HindsightCallback()
config = HindsightConfig(
bank_id="test",
configure(
hindsight_api_url="http://localhost:8888",
excluded_models=["gpt-3.5*", "claude-instant-*"],
)
set_defaults(bank_id="test")
config = get_config()
assert callback._should_skip_model("gpt-3.5-turbo", config) is True
assert callback._should_skip_model("gpt-3.5-turbo-16k", config) is True
@ -414,7 +458,8 @@ class TestContextManager:
with hindsight_memory(bank_id="test-agent"):
assert is_enabled() is True
assert get_config().bank_id == "test-agent"
defaults = get_defaults()
assert defaults.bank_id == "test-agent"
assert is_enabled() is False
@ -423,16 +468,17 @@ class TestContextManager:
from hindsight_litellm import hindsight_memory
# Set up initial config
configure(bank_id="original-agent")
configure(hindsight_api_url="http://localhost:8888")
set_defaults(bank_id="original-agent")
enable()
assert get_config().bank_id == "original-agent"
assert get_defaults().bank_id == "original-agent"
# Use context manager with different config
with hindsight_memory(bank_id="temporary-agent"):
assert get_config().bank_id == "temporary-agent"
assert get_defaults().bank_id == "temporary-agent"
# Should restore original config
assert get_config().bank_id == "original-agent"
assert get_defaults().bank_id == "original-agent"
assert is_enabled() is True
def test_context_manager_with_fact_types(self):
@ -440,8 +486,8 @@ class TestContextManager:
from hindsight_litellm import hindsight_memory
with hindsight_memory(bank_id="test-agent", fact_types=["world", "opinion"]):
config = get_config()
assert config.fact_types == ["world", "opinion"]
defaults = get_defaults()
assert defaults.fact_types == ["world", "opinion"]
class TestFactTypes:
@ -457,15 +503,75 @@ class TestFactTypes:
def test_configure_with_fact_types(self):
"""Test configuring with fact_types."""
config = configure(
configure(hindsight_api_url="http://localhost:8888")
defaults = set_defaults(
bank_id="test-agent",
fact_types=["world", "agent", "opinion"],
)
assert config.fact_types == ["world", "agent", "opinion"]
assert defaults.fact_types == ["world", "agent", "opinion"]
def test_configure_without_fact_types(self):
"""Test configuring without fact_types defaults to None."""
config = configure(bank_id="test-agent")
configure(hindsight_api_url="http://localhost:8888")
defaults = set_defaults(bank_id="test-agent")
assert config.fact_types is None
assert defaults.fact_types is None
class TestSetDefaults:
"""Tests for set_defaults functionality."""
def setup_method(self):
"""Reset config before each test."""
reset_config()
def teardown_method(self):
"""Clean up after each test."""
cleanup()
def test_set_defaults_creates_defaults(self):
"""Test set_defaults creates a defaults object."""
defaults = set_defaults(bank_id="test-agent")
assert defaults is not None
assert defaults.bank_id == "test-agent"
def test_set_defaults_with_all_options(self):
"""Test set_defaults with all options."""
defaults = set_defaults(
bank_id="test-agent",
document_id="doc-123",
budget="high",
fact_types=["world", "opinion"],
max_memories=10,
max_memory_tokens=2048,
use_reflect=True,
reflect_include_facts=True,
reflect_context="I am a helpful assistant.",
include_entities=False,
trace=True,
)
assert defaults.bank_id == "test-agent"
assert defaults.document_id == "doc-123"
assert defaults.budget == "high"
assert defaults.fact_types == ["world", "opinion"]
assert defaults.max_memories == 10
assert defaults.max_memory_tokens == 2048
assert defaults.use_reflect is True
assert defaults.reflect_include_facts is True
assert defaults.reflect_context == "I am a helpful assistant."
assert defaults.include_entities is False
assert defaults.trace is True
def test_set_defaults_updates_existing(self):
"""Test set_defaults updates existing defaults."""
set_defaults(bank_id="first-agent", budget="low")
defaults = set_defaults(budget="high") # Only update budget
assert defaults.bank_id == "first-agent" # Preserved
assert defaults.budget == "high" # Updated
def test_get_defaults_returns_none_initially(self):
"""Test get_defaults returns None when not set."""
assert get_defaults() is None