fleet-memory/hindsight-api
Nicolò Boschi 67c47881cb
fix: add defensive error handling to PyTorch device detection (#221)
* fix: include correct __version__ in python packages

* fix(embed): force CPU mode for local models in daemon to prevent XPC crashes

Adds HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU and HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU
environment variables to force CPU-only operation for local sentence-transformer models.

This prevents XPC_ERROR_CONNECTION_INVALID crashes on macOS when running in daemon mode.
The issue occurs because PyTorch's MPS (Metal Performance Shaders) backend has unstable
XPC connections in background processes, leading to C++ assertion failures that Python
exception handlers cannot catch.

Changes:
- config.py: Add ENV_*_FORCE_CPU constants and config dataclass fields
- embeddings.py: Add force_cpu parameter to LocalSTEmbeddings constructor
- cross_encoder.py: Add force_cpu parameter to LocalSTCrossEncoder constructor
- main.py: Set force CPU env vars in daemon mode, add fields to config constructor

The daemon mode automatically enables force CPU for both embeddings and reranker,
while normal mode allows hardware acceleration (GPU/MPS) as before.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix: add defensive error handling to PyTorch device detection

Wraps all PyTorch device detection code (torch.cuda.is_available()
and torch.backends.mps.is_available()) in try-except blocks that
gracefully fall back to CPU if any errors occur.

This complements PR #218's force_cpu configuration by ensuring the
code works reliably in all environments without configuration:
- CI environments with CPU-only PyTorch builds
- Systems without proper GPU/MPS support
- Partial or misconfigured PyTorch installations

The defensive approach prevents startup failures while still taking
advantage of GPU/MPS acceleration when available and force_cpu is
not explicitly set.

Changes:
- embeddings.py: Added try-except in initialize() and _reinitialize_model_sync()
- cross_encoder.py: Added try-except in initialize() and _reinitialize_model_sync()

* refactor: use get_config() for embeddings and reranker force_cpu

Changes create_embeddings_from_env() and create_cross_encoder_from_env()
to read configuration via get_config() instead of directly accessing
os.environ. This ensures consistency across the codebase and properly
respects the force_cpu configuration set by daemon mode.

Changes:
- embeddings.py: Use config.embeddings_local_model and config.embeddings_local_force_cpu
- cross_encoder.py: Use config.reranker_local_model and config.reranker_local_force_cpu
- Both: Use get_config() for provider, tei_url, and other config fields
- Note: Some fields not in config (like max_concurrent for local reranker) still read from os.environ

This fixes the issue where force_cpu was read inconsistently from environment
variables instead of using the centralized config system.

* test: clear config cache in test_create_from_env

Fixes test failure caused by cached config not picking up
environment variable changes in test. The test now calls
clear_config_cache() before and after patching os.environ
to ensure the factory function reads the test's env vars.

* refactor: add reranker_local_max_concurrent to config system

Adds reranker_local_max_concurrent to HindsightConfig dataclass
and removes the workaround in create_cross_encoder_from_env() that
was reading it directly from os.environ.

Changes:
- config.py: Add reranker_local_max_concurrent field to dataclass and from_env()
- main.py: Add reranker_local_max_concurrent to manual config constructor
- cross_encoder.py: Use config.reranker_local_max_concurrent instead of os.environ

This completes the refactoring to use the centralized config system
for all reranker configuration.

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-28 18:14:54 +01:00
..
hindsight_api fix: add defensive error handling to PyTorch device detection (#221) 2026-01-28 18:14:54 +01:00
tests fix: add defensive error handling to PyTorch device detection (#221) 2026-01-28 18:14:54 +01:00
pyproject.toml Release v0.4.0 2026-01-28 15:04:43 +01:00
README.md feat(doc): add new config options and supported providers (#84) 2026-01-01 17:09:05 +01:00

Hindsight API

Memory System for AI Agents — Temporal + Semantic + Entity Memory Architecture using PostgreSQL with pgvector.

Hindsight gives AI agents persistent memory that works like human memory: it stores facts, tracks entities and relationships, handles temporal reasoning ("what happened last spring?"), and forms opinions based on configurable disposition traits.

Installation

pip install hindsight-api

Quick Start

Run the Server

# Set your LLM provider
export HINDSIGHT_API_LLM_PROVIDER=openai
export HINDSIGHT_API_LLM_API_KEY=sk-xxxxxxxxxxxx

# Start the server (uses embedded PostgreSQL by default)
hindsight-api

The server starts at http://localhost:8888 with:

  • REST API for memory operations
  • MCP server at /mcp for tool-use integration

Use the Python API

from hindsight_api import MemoryEngine

# Create and initialize the memory engine
memory = MemoryEngine()
await memory.initialize()

# Create a memory bank for your agent
bank = await memory.create_memory_bank(
    name="my-assistant",
    background="A helpful coding assistant"
)

# Store a memory
await memory.retain(
    memory_bank_id=bank.id,
    content="The user prefers Python for data science projects"
)

# Recall memories
results = await memory.recall(
    memory_bank_id=bank.id,
    query="What programming language does the user prefer?"
)

# Reflect with reasoning
response = await memory.reflect(
    memory_bank_id=bank.id,
    query="Should I recommend Python or R for this ML project?"
)

CLI Options

hindsight-api --help

# Common options
hindsight-api --port 9000          # Custom port (default: 8888)
hindsight-api --host 127.0.0.1     # Bind to localhost only
hindsight-api --workers 4          # Multiple worker processes
hindsight-api --log-level debug    # Verbose logging

Configuration

Configure via environment variables:

Variable Description Default
HINDSIGHT_API_DATABASE_URL PostgreSQL connection string pg0 (embedded)
HINDSIGHT_API_LLM_PROVIDER openai, anthropic, gemini, groq, ollama, lmstudio openai
HINDSIGHT_API_LLM_API_KEY API key for LLM provider -
HINDSIGHT_API_LLM_MODEL Model name gpt-4o-mini
HINDSIGHT_API_HOST Server bind address 0.0.0.0
HINDSIGHT_API_PORT Server port 8888

Example with External PostgreSQL

export HINDSIGHT_API_DATABASE_URL=postgresql://user:pass@localhost:5432/hindsight
export HINDSIGHT_API_LLM_PROVIDER=groq
export HINDSIGHT_API_LLM_API_KEY=gsk_xxxxxxxxxxxx

hindsight-api

Docker

docker run --rm -it -p 8888:8888 \
  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
  -v $HOME/.hindsight-docker:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

MCP Server

For local MCP integration without running the full API server:

hindsight-local-mcp

This runs a stdio-based MCP server that can be used directly with MCP-compatible clients.

Key Features

  • Multi-Strategy Retrieval (TEMPR) — Semantic, keyword, graph, and temporal search combined with RRF fusion
  • Entity Graph — Automatic entity extraction and relationship tracking
  • Temporal Reasoning — Native support for time-based queries
  • Disposition Traits — Configurable skepticism, literalism, and empathy influence opinion formation
  • Three Memory Types — World facts, bank actions, and formed opinions with confidence scores

Documentation

Full documentation: https://hindsight.vectorize.io

License

Apache 2.0