fleet-memory/hindsight-api-slim/hindsight_api/engine/providers
Nicolò Boschi f74b577e02
feat: add built-in llama.cpp LLM provider for local inference (#933)
* feat: add built-in llama.cpp LLM provider for fully local inference

Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server
subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and
runs inference locally via Metal/CUDA with no external services needed.

- New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp`
- Singleton server shared across retain/reflect/consolidation
- Configurable: model path, GPU layers, context size, grammar enforcement
- User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS`
- Flash attention + prompt caching enabled by default
- LLM provider cleanup on shutdown (stops subprocess)
- hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS
- Docs: configuration.md, models.mdx, providers grid updated

* chore: regenerate docs skill and update lockfile for local-llm dep
2026-04-08 15:22:10 +02:00
..
__init__.py feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
anthropic_llm.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
claude_code_llm.py fix(claude-code): disable built-in tools to prevent MCP tool deferral (#784) 2026-03-31 14:20:17 +02:00
codex_llm.py fix(codex): don't crash on startup when quota is exhausted (429) (#744) 2026-03-30 10:39:12 +02:00
gemini_llm.py fix(gemini): thought_signature read from wrong object and type in 3.1+ tool calls (#785) 2026-03-31 09:09:12 +02:00
litellm_llm.py feat: add LiteLLM LLM provider for Bedrock and 100+ providers (#679) 2026-03-25 14:17:38 +01:00
llamacpp_llm.py feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
mock_llm.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
none_llm.py feat: add 'none' LLM provider for chunk-only storage mode (#691) 2026-03-25 18:01:20 +01:00
openai_compatible_llm.py feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00