fleet-memory/hindsight-api-slim/hindsight_api/engine
Nicolò Boschi f74b577e02
feat: add built-in llama.cpp LLM provider for local inference (#933)
* feat: add built-in llama.cpp LLM provider for fully local inference

Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server
subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and
runs inference locally via Metal/CUDA with no external services needed.

- New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp`
- Singleton server shared across retain/reflect/consolidation
- Configurable: model path, GPU layers, context size, grammar enforcement
- User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS`
- Flash attention + prompt caching enabled by default
- LLM provider cleanup on shutdown (stops subprocess)
- hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS
- Docs: configuration.md, models.mdx, providers grid updated

* chore: regenerate docs skill and update lockfile for local-llm dep
2026-04-08 15:22:10 +02:00
..
consolidation fix(consolidation): improve observation quality with structured processing rules (#814) 2026-04-01 12:44:52 +02:00
directives feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
parsers feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
providers feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
reflect feat(reflect): make source facts in search_observations configurable (#688) 2026-03-25 17:54:49 +01:00
retain feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
search feat: include occurred_end and mentioned_at in think-prompt fact serialization (#929) 2026-04-08 10:33:14 +02:00
storage feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
__init__.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
audit.py feat: add audit log for feature usage tracking (#717) 2026-03-27 09:52:03 +01:00
cross_encoder.py feat: add OpenRouter support for LLM, embeddings, and reranking (#930) 2026-04-08 11:24:21 +02:00
db_budget.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
db_utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
embeddings.py feat: add OpenRouter support for LLM, embeddings, and reranking (#930) 2026-04-08 11:24:21 +02:00
entity_resolver.py perf: 3-phase retain pipeline — fix deadlocks, cap temporal links, query-time entity expansion (#722) 2026-04-01 12:52:49 +02:00
interface.py fix(api): clear memories endpoint no longer deletes the bank profile (#837) 2026-04-01 18:34:39 +02:00
jina_mlx_reranker.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
llm_interface.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
llm_wrapper.py feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
memory_engine.py feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
operation_metadata.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
query_analyzer.py fix(query_analyzer): handle dateparser internal crashes gracefully (#893) 2026-04-07 09:26:27 +02:00
response_models.py fix(retain): preserve normalized experience fact types (#848) 2026-04-02 12:20:37 +02:00
task_backend.py fix(ci): resolve all CI failures (#847) 2026-04-02 17:17:42 +02:00
utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00