fleet-memory/skills
Nicolò Boschi f74b577e02
feat: add built-in llama.cpp LLM provider for local inference (#933)
* feat: add built-in llama.cpp LLM provider for fully local inference

Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server
subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and
runs inference locally via Metal/CUDA with no external services needed.

- New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp`
- Singleton server shared across retain/reflect/consolidation
- Configurable: model path, GPU layers, context size, grammar enforcement
- User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS`
- Flash attention + prompt caching enabled by default
- LLM provider cleanup on shutdown (stops subprocess)
- hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS
- Docs: configuration.md, models.mdx, providers grid updated

* chore: regenerate docs skill and update lockfile for local-llm dep
2026-04-08 15:22:10 +02:00
..
hindsight-cloud docs(skills): encourage rich context over pre-summarized strings in retain (#594) 2026-03-16 18:37:12 +01:00
hindsight-docs feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
hindsight-local docs(skills): encourage rich context over pre-summarized strings in retain (#594) 2026-03-16 18:37:12 +01:00
hindsight-self-hosted docs(skills): encourage rich context over pre-summarized strings in retain (#594) 2026-03-16 18:37:12 +01:00