fleet-memory/skills/hindsight-docs/references/developer
Nicolò Boschi f74b577e02
feat: add built-in llama.cpp LLM provider for local inference (#933)
* feat: add built-in llama.cpp LLM provider for fully local inference

Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server
subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and
runs inference locally via Metal/CUDA with no external services needed.

- New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp`
- Singleton server shared across retain/reflect/consolidation
- Configurable: model path, GPU layers, context size, grammar enforcement
- User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS`
- Flash attention + prompt caching enabled by default
- LLM provider cleanup on shutdown (stops subprocess)
- hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS
- Docs: configuration.md, models.mdx, providers grid updated

* chore: regenerate docs skill and update lockfile for local-llm dep
2026-04-08 15:22:10 +02:00
..
api fix(ci): resolve all CI failures (#847) 2026-04-02 17:17:42 +02:00
admin-cli.md Fix run-db-migration for all-tenant upgrades (#530) 2026-03-10 10:10:03 +01:00
configuration.md feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
development.md feat: add docs skill (#335) 2026-02-10 14:41:51 +01:00
extensions.md feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
index.md doc: add langgraph and nemoclaw (#633) 2026-03-20 16:18:05 +01:00
installation.md docs: add supported platforms and Windows installation guide (#700) 2026-03-26 12:01:30 +01:00
mcp-server.md fix: doc build 2026-02-26 18:01:56 +01:00
models.md feat: add built-in llama.cpp LLM provider for local inference (#933) 2026-04-08 15:22:10 +02:00
monitoring.md feat: add LiteLLM LLM provider for Bedrock and 100+ providers (#679) 2026-03-25 14:17:38 +01:00
multilingual.md feat: add docs skill (#335) 2026-02-10 14:41:51 +01:00
observations.md feat(skill): validate links, strip images, include openapi.json and changelog (#614) 2026-03-19 12:32:33 +01:00
performance.md Release v0.4.13 2026-02-19 18:46:07 +01:00
rag-vs-hindsight.md feat: add docs skill (#335) 2026-02-10 14:41:51 +01:00
reflect.md feat(skill): validate links, strip images, include openapi.json and changelog (#614) 2026-03-19 12:32:33 +01:00
retain.md feat(skill): validate links, strip images, include openapi.json and changelog (#614) 2026-03-19 12:32:33 +01:00
retrieval.md feat: add docs skill (#335) 2026-02-10 14:41:51 +01:00
services.md feat: add docs skill (#335) 2026-02-10 14:41:51 +01:00
storage.md feat: add docs skill (#335) 2026-02-10 14:41:51 +01:00