9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Nicolò Boschi
|
f74b577e02
|
feat: add built-in llama.cpp LLM provider for local inference (#933)
* feat: add built-in llama.cpp LLM provider for fully local inference Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and runs inference locally via Metal/CUDA with no external services needed. - New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp` - Singleton server shared across retain/reflect/consolidation - Configurable: model path, GPU layers, context size, grammar enforcement - User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS` - Flash attention + prompt caching enabled by default - LLM provider cleanup on shutdown (stops subprocess) - hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS - Docs: configuration.md, models.mdx, providers grid updated * chore: regenerate docs skill and update lockfile for local-llm dep |
||
|
Daniyar
|
6cb309f72b
|
Fix AttributeError when event_date is None in fact_extraction (#875)
* Fix AttributeError when event_date is None in fact_extraction `_extract_facts_from_chunk` crashes with `'NoneType' object has no attribute 'isoformat'` when retaining documents without a timestamp. Two locations fixed: - Line 1058: debug log called `event_date.isoformat()` without a None check - Line 921: `parse_datetime_flexible()` can return None, so re-check before calling `.strftime()` / `.isoformat()` Fixes #874 * Revert unnecessary None guard on line 921 The original `if event_date is not None:` already guards that block. Only line 1058 needed the fix. |
||
|
Nicolò Boschi
|
9cfdd464a9
|
fix(retain): preserve normalized experience fact types (#848)
* fix(retain): preserve normalized experience fact types and remove deprecated opinion type The ExtractedFactType conversion was re-checking for raw "assistant" fact_type after the parsing layer had already normalized it to "experience". Since fact_from_llm.fact_type was always "experience" (never "assistant"), the ternary always fell through to "world", silently losing experience classification. Also removes the deprecated "opinion" fact type from internal extraction models, database constraints/indexes (via migration), and dead code paths. The public API surface (descriptions, response models, backwards-compat filter) is unchanged. * refactor(retain): drop unused confidence_score column The confidence_score column was only ever non-null for opinion facts (which are now removed). It was always written as NULL and never read back from the database. Remove it from: - DB model and migration (DROP COLUMN) - INSERT queries in fact_storage.py - retain_async/retain_batch_async parameters - RetainContext/RetainResult extension models - RetainBatch dataclass |
||
|
Nicolò Boschi
|
914ba7962c
|
perf: 3-phase retain pipeline — fix deadlocks, cap temporal links, query-time entity expansion (#722)
* perf: 3-phase retain pipeline — fix deadlocks, cap temporal links, query-time entity expansion
Major retain pipeline overhaul addressing deadlocks, write amplification,
and TimeoutErrors. Restructures retain into three phases:
Phase 1: Entity resolution on separate connection (read-heavy)
Phase 2: Core write transaction (atomic) — facts, unit_entities, links
Phase 3: Best-effort display data (error-isolated) — entity viz links, stats
Key changes:
- Sorted bulk INSERT FROM unnest() prevents deadlocks
- Temporal links capped to top-20 per unit (95% reduction)
- Batched semantic ANN via temp table + LATERAL
- Query-time entity expansion via unit_entities self-join
- Entity viz links moved to Phase 3 (post-transaction)
- HINDSIGHT_API_RETAIN_MAX_CONCURRENT config (default: 32)
* fix: increase semantic link top_k from 5 to 20
The hardcoded top_k=5 was artificially limiting semantic link creation.
Link expansion retrieval can consume up to budget (50-200) semantic
neighbors per seed set, but each fact only had 5 outgoing edges — making
the bidirectional graph very sparse.
Increasing to 20 gives retrieval 4x more edges to work with. The ANN
probe cost is unchanged (same HNSW traversal per fact, just returning
more rows). INSERT cost is negligible (~14k rows via bulk INSERT).
Also: all 18 TimeoutErrors in the latest benchmark (beam-1m-u20) were
from Gemini LLM calls, zero from the database — confirming the entity
resolution split eliminated DB timeouts entirely.
* perf: move semantic ANN search to Phase 1 to avoid transaction timeouts
The batched LATERAL ANN query (700 HNSW probes) was the last remaining
source of DB TimeoutErrors — all 29 in the latest benchmark were from
create_semantic_links_batch inside the Phase 2 write transaction.
Split semantic link creation into three phases:
- Phase 1 (separate conn, autocommit): ANN search via temp table + LATERAL.
No transaction locks, no contention with concurrent writers.
- Phase 2 (write transaction): within-batch numpy similarities (instant) +
INSERT of both within-batch and Phase 1 ANN results. No DB reads.
- Phase 3 (flush_pending_stats): future hook point for re-checking ANN
results after commit to catch links missed by concurrent batches.
Also adds 7 unit tests for compute_semantic_links_within_batch covering
empty input, identical/orthogonal embeddings, threshold filtering, top_k
cap, and tuple structure validation.
* fix: handle placeholder unit_ids in Phase 1 ANN search (not valid UUIDs)
* test: add Phase 1 ANN cross-batch test + configurable test PG port
- New test_semantic_links_phase1_ann_cross_batch verifies that the Phase 1
ANN search with placeholder unit IDs correctly creates cross-batch
semantic links after remapping to real IDs.
- Test PG port now configurable via HINDSIGHT_TEST_PG_PORT env var
(default: 5556) to avoid conflicts with running benchmark daemons.
* perf: remove retry_with_backoff from retain, set semaphore default to 4
Remove retry_with_backoff from _run_db_work and _run_delta_db_work:
- Deadlocks are prevented by sorted bulk INSERT (no need for retry)
- Transient timeouts are handled by the worker poller's task-level retry
(3 attempts, 60s spacing) which is better than rapid internal retries
that amplify I/O pressure during contention storms
Set HINDSIGHT_API_RETAIN_MAX_CONCURRENT default from 32 to 4:
- The semaphore gates Phase 1 (ANN + entity resolution) + Phase 2 (writes)
- At 4 concurrent, HNSW index I/O is manageable; at 10+ concurrent the
probes saturate disk and cause cascading timeouts
- LLM extraction still runs at full parallelism (semaphore acquired after)
* fix: add fact_type filter to Phase 1 ANN query to use per-bank HNSW indexes
The LATERAL ANN query was falling back to sequential scan + sort (90ms/probe)
because the per-bank HNSW indexes are partial indexes filtered on fact_type.
Without fact_type in the WHERE clause, PostgreSQL couldn't use them.
Fix: iterate over ('world', 'experience') and run one HNSW-indexed ANN per
type. EXPLAIN shows 8ms/probe (was 90ms) — 11x faster.
700 probes × 8ms × 2 types = ~11s total (was ~63s via seq scan).
* fix: scope temporal links by fact_type + add integration tests
Temporal links now filter by fact_type in the LATERAL query — world facts
only link to world facts, experience to experience. This matches how
retrieval filters results and avoids wasted cross-type link rows.
New integration tests:
- test_semantic_ann_uses_hnsw_index: verifies Phase 1 ANN creates
cross-batch semantic links (tests fact_type filter + placeholder remap)
- test_temporal_links_scoped_by_fact_type: verifies world facts get
temporal links to other world facts but NOT to experience facts
* fix: tolerate individual chunk LLM failures instead of failing entire batch
Changed asyncio.gather(*tasks) to asyncio.gather(*tasks, return_exceptions=True)
in both chunk-level and content-level fact extraction. A single chunk timeout
(e.g., Gemini >90s) no longer discards all other successfully extracted facts.
For a 50MB document with 17k chunks, even a 2% chunk failure rate previously
caused 0 completions (entire batch discarded). Now 16,700 facts are extracted
and only the 300 failed chunks are skipped with a warning log.
* fix: batch temporal LATERAL query for large documents (16k+ chunks)
The LATERAL query for temporal links passed all unit_ids at once into
unnest(), causing PostgreSQL timeouts on documents with 16k+ chunks.
Split into batches of 500 units per query to keep each under the
command_timeout.
Also identified: HNSW index creation on shared pg0 instances with
50k+ existing units exceeds the 60s command_timeout. This is a
test infrastructure issue (shared pg0 accumulates data) but also
affects production when creating new banks on large instances.
* feat: streaming chunk batching for large documents (RETAIN_CHUNK_BATCH_SIZE)
Process chunks in mini-batches of N (default 500), committing each batch
to the DB before starting the next. This prevents OOM kills on large
documents (50MB / 17k+ chunks) by keeping only ~500 facts + embeddings
in memory at a time instead of 50k+.
Each mini-batch goes through the full Phase 1 → 2 → 3 pipeline
independently, sharing the same document_id. On recovery (process dies
mid-way), delta retain detects already-committed chunks via content_hash
and skips them — only remaining chunks get re-extracted.
Config: HINDSIGHT_API_RETAIN_CHUNK_BATCH_SIZE (default: 500, 0 to disable)
Per-bank configurable via the hierarchical config system.
Tests:
- test_streaming_chunk_batching_produces_same_facts
- test_streaming_chunk_batching_recovery (delta retain skips committed chunks)
- test_streaming_disabled_for_small_docs
* perf(retain): producer-consumer pipeline + deferred semantic ANN
Replace the sequential streaming loop with a producer-consumer pipeline:
- LLM producer fires concurrent chunk extractions (semaphore-bounded)
- DB consumer drains queue in batches, runs Phase 1+2+3 per batch
- LLM and DB work overlap instead of running sequentially
Defer semantic links to a single final ANN pass after all batches commit:
- Remove within-batch semantic links from Phase 2 (was 2.6s/batch)
- Run parallel ANN (4 connections) after all facts committed
- top_k reduced from 50 to 20 (recall uses at most 20 neighbors)
- Recovery via operation result_metadata checkpoint
Additional optimizations:
- skip_exists_check on temporal/causal link INSERT (saves ~0.5s/batch)
- WHERE EXISTS guard on semantic link INSERT (handles document upsert)
- timeout=300s on ANN queries and bulk INSERT for large banks
- Demote [ANN] debug logs to logger.debug()
- Fix docstring typos (agent_id → bank_id)
- Fix content_index remapping in producer-consumer batches
- Fix delta retain passing contents vs delta_contents
50MB benchmark (mock LLM): 9.2 min (was 23 min) — 2.5x faster.
BEAM 10m benchmark: zero deadlocks, zero DB errors.
* refactor(retain): remove legacy fallback code paths
- Remove process_entities_batch (legacy single-connection entity processing)
- Remove extract_entities_batch_optimized (only caller was the above)
- Remove fallback entity processing inside Phase 2 transaction
- Remove legacy ANN inline fallback in create_semantic_links_batch
- Remove fallback entity_links direct-insert path in Phase 3
- Make resolved_entity_ids/entity_to_unit/unit_to_entity_ids required params
* refactor(retain): replace tuple returns with dataclasses, remove dead code
- Add EntityResolutionResult and Phase1Result dataclasses in types.py
- Replace 4-tuple return from _pre_resolve_phase1 with Phase1Result
- Remove dead `entity_links = []` variables in retain_batch and _try_delta_retain
- Remove unused `confidence_score` parameter from orchestrator.retain_batch
and _retain_batch_async_internal (was accepted but never used)
* fix(entity-resolver): remove LIKE full-scan fallbacks, use index-only trigram matching
The entity resolution query had LIKE '%...' substring conditions that bypassed
the GIN trigram index, causing full sequential scans of the entities table.
On banks with 10k+ entities, this caused TimeoutErrors (observed in BEAM 10m).
Changes:
- Remove LIKE fallbacks, use trigram % operator only (GIN index-based)
- Lower similarity threshold from 0.3 to 0.15 to catch substring relationships
- Use LOWER() on both sides for case-insensitive matching
- Migration: recreate GIN trigram index on LOWER(canonical_name)
* fix: remove schema prefix from index names in trigram migration
* fix(delta-retain): use same chunk_size as streaming path (3000 vs 120000)
_chunk_contents_for_delta defaulted to chunk_size=120000 while the streaming
path used 3000. On retry, delta re-chunked the document with different
boundaries, found 0 matching chunks, and fell through to full re-extraction.
This wasted all LLM calls on already-committed chunks.
Fix: use the same default (3000) so chunk hashes match on recovery.
* fix(retain): persist generated document_id in operation metadata for retry recovery
When no document_id is provided, retain generates a UUID. On retry, a new UUID
was generated, making delta retain and streaming chunk-hash recovery unable to
find previously committed chunks. All LLM extraction was wasted on retry.
Fix: resolve document_id early in retain_batch (before delta), persist it to
operation result_metadata, and recover it on retry. Both delta and streaming
paths now see the same document_id across attempts.
* refactor(retain): unify into single streaming pipeline, remove non-streaming path
All retains now go through the producer-consumer streaming pipeline,
regardless of document size. Small documents are processed as a single batch.
This eliminates the maintenance burden of two separate code paths.
Also fix document upsert: compare content hash to distinguish recovery
(same content, partially committed) from update (different content, needs
cascade-delete). Previously, existing chunks always triggered recovery mode.
* refactor(retain): remove dead code, replace raw dicts with Phase3Context dataclass
- Remove dead _handle_zero_facts_documents (no callers after path unification)
- Remove unused imports: defaultdict, EntityLink
- Replace raw dict phase3_context with typed Phase3Context dataclass
- Update _build_and_insert_entity_links_phase3 to use typed parameter
|
||
|
Nicolò Boschi
|
0096115678
|
fix(engine): classify first-person agent experiences as 'experience' fact type (#775)
* fix(engine): classify first-person agent experiences as 'experience' fact type The extraction prompt defined "assistant" too narrowly as only "interactions with assistant (requests, recommendations)", causing the LLM to classify first-person agent actions (code changes, debugging, discoveries) as "world". Broadened the fact_type definition in the prompt and Pydantic model descriptions to cover all first-person actions, experiences, and observations by the speaker. * style: fix line length in fact_extraction.py |
||
|
Nicolò Boschi
|
d2965e64e6
|
fix(retain): inject retain_mission into verbose extraction mode (#745) (#754)
Verbose mode was the only extraction mode that skipped injecting the retain_mission FOCUS section into its prompt template. Users who set a retain_mission got no filtering when using verbose mode. |
||
|
Nicolò Boschi
|
446c75f3e2
|
fix: correctly map LLM fact_type \"assistant\" to \"experience\" for DB storage (#609)
The Pydantic model extraction paths (batch API and parallel extraction) used fact_from_llm.fact_type directly, bypassing the \"assistant\" → \"experience\" conversion and causing DB CHECK constraint violations. Unified the conversion logic across all paths: - \"assistant\" → \"experience\" - \"world\" → \"world\" - anything else: fall back to fact_kind (\"assistant\" → \"experience\"), else \"world\" |
||
|
Nicolò Boschi
|
e4f8a157c2
|
feat(retain): verbatim, chunks modes and named retain strategies (#593)
* feat(retain): add verbatim extraction mode Adds retain_extraction_mode="verbatim" that stores each chunk as-is without LLM summarization. The LLM still runs to extract entities, temporal info, and location for full indexability — only the fact text is replaced with the original chunk content (one memory per chunk). Useful for RAG-style indexing and benchmarks where original text must be preserved in memory. - Add "verbatim" to RETAIN_EXTRACTION_MODES in config.py - Add VERBATIM_FACT_EXTRACTION_PROMPT with instructions to preserve text - Add _collapse_to_verbatim() post-processing to enforce 1 fact/chunk - Expose in bank config UI dropdown with updated description - Update configuration.md docs with verbatim mode description - Add unit test for _collapse_to_verbatim and integration test via LLM - Fix pre-existing main.py CLI override missing new reranker fields - Fix pre-existing cross_encoder.py ty type error via setattr * refactor(retain): verbatim mode skips 'what' field entirely Instead of asking the LLM to echo the chunk text back into 'what' and then discarding it, verbatim mode now uses a dedicated schema (VerbatimExtractedFact) that omits the 'what' field altogether. The LLM only returns metadata (entities, temporal info, location, who), saving output tokens and avoiding any risk of paraphrasing before the backfill. - Add VerbatimExtractedFact / VerbatimFactExtractionResponse models - Verbatim mode skips causal-relations section (nothing to relate causally) - _extract_facts_from_chunk: allow missing 'what' in verbatim mode, set combined_text="" (backfilled by _collapse_to_verbatim) - Update verbatim prompt to say DO NOT include 'what' * feat(retain): add index_only extraction mode Zero-LLM retain mode: chunks are stored as-is with no LLM call, no entity extraction, and no temporal indexing. Embeddings still run for semantic search. User-provided entities via RetainContent.entities are the sole source of entity data. Early return placed before the batch-API check so no LLM queue or concurrency locks are acquired. - Add "index_only" to RETAIN_EXTRACTION_MODES - Add _extract_facts_index_only() with pure Python chunking path - Add to UI dropdown and update description - Update configuration.md with index_only docs and table entry - Add unit test asserting zero token usage and exact text preservation * feat(retain): add named retain strategies Allows mixing extraction modes in a single bank via named strategies. Each strategy is a set of hierarchical config overrides (extraction_mode, chunk_size, entity_labels, entities_allow_free_form, etc.) applied on top of the resolved bank config at retain time. - retain_strategies: dict of strategy_name → config overrides (bank config) - retain_default_strategy: default strategy when none specified (bank config) - strategy field on /retain request: per-call override - apply_strategy() in config_resolver applies overrides via dataclasses.replace() - strategy propagates through retain_batch_async → _retain_batch_async_internal and through the async worker task payload - Any hierarchical field is overridable per strategy, including entity_labels and entities_allow_free_form - Docs updated with strategy configuration example and RRF fairness note - Unit test for apply_strategy covering overrides, unknown strategy, and non-hierarchical field filtering * feat(retain): add per-item strategy and strategy tests - Add `strategy` field to `MemoryItem` so individual items in a retain request can override the request-level strategy - Add `strategy` field to `FileRetainMetadata` for per-file strategy override in file retain requests - Group memory items by effective strategy in `api_retain`; each group is processed as a separate batch, results are aggregated - Thread strategy through `submit_async_file_retain` → `_handle_file_convert_retain` → retain task payload - Add `operation_ids` to `RetainResponse` for async requests with mixed per-item strategies - Add `test_strategy_overrides_extraction_mode_for_index_only`: unit test verifying a named strategy with index_only bypasses the LLM - Add `test_retain_request_per_item_strategy_field`: unit test for per-item strategy grouping logic * feat(ui): add retain strategies and default strategy to bank config UI - Add StrategiesEditor component: per-strategy cards with name input and JSON overrides textarea; supports add/remove; validates JSON inline - Add Default Strategy text input (retain_default_strategy) - Update RetainEdits type and retainSlice() to include both new fields - Regenerate OpenAPI spec (retain_strategies, retain_default_strategy, per-item strategy on MemoryItem/FileRetainMetadata, operation_ids on RetainResponse) * refactor(ui): move retain strategies into its own dedicated config section * feat(ui): improve retain strategies UX and add strategy to document dialog - Strategy form now includes entity section (free form toggle + entity labels editor) - Default strategy selector moved outside tab panel, above strategy chips - Strategy tabs redesigned with underline indicator style for clarity - Remove strategy confirms with AlertDialog - Fix tab re-render bug when typing strategy name (skipSyncRef) - Add strategy field to Add New Document dialog (text + per-file for uploads) - File upload collapsible uses same Document/Tags/Source tabbed layout - API: validate empty strategy names in config_resolver - api.ts: add strategy field to retain and uploadFiles types * fix: forward strategy through HTTP layer and SDK; add integration test - route.ts: extract and forward `strategy` from request body to retainBatch - TypeScript SDK: accept and forward `strategy` in retainBatch options and per-item - config_resolver.py: validate empty strategy name keys on update - bank-config-view.tsx: merge entity fields into RetainStrategyForm, redesign strategy tabs with underline style, add confirmation dialog for removal, fix tab-reset-on-typing with skipSyncRef, move default strategy selector outside panel - bank-selector.tsx: add strategy field to Add Document dialog (per-file in tabbed collapsible) - test_retain.py: add end-to-end integration test verifying named strategy application (index_only = 0 LLM tokens) * fix: regenerate TypeScript client with strategy field in RetainRequest/MemoryItem - Regenerate OpenAPI spec to include strategy field in RetainRequest and MemoryItem - Regenerate TypeScript client from updated spec - Add strategy to MemoryItemInput interface - Remove (item as any) cast now that strategy is properly typed * rename: index_only extraction mode → chunks * remove top-level strategy from RetainRequest; strategy is per-item only * fix(clients): update Go and Python generated clients with strategy/operation_ids fields * fix(ci): update hierarchical field count, add strategy to Rust MemoryItem initializers * fix(go-client): minimal targeted YAML updates for strategy/operation_ids fields |
||
|
Nicolò Boschi
|
15ea23d5d6
|
feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560)
* feat: introduce hindsight-api-slim and hindsight-all-slim packages Closes #552 - Move all source code from hindsight-api/ to new hindsight-api-slim/ - hindsight-api-slim has heavy ML deps (torch, sentence-transformers, transformers, einops, flashrank, mlx, mlx-lm, safetensors) and pg0-embedded as optional extras: [local-ml], [embedded-db], [all] - hindsight-api becomes a zero-code meta-package depending on hindsight-api-slim[all] for full backward compatibility - Add hindsight-all-slim meta-package: hindsight-api-slim + client + embed - hindsight-all updated to depend on hindsight-api-slim[all] - pg0.py: lazy-import pg0 with clear ImportError pointing to [embedded-db] - Dockerfile: replace sed hack with proper uv sync --extra flags - Update release.yml, test.yml, lint.sh, release.sh, CLAUDE.md and all path references throughout the repo * refactor: rename hindsight/ directory to hindsight-all/ * docs: document hindsight-api-slim and hindsight-all-slim package variants Add package variants table and extras explanation to installation.md * docs: remove emojis from installation.md, use professional tone * docs: link Docker slim variant to pip package variants section * docs: consolidate Docker image variants into single table * ci: fix working-directory paths after package restructure - Replace all hindsight-api → hindsight-api-slim in test.yml - Replace hindsight → hindsight-all in test.yml - Add --extra embedded-db to test-embed API install step * ci: add local-ml and embedded-db extras to API sync steps These extras were previously implicit in the old hindsight-api package (which bundled everything). Now that hindsight-api-slim uses optional extras, we must explicitly request local-ml and embedded-db in CI. * ci: add API install step with embedded-db to test-embed smoke test The smoke test starts hindsight-api as a daemon, which requires pg0-embedded. Add a dedicated install step for hindsight-api-slim with embedded-db extra so the daemon can start successfully. * ci: remove --no-install-project when using optional extras When --no-install-project is combined with --extra, the optional deps are not installed because extras require the project to be active. Remove --no-install-project from steps that need local-ml or embedded-db. * ci: fix ordering of uv sync steps to preserve optional extras When uv sync runs for a different workspace member, it removes optional extras installed for other members. Fix by always running extra-requiring API sync last, after other workspace member syncs. Also remove --no-install-project from embedded-db sync in test-embed, as --no-install-project prevents optional extras from being active. * ci: add local-ml extra to test-embed API install for smoke test The smoke test starts the full API server which needs sentence-transformers for local embeddings (default provider). Add local-ml extra to the install. * ci: simplify extras with --all-extras and add slim pip smoke test - Replace explicit --extra local-ml --extra embedded-db with --all-extras for cleaner, more maintainable sync steps - Add test-pip-slim job: tests hindsight-api-slim[embedded-db] without local ML models, using Cohere for embeddings/reranking (mirrors Docker slim smoke test approach) * ci: simplify slim smoke test to health check only (mirrors Docker test) |
Renamed from hindsight-api/hindsight_api/engine/retain/fact_extraction.py (Browse further)