fleet-memory/hindsight-api/hindsight_api/engine
Nicolò Boschi ad2cf72aab
perf: add GIN index on source_memory_ids for observation lookup (#485)
* perf: add GIN index on source_memory_ids for observation lookup

Addresses a 927x performance regression (45ms → 0.049ms) reported by a
user with ~77k observations. The array overlap operator (&&) on
source_memory_ids was doing a full sequential scan over all observations,
causing recall timeouts (57-64s) and slow user recall (18-27s avg).

The partial GIN index reduces consolidation recall from timeout to ~15s
and user recall to ~6s.

* fix: use pre-bounded memory_links for observation graph expansion

Replace raw unit_entities join in _expand_observations() with the same
memory_links entity graph used by non-observation fact types. The previous
approach joined unit_entities twice (seeds→entities→connected_sources),
which explodes at scale (30-70s at 100k observations). The LIMIT 500
workaround was non-deterministic and dropped valid results.

Using memory_links (pre-bounded to MAX_LINKS_PER_ENTITY=50 at retain time)
is algorithmically identical to the non-observation entity expansion and
keeps graph retrieval at ~2s p50 even at 100k observations.

Also fix migration down_revision (z1u2v3w4x5y6 → d2e3f4a5b6c7) and add
observation generation + fact-type filtering to the recall perf benchmark.
2026-03-05 10:15:40 +01:00
..
consolidation feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
directives feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
mental_models feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
parsers feat: add iris as file parser (#395) 2026-02-18 14:09:56 +01:00
providers refactor: replace set_gemini_safety_settings() with LLMProvider.with_config() (#474) 2026-03-03 15:00:32 +01:00
reflect feat: entity labels — optional, free_values, multi_value, UI polish (#450) 2026-03-02 13:05:25 +01:00
retain fix: resolve TypeError when LLM returns invalid JSON across all retries (#488) (#490) 2026-03-04 14:17:55 +01:00
search perf: add GIN index on source_memory_ids for observation lookup (#485) 2026-03-05 10:15:40 +01:00
storage fix(storage): use dynamic schema_getter in PostgreSQLFileStorage for multi-tenant (#440) 2026-02-25 15:19:08 +01:00
__init__.py feat: extensions (#54) 2025-12-22 11:05:23 +01:00
cross_encoder.py fix: zeroentropy rerank URL missing /v1 prefix and MCP retain async_processing param (#460) 2026-03-02 10:32:01 +01:00
db_budget.py fix: batch queries on recall (#149) 2026-01-13 13:20:22 +01:00
db_utils.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
embeddings.py fix: pass encoding_format="float" in LiteLLM embedding calls (#434) 2026-02-25 10:32:05 +01:00
entity_resolver.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
interface.py feat: add tags filtering and q description fix for list documents API (#468) 2026-03-02 17:03:16 +01:00
llm_interface.py feat: support Batch API for retain (openai/groq) (#365) 2026-02-16 13:31:50 +01:00
llm_wrapper.py refactor: replace set_gemini_safety_settings() with LLMProvider.with_config() (#474) 2026-03-03 15:00:32 +01:00
memory_engine.py feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
operation_metadata.py fix: improve async batch retain with large payloads (#366) 2026-02-16 12:51:42 +01:00
query_analyzer.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
response_models.py feat: include source facts in observation recall (#404) 2026-02-19 14:54:19 +01:00
task_backend.py fix: retain async fails if timestamp is set (#251) 2026-01-30 13:04:33 +01:00
utils.py feat: implement hierarchical configuration (system, tenant, bank) (#329) 2026-02-12 13:14:57 +01:00