fleet-memory/hindsight-api/hindsight_api
Nicolò Boschi ad2cf72aab
perf: add GIN index on source_memory_ids for observation lookup (#485)
* perf: add GIN index on source_memory_ids for observation lookup

Addresses a 927x performance regression (45ms → 0.049ms) reported by a
user with ~77k observations. The array overlap operator (&&) on
source_memory_ids was doing a full sequential scan over all observations,
causing recall timeouts (57-64s) and slow user recall (18-27s avg).

The partial GIN index reduces consolidation recall from timeout to ~15s
and user recall to ~6s.

* fix: use pre-bounded memory_links for observation graph expansion

Replace raw unit_entities join in _expand_observations() with the same
memory_links entity graph used by non-observation fact types. The previous
approach joined unit_entities twice (seeds→entities→connected_sources),
which explodes at scale (30-70s at 100k observations). The LIMIT 500
workaround was non-deterministic and dropped valid results.

Using memory_links (pre-bounded to MAX_LINKS_PER_ENTITY=50 at retain time)
is algorithmically identical to the non-observation entity expansion and
keeps graph retrieval at ~2s p50 even at 100k observations.

Also fix migration down_revision (z1u2v3w4x5y6 → d2e3f4a5b6c7) and add
observation generation + fact-type filtering to the recall perf benchmark.
2026-03-05 10:15:40 +01:00
..
admin feat: new 'worker' service (#176) 2026-01-20 10:17:56 +01:00
alembic perf: add GIN index on source_memory_ids for observation lookup (#485) 2026-03-05 10:15:40 +01:00
api feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
engine perf: add GIN index on source_memory_ids for observation lookup (#485) 2026-03-05 10:15:40 +01:00
extensions feat: add extension hooks for root routing and error headers (#470) 2026-03-03 10:50:03 +01:00
webhooks feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
worker feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
__init__.py Release v0.4.15 2026-03-03 15:03:42 +01:00
banner.py feat: support for pgvectorscale (DiskANN) (#378) 2026-02-16 14:19:56 +01:00
config.py feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
config_resolver.py Fix bank config API for multi-tenant schema isolation (#417) 2026-02-20 23:43:52 +01:00
daemon.py feat: improve openclaw and hindisght-embed params (#279) 2026-02-03 09:39:04 +01:00
main.py feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
mcp_local.py fix(mcp): unify hindsight-mcp-local and server mcp (#407) 2026-02-19 17:57:17 +01:00
mcp_tools.py Fix bank-level MCP tool filtering for FastMCP 3.x (#491) 2026-03-04 10:29:29 -05:00
metrics.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
migrations.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
models.py Add bank-scoped validation to engine and HTTP handlers (#454) 2026-03-02 09:55:21 +01:00
pg0.py feat: support vertex as llm provider (#233) 2026-01-29 16:13:57 -05:00
server.py Fix: Load extensions in server.py for multi-worker deployments (#155) 2026-01-13 17:55:33 +01:00
tracing.py feat: add otel traceability (#330) 2026-02-10 12:20:48 +01:00
utils.py fix(helm): improve appVersion usage (#326) 2026-02-09 11:35:08 +01:00