* fix(recall): cap entity fanout in graph expansion to prevent slow queries
On large banks, the entity co-occurrence self-join in _expand_combined()
produces massive intermediate row counts when seeds reference high-fanout
entities (e.g. an entity with 25K+ mentions). This causes recall latency
to degrade significantly.
Changes:
- Replace unbounded entity self-join with LATERAL per-entity cap
(graph_per_entity_limit, default 200), reducing intermediate rows
from potentially millions to at most num_entities * 200
- Add ORDER BY unit_id DESC in LATERAL subquery for deterministic
recency-biased sampling (rides the PK index, no extra sort)
- Add timeout fallback (graph_expansion_timeout, default 10s) that
drops entity expansion and falls back to semantic+causal only
- Add composite index (entity_id, unit_id) on unit_entities for
index-only scans in the LATERAL subquery
- Merge 3 unmerged migration heads into one
- Fix recall_perf.py dotenv override issue
Unlike the approach in #895, this does NOT filter out hub entities
entirely — all entities are kept but capped equally, preserving
retrieval quality for queries about frequently-mentioned entities.
Benchmarked on a 67K-unit bank (top entity = 25K mentions):
- retrieval_graph: 0.337s → 0.055s (84% faster)
- end-to-end recall: 0.912s → 0.519s (43% faster)
* fix(tests): fix broken test_combined_scoring and test_reranking_proof_count
- test_combined_scoring: replace MagicMock(spec=RetrievalResult) with real
dataclass instances — MagicMock attributes returned nested mocks that
failed on >= comparisons with int
- test_reranking_proof_count: remove deleted `embedding` param from
RetrievalResult constructor, use None for occurred_start/end to get
neutral recency (datetime.now gave recency=1.0 which boosted scores)
* refactor: rename config to link_expansion_ prefix, fix observation fanout
- Rename GRAPH_PER_ENTITY_LIMIT → LINK_EXPANSION_PER_ENTITY_LIMIT and
GRAPH_EXPANSION_TIMEOUT → LINK_EXPANSION_TIMEOUT to follow the
convention that these are specific to the link_expansion graph retriever
- Apply the same LATERAL per-entity cap to _expand_observations(), which
had the same unbounded self-join through unit_entities
* style: fix formatting in config.py
* perf: add GIN index on source_memory_ids for observation lookup
Addresses a 927x performance regression (45ms → 0.049ms) reported by a
user with ~77k observations. The array overlap operator (&&) on
source_memory_ids was doing a full sequential scan over all observations,
causing recall timeouts (57-64s) and slow user recall (18-27s avg).
The partial GIN index reduces consolidation recall from timeout to ~15s
and user recall to ~6s.
* fix: use pre-bounded memory_links for observation graph expansion
Replace raw unit_entities join in _expand_observations() with the same
memory_links entity graph used by non-observation fact types. The previous
approach joined unit_entities twice (seeds→entities→connected_sources),
which explodes at scale (30-70s at 100k observations). The LIMIT 500
workaround was non-deterministic and dropped valid results.
Using memory_links (pre-bounded to MAX_LINKS_PER_ENTITY=50 at retain time)
is algorithmically identical to the non-observation entity expansion and
keeps graph retrieval at ~2s p50 even at 100k observations.
Also fix migration down_revision (z1u2v3w4x5y6 → d2e3f4a5b6c7) and add
observation generation + fact-type filtering to the recall perf benchmark.
* doc: update cookbook
* fix(cookbook): preserve tag keys during sync, strip local .md links
- Fix extract_tags_from_readme/notebook to return dict[str,str] preserving
sdk/topic keys instead of bare values, preventing topics like
"Customer Service" from being misclassified as SDK
- Add strip_local_md_links() to remove relative .md references that
would cause broken link errors in Docusaurus build
* ci: run test-doc-examples independently without waiting for test-rust-cli
Build the CLI directly in the job instead of downloading the artifact,
so test-doc-examples can start at the beginning in parallel with all other jobs.
* feat: webhook system with task-owned retry, retain.completed event, and UI
- New webhook system: register per-bank webhooks with HMAC signing, configurable
HTTP method/timeout/headers/params (http_config JSONB), and PATCH support
- Webhook deliveries run as async_operations (webhook_delivery type) with
task-owned retry via RetryTaskAt exception and exponential backoff
(60s / 5m / 30m / 2h / 8h, max 6 attempts)
- New retain.completed event fires per-document for both sync and async retain
- Delivery debug info (status code, response body) stored in result_metadata
- Control plane UI: webhooks tab per bank with create/edit/delete and a
deliveries table with cursor pagination and expandable response details
- 28 webhook tests covering HMAC signing, delivery retries, CRUD endpoints,
PATCH update, and retain.completed queuing
- Docs page at developer/api/webhooks documenting event payloads and delivery
- OpenAPI spec and all client SDKs (Python, TypeScript, Rust, Go) regenerated
* fix: update tests for task-owned retry model and guard _webhook_manager attribute
- test_worker.py: test_executor_exception_triggers_retry now raises RetryTaskAt
(plain exceptions are immediate failures in the new system); rename
test_executor_exception_marks_failed_after_max_retries to
test_executor_exception_marks_failed_immediately to reflect new semantics
- test_batch_api.py: remove max_retries kwarg from WorkerPoller constructor
- memory_engine.py: use getattr for _webhook_manager in _fire_retain_webhook
to avoid AttributeError when engine is created without __init__ (tests)
* fix: remove max_retries from benchmark WorkerPoller call
* fix(webhooks): transactional outbox, observations_deleted tracking, sidebar
- Queue webhook delivery rows atomically with the primary operation using the
transactional outbox pattern — prevents lost events on process crash:
- Retain (sync + async): outbox_callback passed into orchestrator.retain_batch
and called inside the DB transaction, replacing the post-commit fire call
- Consolidation: new _mark_operation_completed_and_fire_webhook combines the
status UPDATE and webhook INSERT in one transaction
- Added fire_event_with_conn() to WebhookManager for in-connection delivery
- Track observations_deleted count in consolidation stats and expose it in the
consolidation.completed webhook payload (was always None)
- Add Webhooks page to docs sidebar
- Document at-least-once delivery guarantee with operation_id dedup guidance
* fix(ui): add retain.completed to available webhook event types
* feat(ui): add delete confirmation dialog for webhooks
* fix(webhooks): include operation_id in task_payload so delivery is marked completed
The task_payload JSON was missing the operation_id field, causing execute_task
to see operation_id=None and skip _mark_operation_completed — leaving every
delivery row stuck in 'pending' forever.
Added a test that inserts a real async_operations row and verifies the status
transitions to 'completed' after a successful execute_task call.
* style: fix prettier formatting in webhooks-view
Replace the multi-round-trip while-loop in step 5.5 of recall_async with a
single WHERE chunk_id = ANY($1) query covering all candidate chunk IDs.
Token-budget accounting happens in Python after the single fetch.
Measured on a 97K-unit / 98M-link bank (budget=HIGH, include_chunks,
include_entities):
p50: 1.209s → 0.611s (−49%)
mean: 1.534s → 0.772s (−50%)
p95: 3.366s → 2.316s (−31%)
Also update recall_perf.py benchmark to use Budget.HIGH, include_chunks,
include_entities, and a realistic mixed fact_type distribution.
