fleet-memory/hindsight-api/hindsight_api/alembic/versions
Nicolò Boschi abbf874d84
feat: webhook system with retain.completed event, UI, and docs (#487)
* doc: update cookbook

* fix(cookbook): preserve tag keys during sync, strip local .md links

- Fix extract_tags_from_readme/notebook to return dict[str,str] preserving
  sdk/topic keys instead of bare values, preventing topics like
  "Customer Service" from being misclassified as SDK
- Add strip_local_md_links() to remove relative .md references that
  would cause broken link errors in Docusaurus build

* ci: run test-doc-examples independently without waiting for test-rust-cli

Build the CLI directly in the job instead of downloading the artifact,
so test-doc-examples can start at the beginning in parallel with all other jobs.

* feat: webhook system with task-owned retry, retain.completed event, and UI

- New webhook system: register per-bank webhooks with HMAC signing, configurable
  HTTP method/timeout/headers/params (http_config JSONB), and PATCH support
- Webhook deliveries run as async_operations (webhook_delivery type) with
  task-owned retry via RetryTaskAt exception and exponential backoff
  (60s / 5m / 30m / 2h / 8h, max 6 attempts)
- New retain.completed event fires per-document for both sync and async retain
- Delivery debug info (status code, response body) stored in result_metadata
- Control plane UI: webhooks tab per bank with create/edit/delete and a
  deliveries table with cursor pagination and expandable response details
- 28 webhook tests covering HMAC signing, delivery retries, CRUD endpoints,
  PATCH update, and retain.completed queuing
- Docs page at developer/api/webhooks documenting event payloads and delivery
- OpenAPI spec and all client SDKs (Python, TypeScript, Rust, Go) regenerated

* fix: update tests for task-owned retry model and guard _webhook_manager attribute

- test_worker.py: test_executor_exception_triggers_retry now raises RetryTaskAt
  (plain exceptions are immediate failures in the new system); rename
  test_executor_exception_marks_failed_after_max_retries to
  test_executor_exception_marks_failed_immediately to reflect new semantics
- test_batch_api.py: remove max_retries kwarg from WorkerPoller constructor
- memory_engine.py: use getattr for _webhook_manager in _fire_retain_webhook
  to avoid AttributeError when engine is created without __init__ (tests)

* fix: remove max_retries from benchmark WorkerPoller call

* fix(webhooks): transactional outbox, observations_deleted tracking, sidebar

- Queue webhook delivery rows atomically with the primary operation using the
  transactional outbox pattern — prevents lost events on process crash:
  - Retain (sync + async): outbox_callback passed into orchestrator.retain_batch
    and called inside the DB transaction, replacing the post-commit fire call
  - Consolidation: new _mark_operation_completed_and_fire_webhook combines the
    status UPDATE and webhook INSERT in one transaction
  - Added fire_event_with_conn() to WebhookManager for in-connection delivery

- Track observations_deleted count in consolidation stats and expose it in the
  consolidation.completed webhook payload (was always None)

- Add Webhooks page to docs sidebar

- Document at-least-once delivery guarantee with operation_id dedup guidance

* fix(ui): add retain.completed to available webhook event types

* feat(ui): add delete confirmation dialog for webhooks

* fix(webhooks): include operation_id in task_payload so delivery is marked completed

The task_payload JSON was missing the operation_id field, causing execute_task
to see operation_id=None and skip _mark_operation_completed — leaving every
delivery row stuck in 'pending' forever.

Added a test that inserts a real async_operations row and verifies the status
transitions to 'completed' after a successful execute_task call.

* style: fix prettier formatting in webhooks-view
2026-03-04 14:17:01 +01:00
..
5a366d414dce_initial_schema.py feat: support azure pg_diskann (#381) 2026-02-16 16:08:54 +01:00
a1b2c3d4e5f6_add_file_storage_table.py feat: accept pdf, images and office files (#390) 2026-02-17 18:15:03 +01:00
a2b3c4d5e6f7_add_text_signals_column.py feat: entity labels — optional, free_values, multi_value, UI polish (#450) 2026-03-02 13:05:25 +01:00
aa2b3c4d5e6f_nullable_event_date.py feat: support timestamp="unset" to retain content without a date (#465) 2026-03-02 12:03:23 +01:00
b3c4d5e6f7g8_add_temporal_date_indexes.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
b4c5d6e7f8a9_backfill_observation_scopes.py feat: entity labels — optional, free_values, multi_value, UI polish (#450) 2026-03-02 13:05:25 +01:00
b7c4d8e9f1a2_add_chunks_table.py misc: add mcp integration tests and increase test coverage (#98) 2026-01-05 11:16:55 +01:00
c1a2b3d4e5f6_enable_pg_trgm_and_entities_trgm_index.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
c8e5f2a3b4d1_add_retain_params_to_documents.py misc: add mcp integration tests and increase test coverage (#98) 2026-01-05 11:16:55 +01:00
d2e3f4a5b6c7_add_memory_links_expansion_indexes.py fix(performance): improve recall and retain performance on large banks (#469) 2026-03-03 13:35:22 +01:00
d9f6a3b4c5e2_rename_bank_to_interactions.py feat: extensions (#54) 2025-12-22 11:05:23 +01:00
e0a1b2c3d4e5_disposition_to_3_traits.py misc: add mcp integration tests and increase test coverage (#98) 2026-01-05 11:16:55 +01:00
e4f5a6b7c8d9_add_webhooks_tables.py feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
f1a2b3c4d5e6_add_memory_links_composite_index.py fix: improve mpfp retrieval (#146) 2026-01-12 18:58:05 +01:00
f7g8h9i0j1k2_add_webhook_http_config.py feat: webhook system with retain.completed event, UI, and docs (#487) 2026-03-04 14:17:01 +01:00
g2a3b4c5d6e7_add_tags_column.py feat: add memory tags (#152) 2026-01-13 18:28:40 +01:00
h3c4d5e6f7g8_mental_models_v4.py feat: introduce mental models (#132) 2026-01-16 11:16:41 +01:00
i4d5e6f7g8h9_delete_opinions.py feat: introduce mental models (#132) 2026-01-16 11:16:41 +01:00
j5e6f7g8h9i0_mental_model_versions.py feat: improve mental model refresh and add directives (#166) 2026-01-19 11:38:35 +01:00
k6f7g8h9i0j1_add_directive_subtype.py feat: improve mental model refresh and add directives (#166) 2026-01-19 11:38:35 +01:00
l7g8h9i0j1k2_add_worker_columns.py feat: new 'worker' service (#176) 2026-01-20 10:17:56 +01:00
m8h9i0j1k2l3_mental_model_id_to_text.py feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
n9i0j1k2l3m4_learnings_and_pinned_reflections.py feat: support azure pg_diskann (#381) 2026-02-16 16:08:54 +01:00
o0j1k2l3m4n5_migrate_mental_models_data.py feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
p1k2l3m4n5o6_new_knowledge_architecture.py feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
q2l3m4n5o6p7_fix_mental_model_fact_type.py feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
r3m4n5o6p7q8_add_reflect_response_to_reflections.py feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
rename_personality_to_disposition.py misc: add mcp integration tests and increase test coverage (#98) 2026-01-05 11:16:55 +01:00
s4n5o6p7q8r9_add_consolidated_at_to_memory_units.py fix: improve mental model consolidation (#197) 2026-01-26 09:54:25 +01:00
t5o6p7q8r9s0_rename_mental_models_to_observations.py chore: internal renames (#204) 2026-01-27 09:53:28 +01:00
u6p7q8r9s0t1_mental_models_text_id.py fix: misc fixes for observations and mental models (#209) 2026-01-27 15:37:57 +01:00
v7q8r9s0t1u2_add_max_tokens_to_mental_models.py fix: misc fixes for observations and mental models (#209) 2026-01-27 15:37:57 +01:00
w8r9s0t1u2v3_fix_mental_models_pk_isolation.py feat: support markdown in reflect and mental models (#307) 2026-02-06 10:49:13 +01:00
x9s0t1u2v3w4_add_bank_config_column.py feat: implement hierarchical configuration (system, tenant, bank) (#329) 2026-02-12 13:14:57 +01:00
y0t1u2v3w4x5_add_result_metadata_gin_index.py fix: improve async batch retain with large payloads (#366) 2026-02-16 12:51:42 +01:00
z1u2v3w4x5y6_add_observation_tags_to_memory_units.py feat: observation_scopes field to drive observations granularity (#447) 2026-02-28 10:46:21 +01:00