fleet-memory/hindsight-api-slim/hindsight_api/engine/retain
Nicolò Boschi fd88c0efa5
feat(retain): delta retain — skip LLM for unchanged chunks on upsert (#701)
* feat(retain): delta retain — skip LLM re-extraction for unchanged chunks on upsert

When upserting a document (same document_id), instead of deleting all
facts and re-extracting from scratch, compare chunk content hashes
and only process changed/new chunks. Unchanged chunks keep their
existing facts, entities, and links.

- Add content_hash column to chunks table (migration b3c4d5e6f7a8)
- Add chunk delta comparison functions in chunk_storage.py
- Add delta_mode to fact_storage.handle_document_tracking (skip full delete)
- Add update_memory_units_tags for propagating tag changes to existing facts
- Refactor orchestrator into _try_delta_retain and _full_retain paths
- Automatic fallback to full retain for pre-migration data or all-changed scenarios
- Fix ty type error in metrics.py (resource module import on Windows)
- 16 new tests covering entities, links, tags, metadata, edge cases

* refactor(retain): deduplicate delta and full retain paths

Extract shared _insert_facts_and_links() and _extract_and_embed()
functions used by both the full retain and delta retain paths.
Remove delta_mode flag from handle_document_tracking — delta path
uses dedicated upsert_document_metadata() instead.

* chore: regenerate clients, openapi spec, and lockfile

* chore: regenerate docs skill
2026-03-26 13:50:55 +01:00
..
__init__.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
bank_utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
chunk_storage.py feat(retain): delta retain — skip LLM for unchanged chunks on upsert (#701) 2026-03-26 13:50:55 +01:00
embedding_processing.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
embedding_utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
entity_labels.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
entity_processing.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
fact_extraction.py fix: correctly map LLM fact_type \"assistant\" to \"experience\" for DB storage (#609) 2026-03-19 10:32:08 +01:00
fact_storage.py feat(retain): delta retain — skip LLM for unchanged chunks on upsert (#701) 2026-03-26 13:50:55 +01:00
link_creation.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
link_utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
orchestrator.py feat(retain): delta retain — skip LLM for unchanged chunks on upsert (#701) 2026-03-26 13:50:55 +01:00
types.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00