fleet-memory/hindsight-api-slim/hindsight_api/engine
Nicolò Boschi fd88c0efa5
feat(retain): delta retain — skip LLM for unchanged chunks on upsert (#701)
* feat(retain): delta retain — skip LLM re-extraction for unchanged chunks on upsert

When upserting a document (same document_id), instead of deleting all
facts and re-extracting from scratch, compare chunk content hashes
and only process changed/new chunks. Unchanged chunks keep their
existing facts, entities, and links.

- Add content_hash column to chunks table (migration b3c4d5e6f7a8)
- Add chunk delta comparison functions in chunk_storage.py
- Add delta_mode to fact_storage.handle_document_tracking (skip full delete)
- Add update_memory_units_tags for propagating tag changes to existing facts
- Refactor orchestrator into _try_delta_retain and _full_retain paths
- Automatic fallback to full retain for pre-migration data or all-changed scenarios
- Fix ty type error in metrics.py (resource module import on Windows)
- 16 new tests covering entities, links, tags, metadata, edge cases

* refactor(retain): deduplicate delta and full retain paths

Extract shared _insert_facts_and_links() and _extract_and_embed()
functions used by both the full retain and delta retain paths.
Remove delta_mode flag from handle_document_tracking — delta path
uses dedicated upsert_document_metadata() instead.

* chore: regenerate clients, openapi spec, and lockfile

* chore: regenerate docs skill
2026-03-26 13:50:55 +01:00
..
consolidation fix: prevent silent memory loss on consolidation LLM failure (#601) 2026-03-17 20:15:33 +01:00
directives feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
parsers feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
providers feat: add 'none' LLM provider for chunk-only storage mode (#691) 2026-03-25 18:01:20 +01:00
reflect feat(reflect): make source facts in search_observations configurable (#688) 2026-03-25 17:54:49 +01:00
retain feat(retain): delta retain — skip LLM for unchanged chunks on upsert (#701) 2026-03-26 13:50:55 +01:00
search fix: control plane UI fixes for recall and data view (#693) 2026-03-25 18:42:57 +01:00
storage feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
__init__.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
cross_encoder.py docs: add config vars for local reranker FP16 and bucket batching (#589) 2026-03-16 17:35:09 +01:00
db_budget.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
db_utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
embeddings.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
entity_resolver.py fix(entity_resolver): prevent _pending_stats/_pending_cooccurrences memory leak (#662) 2026-03-23 16:06:04 +01:00
interface.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
jina_mlx_reranker.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
llm_interface.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
llm_wrapper.py feat: add 'none' LLM provider for chunk-only storage mode (#691) 2026-03-25 18:01:20 +01:00
memory_engine.py feat: add 'none' LLM provider for chunk-only storage mode (#691) 2026-03-25 18:01:20 +01:00
operation_metadata.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
query_analyzer.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
response_models.py fix: return metadata in recall responses (#680) 2026-03-25 11:24:18 +01:00
task_backend.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00
utils.py feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560) 2026-03-13 13:50:03 +01:00