compute_semantic_links_ann created a TEMP TABLE outside any transaction,
then ran a TRUNCATE / COPY / SELECT / DROP sequence as separate statements
on the same asyncpg connection. This is fine against a direct Postgres
connection but fails intermittently when the caller is routed through
PgBouncer in transaction pool mode:
CREATE TEMP TABLE IF NOT EXISTS _ann_seeds (...) -- backend A
TRUNCATE _ann_seeds -- backend B -> FAILS
Temp tables are session-scoped to the backend that created them. In
PgBouncer transaction mode the backend is only pinned to the client for
the duration of an actual transaction, so between standalone statements
the pooler can (and under concurrency, will) rebind the client to a
different backend. When that happens the _ann_seeds table disappears
and the follow-up statement fails with:
relation "_ann_seeds" does not exist
Symptom: ~3% of sync retain calls (2 of 61) failed the Hindsight Cloud
smoke test on a recent hindsight-dev deploy. Async retains are masked
by the 3-attempt retry loop so they usually eventually succeed.
Fix: wrap the CREATE TEMP TABLE -> COPY -> SELECT sequence in a single
`async with conn.transaction():` block, and use ON COMMIT DROP so the
temp table is transaction-scoped and auto-cleaned at commit. Also
switch `SET hnsw.ef_search = 60` to `SET LOCAL` so the tuning is
transaction-scoped and no longer leaks onto the pooled backend for
subsequent recall queries. Drop the now-unnecessary manual TRUNCATE,
explicit DROP TABLE, and RESET hnsw.ef_search.
The function docstring still correctly describes this as running on a
separate connection outside the surrounding write transaction — this
change only adds an inner transaction around the ANN work itself to
keep the temp table visible to PgBouncer.
Tests:
- Add TestComputeSemanticLinksAnnPgBouncerSafety with 5 regression
tests using a mocked connection. These are structural asserts — they
check that the function enters conn.transaction(), uses ON COMMIT DROP,
uses SET LOCAL, and does not reintroduce manual TRUNCATE / DROP /
RESET calls. They would have caught the original bug if they had
existed, and will catch any future reversion.
* feat: add HINDSIGHT_API_DEFAULT_BANK_TEMPLATE env var
Server-level default bank template applied automatically to every
newly-created bank. Holds an inline JSON BankTemplateManifest with the
same shape as the /import endpoint body. Fields set by the template
become per-bank overrides so they take precedence over equivalent
HINDSIGHT_API_* env defaults. The template is applied once on first
creation and never reapplied, so user overrides via PATCH /config are
never clobbered. Malformed manifests are logged and ignored so a broken
server-level setting cannot wedge bank creation.
* chore: regenerate docs skill
* test: update async_retain test mock for renamed bank_profile helper
* feat: add built-in llama.cpp LLM provider for fully local inference
Add `llamacpp` as a new LLM provider that manages a llama-cpp-python server
subprocess. Auto-downloads Gemma 4 E2B Q4_K_M (~3.5 GB) on first use and
runs inference locally via Metal/CUDA with no external services needed.
- New provider: `HINDSIGHT_API_LLM_PROVIDER=llamacpp`
- Singleton server shared across retain/reflect/consolidation
- Configurable: model path, GPU layers, context size, grammar enforcement
- User-extensible via `HINDSIGHT_API_LLAMACPP_EXTRA_ARGS`
- Flash attention + prompt caching enabled by default
- LLM provider cleanup on shutdown (stops subprocess)
- hindsight-embed: `--ui` flag on `daemon start`, removed FORCE_CPU on macOS
- Docs: configuration.md, models.mdx, providers grid updated
* chore: regenerate docs skill and update lockfile for local-llm dep
* feat: add update_mode='append' for retain to concatenate content to existing documents
When retaining with update_mode='append' and a document_id that already exists,
the new content is appended to the existing document text and the full document
is reprocessed. Delta retain automatically skips unchanged chunks, so only the
new content triggers LLM extraction.
- Add update_mode field to MemoryItem (API), RetainContentDict (internal), MCP tools
- Validate that update_mode='append' requires a document_id
- Fetch existing document content and prepend before processing in orchestrator
- Update Python, TypeScript, Go generated clients and top-level client wrappers
- Add tests for append, multiple appends, no-existing-doc, validation, and default replace
* fix: add update_mode field to Rust CLI and client MemoryItem initializers
* chore: regenerate docs skill references for update_mode
Some LLM providers (e.g. Anthropic Haiku) return 1-indexed
content_index values. When only one content item is provided,
this causes KeyError: 1 since the dict only has key 0.
Clamp content_index to the valid range instead of crashing.
Fixes#873
Co-authored-by: easonysliu <easonysliu@tencent.com>
* Fix AttributeError when event_date is None in fact_extraction
`_extract_facts_from_chunk` crashes with `'NoneType' object has no
attribute 'isoformat'` when retaining documents without a timestamp.
Two locations fixed:
- Line 1058: debug log called `event_date.isoformat()` without a None
check
- Line 921: `parse_datetime_flexible()` can return None, so re-check
before calling `.strftime()` / `.isoformat()`
Fixes#874
* Revert unnecessary None guard on line 921
The original `if event_date is not None:` already guards that block.
Only line 1058 needed the fix.
* fix(retain): preserve normalized experience fact types and remove deprecated opinion type
The ExtractedFactType conversion was re-checking for raw "assistant" fact_type
after the parsing layer had already normalized it to "experience". Since
fact_from_llm.fact_type was always "experience" (never "assistant"), the ternary
always fell through to "world", silently losing experience classification.
Also removes the deprecated "opinion" fact type from internal extraction models,
database constraints/indexes (via migration), and dead code paths. The public API
surface (descriptions, response models, backwards-compat filter) is unchanged.
* refactor(retain): drop unused confidence_score column
The confidence_score column was only ever non-null for opinion facts
(which are now removed). It was always written as NULL and never read
back from the database. Remove it from:
- DB model and migration (DROP COLUMN)
- INSERT queries in fact_storage.py
- retain_async/retain_batch_async parameters
- RetainContext/RetainResult extension models
- RetainBatch dataclass
The 3-phase retain pipeline (914ba796) introduced several regressions:
1. **Per-content tags lost** — streaming pipeline used `contents[0].tags`
for ALL chunks, breaking tag-based visibility. Fixed by tracking
chunk-to-content mapping so each chunk uses its source content's tags.
2. **Multi-document batches broken** — batches with per-content
`document_id` values were merged into a single document. Fixed by
grouping by document_id and processing each group independently.
3. **Migration ID collision** — `d6e7f8a9b0c1` was used by both
`drop_documents_metadata` and `case_insensitive_entities_trgm_index`.
Renamed trgm migration to `e8f9a0b1c2d3`, fixed chain, added missing
schema prefix on DROP INDEX.
4. **Graph entity inheritance** — `get_graph_data` queried entities for
observation IDs only, but observations inherit entities from source
memories. Fixed by querying `all_relevant_ids`.
5. **Docstring false positives** — link_utils.py docstrings triggered
the SQL schema safety test's unqualified table reference check.
6. **Config test count** — `retain_chunk_batch_size` added to
`_CONFIGURABLE_FIELDS` without updating the test assertion.
* perf: 3-phase retain pipeline — fix deadlocks, cap temporal links, query-time entity expansion
Major retain pipeline overhaul addressing deadlocks, write amplification,
and TimeoutErrors. Restructures retain into three phases:
Phase 1: Entity resolution on separate connection (read-heavy)
Phase 2: Core write transaction (atomic) — facts, unit_entities, links
Phase 3: Best-effort display data (error-isolated) — entity viz links, stats
Key changes:
- Sorted bulk INSERT FROM unnest() prevents deadlocks
- Temporal links capped to top-20 per unit (95% reduction)
- Batched semantic ANN via temp table + LATERAL
- Query-time entity expansion via unit_entities self-join
- Entity viz links moved to Phase 3 (post-transaction)
- HINDSIGHT_API_RETAIN_MAX_CONCURRENT config (default: 32)
* fix: increase semantic link top_k from 5 to 20
The hardcoded top_k=5 was artificially limiting semantic link creation.
Link expansion retrieval can consume up to budget (50-200) semantic
neighbors per seed set, but each fact only had 5 outgoing edges — making
the bidirectional graph very sparse.
Increasing to 20 gives retrieval 4x more edges to work with. The ANN
probe cost is unchanged (same HNSW traversal per fact, just returning
more rows). INSERT cost is negligible (~14k rows via bulk INSERT).
Also: all 18 TimeoutErrors in the latest benchmark (beam-1m-u20) were
from Gemini LLM calls, zero from the database — confirming the entity
resolution split eliminated DB timeouts entirely.
* perf: move semantic ANN search to Phase 1 to avoid transaction timeouts
The batched LATERAL ANN query (700 HNSW probes) was the last remaining
source of DB TimeoutErrors — all 29 in the latest benchmark were from
create_semantic_links_batch inside the Phase 2 write transaction.
Split semantic link creation into three phases:
- Phase 1 (separate conn, autocommit): ANN search via temp table + LATERAL.
No transaction locks, no contention with concurrent writers.
- Phase 2 (write transaction): within-batch numpy similarities (instant) +
INSERT of both within-batch and Phase 1 ANN results. No DB reads.
- Phase 3 (flush_pending_stats): future hook point for re-checking ANN
results after commit to catch links missed by concurrent batches.
Also adds 7 unit tests for compute_semantic_links_within_batch covering
empty input, identical/orthogonal embeddings, threshold filtering, top_k
cap, and tuple structure validation.
* fix: handle placeholder unit_ids in Phase 1 ANN search (not valid UUIDs)
* test: add Phase 1 ANN cross-batch test + configurable test PG port
- New test_semantic_links_phase1_ann_cross_batch verifies that the Phase 1
ANN search with placeholder unit IDs correctly creates cross-batch
semantic links after remapping to real IDs.
- Test PG port now configurable via HINDSIGHT_TEST_PG_PORT env var
(default: 5556) to avoid conflicts with running benchmark daemons.
* perf: remove retry_with_backoff from retain, set semaphore default to 4
Remove retry_with_backoff from _run_db_work and _run_delta_db_work:
- Deadlocks are prevented by sorted bulk INSERT (no need for retry)
- Transient timeouts are handled by the worker poller's task-level retry
(3 attempts, 60s spacing) which is better than rapid internal retries
that amplify I/O pressure during contention storms
Set HINDSIGHT_API_RETAIN_MAX_CONCURRENT default from 32 to 4:
- The semaphore gates Phase 1 (ANN + entity resolution) + Phase 2 (writes)
- At 4 concurrent, HNSW index I/O is manageable; at 10+ concurrent the
probes saturate disk and cause cascading timeouts
- LLM extraction still runs at full parallelism (semaphore acquired after)
* fix: add fact_type filter to Phase 1 ANN query to use per-bank HNSW indexes
The LATERAL ANN query was falling back to sequential scan + sort (90ms/probe)
because the per-bank HNSW indexes are partial indexes filtered on fact_type.
Without fact_type in the WHERE clause, PostgreSQL couldn't use them.
Fix: iterate over ('world', 'experience') and run one HNSW-indexed ANN per
type. EXPLAIN shows 8ms/probe (was 90ms) — 11x faster.
700 probes × 8ms × 2 types = ~11s total (was ~63s via seq scan).
* fix: scope temporal links by fact_type + add integration tests
Temporal links now filter by fact_type in the LATERAL query — world facts
only link to world facts, experience to experience. This matches how
retrieval filters results and avoids wasted cross-type link rows.
New integration tests:
- test_semantic_ann_uses_hnsw_index: verifies Phase 1 ANN creates
cross-batch semantic links (tests fact_type filter + placeholder remap)
- test_temporal_links_scoped_by_fact_type: verifies world facts get
temporal links to other world facts but NOT to experience facts
* fix: tolerate individual chunk LLM failures instead of failing entire batch
Changed asyncio.gather(*tasks) to asyncio.gather(*tasks, return_exceptions=True)
in both chunk-level and content-level fact extraction. A single chunk timeout
(e.g., Gemini >90s) no longer discards all other successfully extracted facts.
For a 50MB document with 17k chunks, even a 2% chunk failure rate previously
caused 0 completions (entire batch discarded). Now 16,700 facts are extracted
and only the 300 failed chunks are skipped with a warning log.
* fix: batch temporal LATERAL query for large documents (16k+ chunks)
The LATERAL query for temporal links passed all unit_ids at once into
unnest(), causing PostgreSQL timeouts on documents with 16k+ chunks.
Split into batches of 500 units per query to keep each under the
command_timeout.
Also identified: HNSW index creation on shared pg0 instances with
50k+ existing units exceeds the 60s command_timeout. This is a
test infrastructure issue (shared pg0 accumulates data) but also
affects production when creating new banks on large instances.
* feat: streaming chunk batching for large documents (RETAIN_CHUNK_BATCH_SIZE)
Process chunks in mini-batches of N (default 500), committing each batch
to the DB before starting the next. This prevents OOM kills on large
documents (50MB / 17k+ chunks) by keeping only ~500 facts + embeddings
in memory at a time instead of 50k+.
Each mini-batch goes through the full Phase 1 → 2 → 3 pipeline
independently, sharing the same document_id. On recovery (process dies
mid-way), delta retain detects already-committed chunks via content_hash
and skips them — only remaining chunks get re-extracted.
Config: HINDSIGHT_API_RETAIN_CHUNK_BATCH_SIZE (default: 500, 0 to disable)
Per-bank configurable via the hierarchical config system.
Tests:
- test_streaming_chunk_batching_produces_same_facts
- test_streaming_chunk_batching_recovery (delta retain skips committed chunks)
- test_streaming_disabled_for_small_docs
* perf(retain): producer-consumer pipeline + deferred semantic ANN
Replace the sequential streaming loop with a producer-consumer pipeline:
- LLM producer fires concurrent chunk extractions (semaphore-bounded)
- DB consumer drains queue in batches, runs Phase 1+2+3 per batch
- LLM and DB work overlap instead of running sequentially
Defer semantic links to a single final ANN pass after all batches commit:
- Remove within-batch semantic links from Phase 2 (was 2.6s/batch)
- Run parallel ANN (4 connections) after all facts committed
- top_k reduced from 50 to 20 (recall uses at most 20 neighbors)
- Recovery via operation result_metadata checkpoint
Additional optimizations:
- skip_exists_check on temporal/causal link INSERT (saves ~0.5s/batch)
- WHERE EXISTS guard on semantic link INSERT (handles document upsert)
- timeout=300s on ANN queries and bulk INSERT for large banks
- Demote [ANN] debug logs to logger.debug()
- Fix docstring typos (agent_id → bank_id)
- Fix content_index remapping in producer-consumer batches
- Fix delta retain passing contents vs delta_contents
50MB benchmark (mock LLM): 9.2 min (was 23 min) — 2.5x faster.
BEAM 10m benchmark: zero deadlocks, zero DB errors.
* refactor(retain): remove legacy fallback code paths
- Remove process_entities_batch (legacy single-connection entity processing)
- Remove extract_entities_batch_optimized (only caller was the above)
- Remove fallback entity processing inside Phase 2 transaction
- Remove legacy ANN inline fallback in create_semantic_links_batch
- Remove fallback entity_links direct-insert path in Phase 3
- Make resolved_entity_ids/entity_to_unit/unit_to_entity_ids required params
* refactor(retain): replace tuple returns with dataclasses, remove dead code
- Add EntityResolutionResult and Phase1Result dataclasses in types.py
- Replace 4-tuple return from _pre_resolve_phase1 with Phase1Result
- Remove dead `entity_links = []` variables in retain_batch and _try_delta_retain
- Remove unused `confidence_score` parameter from orchestrator.retain_batch
and _retain_batch_async_internal (was accepted but never used)
* fix(entity-resolver): remove LIKE full-scan fallbacks, use index-only trigram matching
The entity resolution query had LIKE '%...' substring conditions that bypassed
the GIN trigram index, causing full sequential scans of the entities table.
On banks with 10k+ entities, this caused TimeoutErrors (observed in BEAM 10m).
Changes:
- Remove LIKE fallbacks, use trigram % operator only (GIN index-based)
- Lower similarity threshold from 0.3 to 0.15 to catch substring relationships
- Use LOWER() on both sides for case-insensitive matching
- Migration: recreate GIN trigram index on LOWER(canonical_name)
* fix: remove schema prefix from index names in trigram migration
* fix(delta-retain): use same chunk_size as streaming path (3000 vs 120000)
_chunk_contents_for_delta defaulted to chunk_size=120000 while the streaming
path used 3000. On retry, delta re-chunked the document with different
boundaries, found 0 matching chunks, and fell through to full re-extraction.
This wasted all LLM calls on already-committed chunks.
Fix: use the same default (3000) so chunk hashes match on recovery.
* fix(retain): persist generated document_id in operation metadata for retry recovery
When no document_id is provided, retain generates a UUID. On retry, a new UUID
was generated, making delta retain and streaming chunk-hash recovery unable to
find previously committed chunks. All LLM extraction was wasted on retry.
Fix: resolve document_id early in retain_batch (before delta), persist it to
operation result_metadata, and recover it on retry. Both delta and streaming
paths now see the same document_id across attempts.
* refactor(retain): unify into single streaming pipeline, remove non-streaming path
All retains now go through the producer-consumer streaming pipeline,
regardless of document size. Small documents are processed as a single batch.
This eliminates the maintenance burden of two separate code paths.
Also fix document upsert: compare content hash to distinguish recovery
(same content, partially committed) from update (different content, needs
cascade-delete). Previously, existing chunks always triggered recovery mode.
* refactor(retain): remove dead code, replace raw dicts with Phase3Context dataclass
- Remove dead _handle_zero_facts_documents (no callers after path unification)
- Remove unused imports: defaultdict, EntityLink
- Replace raw dict phase3_context with typed Phase3Context dataclass
- Update _build_and_insert_entity_links_phase3 to use typed parameter
* fix(db): respect vector extension config in per-bank index migration
Migration d5e6f7a8b9c0 hardcoded HNSW when creating per-bank partial
vector indexes, ignoring HINDSIGHT_API_VECTOR_EXTENSION. This caused
banks migrated from pre-v0.4.18 to get HNSW indexes even when
pgvectorscale (DiskANN) or vchord was configured.
- Fix the original migration to read the vector extension config
- Add migration a4b5c6d7e8f9 to detect and recreate mismatched indexes
(skipped entirely when extension is pgvector, since those are correct)
* chore: regenerate openapi.json for v0.4.22 version bump
* feat: expose document_metadata in API and control plane
Add document_metadata (sourced from retain_params.metadata) to both
list and get document endpoints. Display it in the control plane
documents table and detail panel. Drop the unused metadata column
from the documents table (was always stored as empty {}).
* fix: code review fixes for document_metadata feature
- Remove unnecessary `import json as _json` (json already imported at module level)
- Simplify redundant truthiness checks in retain_params parsing
- Regenerate OpenAPI spec and client SDKs (Python, TypeScript, Go)
- Add tests for document_metadata in get_document and list_documents
* feat(ui): improve documents table and detail panel
- Relative timestamps with full date on hover
- Remove context column from table
- Metadata shown as k=v badges (blue, like tags)
- Size in bytes instead of chars
- Document IDs wrap instead of truncating
- Detail panel wider (560px)
- Retain params: context, event_date, metadata badges
* fix(engine): classify first-person agent experiences as 'experience' fact type
The extraction prompt defined "assistant" too narrowly as only "interactions
with assistant (requests, recommendations)", causing the LLM to classify
first-person agent actions (code changes, debugging, discoveries) as "world".
Broadened the fact_type definition in the prompt and Pydantic model descriptions
to cover all first-person actions, experiences, and observations by the speaker.
* style: fix line length in fact_extraction.py
create_bank_hnsw_indexes() hardcoded USING hnsw regardless of the configured
vector extension, causing "column cannot have more than 2000 dimensions for
hnsw index" when using pgvectorscale or vchord with high-dimensional embeddings.
Now reads get_config().vector_extension and uses the appropriate index type:
- pgvector → USING hnsw
- pgvectorscale → USING diskann
- vchord → USING vchordrq
Closes#738
Verbose mode was the only extraction mode that skipped injecting the
retain_mission FOCUS section into its prompt template. Users who set a
retain_mission got no filtering when using verbose mode.
The stats endpoint JOINs memory_links to memory_units just to filter
by bank_id. With 8.2M+ links per bank this takes 18+ seconds, and
the control plane polls every 10s — perpetually blocking the server.
Add bank_id column directly to memory_links so the query can filter
on ml.bank_id instead of mu.bank_id, letting Postgres push the filter
down before the JOIN.
- Migration: add bank_id TEXT NOT NULL, backfill from memory_units
- All 4 INSERT paths (temporal, semantic, entity, causal) now write bank_id
- Stats query filters on ml.bank_id instead of mu.bank_id
* docs(python-client): improve pydoc strings for async-first usage and low-level API access
- Class docstring now clearly documents async-first pattern: a* methods
preferred, sync wrappers for scripts/REPLs only
- Every sync method docstring points to its async counterpart
- Every async method docstring says "preferred"
- Expose 10 low-level API properties (documents, entities, operations,
webhooks, monitoring, etc.) so agents/users can discover the full API
surface without guessing at _-prefixed internals
- Add missing API parameters: tag_groups (recall/reflect), fact_types,
exclude_mental_models, exclude_mental_model_ids (reflect),
observation_scopes/strategy (retain items), background (create_bank)
- Fix areflect missing include_facts param that sync reflect already had
- Sync recall/reflect now delegate to async counterparts (no logic duplication)
* style(retain): format long function call arguments one-per-line
* feat(retain): delta retain — skip LLM re-extraction for unchanged chunks on upsert
When upserting a document (same document_id), instead of deleting all
facts and re-extracting from scratch, compare chunk content hashes
and only process changed/new chunks. Unchanged chunks keep their
existing facts, entities, and links.
- Add content_hash column to chunks table (migration b3c4d5e6f7a8)
- Add chunk delta comparison functions in chunk_storage.py
- Add delta_mode to fact_storage.handle_document_tracking (skip full delete)
- Add update_memory_units_tags for propagating tag changes to existing facts
- Refactor orchestrator into _try_delta_retain and _full_retain paths
- Automatic fallback to full retain for pre-migration data or all-changed scenarios
- Fix ty type error in metrics.py (resource module import on Windows)
- 16 new tests covering entities, links, tags, metadata, edge cases
* refactor(retain): deduplicate delta and full retain paths
Extract shared _insert_facts_and_links() and _extract_and_embed()
functions used by both the full retain and delta retain paths.
Remove delta_mode flag from handle_document_tracking — delta path
uses dedicated upsert_document_metadata() instead.
* chore: regenerate clients, openapi spec, and lockfile
* chore: regenerate docs skill
* feat: Windows native support — run Hindsight without Docker on Windows
Four compatibility fixes that allow Hindsight to run natively on Windows
with an external PostgreSQL + pgvector installation:
1. **pyproject.toml**: Conditional event loop dependency
- `winloop` on Windows (sys_platform == 'win32')
- `uvloop` on Linux/macOS (sys_platform != 'win32')
2. **main.py**: winloop integration via `winloop.install()`
- Patches asyncio event loop policy globally before uvicorn starts
- uvicorn sees "asyncio" but runs winloop underneath (same perf as uvloop)
- Falls back to default asyncio if winloop unavailable
3. **metrics.py**: Guard `resource` module import
- `resource` is Unix-only (getrusage, getrlimit)
- Conditional import with None fallback
- Skip process metrics collection on Windows
4. **fact_storage.py**: Cross-platform strftime
- `%-d` (no-padding day) is glibc-only, fails on Windows
- Replaced with `%d` + `.replace(" 0", " ")` for same output
## Windows Setup Guide
### Prerequisites
- Python 3.11+
- PostgreSQL 17 with pgvector extension
- Ollama (for local embeddings) or external embedding provider
### Install PostgreSQL + pgvector on Windows
```bash
winget install PostgreSQL.PostgreSQL.17
# Build pgvector from source (requires Visual Studio Build Tools)
git clone https://github.com/pgvector/pgvector.git
# In x64 Native Tools Command Prompt:
set PGROOT=C:\Program Files\PostgreSQL\17
nmake /F Makefile.win
nmake /F Makefile.win install
# Enable extension
psql -U postgres -d hindsight -c "CREATE EXTENSION IF NOT EXISTS vector;"
```
### Install and Run Hindsight
```bash
pip install -e ".[embedded-db]"
# Set environment variables
set HINDSIGHT_API_LLM_PROVIDER=openai
set HINDSIGHT_API_LLM_API_KEY=your-api-key
set HINDSIGHT_API_LLM_BASE_URL=https://your-llm-endpoint/v1
set HINDSIGHT_API_LLM_MODEL=your-model
set HINDSIGHT_API_DATABASE_URL=postgresql://postgres@localhost:5432/hindsight
set HINDSIGHT_API_EMBEDDING_PROVIDER=ollama
set HINDSIGHT_API_PORT=8889
hindsight-api
```
Data persists in PostgreSQL on your local disk — survives reboots,
updates, and anything that would wipe a Docker volume.
Tested on Windows 11 with PostgreSQL 17.9, pgvector 0.8.2,
Python 3.11, RTX 5080 (CUDA embeddings + reranking).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: handle strftime ValueError on Windows in fact_storage
The strftime call on occurred_start/occurred_end can raise ValueError
on Windows when the datetime object has unexpected format properties.
Wrap in try/except to gracefully skip date signal rather than crash
the entire retain batch.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(entity_resolver): prevent _pending_stats/_pending_cooccurrences memory leak
Add discard_pending_stats() to EntityResolver to clean up both pending dicts
for the current task key. Call it at the start of each _run_db_work attempt so
that exceptions between accumulation and flush_pending_stats() — including
deadlock retries — never leave stale entries keyed by recycled task IDs.
Fixes#660
* test(entity_resolver): add unit tests for discard_pending_stats()
Covers: clears both dicts for current task, is idempotent when empty,
and does not touch entries belonging to other task keys.
No database required — purely in-memory logic.
The Pydantic model extraction paths (batch API and parallel extraction) used
fact_from_llm.fact_type directly, bypassing the \"assistant\" → \"experience\"
conversion and causing DB CHECK constraint violations.
Unified the conversion logic across all paths:
- \"assistant\" → \"experience\"
- \"world\" → \"world\"
- anything else: fall back to fact_kind (\"assistant\" → \"experience\"), else \"world\"
* feat(retain): add verbatim extraction mode
Adds retain_extraction_mode="verbatim" that stores each chunk as-is
without LLM summarization. The LLM still runs to extract entities,
temporal info, and location for full indexability — only the fact text
is replaced with the original chunk content (one memory per chunk).
Useful for RAG-style indexing and benchmarks where original text
must be preserved in memory.
- Add "verbatim" to RETAIN_EXTRACTION_MODES in config.py
- Add VERBATIM_FACT_EXTRACTION_PROMPT with instructions to preserve text
- Add _collapse_to_verbatim() post-processing to enforce 1 fact/chunk
- Expose in bank config UI dropdown with updated description
- Update configuration.md docs with verbatim mode description
- Add unit test for _collapse_to_verbatim and integration test via LLM
- Fix pre-existing main.py CLI override missing new reranker fields
- Fix pre-existing cross_encoder.py ty type error via setattr
* refactor(retain): verbatim mode skips 'what' field entirely
Instead of asking the LLM to echo the chunk text back into 'what' and
then discarding it, verbatim mode now uses a dedicated schema
(VerbatimExtractedFact) that omits the 'what' field altogether.
The LLM only returns metadata (entities, temporal info, location, who),
saving output tokens and avoiding any risk of paraphrasing before the
backfill.
- Add VerbatimExtractedFact / VerbatimFactExtractionResponse models
- Verbatim mode skips causal-relations section (nothing to relate causally)
- _extract_facts_from_chunk: allow missing 'what' in verbatim mode,
set combined_text="" (backfilled by _collapse_to_verbatim)
- Update verbatim prompt to say DO NOT include 'what'
* feat(retain): add index_only extraction mode
Zero-LLM retain mode: chunks are stored as-is with no LLM call, no
entity extraction, and no temporal indexing. Embeddings still run for
semantic search. User-provided entities via RetainContent.entities
are the sole source of entity data.
Early return placed before the batch-API check so no LLM queue or
concurrency locks are acquired.
- Add "index_only" to RETAIN_EXTRACTION_MODES
- Add _extract_facts_index_only() with pure Python chunking path
- Add to UI dropdown and update description
- Update configuration.md with index_only docs and table entry
- Add unit test asserting zero token usage and exact text preservation
* feat(retain): add named retain strategies
Allows mixing extraction modes in a single bank via named strategies.
Each strategy is a set of hierarchical config overrides (extraction_mode,
chunk_size, entity_labels, entities_allow_free_form, etc.) applied on
top of the resolved bank config at retain time.
- retain_strategies: dict of strategy_name → config overrides (bank config)
- retain_default_strategy: default strategy when none specified (bank config)
- strategy field on /retain request: per-call override
- apply_strategy() in config_resolver applies overrides via dataclasses.replace()
- strategy propagates through retain_batch_async → _retain_batch_async_internal
and through the async worker task payload
- Any hierarchical field is overridable per strategy, including entity_labels
and entities_allow_free_form
- Docs updated with strategy configuration example and RRF fairness note
- Unit test for apply_strategy covering overrides, unknown strategy, and
non-hierarchical field filtering
* feat(retain): add per-item strategy and strategy tests
- Add `strategy` field to `MemoryItem` so individual items in a retain
request can override the request-level strategy
- Add `strategy` field to `FileRetainMetadata` for per-file strategy
override in file retain requests
- Group memory items by effective strategy in `api_retain`; each group
is processed as a separate batch, results are aggregated
- Thread strategy through `submit_async_file_retain` →
`_handle_file_convert_retain` → retain task payload
- Add `operation_ids` to `RetainResponse` for async requests with
mixed per-item strategies
- Add `test_strategy_overrides_extraction_mode_for_index_only`: unit
test verifying a named strategy with index_only bypasses the LLM
- Add `test_retain_request_per_item_strategy_field`: unit test for
per-item strategy grouping logic
* feat(ui): add retain strategies and default strategy to bank config UI
- Add StrategiesEditor component: per-strategy cards with name input and
JSON overrides textarea; supports add/remove; validates JSON inline
- Add Default Strategy text input (retain_default_strategy)
- Update RetainEdits type and retainSlice() to include both new fields
- Regenerate OpenAPI spec (retain_strategies, retain_default_strategy,
per-item strategy on MemoryItem/FileRetainMetadata, operation_ids on
RetainResponse)
* refactor(ui): move retain strategies into its own dedicated config section
* feat(ui): improve retain strategies UX and add strategy to document dialog
- Strategy form now includes entity section (free form toggle + entity labels editor)
- Default strategy selector moved outside tab panel, above strategy chips
- Strategy tabs redesigned with underline indicator style for clarity
- Remove strategy confirms with AlertDialog
- Fix tab re-render bug when typing strategy name (skipSyncRef)
- Add strategy field to Add New Document dialog (text + per-file for uploads)
- File upload collapsible uses same Document/Tags/Source tabbed layout
- API: validate empty strategy names in config_resolver
- api.ts: add strategy field to retain and uploadFiles types
* fix: forward strategy through HTTP layer and SDK; add integration test
- route.ts: extract and forward `strategy` from request body to retainBatch
- TypeScript SDK: accept and forward `strategy` in retainBatch options and per-item
- config_resolver.py: validate empty strategy name keys on update
- bank-config-view.tsx: merge entity fields into RetainStrategyForm, redesign strategy tabs with underline style, add confirmation dialog for removal, fix tab-reset-on-typing with skipSyncRef, move default strategy selector outside panel
- bank-selector.tsx: add strategy field to Add Document dialog (per-file in tabbed collapsible)
- test_retain.py: add end-to-end integration test verifying named strategy application (index_only = 0 LLM tokens)
* fix: regenerate TypeScript client with strategy field in RetainRequest/MemoryItem
- Regenerate OpenAPI spec to include strategy field in RetainRequest and MemoryItem
- Regenerate TypeScript client from updated spec
- Add strategy to MemoryItemInput interface
- Remove (item as any) cast now that strategy is properly typed
* rename: index_only extraction mode → chunks
* remove top-level strategy from RetainRequest; strategy is per-item only
* fix(clients): update Go and Python generated clients with strategy/operation_ids fields
* fix(ci): update hierarchical field count, add strategy to Rust MemoryItem initializers
* fix(go-client): minimal targeted YAML updates for strategy/operation_ids fields
* feat: introduce hindsight-api-slim and hindsight-all-slim packages
Closes#552
- Move all source code from hindsight-api/ to new hindsight-api-slim/
- hindsight-api-slim has heavy ML deps (torch, sentence-transformers,
transformers, einops, flashrank, mlx, mlx-lm, safetensors) and
pg0-embedded as optional extras: [local-ml], [embedded-db], [all]
- hindsight-api becomes a zero-code meta-package depending on
hindsight-api-slim[all] for full backward compatibility
- Add hindsight-all-slim meta-package: hindsight-api-slim + client + embed
- hindsight-all updated to depend on hindsight-api-slim[all]
- pg0.py: lazy-import pg0 with clear ImportError pointing to [embedded-db]
- Dockerfile: replace sed hack with proper uv sync --extra flags
- Update release.yml, test.yml, lint.sh, release.sh, CLAUDE.md and
all path references throughout the repo
* refactor: rename hindsight/ directory to hindsight-all/
* docs: document hindsight-api-slim and hindsight-all-slim package variants
Add package variants table and extras explanation to installation.md
* docs: remove emojis from installation.md, use professional tone
* docs: link Docker slim variant to pip package variants section
* docs: consolidate Docker image variants into single table
* ci: fix working-directory paths after package restructure
- Replace all hindsight-api → hindsight-api-slim in test.yml
- Replace hindsight → hindsight-all in test.yml
- Add --extra embedded-db to test-embed API install step
* ci: add local-ml and embedded-db extras to API sync steps
These extras were previously implicit in the old hindsight-api package
(which bundled everything). Now that hindsight-api-slim uses optional
extras, we must explicitly request local-ml and embedded-db in CI.
* ci: add API install step with embedded-db to test-embed smoke test
The smoke test starts hindsight-api as a daemon, which requires pg0-embedded.
Add a dedicated install step for hindsight-api-slim with embedded-db extra
so the daemon can start successfully.
* ci: remove --no-install-project when using optional extras
When --no-install-project is combined with --extra, the optional deps
are not installed because extras require the project to be active.
Remove --no-install-project from steps that need local-ml or embedded-db.
* ci: fix ordering of uv sync steps to preserve optional extras
When uv sync runs for a different workspace member, it removes optional
extras installed for other members. Fix by always running extra-requiring
API sync last, after other workspace member syncs.
Also remove --no-install-project from embedded-db sync in test-embed,
as --no-install-project prevents optional extras from being active.
* ci: add local-ml extra to test-embed API install for smoke test
The smoke test starts the full API server which needs sentence-transformers
for local embeddings (default provider). Add local-ml extra to the install.
* ci: simplify extras with --all-extras and add slim pip smoke test
- Replace explicit --extra local-ml --extra embedded-db with --all-extras
for cleaner, more maintainable sync steps
- Add test-pip-slim job: tests hindsight-api-slim[embedded-db] without
local ML models, using Cohere for embeddings/reranking (mirrors Docker
slim smoke test approach)
* ci: simplify slim smoke test to health check only (mirrors Docker test)
