Commit graph

643 commits

Author SHA1 Message Date
dependabot[bot]
7eabe5e168
chore(deps): bump actions/checkout from 4 to 6 (#581)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-16 12:01:07 +01:00
dependabot[bot]
33565a8236
chore(deps): bump actions/download-artifact from 4 to 8 (#582)
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 4 to 8.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v4...v8)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-16 12:00:56 +01:00
dependabot[bot]
51f05365a1
chore(deps): bump actions/setup-python from 5 to 6 (#583)
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-16 12:00:46 +01:00
Salman Chishti
1e6cb15e99
Upgrade GitHub Actions to latest versions (#576)
Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>
2026-03-14 12:22:05 +01:00
Ben
bd6348aa08
blog: add disposition-aware agents post (#566)
* blog: add disposition-aware agents post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:53:27 -04:00
DK09876
836fd81e19
fix: inject Accept header in MCP middleware to prevent 406 errors (#571)
Some MCP clients (e.g., Claude Code) don't send an Accept header,
causing the MCP SDK to reject requests with 406 Not Acceptable. The
middleware now ensures Accept includes application/json and
text/event-stream when missing.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 21:33:30 +01:00
陈家名
32b00cea4f
docs: improve type hints and documentation in client_wrapper (#570)
- Add comprehensive docstrings to all API namespace classes
- Add return type annotations (Any) to all methods
- Add detailed Args and Returns sections to method docstrings
- Improve HindsightClient class docstring with Attributes section
- Add type annotations to __init__ parameters

Co-authored-by: 陈家名 <chenjiaming@icloud.com>
2026-03-13 17:42:54 +01:00
Nicolò Boschi
21f9f46ca3
fix: support gemini-3.1-flash-lite-preview by preserving thought_signature in tool calls (#568)
Gemini 3.1+ thinking models include a thought_signature field in functionCall
parts. When reconstructing conversation history for subsequent turns, this
signature must be preserved or the API returns 400 INVALID_ARGUMENT.

- Add optional thought_signature field to LLMToolCall
- Capture thought_signature from Gemini response parts
- Pass thought_signature back when reconstructing multi-turn history
- Add gemini-3.1-flash-lite-preview to the LLM provider test matrix
2026-03-13 16:43:01 +01:00
Nicolò Boschi
c7db770281
doc: add 0.4.18 release blog post (#567)
* doc: add 0.4.18 release blog post

* doc: include changelog and blog image for 0.4.18
2026-03-13 16:00:59 +01:00
Nicolò Boschi
5fdb0e863f Release v0.4.18
- Update version to 0.4.18 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-litellm, hindsight-crewai, hindsight-pydantic-ai, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- OpenClaw integration: hindsight-integrations/openclaw
- AI SDK integration: hindsight-integrations/ai-sdk
- Chat SDK integration: hindsight-integrations/chat
- Helm chart
- Sync documentation to version-0.4
2026-03-13 15:21:03 +01:00
Nicolò Boschi
4a69a422a0 doc: fix build 2026-03-13 15:19:56 +01:00
Nicolò Boschi
26472df166 doc: improve link icons and structure 2026-03-13 15:09:13 +01:00
Nicolò Boschi
5de793eec7
feat: compound tag filtering via tag_groups (#562)
* feat: add compound tag filtering via tag_groups

Adds tag_groups to RecallRequest and ReflectRequest to express arbitrary
boolean tag predicates: leaf {tags, match}, and/or/not compounds.
Top-level groups are AND-ed. Existing tags/tags_match unchanged.

Examples:
  Step filter AND user scope:
    tag_groups: [{tags: ["step:5","step:8"], match: "any_strict"},
                 {tags: ["user:alice"], match: "all_strict"}]
  Exclusion:
    tag_groups: [{tags: ["user:alice"], match: "all_strict"},
                 {not: {tags: ["archived"], match: "any_strict"}}]

- Recursive SQL builder (build_tag_groups_where_clause) threads through
  all 4 retrieval strategies (semantic/BM25, temporal, graph, MPFP)
- Python-side filter (filter_results_by_tag_groups) for post-traversal
- 22 new unit tests
- OpenAPI spec + all clients regenerated (Rust, Python, TypeScript, Go)

* fix: add tag_groups: None to Rust CLI struct initializers

* fix: add tag_groups: None to Rust client test RecallRequest initializer

* feat: reject tags+tag_groups together, add tag_groups integration tests

- Add model_validator to RecallRequest and ReflectRequest that returns 422
  when both `tags` and `tag_groups` are set (mutually exclusive)
- Add 5 integration tests for tag_groups compound filtering:
  * validation: 422 when both fields are set
  * AND filter: two leaf groups (step scope AND user scope)
  * OR compound: user:alice OR user:bob
  * NOT compound: user:alice AND NOT archived
  * Nested: user:alice AND (step:5 OR step:8)

* ci: trigger CI run
2026-03-13 14:30:11 +01:00
Nicolò Boschi
06200f1752
docs: revamp sidebar with icon grids and language support (#563)
* docs: revamp sidebar with icon grid components and language support

- Merge Clients and Integrations sections into the developer sidebar
  (removed top-level SDKs navbar item)
- Reorder sidebar: Architecture → API → Clients → Integrations → Hosting
- Unify icon system using react-icons (LuXxx/SiXxx) via customProps.icon
- Add uppercase section titles with increased spacing and reduced indentation
- Rename Node.js → "JavaScript / TypeScript" with TypeScript icon
- Add reusable IconGrid and SupportedGrids components (ClientsGrid,
  IntegrationsGrid, LLMProvidersGrid)
- Use grids in FAQ, Models, Overview, and Quick Start pages
- Convert developer/index.md, models.md, faq.md to MDX for JSX support

* fix: use inline style for label color to prevent link color inheritance

* fix: label visibility and rename JavaScript/TypeScript to TypeScript

* feat: add HTTP client to grid and OpenAI Compatible to LLM providers grid
2026-03-13 14:12:42 +01:00
dependabot[bot]
d4131f88fa
chore(deps): bump actions/setup-node from 4 to 6 (#557)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 6.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 14:10:39 +01:00
Nicolò Boschi
94598fbd25
fix: remove broken minimax test and enhance slim smoke test with retain/recall (#564)
- Delete test_minimax_provider.py which imports non-existent `create_llm`
  function (should be `create_llm_provider`), causing pytest collection errors
- Add scripts/smoke-test-slim.sh: shared retain + recall validation script
  used by both Docker slim and pip slim CI jobs
- Update docker/test-image.sh to run retain/recall after health check for
  all API targets
- Update test-pip-slim CI job to run the shared smoke test script
2026-03-13 14:10:32 +01:00
Nicolò Boschi
15ea23d5d6
feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560)
* feat: introduce hindsight-api-slim and hindsight-all-slim packages

Closes #552

- Move all source code from hindsight-api/ to new hindsight-api-slim/
- hindsight-api-slim has heavy ML deps (torch, sentence-transformers,
  transformers, einops, flashrank, mlx, mlx-lm, safetensors) and
  pg0-embedded as optional extras: [local-ml], [embedded-db], [all]
- hindsight-api becomes a zero-code meta-package depending on
  hindsight-api-slim[all] for full backward compatibility
- Add hindsight-all-slim meta-package: hindsight-api-slim + client + embed
- hindsight-all updated to depend on hindsight-api-slim[all]
- pg0.py: lazy-import pg0 with clear ImportError pointing to [embedded-db]
- Dockerfile: replace sed hack with proper uv sync --extra flags
- Update release.yml, test.yml, lint.sh, release.sh, CLAUDE.md and
  all path references throughout the repo

* refactor: rename hindsight/ directory to hindsight-all/

* docs: document hindsight-api-slim and hindsight-all-slim package variants

Add package variants table and extras explanation to installation.md

* docs: remove emojis from installation.md, use professional tone

* docs: link Docker slim variant to pip package variants section

* docs: consolidate Docker image variants into single table

* ci: fix working-directory paths after package restructure

- Replace all hindsight-api → hindsight-api-slim in test.yml
- Replace hindsight → hindsight-all in test.yml
- Add --extra embedded-db to test-embed API install step

* ci: add local-ml and embedded-db extras to API sync steps

These extras were previously implicit in the old hindsight-api package
(which bundled everything). Now that hindsight-api-slim uses optional
extras, we must explicitly request local-ml and embedded-db in CI.

* ci: add API install step with embedded-db to test-embed smoke test

The smoke test starts hindsight-api as a daemon, which requires pg0-embedded.
Add a dedicated install step for hindsight-api-slim with embedded-db extra
so the daemon can start successfully.

* ci: remove --no-install-project when using optional extras

When --no-install-project is combined with --extra, the optional deps
are not installed because extras require the project to be active.
Remove --no-install-project from steps that need local-ml or embedded-db.

* ci: fix ordering of uv sync steps to preserve optional extras

When uv sync runs for a different workspace member, it removes optional
extras installed for other members. Fix by always running extra-requiring
API sync last, after other workspace member syncs.

Also remove --no-install-project from embedded-db sync in test-embed,
as --no-install-project prevents optional extras from being active.

* ci: add local-ml extra to test-embed API install for smoke test

The smoke test starts the full API server which needs sentence-transformers
for local embeddings (default provider). Add local-ml extra to the install.

* ci: simplify extras with --all-extras and add slim pip smoke test

- Replace explicit --extra local-ml --extra embedded-db with --all-extras
  for cleaner, more maintainable sync steps
- Add test-pip-slim job: tests hindsight-api-slim[embedded-db] without
  local ML models, using Cohere for embeddings/reranking (mirrors Docker
  slim smoke test approach)

* ci: simplify slim smoke test to health check only (mirrors Docker test)
2026-03-13 13:50:03 +01:00
dependabot[bot]
720d42c576
chore(deps): bump actions/checkout from 4 to 6 (#556)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 13:47:18 +01:00
dependabot[bot]
8cbad0ef7a
chore(deps): bump actions/upload-artifact from 4 to 7 (#555)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 13:47:09 +01:00
dependabot[bot]
3d2c62ef09
chore(deps): bump actions/setup-go from 5 to 6 (#558)
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 6.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 13:47:00 +01:00
dependabot[bot]
56462a30cd
chore(deps): bump actions/cache from 4 to 5 (#559)
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-13 13:46:52 +01:00
Nicolò Boschi
067acf1ba5
chore: add dependabot config for GitHub Actions updates (#554) 2026-03-13 10:32:29 +01:00
Salman Chishti
4eaa2f3566
Upgrade GitHub Actions to latest versions (#553)
Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>
2026-03-13 10:32:22 +01:00
Nicolò Boschi
eeb938fc65
fix: truncate documents exceeding LiteLLM reranker context limit (#549)
* fix: register embedded profiles in CLI metadata on daemon start

When HindsightEmbedded(profile="myapp") starts a daemon, the profile
was never written to metadata.json or given a .env file, making it
invisible to `hindsight-embed profile list` and other CLI commands.

Add _register_profile() to DaemonEmbedManager which saves HINDSIGHT_API_*
config to ~/.hindsight/profiles/{name}.env and registers the port in
metadata.json. Called after a successful new daemon start and when the
daemon is already running, so orphaned profiles also get registered on
next use.

* fix: truncate documents exceeding LiteLLM reranker context limit

Add HINDSIGHT_API_RERANKER_LITELLM_MAX_TOKENS_PER_DOC env var for both
litellm and litellm-sdk reranker providers. When set, documents are
truncated to the configured token limit using tiktoken (cl100k_base)
before being sent to the reranker, preventing BadRequestError for
models with small context windows (e.g. 1024-token limit).

* refactor: use shared _tiktoken_encoder for doc truncation in LiteLLM reranker

* refactor: use _get_tiktoken_encoding() consistently, remove eager module-level encoder instance

* doc: add HINDSIGHT_API_RERANKER_LITELLM_MAX_TOKENS_PER_DOC to configuration reference
2026-03-13 10:18:18 +01:00
Ethan Clarke
2344484f77
feat: add MiniMax LLM provider support (#550)
Add MiniMax as a supported LLM provider via the OpenAI-compatible interface.

- Register MiniMax in the provider factory and valid providers list
- Set default base URL to https://api.minimax.io/v1
- Set default model to MiniMax-M2.5 in PROVIDER_DEFAULT_MODELS
- Add temperature clamping for MiniMax (must be >0, ≤1.0)
- Add API key validation (MiniMax requires an API key)
- Add MiniMax configuration example to .env.example
- Update documentation (models.md, configuration.md, embed.md, CLAUDE.md, README.md)
- Add unit and integration tests for MiniMax provider

Co-authored-by: octo-patch <octo-patch@users.noreply.github.com>
2026-03-13 10:17:55 +01:00
Ben
a01bb18bc4
blog: Time-Aware Spreading Activation for Memory Graphs (#547)
doc: add blog post on time-aware spreading activation for memory graphs
2026-03-12 12:55:14 -04:00
Stable Genius
b17f338e17
fix(openclaw): inject recalled memories as system context (#548)
Co-authored-by: Stable Genius <259448942+stablegenius49@users.noreply.github.com>
2026-03-12 17:09:13 +01:00
Nicolò Boschi
e210953d05
add trending badge HTML in README.md 2026-03-12 16:26:17 +01:00
Nicolò Boschi
06b0f74a48
fix: register embedded profiles in CLI metadata on daemon start (#546)
When HindsightEmbedded(profile="myapp") starts a daemon, the profile
was never written to metadata.json or given a .env file, making it
invisible to `hindsight-embed profile list` and other CLI commands.

Add _register_profile() to DaemonEmbedManager which saves HINDSIGHT_API_*
config to ~/.hindsight/profiles/{name}.env and registers the port in
metadata.json. Called after a successful new daemon start and when the
daemon is already running, so orphaned profiles also get registered on
next use.
2026-03-12 09:39:47 +01:00
Nicolò Boschi
0560f6260d
fix: cancel in-flight async ops when bank is deleted (#545)
* fix: cancel async ops on bank delete via CASCADE FK + heartbeat checkpoints

- Add migration e5f6g7h8i9j0: FK ON DELETE CASCADE from async_operations
  and webhooks to banks, so deleting a bank auto-removes all its ops/webhooks
- Add _check_op_alive() helper: returns False if op row was deleted (cascade)
- Add consolidation checkpoint: after each LLM batch commit, abort early if
  op was deleted mid-run (returns status='cancelled')
- Add retain checkpoint: between sub-batches, abort early if op was deleted
- _mark_operation_completed/failed/completed_and_fire_webhook: gracefully
  handle missing row (UPDATE 0) with log instead of silent error
- Thread operation_id into run_consolidation_job() for checkpoint access
- Fix y0t1u2v3w4x5 and a1b2c3d4e5f6 migrations: add IF NOT EXISTS to prevent
  failure on idempotent re-runs
- Add 10 tests covering cascade delete, _check_op_alive, graceful mark methods,
  consolidation checkpoint, and retain checkpoint

* refactor: use RETURNING + fetchrow instead of execute + string comparison

* fix: add bank upsert before async_operations FK inserts and update tests

- memory_engine.py: upsert bank in submit_async_retain before async_operations INSERT
- http.py: upsert bank in api_create_webhook before webhooks INSERT
- test_worker.py, test_async_batch_retain.py, test_webhooks.py: add _ensure_bank
  helper calls before direct async_operations/webhooks inserts to satisfy FK constraint

* fix: mock bank_utils.get_bank_profile in unit test with mocked pool
2026-03-12 09:39:31 +01:00
Ben
220851e6f4
doc: What's New in Hindsight Cloud — Programmatic API Key Management (#543)
* doc: What's New in Hindsight Cloud — Programmatic API Key Management

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 11:41:21 -04:00
Ben
5b360c83d2
doc: Run Hindsight with Ollama: Local AI Memory, No API Keys Needed (#536)
* doc: add run-hindsight-with-ollama blog post
2026-03-11 11:39:59 -04:00
Nicolò Boschi
1caf5ec9ee
feat: add jina-mlx reranker provider for Apple Silicon (#542)
* feat: add JinaMLXCrossEncoder for native Apple Silicon reranking

Adds a new `jina-mlx` reranker provider backed by jinaai/jina-reranker-v3-mlx,
a 0.6B multilingual listwise reranker running via the MLX framework on Apple Silicon.
The model is downloaded automatically from HuggingFace Hub on first use.

Benchmarked latencies (Apple Silicon): 1 doc→32ms, 5→45ms, 10→60ms, 20→94ms.
Sub-linear scaling because all docs are ranked in a single forward pass.

- Embeds the MLX reranker implementation (_MLXReranker / _MLPProjector) directly
  in cross_encoder.py with no transformers/PyTorch dependency
- Adds `mlx`, `mlx-lm`, `safetensors` to pyproject.toml optional deps (uv add)
- Updates configuration.md with provider docs and benchmark table

* refactor: import MLXReranker from repo rerank.py instead of duplicating code

Use importlib to load MLXReranker directly from the model repo's own rerank.py
(downloaded via snapshot_download). Also pin exact minimum versions for
mlx>=0.31.0, mlx-lm>=0.31.1, safetensors>=0.6.2 (verified against installed versions).

* refactor: move MLX reranker impl to dedicated jina_mlx_reranker.py

Replaces the importlib hack with a proper module. jina_mlx_reranker.py is
adapted from jinaai/jina-reranker-v3-mlx/rerank.py (CC BY-NC 4.0) with the
source clearly documented at the top of the file.

* docs: simplify jina-mlx reranker docs

* fix: disable GIN fastupdate on source_memory_ids index to prevent deadlocks

GIN fastupdate buffers inserts in a pending list and flushes it with
AccessExclusiveLock when full. Under concurrent test load (8 xdist workers
all running retain_async), two workers can trigger a flush simultaneously
and deadlock. Recreating the index with fastupdate=off eliminates the
flush/lock cycle at the cost of slightly slower individual inserts.

* fix: drop per-bank HNSW indexes after transaction to avoid AccessExclusiveLock deadlock

When deleting a bank, the previous code dropped HNSW indexes inside the
same transaction as the DELETE FROM memory_units. Since DROP INDEX needs
AccessExclusiveLock on the parent table and DELETE holds RowExclusiveLock,
two concurrent bank deletions deadlocked on the same table lock.

Fix: capture internal_id inside the transaction, commit, then drop the
indexes outside the transaction so no row-level locks are held.
2026-03-11 15:15:58 +01:00
Nicolò Boschi
66dedb8d41
feat: make recall max query tokens configurable via env var (#544)
* doc: add 0.4.17 release blog post

* feat: make recall max query tokens configurable via env var

Add HINDSIGHT_API_RECALL_MAX_QUERY_TOKENS env var (default: 500) to
replace the hardcoded MAX_QUERY_TOKENS constant in http.py.
2026-03-11 14:56:59 +01:00
Nicolò Boschi
43b3efc494
perf: replace window-function retrieval with UNION ALL + per-bank HNSW indexes (#541)
* perf: replace window-function retrieval with UNION ALL + per-bank HNSW indexes

The previous retrieve_semantic_bm25_combined() used ROW_NUMBER() OVER (PARTITION
BY fact_type ...) which forced a full sequential scan — pgvector cannot use HNSW
indexes when a window function partitions on the same column as the ORDER BY.

Changes:
- retrieval.py: rewrite to UNION ALL of per-fact_type subqueries; each arm has
  its own ORDER BY embedding <=> $1 LIMIT n, enabling partial HNSW index scans.
  Semantic arms over-fetch 5x (min 100) for HNSW approximation; trimmed in Python.
- memory_engine.py: set hnsw.ef_search=200 at pool init (persistent per-connection,
  no per-query SET/RESET overhead).
- bank_utils.py: add create_bank_hnsw_indexes / drop_bank_hnsw_indexes for
  per-(bank_id, fact_type) partial HNSW index lifecycle management.
- fact_storage.py / bank_utils.py: create per-bank indexes on fresh bank insert.
- memory_engine.py delete_bank: drop per-bank indexes via DELETE...RETURNING to
  avoid a separate round-trip.
- Migration a3b4c5d6e7f8: add interim fact_type-only partial indexes.
- Migration d5e6f7a8b9c0: add internal_id UUID UNIQUE to banks, replace
  fact_type-only indexes with per-(bank, fact_type) partial HNSW indexes, drop
  the global idx_memory_units_embedding that competed with them.

Why per-(bank, fact_type) not just per-fact_type:
The idx_memory_units_bank_id B-tree index always wins over fact_type-only partial
indexes when bank_id appears in the WHERE clause. Including bank_id in the partial
index predicate removes the B-tree from consideration and lets the planner choose
HNSW. The global HNSW index must also be dropped to avoid competing for the larger
fact_type partitions (world, observation).

* refactor: collapse two HNSW migrations into one

* refactor: generate bank internal_id in Python before insert

Instead of relying on DEFAULT gen_random_uuid() and RETURNING internal_id,
generate the UUID in application code before the INSERT. This means we
always know the value upfront and can call create_bank_hnsw_indexes
immediately without needing a DB round-trip to retrieve the assigned ID.

Also adds tests for HNSW index lifecycle and retrieve_semantic_bm25_combined.

* fix: correct migration and prevent global HNSW index recreation

Migration fixes:
- Add text() wrappers for raw SQL in d5e6f7a8b9c0 (SQLAlchemy 2.0 compat)
- Drop stale fact_type-only partial indexes (idx_mu_emb_world/observation/experience)
  that may exist from prior migrations on the same DB

migrations.py fix:
- Skip global HNSW index creation when per-bank partial HNSW indexes already
  exist on memory_units (idx_mu_emb_* pattern). Without this, the post-migration
  vector index check detects no %embedding% named index and recreates the global
  idx_memory_units_embedding, which defeats the per-bank index strategy.

Verified with EXPLAIN ANALYZE on 66K-row bank: all three fact_type arms use
their per-bank HNSW index scan (idx_mu_emb_worl/expr/obsv_<uid16>).

* fix: use correct embeddings.encode() in test
2026-03-11 12:09:50 +01:00
Nicolò Boschi
00ac3d8834
doc: add 0.4.17 release blog post (#538) 2026-03-10 17:40:10 +01:00
Nicolò Boschi
2191654b1f Release v0.4.17
- Update version to 0.4.17 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-litellm, hindsight-crewai, hindsight-pydantic-ai, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- OpenClaw integration: hindsight-integrations/openclaw
- AI SDK integration: hindsight-integrations/ai-sdk
- Chat SDK integration: hindsight-integrations/chat
- Helm chart
- Sync documentation to version-0.4
2026-03-10 17:18:35 +01:00
Nicolò Boschi
dcaacbe407
feat: add manual retry for failed async operations (#537)
- API: POST /v1/default/banks/{bank_id}/operations/{operation_id}/retry
  resets status to pending so the worker re-executes the task
- UI: Retry button on failed operations in the operations view
- Control plane proxy route + ControlPlaneClient.retryOperation()
- Updated OpenAPI spec, all generated clients, and operations docs
2026-03-10 17:16:11 +01:00
And#ocean
32a4882a10
fix: resolve remaining webhook schema issues in multi-tenant retain (#533)
Follow-up to #499 which fixed the worker path and http.py but missed
two code paths in memory_engine.py:

1. `_retain_batch_async_internal` (line ~2185) still passed
   `request_context.tenant_id` which is always None for HTTP requests
   (tenant_id is never populated by the HTTP layer — the schema is
   stored in the _current_schema contextvar by _authenticate_tenant).

2. `_build_retain_outbox_callback._callback` captured the `schema`
   parameter at closure creation time. In the HTTP path, http.py builds
   the callback *before* calling retain_batch_async, but _current_schema
   is only set inside retain_batch_async by _authenticate_tenant — so
   the captured schema is always None. Fixed by resolving schema at
   callback invocation time via `schema or _current_schema.get()`.

Both issues cause `relation "webhooks" does not exist` errors that
abort the entire retain transaction in multi-tenant deployments,
silently rolling back all inserted memory data.
2026-03-10 16:44:04 +01:00
Nicolò Boschi
cd3a6a227b
fix: strip null bytes from parsed file content before retain (#535)
* doc: split blog index into Hindsight and Hindsight Cloud sections

- Tag the document upload post with `hindsight-cloud`
- BlogListPage renders two sections, capping Cloud at 3 posts with a "View all →" link
- Swizzle BlogTagsPostsPage so /blog/tags/hindsight-cloud uses the custom grid layout

* doc: attribute blog posts to Nicolò Boschi with GitHub profile image

Replace the generic "Hindsight Team" author with the real author entry
(nicoloboschi) across all 15 blog posts. GitHub profile image is loaded
from https://github.com/nicoloboschi.png.

* doc: add Hindsight Team title to nicoloboschi author

* doc: assign blog posts to correct authors based on git blame

- Add benfrank241 (Ben Bartholomew) and chrislatimer (Chris Latimer) to authors.yml
- Assign 7 posts to Ben, 1 post to Chris, remainder stay with Nicolò

* fix: strip null bytes from parsed file content before retain

* test: add tests for sanitize_llm_output

* fix: retry retain DB transaction on deadlock during parallel document processing
2026-03-10 16:24:11 +01:00
Nicolò Boschi
28308a14d6
doc: split blog index into Hindsight and Hindsight Cloud sections (#534)
* doc: split blog index into Hindsight and Hindsight Cloud sections

- Tag the document upload post with `hindsight-cloud`
- BlogListPage renders two sections, capping Cloud at 3 posts with a "View all →" link
- Swizzle BlogTagsPostsPage so /blog/tags/hindsight-cloud uses the custom grid layout

* doc: attribute blog posts to Nicolò Boschi with GitHub profile image

Replace the generic "Hindsight Team" author with the real author entry
(nicoloboschi) across all 15 blog posts. GitHub profile image is loaded
from https://github.com/nicoloboschi.png.

* doc: add Hindsight Team title to nicoloboschi author

* doc: assign blog posts to correct authors based on git blame

- Add benfrank241 (Ben Bartholomew) and chrislatimer (Chris Latimer) to authors.yml
- Assign 7 posts to Ben, 1 post to Chris, remainder stay with Nicolò
2026-03-10 13:32:43 +01:00
Ben
fc71664b5f
doc: What's New in Hindsight — Document File Upload (#532)
* doc: add Hindsight document file upload blog post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* doc: clarify document upload is a Hindsight Cloud feature

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* doc: fix Iris billing claim to be more accurate

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-10 10:11:10 +01:00
Chris Bartholomew
9a694f64b8
Fix run-db-migration for all-tenant upgrades (#530)
* Add release-scoped migration admin command

* Fix run-db-migration for all-tenant upgrades
2026-03-10 10:10:03 +01:00
Ben
7bcf26097c
doc: Your Pydantic AI Agent Forgets You After Every Run. Fix It in 5 Lines. (#531)
* doc: add pydantic-ai-persistent-memory blog post

* doc: update Pydantic AI blog cover image

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* doc: SEO-optimized rewrite of Pydantic AI blog post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:39:20 -04:00
Nicolò Boschi
1cdfb7c2e2
fix: normalize named tool_choice to required + filtered tools for OpenAI-compatible providers (#528)
LM Studio (and Ollama) reject the named tool_choice dict format
{"type": "function", "function": {"name": "..."}} with HTTP 400.

The reflect agent uses this format on iterations 0-2 to force sequential
tool selection, causing reflect to fail entirely on LM Studio.

The fix converts named tool_choice dicts to tool_choice="required" with
the tools list filtered to just the requested tool — semantically identical
and accepted by all providers including LM Studio and Ollama.

Closes #520
2026-03-09 15:47:51 +01:00
Nicolò Boschi
3e967add78
docs: add FAQ entry for conversation retain format (#529)
Addresses common questions from community discussions on the recommended
format and flow for retaining conversations (JSON array vs plain text,
upsert pattern, avoiding pre-summarization).
2026-03-09 15:47:25 +01:00
Nicolò Boschi
00ccf0b218
fix(consolidation): respect bank mission over ephemeral-state heuristic (#525)
* Add Hindsight as git subtree + BCGU noise filtering tests

Adds hindsight server source as a subtree under hindsight-api/ so we
can iterate on server-side fixes directly.

test_bcgu_noise_filtering.py proves that a well-crafted
retain_custom_instructions (BCGU_RETAIN_MISSION) can suppress
talking-head noise at fact extraction time — eliminating the need for
client-side --filter-vision-noise preprocessing.

Tests cover:
- Default mode extracts 3 noise facts from talking-head frame (problem documented)
- BCGU mission produces 0 noise facts from same talking-head frame
- BCGU mission still extracts 2 high-value ChatGPT screen facts correctly
- Mixed doc (2 talking-head + 2 screen): 0% noise ratio with BCGU mission
- Pure talking-head doc: 0 facts extracted

All 5 tests pass in ~32s using gpt-4o-mini.

* fix(consolidation): respect mission context over ephemeral-state heuristic

Two related fixes for the consolidation engine when a bank mission is
configured:

1. **Mission override for ephemeral-state filter** (`prompts.py`):
   The system prompt previously instructed the LLM to discard any fact
   that looked like "ephemeral state" (e.g. current position, transient
   actions).  When a mission is active the mission itself defines what is
   valuable — timestamped screen actions, session events, tool interactions
   may all be mission-critical even though they look ephemeral.  Added a
   MISSION OVERRIDE block that explicitly tells the LLM the mission takes
   priority over the generic ephemeral-state guidance.

2. **Remove contradictory durable-knowledge nudge** (`consolidator.py`):
   The user-prompt builder was injecting "Focus on DURABLE knowledge that
   serves this mission, not ephemeral state" alongside the mission text.
   This phrasing contradicted missions that intentionally capture
   timestamped events.  Replaced with a neutral directive that simply
   signals the mission overrides general rules.

3. **JSON control-character sanitisation** (`consolidator.py`):
   LLMs occasionally embed literal ASCII control characters (0x00–0x1f)
   inside JSON string values, causing `json.loads` to raise a
   JSONDecodeError.  Added a try/except that strips control characters
   and retries the parse before re-raising, preventing spurious failures.

* refactor(consolidation): move sanitize_llm_output to llm_wrapper, reuse in consolidator

- Add `sanitize_llm_output()` to `llm_wrapper.py` as the single canonical
  function for stripping characters that break downstream systems
  (ASCII control chars 0x00-0x08/0x0B-0x0C/0x0E-0x1F/0x7F and Unicode
  surrogates). Tab, newline, and carriage-return are preserved.
- Reduce `_sanitize_text()` in `fact_extraction.py` to a thin wrapper
  that delegates to `sanitize_llm_output()`.
- Update `consolidator.py` to import and call `sanitize_llm_output()`
  directly instead of reimplementing the logic inline.
- Remove test_bcgu_noise_filtering.py (should not have been committed).

* fix(consolidation): apply sanitize_llm_output to observation text fields

sanitize_llm_output was imported but unused after the old _call_llm_once
path was removed. The batch flow uses structured Pydantic output so
there's no raw json.loads call — instead, apply sanitization via
field_validator on _CreateAction.text and _UpdateAction.text so control
characters are stripped before observation text reaches the database.

* fix(entity-resolver): correct mention_count for new entities in batch retain

When the same entity (e.g. "Bob") appears across N items in a single batch
retain, _resolve_entities_batch_impl deduplicates them into one name group
before inserting, then queued only ONE _EntityStat regardless of N. The
flush therefore always incremented mention_count by 1 beyond the INSERT
value — giving 2 for any number of mentions.

Two-part fix:
- INSERT with mention_count=0 so the post-transaction flush is the single
  source of truth for the count (avoids an off-by-one for N=1 as well).
- Append one _EntityStat per original mention (len(g.indices)) instead of
  one per unique name, so flush_pending_stats() adds the correct total N.

This makes the batch path consistent with the single-entity path, which
already accumulates one stat per mention via entities_to_update.
2026-03-09 15:04:36 +01:00
Nicolò Boschi
f7a60f898d
feat: filter operations by type + fix stale auto-refresh closure (#522) (#527)
* feat: filter operations by type + fix stale closure in auto-refresh

- Add `type` query param to GET /operations endpoint and engine layer
- Add operation type dropdown filter in Background Operations UI
- Fix auto-refresh interval using stale statusFilter/offset closure by
  adding filter state to useEffect deps and wrapping loadOperations in
  useCallback (fixes #522)
- Regenerate OpenAPI spec and all SDK clients

* fix: update Rust CLI list_operations call with new type parameter
2026-03-09 13:17:00 +01:00
Nicolò Boschi
7accac94b2
fix: migrate mental_models.embedding dimension alongside memory_units (#526)
ensure_embedding_dimension() now also checks and migrates mental_models.embedding,
fixing silent failures when changing embedding model dimensions. Extracted shared
per-table logic into _migrate_table_embedding_dimension() to avoid duplication.
Adds test coverage for the mental_models dimension migration path.

Fixes #523
2026-03-09 12:23:50 +01:00
Chris Bartholomew
fa3501d448
Fix Iris parser httpx read timeout for file uploads (#524)
The httpx.AsyncClient was created without a timeout parameter,
defaulting to 5 seconds for reads. This is too short for uploading
PDFs to presigned URLs and waiting for Iris API responses. Set
explicit timeouts: 30s default, 120s for reads.
2026-03-09 11:22:44 +01:00