Commit graph

151 commits

Author SHA1 Message Date
Nicolò Boschi
627ec5d524
feat: expose document_metadata in API and control plane (#798)
* feat: expose document_metadata in API and control plane

Add document_metadata (sourced from retain_params.metadata) to both
list and get document endpoints. Display it in the control plane
documents table and detail panel. Drop the unused metadata column
from the documents table (was always stored as empty {}).

* fix: code review fixes for document_metadata feature

- Remove unnecessary `import json as _json` (json already imported at module level)
- Simplify redundant truthiness checks in retain_params parsing
- Regenerate OpenAPI spec and client SDKs (Python, TypeScript, Go)
- Add tests for document_metadata in get_document and list_documents

* feat(ui): improve documents table and detail panel

- Relative timestamps with full date on hover
- Remove context column from table
- Metadata shown as k=v badges (blue, like tags)
- Size in bytes instead of chars
- Document IDs wrap instead of truncating
- Detail panel wider (560px)
- Retain params: context, event_date, metadata badges
2026-03-31 11:42:02 +02:00
Nicolò Boschi
b104bad02c
fix(codex): merge new settings on upgrade instead of skipping (#780)
The installer skipped settings.json entirely if it already existed,
leaving version and new config keys stale. Now merges: updates version,
adds new upstream keys, preserves user customizations.

Also fixes pre-existing typo: RERANK_URL → rerank_url in ZeroEntropy
cross-encoder.
2026-03-31 09:05:49 +02:00
Ben
df17570d8a
blog: What's New in Hindsight Cloud — Native OAuth for MCP Clients (#731)
* Add MCP OAuth blog post
2026-03-30 09:33:58 -04:00
Nicolò Boschi
7a9e99998a
docs: 0.4.21 release blog post and changelog (#765)
* docs: 0.4.21 release blog post and changelog

* fix(blog): align 0.4.21 code snippets with docs, add release image

* chore: regenerate docs skill references
2026-03-30 15:28:27 +02:00
Nicolò Boschi
cc3cdc2f83 Release v0.4.21
- Update version to 0.4.21 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- Helm chart
- Sync documentation to version-0.4
2026-03-30 14:52:50 +02:00
Nicolò Boschi
56489a58d1
refactor(llamaindex): merge into single hindsight-llamaindex package (#760)
* refactor(llamaindex): merge two packages into single hindsight-llamaindex

Merge `llama-index-tools-hindsight` and `llama-index-memory-hindsight` into
a single `hindsight-llamaindex` package following our naming convention.

- Rename package to `hindsight-llamaindex` (Python module: `hindsight_llamaindex`)
- Move HindsightToolSpec and HindsightMemory into the same package
- Delete `llamaindex-memory/` directory
- Add CI test job for llamaindex integration
- Update docs, blog post, and integrations.json

* fix(blog): update llamaindex blog post for merged package

- Move date to 2026-03-30
- Add HindsightMemory (automatic BaseMemory) pattern
- Fix "bank must exist first" pitfall — mission auto-creates
- Align all code examples with docs page
- Update architecture diagram to show both patterns

* fix(docs): add llamaindex/openai icons, rename Codex

- Add llamaindex.png and openai.png icons
- Rename "OpenAI Codex CLI" to "Codex" in integrations.json and docs
- Use openai.png icon for Codex integration
2026-03-30 14:30:48 +02:00
Nicolò Boschi
a8a63818c7
feat(api): add duration_ms to audit log entries (#758)
* feat(api): add duration_ms to audit log entries

Server-computed duration in milliseconds (started_at → ended_at) on
the list audit logs endpoint. Null when ended_at is not set.

Closes #749

* feat(api): add duration_ms to audit log entries and type audit endpoints

- Add server-computed duration_ms (started_at → ended_at) to audit log
  list response. Null when ended_at is not set.
- Add typed Pydantic response models for both audit log endpoints
  (list and stats) so they appear in the OpenAPI spec.
- Regenerate OpenAPI spec and all client SDKs.

Closes #749

* chore: regenerate docs skill after audit log response models
2026-03-30 12:32:02 +02:00
Nicolò Boschi
ecf16ea1e6
fix(codex): cleanup dead code, add release lifecycle and docs (#753)
* fix(codex): cleanup dead code and add to release lifecycle

- Remove orphaned reflect() method from client.py (leftover from dropped auto-mode)
- Remove dead retainToolCalls config default (never wired through)
- Add codex to release-integration.sh valid integrations
- Add settings.json version fallback to release script
- Add codex CI test job in test.yml
- Add codex to integrations.json registry

* docs(codex): add changelog page and link from integration docs

* feat(codex): add hosted installer script (get-codex)

Add self-contained installer at hindsight.vectorize.io/get-codex that
downloads scripts from GitHub, configures hooks, and supports local/cloud
mode selection — no git clone required.

Update docs and README to use the one-liner install:
  curl -fsSL https://hindsight.vectorize.io/get-codex | bash

* chore(codex): remove install.sh in favor of hosted get-codex

* fix(docs): use /next/ prefix for codex changelog link

* fix(docs): use GitHub link for codex changelog back-link
2026-03-30 11:34:43 +02:00
Ben
a69bdbb55f
How We Built a 4-Way Hybrid Search System That Actually Runs in Parallel (#708)
* Add blog: How We Built a 4-Way Parallel Hybrid Search System

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add cover image for parallel hybrid search post

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update parallel hybrid search post date to 2026-03-27

* Set author to chrislatimer

* Update recall docs link

* review: align blog post to actual retrieval code

- Reframe as evolutionary narrative (V1 asyncio.gather → connection sharing)
- Add missing reranker section (cross-encoder + multiplicative boost scoring)
- Replace MPFP references with LinkExpansion (3-signal CTE)
- Fix SQL to match actual UNION ALL approach, explain CTE planner issue
- Fix acquire_with_retry, index types (ivfflat→HNSW), fusion code
- Remove fabricated perf numbers
- Add alpha calibration rationale and connection contention insight

* add nicoloboschi and benfrank241 as co-authors

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>
2026-03-27 11:21:01 -04:00
Nicolò Boschi
3c78b717b0
docs: add AG2 integration page (#723)
- Add AG2 integration doc with quick start, configuration, GroupChat example, and API reference
- Add to sidebar, versioned sidebar, and integrations hub
- Add AG2 icon
2026-03-27 10:58:42 +01:00
Nicolò Boschi
083295dc6f
feat: add audit log for feature usage tracking (#717)
* feat: add audit log for feature usage tracking

Add full auditability for all mutating and core API operations across
HTTP, MCP, and system (worker) transports. Audit entries record raw
request/response as JSONB, timing (started_at/ended_at), action, and
transport type.

Backend:
- New audit_log table with JSONB columns for expandability without
  future migrations (merge migration of 3 existing heads)
- AuditLogger with fire-and-forget writes via asyncio.create_task
- @audited decorator on 28 HTTP route handlers
- MCP tool audit wrapping for 16 auditable tools
- Worker task execution wrapped with audit_context
- List endpoint with action, transport, date range filters + pagination
- Stats endpoint with per-day counts for charting
- Configurable retention sweep (concurrent-safe DELETE)

Config (env-only, static):
- HINDSIGHT_API_AUDIT_LOG_ENABLED (default: false)
- HINDSIGHT_API_AUDIT_LOG_ACTIONS (comma-separated allowlist, empty=all)
- HINDSIGHT_API_AUDIT_LOG_RETENTION_DAYS (default: -1, keep forever)

Control Plane:
- New "Audit Logs" tab on bank configuration page
- Line chart showing request volume (today/7d/30d) with action filter
- Filterable table with action, transport, date range filters
- Paginated list with detail dialog showing raw request/response JSON

Tests:
- 13 tests covering list, filters, pagination, stats, disabled mode,
  action allowlist, and ordering

* fix: split 3-way merge migration into two 2-way merges

Alembic doesn't support 3-parent merge migrations. Split into a no-op
merge of 2 heads (b1c2d3e4f5g6) followed by the audit_log table
migration merging the third head.

* fix: correct merge migration to merge actual 2 heads

The original analysis incorrectly identified 3 heads. There were only 2
(a3b4c5d6e7f8 and c8e5f2a3b4d1). Remove the unnecessary intermediate
merge migration and fix the audit_log migration to merge these 2 heads.

* fix: use 'heads' instead of 'head' in migration runner

Alembic's upgrade('head') fails when multiple heads exist (e.g. from
namespace package overlaps between hindsight-api and hindsight-api-slim).
Using 'heads' (plural) handles this gracefully by upgrading all branches.

* chore: regenerate OpenAPI spec with audit log endpoints

* chore: regenerate TypeScript client and docs skill OpenAPI spec

Python and Go clients still need regeneration (requires Docker).

* chore: regenerate all client SDKs (Python, Go, TypeScript)

Adds generated audit log API clients for Python (audit_api.py),
Go (api_audit.go), and TypeScript client type updates.
2026-03-27 09:52:03 +01:00
Ben
62c0992075
Teaching the Llama to Remember (#707)
Llama index blog
2026-03-26 15:44:18 -04:00
Mr. Khachaturov
939cb40a73
fix: include Pydantic v2 fields in ValidationError OpenAPI schema (#697)
FastAPI generates the ValidationError schema with only loc, msg, and
type, but Pydantic v2 actually returns input, ctx, and url as well.
Generated clients with strict JSON decoding (Go's DisallowUnknownFields)
cannot parse real 422 responses — the actual validation message gets
replaced by a confusing JSON decoding error.

- Patch the OpenAPI schema in create_app() to add input, ctx, url
- Regenerate spec and Go client
2026-03-26 11:25:46 +01:00
Ben
d344ef26da
blog: Your AWS Strands Agent Forgets Everything Between Runs. Here's the Fix. (#685)
* blog: add Strands persistent memory post
2026-03-25 10:37:37 -04:00
Ben
0ad6ee3156
Blog: Adding Long-Term Memory to LangGraph and LangChain Agents (#637)
* Add blog post: Adding Long-Term Memory to LangGraph and LangChain Agents

* blog: update langgraph post date to 2026-03-24 and add cover image

* blog: fix claude-code-telegram filename to match frontmatter date (2026-03-25)

* blog: set claude-code-telegram date to 2026-03-23

* blog: fix date timezone offset by adding T12:00 to all post dates

* ci: trigger fresh CI run

* blog: fix broken docs link (routeBasePath is /)
2026-03-24 13:51:27 -04:00
Ben
7fe773c0ee
feat: add Strands Agents SDK integration with Hindsight memory tools (#659)
* feat: add Strands Agents SDK integration with Hindsight memory tools

* fix: add strands docs to versioned docs so build link check passes

* fix(strands): run hindsight client calls in thread pool to avoid event loop conflict with Strands
2026-03-24 17:21:30 +01:00
Nicolò Boschi
4f533dde94
docs: 0.4.20 release blog post and changelog (#671)
* docs: add 0.4.20 release blog post and changelog

Add release notes blog post covering Claude Code integration, LangGraph
integration, NemoClaw integration, independent integration versioning,
and reflect improvements. Auto-generated changelog entry included.

* docs: add 0.4.20 release blog cover image
2026-03-24 10:03:33 +01:00
Nicolò Boschi
08d2c78ae7 Release v0.4.20
- Update version to 0.4.20 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- Helm chart
- Sync documentation to version-0.4
2026-03-24 09:19:14 +01:00
Nicolò Boschi
742f212b2f docs: update blog 2026-03-23 18:14:04 +01:00
Nicolò Boschi
546d595c9f
feat(blog): Agent Memory Benchmark launch post (#657)
* feat(blog): launch Agent Memory Benchmark post and ImageCarousel component

* feat(blog): remove RAG terminology, add agentic eval framing
2026-03-23 17:51:51 +01:00
Nicolò Boschi
d886d3acb9
doc: Claude Code + Telegram + Hindsight blog post (#656)
* doc: add Claude Code + Telegram + Hindsight blog post

* doc: add fabioscarsi to blog authors

* doc: update fabioscarsi title to Contributor

* doc: remove horizontal rule dividers from blog post

* doc: update cover image and add image frontmatter for claude-code-telegram blog post

* doc: remove horizontal rule dividers

* doc: align Hindsight setup steps with PR #661 README

* fix: move marketplace.json to repo root and update source path

* doc: add Claude Code integration page, sidebar, and integrations hub entry

* doc: update versioned docs to 0.4.19

---------

Co-authored-by: Ben <ben.bartholomew@vectorize.io>
2026-03-23 15:44:07 +01:00
Mr. Khachaturov
e0f0da5d2d
docs: update HindClaw integration listing (#653)
Rename hindsight-openclaw-pro → HindClaw and update description to
reflect the current architecture: server-side Hindsight extensions
(hindclaw-extension on PyPI), Terraform provider for infrastructure
management, and the hindclaw-openclaw gateway plugin.

Link points to https://github.com/mrkhachaturov/hindclaw.
2026-03-23 11:11:00 +01:00
Ben
d2c32cb8e4
blog: Give NemoClaw the Best Agent Memory Available In One Command (#631)
* docs(blog): add NemoClaw persistent memory blog post

Covers external API mode, OpenShell network egress policy pattern,
and the LaunchAgent symlink gotcha from the live test run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(blog): update NemoClaw blog post with SEO-optimized draft

- Add slug, TL;DR, pitfalls, tradeoffs table, recap, next steps sections
- Restructure into numbered implementation steps
- Remove internal blog links that don't exist yet

* docs(blog): fix docs link to include /recall/ path

* docs(blog): add correct internal links to NemoClaw blog post

* docs(blog): make hindsight-nemoclaw setup command the primary path

One-command setup is now the default; manual 4-step process moved to
'Manual Alternative' section for reference.

* docs(blog): update title to lead with NemoClaw and best-in-class memory

* Add cover image to NemoClaw memory blog post

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-20 16:29:08 +01:00
Nicolò Boschi
ce691549ba
doc: add langgraph and nemoclaw (#633) 2026-03-20 16:18:05 +01:00
Nicolò Boschi
ea662d062e
feat: fact_types and mental model exclusion filters for reflect (#615)
* feat: add fact_types and mental model exclusion filters to reflect and mental models

Adds three new filtering options to both the reflect endpoint and mental model creation/refresh:

- `fact_types`: restrict which fact types (world, experience, observation) are retrieved.
  Disables irrelevant agent tools entirely (no wasted tokens).
- `exclude_mental_models`: skip the search_mental_models tool altogether.
- `exclude_mental_model_ids`: exclude specific mental models by ID (merged with the
  existing self-exclusion logic during mental model refresh).

For mental models, options are persisted in the existing `trigger` JSONB column so they
are automatically applied on every refresh. The `UpdateMentalModelRequest` already
proxies `trigger`, so no extra endpoint changes are needed.

Also fixes the test fixture (`pg0_db_url` in conftest.py) to correctly resolve pg0://
URLs and run migrations before tests, which was causing all DB-dependent tests to fail
with "relation public.banks does not exist" when HINDSIGHT_API_DATABASE_URL=pg0://uuuu.

* fix: guard against disabled-tool hallucination and regenerate clients

- Add enabled_tools guard in reflect agent: if an LLM calls a tool that
  was excluded (e.g. recall when fact_types=["observation"]), return an
  error result instead of executing it
- Regenerate OpenAPI spec and all SDK clients (Go, Python, TypeScript)
  to include new fact_types / exclude_mental_models fields

* fix: add missing ReflectRequest fields in Rust CLI struct initializers

* fix: filter hallucinated tool calls before trace to prevent disabled tools appearing in results

* chore: merge main, fix lint formatting and update skills openapi.json

* feat: expose fact_types, exclude_mental_models, exclude_mental_model_ids in control plane UI

* fix: add missing trigger fields to MentalModel type in control plane api.ts

* fix: add missing trigger fields to local MentalModel interface in mental-models-view

* feat: tabbed mental model dialogs (Basic / Options tabs)

* refactor: shared FactTypeFilter component, tabbed mental model dialogs use General tab, clean up labels

* feat: pill-style toggle buttons for fact type filter (blue/emerald/amber per type)

* fix: add spacing between Fact Types label and pills, rename to Exclude all mental models
2026-03-19 17:03:41 +01:00
Ben
276a4ba7e8
blog: Hermes Agent persistent memory (#599)
* blog: add Hermes Agent persistent memory integration post
2026-03-18 17:00:35 -04:00
Nicolò Boschi
c10c9c89e9
docs: add 0.4.19 release blog post, Agno and Hermes integration pages (#608) 2026-03-18 17:35:54 +01:00
Nicolò Boschi
0727f2d069 Release v0.4.19
- Update version to 0.4.19 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-litellm, hindsight-crewai, hindsight-pydantic-ai, hindsight-hermes, hindsight-agno, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- OpenClaw integration: hindsight-integrations/openclaw
- AI SDK integration: hindsight-integrations/ai-sdk
- Chat SDK integration: hindsight-integrations/chat
- Helm chart
- Sync documentation to version-0.4
2026-03-18 14:29:15 +01:00
Ben
8c378b981a
feat: add Agno integration with Hindsight memory toolkit (#596)
* feat: add Agno integration with Hindsight memory toolkit

Add hindsight-agno package providing Hindsight memory tools (retain,
recall, reflect) as an Agno Toolkit, following the same pattern as
Agno's Mem0Tools. Includes per-user bank isolation, global config,
bank auto-creation, and memory_instructions() for system prompt
injection.

Also adds cookbook documentation page with architecture diagrams,
quick start examples, and configuration reference.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove n8n blog post, add Agno icon, bind to release process

- Remove n8n blog post from the agno integration branch
- Add Agno logo icon and map hindsight-agno SDK tag in CookbookGrid
- Add hindsight-agno to release.sh PYTHON_PACKAGES array
- Add build, publish, artifact upload, and release asset steps in release.yml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove cookbook page (moved to hindsight-cookbook repo)

The Agno cookbook application now lives in
vectorize-io/hindsight-cookbook/applications/agno-memory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-18 11:17:23 +01:00
Ben
210a40665d
blog: fix streamlit post slug and add cover image (#604) 2026-03-17 15:16:29 -04:00
Nicolò Boschi
28dac7c7f8
fix: prevent silent memory loss on consolidation LLM failure (#601)
* fix: prevent silent memory loss on consolidation LLM failure

When all LLM retries are exhausted during consolidation, memories were
being marked consolidated_at unconditionally, permanently excluding them
from future consolidation runs without producing any observations.

Fix with two complementary mechanisms:
- Adaptive batch splitting: on LLM failure, the batch is halved and
  retried recursively down to batch_size=1, recovering most transient
  failures (rate limits, Pydantic validation on long prompts) without
  operator intervention
- consolidation_failed_at column: only single-memory batches that still
  fail after all retries are marked here instead of consolidated_at, so
  they remain visible and retryable
- New API endpoint POST /v1/default/banks/{bank_id}/consolidation/retry-failed
  resets these memories for the next consolidation run

* chore: regenerate OpenAPI spec

* fix: rename consolidation endpoint from /retry-failed to /recover

* fix: add consolidation_failed_at column, adaptive batch splitting, and recovery API

- Migration a3b4c5d6e7f8: add consolidation_failed_at TIMESTAMPTZ column to
  memory_units with an index for efficient failure queries; properly chains off
  g7h8i9j0k1l2 (backsweep_orphan_observations)
- Consolidator: filter pending memories with consolidation_failed_at IS NULL
  so failed memories are not re-fetched in an infinite loop
- Consolidator: adaptive batch splitting — when a batch exhausts all 3 LLM
  retries, halve it and retry sub-batches recursively; only single-memory
  batches that also exhaust all retries get consolidation_failed_at set
- New tests (9 total) covering: adaptive splitting recovers all memories,
  larger batch splitting, single-memory permanent failure, exclusion from
  next run, partial batch failure, recover resets columns, recover returns
  0 when none failed, recover-then-consolidate succeeds, HTTP endpoint

* chore: regenerate Go, Python, TypeScript clients with recover consolidation endpoint

* feat: add Recover Consolidation action to bank Actions dropdown

* style: apply ruff formatting to http.py and config.py

* fix: handle consolidation scope in large batch test mock LLM

The mock LLM was returning {"facts": ...} for ALL calls including consolidation.
Consolidation doesn't use skip_validation=True so it expects a _ConsolidationBatchResponse
instance, not a raw dict. Before this PR consolidation silently swallowed the AttributeError
(failed=False was returned); now failed=True triggers adaptive splitting and timeouts.

Fix: return _ConsolidationBatchResponse() when scope=="consolidation".

* fix: restrict claude-agent-sdk to macOS platform only (no Linux wheel available)

Also fix pre-existing type errors: use setattr for XLM-RoBERTa monkey-patch
and add missing reranker_local_fp16/bucket_batching/batch_size fields to main.py config constructor.

* fix: add UV_INDEX_STRATEGY=unsafe-best-match to fix markupsafe cp314 wheel conflict

PyTorch CPU index serves markupsafe==3.0.3 with only cp314 wheels.
uv's default first-index strategy stops at the first index with any version
even if no compatible wheel exists. unsafe-best-match searches all indices
for the best compatible wheel, falling back to PyPI for markupsafe.

* fix: use explicit pytorch index to prevent markupsafe wheel conflict

Configure the pytorch CPU index as explicit=true in pyproject.toml so it is
ONLY used for torch (via [tool.uv.sources]). All other packages (including
markupsafe) are resolved exclusively from PyPI, preventing the pytorch index
from serving incompatible cp314-only wheels for non-pytorch packages.

Remove UV_INDEX and UV_INDEX_STRATEGY from CI workflow (no longer needed
since the index is now configured in pyproject.toml).

* ci: trigger CI run

* ci: retry trigger

* ci: trigger after remote URL fix

* ci: add workflow_dispatch to unblock manual trigger

* fix: remove empty env blocks left after UV_INDEX removal

* fix: add type: ignore for optional claude_agent_sdk imports (macOS-only)

* fix: correct type: ignore rules for claude_agent_sdk and fix utcnow deprecation
2026-03-17 20:15:33 +01:00
Nicolò Boschi
e4f8a157c2
feat(retain): verbatim, chunks modes and named retain strategies (#593)
* feat(retain): add verbatim extraction mode

Adds retain_extraction_mode="verbatim" that stores each chunk as-is
without LLM summarization. The LLM still runs to extract entities,
temporal info, and location for full indexability — only the fact text
is replaced with the original chunk content (one memory per chunk).

Useful for RAG-style indexing and benchmarks where original text
must be preserved in memory.

- Add "verbatim" to RETAIN_EXTRACTION_MODES in config.py
- Add VERBATIM_FACT_EXTRACTION_PROMPT with instructions to preserve text
- Add _collapse_to_verbatim() post-processing to enforce 1 fact/chunk
- Expose in bank config UI dropdown with updated description
- Update configuration.md docs with verbatim mode description
- Add unit test for _collapse_to_verbatim and integration test via LLM
- Fix pre-existing main.py CLI override missing new reranker fields
- Fix pre-existing cross_encoder.py ty type error via setattr

* refactor(retain): verbatim mode skips 'what' field entirely

Instead of asking the LLM to echo the chunk text back into 'what' and
then discarding it, verbatim mode now uses a dedicated schema
(VerbatimExtractedFact) that omits the 'what' field altogether.
The LLM only returns metadata (entities, temporal info, location, who),
saving output tokens and avoiding any risk of paraphrasing before the
backfill.

- Add VerbatimExtractedFact / VerbatimFactExtractionResponse models
- Verbatim mode skips causal-relations section (nothing to relate causally)
- _extract_facts_from_chunk: allow missing 'what' in verbatim mode,
  set combined_text="" (backfilled by _collapse_to_verbatim)
- Update verbatim prompt to say DO NOT include 'what'

* feat(retain): add index_only extraction mode

Zero-LLM retain mode: chunks are stored as-is with no LLM call, no
entity extraction, and no temporal indexing. Embeddings still run for
semantic search. User-provided entities via RetainContent.entities
are the sole source of entity data.

Early return placed before the batch-API check so no LLM queue or
concurrency locks are acquired.

- Add "index_only" to RETAIN_EXTRACTION_MODES
- Add _extract_facts_index_only() with pure Python chunking path
- Add to UI dropdown and update description
- Update configuration.md with index_only docs and table entry
- Add unit test asserting zero token usage and exact text preservation

* feat(retain): add named retain strategies

Allows mixing extraction modes in a single bank via named strategies.
Each strategy is a set of hierarchical config overrides (extraction_mode,
chunk_size, entity_labels, entities_allow_free_form, etc.) applied on
top of the resolved bank config at retain time.

- retain_strategies: dict of strategy_name → config overrides (bank config)
- retain_default_strategy: default strategy when none specified (bank config)
- strategy field on /retain request: per-call override
- apply_strategy() in config_resolver applies overrides via dataclasses.replace()
- strategy propagates through retain_batch_async → _retain_batch_async_internal
  and through the async worker task payload
- Any hierarchical field is overridable per strategy, including entity_labels
  and entities_allow_free_form
- Docs updated with strategy configuration example and RRF fairness note
- Unit test for apply_strategy covering overrides, unknown strategy, and
  non-hierarchical field filtering

* feat(retain): add per-item strategy and strategy tests

- Add `strategy` field to `MemoryItem` so individual items in a retain
  request can override the request-level strategy
- Add `strategy` field to `FileRetainMetadata` for per-file strategy
  override in file retain requests
- Group memory items by effective strategy in `api_retain`; each group
  is processed as a separate batch, results are aggregated
- Thread strategy through `submit_async_file_retain` →
  `_handle_file_convert_retain` → retain task payload
- Add `operation_ids` to `RetainResponse` for async requests with
  mixed per-item strategies
- Add `test_strategy_overrides_extraction_mode_for_index_only`: unit
  test verifying a named strategy with index_only bypasses the LLM
- Add `test_retain_request_per_item_strategy_field`: unit test for
  per-item strategy grouping logic

* feat(ui): add retain strategies and default strategy to bank config UI

- Add StrategiesEditor component: per-strategy cards with name input and
  JSON overrides textarea; supports add/remove; validates JSON inline
- Add Default Strategy text input (retain_default_strategy)
- Update RetainEdits type and retainSlice() to include both new fields
- Regenerate OpenAPI spec (retain_strategies, retain_default_strategy,
  per-item strategy on MemoryItem/FileRetainMetadata, operation_ids on
  RetainResponse)

* refactor(ui): move retain strategies into its own dedicated config section

* feat(ui): improve retain strategies UX and add strategy to document dialog

- Strategy form now includes entity section (free form toggle + entity labels editor)
- Default strategy selector moved outside tab panel, above strategy chips
- Strategy tabs redesigned with underline indicator style for clarity
- Remove strategy confirms with AlertDialog
- Fix tab re-render bug when typing strategy name (skipSyncRef)
- Add strategy field to Add New Document dialog (text + per-file for uploads)
- File upload collapsible uses same Document/Tags/Source tabbed layout
- API: validate empty strategy names in config_resolver
- api.ts: add strategy field to retain and uploadFiles types

* fix: forward strategy through HTTP layer and SDK; add integration test

- route.ts: extract and forward `strategy` from request body to retainBatch
- TypeScript SDK: accept and forward `strategy` in retainBatch options and per-item
- config_resolver.py: validate empty strategy name keys on update
- bank-config-view.tsx: merge entity fields into RetainStrategyForm, redesign strategy tabs with underline style, add confirmation dialog for removal, fix tab-reset-on-typing with skipSyncRef, move default strategy selector outside panel
- bank-selector.tsx: add strategy field to Add Document dialog (per-file in tabbed collapsible)
- test_retain.py: add end-to-end integration test verifying named strategy application (index_only = 0 LLM tokens)

* fix: regenerate TypeScript client with strategy field in RetainRequest/MemoryItem

- Regenerate OpenAPI spec to include strategy field in RetainRequest and MemoryItem
- Regenerate TypeScript client from updated spec
- Add strategy to MemoryItemInput interface
- Remove (item as any) cast now that strategy is properly typed

* rename: index_only extraction mode → chunks

* remove top-level strategy from RetainRequest; strategy is per-item only

* fix(clients): update Go and Python generated clients with strategy/operation_ids fields

* fix(ci): update hierarchical field count, add strategy to Rust MemoryItem initializers

* fix(go-client): minimal targeted YAML updates for strategy/operation_ids fields
2026-03-17 18:08:25 +01:00
Ben
ef90842f87
feat: hindsight-hermes integration for Hermes Agent (#600)
* feat: add hindsight-hermes integration for Hermes Agent

* chore: add Hermes docs page, icon, and release process bindings

- Add cookbook page for Hermes integration (synced with README)
- Add Hermes icon and map hindsight-hermes SDK tag in CookbookGrid
- Add cookbook entry to index.mdx
- Add hindsight-hermes to release.sh PYTHON_PACKAGES array
- Add build, publish, artifact upload, and release asset steps in release.yml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-17 18:06:55 +01:00
Ben
61b01cc040
blog: add n8n persistent memory workflows post (#585)
* blog: add n8n persistent memory workflows post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: add cover image for n8n memory workflows post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: update n8n cover image

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: remove broken screenshot references from n8n post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: add Hindsight Cloud option and n8n Cloud guidance

- Add Cloud vs self-hosted setup paths in Step 1
- Show both Cloud and self-hosted URLs for retain/recall/reflect nodes
- Note that Cloud eliminates the localhost IP gotcha
- Mention n8n Cloud compatibility (requires Hindsight Cloud or public endpoint)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: update n8n post date to 2026-03-16

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: update n8n post with optimized content and fix accuracy

- Use optimized version of the blog post
- Fix blog cross-links to use date-prefixed URLs
- Fix retain response to match actual API (success, bank_id, items_count, async)
- Fix recall response to match actual API (text, type, entities — not confidence/source)
- Update title to "How to Add Persistent Memory to n8n Workflows"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* blog: update n8n post title

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 14:56:56 -04:00
Ben
bd6348aa08
blog: add disposition-aware agents post (#566)
* blog: add disposition-aware agents post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-13 17:53:27 -04:00
Nicolò Boschi
c7db770281
doc: add 0.4.18 release blog post (#567)
* doc: add 0.4.18 release blog post

* doc: include changelog and blog image for 0.4.18
2026-03-13 16:00:59 +01:00
Nicolò Boschi
5fdb0e863f Release v0.4.18
- Update version to 0.4.18 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-litellm, hindsight-crewai, hindsight-pydantic-ai, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- OpenClaw integration: hindsight-integrations/openclaw
- AI SDK integration: hindsight-integrations/ai-sdk
- Chat SDK integration: hindsight-integrations/chat
- Helm chart
- Sync documentation to version-0.4
2026-03-13 15:21:03 +01:00
Nicolò Boschi
5de793eec7
feat: compound tag filtering via tag_groups (#562)
* feat: add compound tag filtering via tag_groups

Adds tag_groups to RecallRequest and ReflectRequest to express arbitrary
boolean tag predicates: leaf {tags, match}, and/or/not compounds.
Top-level groups are AND-ed. Existing tags/tags_match unchanged.

Examples:
  Step filter AND user scope:
    tag_groups: [{tags: ["step:5","step:8"], match: "any_strict"},
                 {tags: ["user:alice"], match: "all_strict"}]
  Exclusion:
    tag_groups: [{tags: ["user:alice"], match: "all_strict"},
                 {not: {tags: ["archived"], match: "any_strict"}}]

- Recursive SQL builder (build_tag_groups_where_clause) threads through
  all 4 retrieval strategies (semantic/BM25, temporal, graph, MPFP)
- Python-side filter (filter_results_by_tag_groups) for post-traversal
- 22 new unit tests
- OpenAPI spec + all clients regenerated (Rust, Python, TypeScript, Go)

* fix: add tag_groups: None to Rust CLI struct initializers

* fix: add tag_groups: None to Rust client test RecallRequest initializer

* feat: reject tags+tag_groups together, add tag_groups integration tests

- Add model_validator to RecallRequest and ReflectRequest that returns 422
  when both `tags` and `tag_groups` are set (mutually exclusive)
- Add 5 integration tests for tag_groups compound filtering:
  * validation: 422 when both fields are set
  * AND filter: two leaf groups (step scope AND user scope)
  * OR compound: user:alice OR user:bob
  * NOT compound: user:alice AND NOT archived
  * Nested: user:alice AND (step:5 OR step:8)

* ci: trigger CI run
2026-03-13 14:30:11 +01:00
Nicolò Boschi
06200f1752
docs: revamp sidebar with icon grids and language support (#563)
* docs: revamp sidebar with icon grid components and language support

- Merge Clients and Integrations sections into the developer sidebar
  (removed top-level SDKs navbar item)
- Reorder sidebar: Architecture → API → Clients → Integrations → Hosting
- Unify icon system using react-icons (LuXxx/SiXxx) via customProps.icon
- Add uppercase section titles with increased spacing and reduced indentation
- Rename Node.js → "JavaScript / TypeScript" with TypeScript icon
- Add reusable IconGrid and SupportedGrids components (ClientsGrid,
  IntegrationsGrid, LLMProvidersGrid)
- Use grids in FAQ, Models, Overview, and Quick Start pages
- Convert developer/index.md, models.md, faq.md to MDX for JSX support

* fix: use inline style for label color to prevent link color inheritance

* fix: label visibility and rename JavaScript/TypeScript to TypeScript

* feat: add HTTP client to grid and OpenAI Compatible to LLM providers grid
2026-03-13 14:12:42 +01:00
Ben
a01bb18bc4
blog: Time-Aware Spreading Activation for Memory Graphs (#547)
doc: add blog post on time-aware spreading activation for memory graphs
2026-03-12 12:55:14 -04:00
Ben
5b360c83d2
doc: Run Hindsight with Ollama: Local AI Memory, No API Keys Needed (#536)
* doc: add run-hindsight-with-ollama blog post
2026-03-11 11:39:59 -04:00
Nicolò Boschi
00ac3d8834
doc: add 0.4.17 release blog post (#538) 2026-03-10 17:40:10 +01:00
Nicolò Boschi
2191654b1f Release v0.4.17
- Update version to 0.4.17 in all components
- Regenerate OpenAPI spec and client SDKs
- Python packages: hindsight-api, hindsight-dev, hindsight-all, hindsight-litellm, hindsight-crewai, hindsight-pydantic-ai, hindsight-embed
- Python client: hindsight-clients/python
- TypeScript client: hindsight-clients/typescript
- Rust CLI: hindsight-cli
- Control Plane: hindsight-control-plane
- OpenClaw integration: hindsight-integrations/openclaw
- AI SDK integration: hindsight-integrations/ai-sdk
- Chat SDK integration: hindsight-integrations/chat
- Helm chart
- Sync documentation to version-0.4
2026-03-10 17:18:35 +01:00
Nicolò Boschi
dcaacbe407
feat: add manual retry for failed async operations (#537)
- API: POST /v1/default/banks/{bank_id}/operations/{operation_id}/retry
  resets status to pending so the worker re-executes the task
- UI: Retry button on failed operations in the operations view
- Control plane proxy route + ControlPlaneClient.retryOperation()
- Updated OpenAPI spec, all generated clients, and operations docs
2026-03-10 17:16:11 +01:00
Ben
7bcf26097c
doc: Your Pydantic AI Agent Forgets You After Every Run. Fix It in 5 Lines. (#531)
* doc: add pydantic-ai-persistent-memory blog post

* doc: update Pydantic AI blog cover image

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* doc: SEO-optimized rewrite of Pydantic AI blog post

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 16:39:20 -04:00
Nicolò Boschi
f7a60f898d
feat: filter operations by type + fix stale auto-refresh closure (#522) (#527)
* feat: filter operations by type + fix stale closure in auto-refresh

- Add `type` query param to GET /operations endpoint and engine layer
- Add operation type dropdown filter in Background Operations UI
- Fix auto-refresh interval using stale statusFilter/offset closure by
  adding filter state to useEffect deps and wrapping loadOperations in
  useCallback (fixes #522)
- Regenerate OpenAPI spec and all SDK clients

* fix: update Rust CLI list_operations call with new type parameter
2026-03-09 13:17:00 +01:00
Nicolò Boschi
1b4ad7f435
feat: change tags for a document (#517)
* feat: add update document tags endpoint with observation invalidation

Adds PATCH /v1/default/banks/{bank_id}/documents/{document_id} to change
tags on a document without re-processing content.

- Updates tags on the document and all associated memory units atomically
- Invalidates observations derived from the document's memory units
- Resets consolidated_at on the document's own units for re-consolidation
- Also resets consolidated_at on co-source memories from other documents
  that shared those observations (matching delete_document behavior)
- Triggers async consolidation when observations are invalidated
- 9 new tests covering all invalidation scenarios

UI: adds inline tag editor to the document detail panel in the control plane
Docs: new "Update Document Tags" section in documents.mdx with Python/JS examples

* refactor: simplify UpdateDocumentTagsResponse to {success: true}

* refactor: make PATCH /documents generic update_document endpoint

Renames update_document_tags → update_document (engine + HTTP + clients + UI).
Currently only tags are supported; the structure is open for future fields.
Tags are the only field with side effects (observation invalidation + re-consolidation).
2026-03-07 09:00:13 +01:00
Ben
d9d7021a49
doc: Upgrading OpenClaw's Memory with Hindsight (#515)
* doc: add adding-memory-to-openclaw-with-hindsight blog post

* doc: update OpenClaw blog cover image

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* doc: update OpenClaw blog title

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* doc: add Hindsight Cloud note to external API section

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 12:35:12 -05:00
Nicolò Boschi
e2baca8bfe
feat: mental model history tracking and UI diff view (#516)
* feat: mental model refresh history tracking and UI diff view

- DB migration: add history JSONB column to mental_models table
- Track previous content on each refresh in update_mental_model
- Add get_mental_model_history() engine method
- New GET /mental-models/{id}/history endpoint
- Control plane proxy route and getMentalModelHistory() in api.ts
- MentalModelDetailModal: add History tab with lazy loading, carousel
  navigation (left=older, right=newer), word-level content diff view

* fix: resolve alembic migration head conflict for mental model history

* feat: mental model history tracking, side-by-side diff UI, and config flag

- Track content changes on every mental model update/refresh (persisted in JSONB history column)
- New GET /mental-models/{id}/history endpoint returning changes most-recent-first
- Side-by-side diff view in History tab (Before/After columns, line-level highlights)
- Actions dropdown in detail panel (Edit, Refresh, View History, Delete)
- HINDSIGHT_API_ENABLE_MENTAL_MODEL_HISTORY config flag (default: true)
- Also adds missing HINDSIGHT_API_ENABLE_OBSERVATION_HISTORY to configuration docs
- Python client wrapper method get_mental_model_history()
- Tests for history persistence (recorded, ordered, name-only skipped, missing returns None)
- Fix NameError: timezone not imported in update_mental_model

* fix: call get_mental_model_history before delete in doc example
2026-03-06 17:50:48 +01:00
Nicolò Boschi
576473b6aa
feat: observation history tracking and diff UI (#513)
* feat: add source facts token limits to consolidation and recall

- Add two new configurable (per-bank) parameters:
  - consolidation_source_facts_max_tokens: total token budget for source
    facts across all observations in the consolidation prompt (-1 = unlimited)
  - consolidation_source_facts_max_tokens_per_observation: per-observation
    cap so each observation gets a fair share of source facts (-1 = unlimited,
    default 256)
- Both are also exposed as recall API parameters via SourceFactsIncludeOptions
  (max_tokens and max_tokens_per_observation)
- Consolidation now uses resolve_full_config to respect bank-level overrides
- Improve consolidation prompt: temporal metadata (occurred_start=, | Involving:)
  is now clearly separated from observation text, with a concrete example showing
  the expected synthesis style and explicit rules not to copy raw fact lines
- Add tests for recall source facts capping and consolidation config forwarding
- Expose all three new fields in the control plane bank config UI
- Document new env vars in configuration.md
- Regenerate OpenAPI spec and all SDK clients

* fix: reorder observations UI fields and rename Label Groups to Entity Labels

* fix: revert Entities section title (only rename inner label)

* doc: add consolidation source facts and batch size fields to memory-banks docs

* feat: add observation history tracking and UI diff view

- Track observation changes over time in a JSONB history column,
  appending each update's previous state (text, tags, dates, sources)
  instead of overwriting
- Add HINDSIGHT_API_ENABLE_OBSERVATION_HISTORY config flag (default: true)
  to toggle history recording
- Expose history field in get_memory_unit for observations
- Fix observations/[modelId] route that was proxying to wrong endpoint
- Add History tab in observation modal and History section in panel,
  showing word-level and tag diffs between each change (newest first)
- Extract shared ObservationHistoryView component used by both modal and panel
- Add --random-port flag to start.sh to run multiple dev instances
- Scope Next.js distDir by port to prevent lock file collisions between instances
- Restyle consolidation pending badge (rounded-md with border) and add
  inline refresh button; fix loading flicker on data refresh

* feat: dedicated observation history endpoint with source facts diff

- Add GET /memories/{id}/history endpoint returning enriched history with
  resolved source fact texts and is_new flags per change
- Deprecate history field in GET /memories/{id} (always returns empty list)
- Reconstruct cumulative source facts per history entry by working backwards
  from current state, marking newly added facts with is_new
- Replace inline history panel with "View History" button opening modal
- History modal fetches from dedicated endpoint lazily on tab switch
- Timeline view now opens MemoryDetailModal instead of side panel
- History view uses prev/next navigation (left = older, right = newer)
- Fix --random-port: pass dynamic API_PORT as HINDSIGHT_CP_DATAPLANE_API_URL
  to control plane, preserving caller values over .env
2026-03-06 16:16:05 +01:00