* feat: mental model refresh history tracking and UI diff view
- DB migration: add history JSONB column to mental_models table
- Track previous content on each refresh in update_mental_model
- Add get_mental_model_history() engine method
- New GET /mental-models/{id}/history endpoint
- Control plane proxy route and getMentalModelHistory() in api.ts
- MentalModelDetailModal: add History tab with lazy loading, carousel
navigation (left=older, right=newer), word-level content diff view
* fix: resolve alembic migration head conflict for mental model history
* feat: mental model history tracking, side-by-side diff UI, and config flag
- Track content changes on every mental model update/refresh (persisted in JSONB history column)
- New GET /mental-models/{id}/history endpoint returning changes most-recent-first
- Side-by-side diff view in History tab (Before/After columns, line-level highlights)
- Actions dropdown in detail panel (Edit, Refresh, View History, Delete)
- HINDSIGHT_API_ENABLE_MENTAL_MODEL_HISTORY config flag (default: true)
- Also adds missing HINDSIGHT_API_ENABLE_OBSERVATION_HISTORY to configuration docs
- Python client wrapper method get_mental_model_history()
- Tests for history persistence (recorded, ordered, name-only skipped, missing returns None)
- Fix NameError: timezone not imported in update_mental_model
* fix: call get_mental_model_history before delete in doc example
* feat: add source facts token limits to consolidation and recall
- Add two new configurable (per-bank) parameters:
- consolidation_source_facts_max_tokens: total token budget for source
facts across all observations in the consolidation prompt (-1 = unlimited)
- consolidation_source_facts_max_tokens_per_observation: per-observation
cap so each observation gets a fair share of source facts (-1 = unlimited,
default 256)
- Both are also exposed as recall API parameters via SourceFactsIncludeOptions
(max_tokens and max_tokens_per_observation)
- Consolidation now uses resolve_full_config to respect bank-level overrides
- Improve consolidation prompt: temporal metadata (occurred_start=, | Involving:)
is now clearly separated from observation text, with a concrete example showing
the expected synthesis style and explicit rules not to copy raw fact lines
- Add tests for recall source facts capping and consolidation config forwarding
- Expose all three new fields in the control plane bank config UI
- Document new env vars in configuration.md
- Regenerate OpenAPI spec and all SDK clients
* fix: reorder observations UI fields and rename Label Groups to Entity Labels
* fix: revert Entities section title (only rename inner label)
* doc: add consolidation source facts and batch size fields to memory-banks docs
* feat: add observation history tracking and UI diff view
- Track observation changes over time in a JSONB history column,
appending each update's previous state (text, tags, dates, sources)
instead of overwriting
- Add HINDSIGHT_API_ENABLE_OBSERVATION_HISTORY config flag (default: true)
to toggle history recording
- Expose history field in get_memory_unit for observations
- Fix observations/[modelId] route that was proxying to wrong endpoint
- Add History tab in observation modal and History section in panel,
showing word-level and tag diffs between each change (newest first)
- Extract shared ObservationHistoryView component used by both modal and panel
- Add --random-port flag to start.sh to run multiple dev instances
- Scope Next.js distDir by port to prevent lock file collisions between instances
- Restyle consolidation pending badge (rounded-md with border) and add
inline refresh button; fix loading flicker on data refresh
* feat: dedicated observation history endpoint with source facts diff
- Add GET /memories/{id}/history endpoint returning enriched history with
resolved source fact texts and is_new flags per change
- Deprecate history field in GET /memories/{id} (always returns empty list)
- Reconstruct cumulative source facts per history entry by working backwards
from current state, marking newly added facts with is_new
- Replace inline history panel with "View History" button opening modal
- History modal fetches from dedicated endpoint lazily on tab switch
- Timeline view now opens MemoryDetailModal instead of side panel
- History view uses prev/next navigation (left = older, right = newer)
- Fix --random-port: pass dynamic API_PORT as HINDSIGHT_CP_DATAPLANE_API_URL
to control plane, preserving caller values over .env
* feat: allow per-request file parser selection with fallback chains
Clients can now specify which parser(s) to use when calling the file
retain endpoint, instead of being locked to the server-side default.
Changes:
- `parser` field added to `FileRetainRequest` (request-level default)
and `FileRetainMetadata` (per-file override); accepts a single name
or an ordered fallback chain (list)
- Resolution priority: per-file > request-level > server default
- `HINDSIGHT_API_FILE_PARSER` now accepts a comma-separated fallback
chain (e.g. `iris,markitdown`); fully backward-compatible
- New `HINDSIGHT_API_FILE_PARSER_ALLOWLIST` env var restricts which
parsers clients may request (defaults to all registered parsers)
- Invalid/disallowed parser names are rejected with HTTP 400
- `FileParserRegistry.convert_with_fallback()` tries each parser in
order, falling back on UnsupportedFileTypeError, empty content, or
any other error
- Worker updated to use the fallback chain stored per-task
- OpenAPI spec and all generated clients regenerated
* fix: handle on_file_convert_complete hook and rebase onto main
- Return ConvertResult dataclass from convert_with_fallback() instead
of a plain str, carrying both the content and the winning parser name
- Use winning_parser_name in the on_file_convert_complete hook so
parser_name reflects the parser that actually succeeded, not the chain
- Update all test calls to submit_async_file_retain() to use the new
per-item parser field instead of the removed top-level parser= kwarg
* docs: document HINDSIGHT_API_FILE_PARSER fallback chain and ALLOWLIST
* doc: update cookbook
* fix(cookbook): preserve tag keys during sync, strip local .md links
- Fix extract_tags_from_readme/notebook to return dict[str,str] preserving
sdk/topic keys instead of bare values, preventing topics like
"Customer Service" from being misclassified as SDK
- Add strip_local_md_links() to remove relative .md references that
would cause broken link errors in Docusaurus build
* ci: run test-doc-examples independently without waiting for test-rust-cli
Build the CLI directly in the job instead of downloading the artifact,
so test-doc-examples can start at the beginning in parallel with all other jobs.
* feat: webhook system with task-owned retry, retain.completed event, and UI
- New webhook system: register per-bank webhooks with HMAC signing, configurable
HTTP method/timeout/headers/params (http_config JSONB), and PATCH support
- Webhook deliveries run as async_operations (webhook_delivery type) with
task-owned retry via RetryTaskAt exception and exponential backoff
(60s / 5m / 30m / 2h / 8h, max 6 attempts)
- New retain.completed event fires per-document for both sync and async retain
- Delivery debug info (status code, response body) stored in result_metadata
- Control plane UI: webhooks tab per bank with create/edit/delete and a
deliveries table with cursor pagination and expandable response details
- 28 webhook tests covering HMAC signing, delivery retries, CRUD endpoints,
PATCH update, and retain.completed queuing
- Docs page at developer/api/webhooks documenting event payloads and delivery
- OpenAPI spec and all client SDKs (Python, TypeScript, Rust, Go) regenerated
* fix: update tests for task-owned retry model and guard _webhook_manager attribute
- test_worker.py: test_executor_exception_triggers_retry now raises RetryTaskAt
(plain exceptions are immediate failures in the new system); rename
test_executor_exception_marks_failed_after_max_retries to
test_executor_exception_marks_failed_immediately to reflect new semantics
- test_batch_api.py: remove max_retries kwarg from WorkerPoller constructor
- memory_engine.py: use getattr for _webhook_manager in _fire_retain_webhook
to avoid AttributeError when engine is created without __init__ (tests)
* fix: remove max_retries from benchmark WorkerPoller call
* fix(webhooks): transactional outbox, observations_deleted tracking, sidebar
- Queue webhook delivery rows atomically with the primary operation using the
transactional outbox pattern — prevents lost events on process crash:
- Retain (sync + async): outbox_callback passed into orchestrator.retain_batch
and called inside the DB transaction, replacing the post-commit fire call
- Consolidation: new _mark_operation_completed_and_fire_webhook combines the
status UPDATE and webhook INSERT in one transaction
- Added fire_event_with_conn() to WebhookManager for in-connection delivery
- Track observations_deleted count in consolidation stats and expose it in the
consolidation.completed webhook payload (was always None)
- Add Webhooks page to docs sidebar
- Document at-least-once delivery guarantee with operation_id dedup guidance
* fix(ui): add retain.completed to available webhook event types
* feat(ui): add delete confirmation dialog for webhooks
* fix(webhooks): include operation_id in task_payload so delivery is marked completed
The task_payload JSON was missing the operation_id field, causing execute_task
to see operation_id=None and skip _mark_operation_completed — leaving every
delivery row stuck in 'pending' forever.
Added a test that inserts a real async_operations row and verifies the status
transitions to 'completed' after a successful execute_task call.
* style: fix prettier formatting in webhooks-view
* feat: add Pydantic AI integration to CI, release pipeline, and docs
- Add test-pydantic-ai-integration job to CI (test.yml)
- Add build, publish, and artifact steps to release workflow (release.yml)
- Add hindsight-integrations/pydantic-ai to release.sh version bumping
- Add Pydantic AI documentation page (sdks/integrations/pydantic-ai.md)
- Add Pydantic AI entry to sidebar with icon
* docs: remove Requirements section from pydantic-ai integration page
* feat: add tags filtering and fix offset pagination docs for list documents API
- Add `tags` and `tags_match` query params to GET /banks/{bank_id}/documents
- Supports any, all, any_strict, all_strict matching modes (default: any_strict)
- Fix `q` param description — it's a case-insensitive substring match on document ID only
- Add tests for offset pagination and all tags_match modes
- Regenerate OpenAPI spec and Python/TypeScript/Go clients
- Document the new filtering options in docs/developer/api/documents.mdx
* fix(cli): pass new tags/tags_match args to list_documents
* feat: filter graph memories with tags
* fix(cli): pass new q/tags/tags_match args to get_graph
* docs: use CodeSnippet for tags_match examples in recall.mdx
* fix: handle observations regeneration when memories get deleted
* feat: add clear_memory_observations endpoint and regenerate clients
- Add DELETE /banks/{id}/memories/{memory_id}/observations endpoint
- Add observations lifecycle/invalidation section to docs
- Regenerate OpenAPI spec and all clients (Python, TypeScript, Go, Rust)
* refactor: use dedicated response model for clear_memory_observations, remove code example from docs
* feat: accept pdf, images and office files
* refactor: rename FileConverter to FileParser, simplify file retain API
- Rename engine/converters/ → engine/parsers/, FileConverter → FileParser,
ConverterRegistry → FileParserRegistry, MarkitdownConverter → MarkitdownParser
- Rename env var HINDSIGHT_API_FILE_CONVERTER → HINDSIGHT_API_FILE_PARSER
- Remove async/document_tags params from FileRetainRequest (always async now)
- Add retain_files() to Python Hindsight client and retainFiles() to TypeScript client
- Add sample.pdf to doc examples for working file upload demonstrations
- Update test_file_retain.py to use new parser names and always-async behavior
- Fix Go client missing os import in api_files.go
- Simplify postgresql.py storage to minimal schema
* fix: update rust CLI tests to use is_supported_file instead of is_text_file
* fix: patch Go api_files.go to add missing 'os' import after generation
* fix: insert 'os' import after 'net/url' in api_files.go patch for correct position
* chore: regenerate OpenAPI spec and clients (converter→parser description update)
* feat: implement hierarchical configuration (system, tenant, bank)
* feat: implement hierarchical configuration (system, tenant, bank)
* docs: add instructions for hierarchical config in CLAUDE.md
* feat: add ENABLE_BANK_CONFIG_API flag (disabled by default)
- Add HINDSIGHT_API_ENABLE_BANK_CONFIG_API env var (default: false)
- Return 403 Forbidden from bank config endpoints when disabled
- Update tests to enable the flag
- Update CLAUDE.md documentation
This provides security control over the bank configuration API,
ensuring it's only accessible when explicitly enabled.
* docs: add hierarchical configuration section
* feat(cli): add bank config commands (config, set-config, reset-config)
- Add 'hindsight bank config' to view bank configuration
- Add 'hindsight bank set-config' to update LLM settings per bank
- Add 'hindsight bank reset-config' to reset to defaults
- Implements client API calls to new bank config endpoints
* fix(cli): fix compilation errors in bank config commands
- Fix type signature: use ApiClient instead of api::Client
- Fix confirmation: use ui::prompt_confirmation instead of ui::confirm
- Fix error handling: use anyhow! macro instead of errors::Error
- Fix type conversion: convert HashMap to serde_json::Map for API call
* feat: implement type-safe hierarchical config with bank overrides
Implements a production-ready hierarchical configuration system that prevents
accidentally using global defaults when bank-specific overrides exist.
- Created StaticConfigProxy that wraps HindsightConfig
- get_config() now returns proxy that blocks access to bank-configurable fields
- Raises ConfigFieldAccessError with clear message when accessing configurable fields
- Added _get_raw_config() for internal use only
- Forces developers to use resolve_full_config(bank_id, context) for bank settings
- Added resolve_full_config() method that returns complete HindsightConfig
- Resolves hierarchy: Global (env) → Tenant → Bank
- No caching to support multi-server deployments (always fresh from DB)
- LLM provider pooling handles expensive operations separately
- Updated entire retain pipeline to pass resolved config through call chain
- memory_engine.py: Resolves config at top level where bank_id/context available
- orchestrator.py: Accepts and passes config to fact_extraction
- fact_extraction.py: Uses passed config instead of get_config()
- utils.py: Added optional config param for backward compatibility
- consolidator.py: Uses resolve_full_config() for enable_observations check
- memory_engine.py: Resolves config before triggering consolidation
- Renamed "Memory Bank" to "Bank Configuration" with tabs
- Combined Stats and Operations into "General" tab
- Consolidated Profile and Configuration into "Configuration" tab
- Moved Actions dropdown to page level (outside tabs)
- Created new component for managing bank-specific config
- Displays configurable fields: retain_chunk_size, retain_extraction_mode, etc.
- Edit via dialog with form validation
- Reset to defaults via AlertDialog confirmation
- Shows field IDs in monospace for clarity
- Visual separation with borders and hover effects
- Removed inline edit mode, switched to dialog-based editing
- Separate dialogs for Disposition and Mission editing
- Read-only display with clear edit buttons
- Removed duplicate stats cards and operations
- bank-stats-view.tsx: Overview statistics (memories, links, documents, pending ops)
- bank-operations-view.tsx: Background operations table with filtering
**Problem**: Consolidation always used global enable_observations, ignoring bank overrides
**Root Cause**: consolidator.py called get_config() instead of resolving bank-specific config
**Solution**: Pass resolved config through the entire pipeline
**Problem**: asyncpg returning JSONB as JSON string instead of parsed dict
**Solution**: Explicit JSON parsing in config_resolver.py with type checking
- All 19 API integration tests pass
- All 10 hierarchical config tests pass
- Retain operations work correctly with bank-specific config
- Consolidation respects bank-specific enable_observations setting
- Updated developer/configuration.md with type-safe config access pattern
- Added examples showing correct usage patterns
- Documented ConfigFieldAccessError and resolution methods
- get_config() now returns StaticConfigProxy (blocks configurable field access)
- Code accessing bank-configurable fields must use resolve_full_config()
- Clear migration path with helpful error messages
Fixes hierarchical configuration to be production-ready with proper type safety.
* refactor: remove LLM client pool and simplify config resolver
Since LLM config (provider, model, api_key) is now static and not
bank-configurable, the LLMClientPool is no longer needed.
Changes:
- Remove hindsight_api/llm_client_pool.py (no longer needed)
- Remove memory_engine._get_bank_llm_config() (dead code, never called)
- Simplify config_resolver.py by eliminating duplication between
resolve_full_config() and get_bank_config()
- get_bank_config() now calls resolve_full_config() and filters results
- Remove outdated "LLM provider pooling" comments from docstrings
All tests pass (10 hierarchical config tests, 19 API integration tests)
* fix: update tests to use _get_raw_config() for configurable fields
Fixed test fixtures that were accessing configurable fields (like
enable_observations) from get_config(), which now raises
ConfigFieldAccessError due to type-safe config access.
Changes:
- test_consolidation.py: Changed enable_observations fixture to use
_get_raw_config() instead of get_config()
- test_consolidation.py: Updated test_consolidation_returns_disabled_status
to set bank config instead of mocking get_config()
- test_link_expansion_retrieval.py: Changed fixture to use _get_raw_config()
- test_observations.py: Changed disable_observations fixture to use
_get_raw_config()
- Regenerated OpenAPI spec and clients
All 39 previously failing tests now pass.
* fix: add missing config parameter to test calls of extract_facts_from_text()
Fixed 45 test failures where tests were calling extract_facts_from_text()
without the new required config parameter.
Changes:
- Added config=_get_raw_config() to all extract_facts_from_text() calls
- Fixed test_main_module.py to patch _get_raw_config instead of get_config
- Updated 6 test files with 37 function call sites
All tests should now pass.
* fix: add missing config parameter to test_skip_podcast_meta_commentary
One more test was missing the config parameter for extract_facts_from_text().
* chore: remove dead code
* chore: remove extract_opinions from test and regenerate openapi
- Remove extract_opinions parameter from test_fact_extraction_analysis
- Regenerate OpenAPI spec after removing entity observations code
* chore: update generated files and apply formatting
- Regenerate Python and TypeScript client SDKs after main merge
- Apply ruff formatting to llm_wrapper.py
* fix: accept and filter deprecated 'opinion' fact type in recall
The dead code removal eliminated support for the 'opinion' fact type,
but existing clients may still pass it. Instead of rejecting it with
a ValueError, silently filter it out before validation to maintain
backward compatibility.
* feat: support vertex as llm provider
* fix
* fix: add uv index-strategy to resolve dependency conflicts with pytorch index
When using pytorch index for faster torch downloads in CI,
filelock dependency resolution was failing because pytorch index
only has older versions. Adding unsafe-best-match strategy allows
uv to search all configured indexes.
Also fix type checking warnings from ty.
* fix: add index-strategy to root pyproject.toml for workspace-level uv resolution
* chore: regenerate client SDKs after Vertex AI support
* fix: misc fixes for observations and mental models
* feat: improve graph retrieval for observations
- Update LinkExpansionRetriever to traverse through source_memory_ids
for observation entity connections (avoiding data duplication)
- Remove entity link copy from world facts to observations in consolidator
- Add tests for link expansion graph retrieval
- Add directives_applied field to ReflectResult
- Include user's other changes (CLI, docs, client updates)
* fix: CI test failures
- Add mental_model_id parameter to create_mental_model function
- Fix ToolCallTrace not including reason field from ToolCall
- Improve test_link_expansion_observation_graph_retrieval to wait for consolidation with retry
* chore: reduce link expansion log verbosity
* Revert "chore: reduce link expansion log verbosity"
This reverts commit 3ce759391cead1012157785fa78fef16ef9bfe3b.
* feat: add semantic/temporal/entity links as fallback in graph retrieval
- Add fallback query for semantic, temporal, and entity links from memory_links
- Check both directions (outgoing and incoming links)
- Weight fallback results at 0.5x to prioritize entity links via unit_entities
- Fixes graph retrieval returning 0 when data has cross-cluster temporal connections
* fix: enable observations fixture for link expansion test
- Add enable_observations fixture to ensure observations are created
- Increase wait time from 10 to 30 seconds for CI reliability
* chore: run benchmarks with reflect mode
* chore: run benchmarks with reflect mode
* fixes
* new mm
* bunch of fixes
* initial commit
* fixes
* fixes
* fixes
* fix: sometimes memories gets extracted in the wrong language
* ci: frozen uv sync
* fix: add missing authorization parameter to get_agent_stats in CLI
The generated Rust client was updated with an authorization header
parameter for get_agent_stats, but the CLI code wasn't updated.
