* Add actual LLM token usage fields to RetainResult
RetainResult now carries llm_input_tokens, llm_output_tokens, and
llm_total_tokens populated from the engine's TokenUsage, so downstream
operation validator extensions can access actual LLM token counts.
* Test that RetainResult includes actual LLM token usage
Move the Supabase tenant extension into the hindsight-api package so users
can enable it with just an environment variable — no file copying or Docker
image modifications needed.
Key improvements over the original submission:
- JWKS-based local JWT verification (no network call per request) with
automatic fallback to /auth/v1/user for legacy HS256 projects
- Service key is now optional (only needed for HS256 or health checks)
- UUID validation on user IDs before schema name construction
- Schema prefix validation against Postgres identifier rules
- Key rotation handling with automatic JWKS cache refresh
- Proper logging via Python logging module
- Tenant extension lifecycle hooks (on_startup/on_shutdown) wired into
the server lifespan
- Public tenant_extension property on MemoryEngine
- 54 unit tests covering both verification modes, cache behavior, error
paths, and the extension loader
- README updated to reflect JWKS-first architecture
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: add TenantExtension auth to MCP endpoint
Replace static MCP_AUTH_TOKEN check with TenantExtension authentication,
making MCP use the same auth path as REST API.
- MCPMiddleware now calls tenant_extension.authenticate()
- Sets _current_schema from TenantContext for multi-tenant isolation
- Returns 401 on AuthenticationError (same as REST API)
- DefaultTenantExtension: no auth (local dev)
- ApiKeyTenantExtension: validates against env var
- CloudTenantExtension: HMAC + DB lookup (production)
Adds tests for middleware auth rejection, acceptance, and schema routing.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Address PR review: backwards compatibility for MCP auth
- Keep MCP_AUTH_TOKEN env var for legacy MCP servers
- Add authenticate_mcp() method to TenantExtension base class
- Default implementation calls authenticate()
- Extensions can override to opt-out of MCP auth
- Add mcp_auth_disabled config option to ApiKeyTenantExtension
- Set HINDSIGHT_API_TENANT_MCP_AUTH_DISABLED=true to skip MCP auth
- Remove CloudTenantExtension from public docstring
- Add tests for legacy auth token and mcp_auth_disabled flag
- Update MCP docs with new auth configuration
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add search_docs MCP tool for documentation search
Implements a new MCP tool that searches Hindsight documentation using
Vectorize RAG pipelines. The tool supports:
- Searching core (OSS) docs, cloud docs, or both
- Configurable number of results (1-10)
- Returns ranked results with URLs, similarity scores, and text snippets
New environment variables:
- HINDSIGHT_API_VECTORIZE_ORG_ID
- HINDSIGHT_API_VECTORIZE_API_TOKEN
- HINDSIGHT_API_VECTORIZE_CORE_PIPELINE_ID
- HINDSIGHT_API_VECTORIZE_CLOUD_PIPELINE_ID
- HINDSIGHT_API_VECTORIZE_API_BASE_URL
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add documentation for search_docs MCP tool
- Add Vectorize environment variables to configuration.md
- Add search_docs tool to MCP server available tools
- Add reflect tool documentation (was missing)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add tests for search_docs MCP tool
Tests cover:
- DocsSource enum values and parsing
- _clean_text HTML stripping helper
- _search_vectorize_pipeline with mocked httpx
- Tool registration and function execution
- Source filtering (core/cloud/all)
- Result sorting by similarity
- Error handling for pipeline failures
- HTML cleaning in results
- Invalid source defaulting to 'all'
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Move search_docs to hindsight-cloud, add MCPExtension pattern
- Add MCPExtension base class for registering additional MCP tools
- Load MCPExtension in create_mcp_server when configured
- Remove search_docs tool (moved to hindsight-cloud CloudMCPExtension)
- Remove Vectorize config from hindsight-core
- Add tests for MCPExtension pattern
- Update docs to remove search_docs references
The MCPExtension pattern allows cloud (or any extension package) to
register additional MCP tools via:
HINDSIGHT_API_MCP_EXTENSION=package.module:ExtensionClass
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Address PR review feedback
- Remove CloudTenantExtension mention from MCPMiddleware docstring
- Fix docs: clarify that ApiKeyTenantExtension must be explicitly enabled
- Revert changes to versioned docs (0.3 and 0.4) - synced automatically on release
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Format mcp.py line length
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix: custom pg schema is not reliable
* fix
* fix
* fix: WorkerPoller now always has tenant extension
Ensures WorkerPoller follows same pattern as MemoryEngine - always
creates a DefaultTenantExtension if none is provided, preventing
NoneType errors when calling list_tenants().
Fixes test failures in test_worker.py
* fix: DefaultTenantExtension honors explicit schema parameter
Allows WorkerPoller's schema parameter to be passed through to
DefaultTenantExtension via config dict, maintaining backward
compatibility for tests that use schema parameter without
providing a tenant extension.
Fixes test_poller_with_custom_schema test failure.
Wire up validate_mental_model_refresh hook in the HTTP routes for both
create and refresh mental model endpoints, allowing extensions to reject
operations (e.g. insufficient credits) before queuing async LLM work.
Add dataclasses and hook methods to OperationValidatorExtension for
tracking mental model operations:
- MentalModelGetContext/Result: context and result for GET operations
- MentalModelRefreshResult: result for refresh operations with token counts
- validate_mental_model_get: pre-operation validation hook
- on_mental_model_get_complete: post-GET completion hook
- on_mental_model_refresh_complete: post-refresh completion hook
Invoke hooks in http.py (GET endpoint) and memory_engine.py (refresh).
Add tests verifying hooks are called with correct parameters.
* chore: run benchmarks with reflect mode
* chore: run benchmarks with reflect mode
* fixes
* new mm
* bunch of fixes
* initial commit
* fixes
* fixes
* fixes
* fix: sometimes memories gets extracted in the wrong language
Call ensure_embedding_dimension after running migrations for tenant
schemas. This ensures the embedding column dimension matches the
model's dimension, which may differ from the default 384 dimensions
used in the initial migration.
Without this fix, using embedding providers with different dimensions
(e.g., Cohere's embed-english-v3.0 with 1024 dims) would fail with
"expected 384 dimensions, not 1024" errors on tenant schemas.
* Load operation validator extension in main entry point
Enable the operation validator extension to be loaded from environment
configuration and passed to MemoryEngine, allowing pre/post operation
hooks for usage metering, rate limiting, and audit logging.
* Fix reflect background task authentication and add internal flag
- Pass API key to background opinion storage task for proper auth
- Add internal flag to RequestContext for tracking internal operations
- Background opinion storage now authenticates correctly with tenant
* Add api_key_id to RequestContext for usage tracking
- Add api_key_id field to RequestContext to track which API key was used
- Enables per-API-key usage analytics in the metering system
* Fix HTTP error handling for authentication and validation errors
- Add status_code parameter to ValidationResult and OperationValidationError
- Convert OperationValidationError to HTTPException with proper status codes
- Fix authentication errors to return 401 instead of raising internal errors
- Re-raise HTTPException in exception handlers to prevent swallowing errors
* Fix AuthenticationError handling in memory engine
- Raise AuthenticationError from memory_engine._authenticate_tenant instead
of HTTPException so unit tests pass
- Add AuthenticationError handling in HTTP layer to convert to 401 responses
- Fixes failing TestMemoryEngineTenantAuth tests
* Add global exception handler for AuthenticationError
Returns proper 401 status code for all authentication failures
across all endpoints, not just the ones with explicit handlers.
* Simplify exception handling: use global AuthenticationError handler
- Remove redundant individual exception handlers
- Add 'except AuthenticationError: raise' before generic Exception handlers
to let global handler process auth errors uniformly
* Refactor background tasks to use tenant_id instead of api_key
This makes the core more generic - it passes tenant_id (which is
extension-agnostic) rather than api_key (which is cloud-specific).
- Add tenant_id field to RequestContext
- Pass tenant_id instead of api_key to background tasks
- Extensions can check internal=True with tenant_id to bypass normal auth
* Fix exception propagation: include HTTPException in re-raise
After cleanup of redundant exception handlers, 404 errors were
returning 500 because HTTPException was caught by the generic
except Exception handler. Fixed by combining AuthenticationError
and HTTPException in the re-raise pattern.
