fleet-memory/hindsight-api/hindsight_api/extensions/tenant.py
Nicolò Boschi 8d731f2e5f
feat: implement hierarchical configuration (system, tenant, bank) (#329)
* feat: implement hierarchical configuration (system, tenant, bank)

* feat: implement hierarchical configuration (system, tenant, bank)

* docs: add instructions for hierarchical config in CLAUDE.md

* feat: add ENABLE_BANK_CONFIG_API flag (disabled by default)

- Add HINDSIGHT_API_ENABLE_BANK_CONFIG_API env var (default: false)
- Return 403 Forbidden from bank config endpoints when disabled
- Update tests to enable the flag
- Update CLAUDE.md documentation

This provides security control over the bank configuration API,
ensuring it's only accessible when explicitly enabled.

* docs: add hierarchical configuration section

* feat(cli): add bank config commands (config, set-config, reset-config)

- Add 'hindsight bank config' to view bank configuration
- Add 'hindsight bank set-config' to update LLM settings per bank
- Add 'hindsight bank reset-config' to reset to defaults
- Implements client API calls to new bank config endpoints

* fix(cli): fix compilation errors in bank config commands

- Fix type signature: use ApiClient instead of api::Client
- Fix confirmation: use ui::prompt_confirmation instead of ui::confirm
- Fix error handling: use anyhow! macro instead of errors::Error
- Fix type conversion: convert HashMap to serde_json::Map for API call

* feat: implement type-safe hierarchical config with bank overrides

Implements a production-ready hierarchical configuration system that prevents
accidentally using global defaults when bank-specific overrides exist.

- Created StaticConfigProxy that wraps HindsightConfig
- get_config() now returns proxy that blocks access to bank-configurable fields
- Raises ConfigFieldAccessError with clear message when accessing configurable fields
- Added _get_raw_config() for internal use only
- Forces developers to use resolve_full_config(bank_id, context) for bank settings

- Added resolve_full_config() method that returns complete HindsightConfig
- Resolves hierarchy: Global (env) → Tenant → Bank
- No caching to support multi-server deployments (always fresh from DB)
- LLM provider pooling handles expensive operations separately

- Updated entire retain pipeline to pass resolved config through call chain
- memory_engine.py: Resolves config at top level where bank_id/context available
- orchestrator.py: Accepts and passes config to fact_extraction
- fact_extraction.py: Uses passed config instead of get_config()
- utils.py: Added optional config param for backward compatibility

- consolidator.py: Uses resolve_full_config() for enable_observations check
- memory_engine.py: Resolves config before triggering consolidation

- Renamed "Memory Bank" to "Bank Configuration" with tabs
- Combined Stats and Operations into "General" tab
- Consolidated Profile and Configuration into "Configuration" tab
- Moved Actions dropdown to page level (outside tabs)

- Created new component for managing bank-specific config
- Displays configurable fields: retain_chunk_size, retain_extraction_mode, etc.
- Edit via dialog with form validation
- Reset to defaults via AlertDialog confirmation
- Shows field IDs in monospace for clarity
- Visual separation with borders and hover effects

- Removed inline edit mode, switched to dialog-based editing
- Separate dialogs for Disposition and Mission editing
- Read-only display with clear edit buttons
- Removed duplicate stats cards and operations

- bank-stats-view.tsx: Overview statistics (memories, links, documents, pending ops)
- bank-operations-view.tsx: Background operations table with filtering

**Problem**: Consolidation always used global enable_observations, ignoring bank overrides
**Root Cause**: consolidator.py called get_config() instead of resolving bank-specific config
**Solution**: Pass resolved config through the entire pipeline

**Problem**: asyncpg returning JSONB as JSON string instead of parsed dict
**Solution**: Explicit JSON parsing in config_resolver.py with type checking

- All 19 API integration tests pass
- All 10 hierarchical config tests pass
- Retain operations work correctly with bank-specific config
- Consolidation respects bank-specific enable_observations setting

- Updated developer/configuration.md with type-safe config access pattern
- Added examples showing correct usage patterns
- Documented ConfigFieldAccessError and resolution methods

- get_config() now returns StaticConfigProxy (blocks configurable field access)
- Code accessing bank-configurable fields must use resolve_full_config()
- Clear migration path with helpful error messages

Fixes hierarchical configuration to be production-ready with proper type safety.

* refactor: remove LLM client pool and simplify config resolver

Since LLM config (provider, model, api_key) is now static and not
bank-configurable, the LLMClientPool is no longer needed.

Changes:
- Remove hindsight_api/llm_client_pool.py (no longer needed)
- Remove memory_engine._get_bank_llm_config() (dead code, never called)
- Simplify config_resolver.py by eliminating duplication between
  resolve_full_config() and get_bank_config()
- get_bank_config() now calls resolve_full_config() and filters results
- Remove outdated "LLM provider pooling" comments from docstrings

All tests pass (10 hierarchical config tests, 19 API integration tests)

* fix: update tests to use _get_raw_config() for configurable fields

Fixed test fixtures that were accessing configurable fields (like
enable_observations) from get_config(), which now raises
ConfigFieldAccessError due to type-safe config access.

Changes:
- test_consolidation.py: Changed enable_observations fixture to use
  _get_raw_config() instead of get_config()
- test_consolidation.py: Updated test_consolidation_returns_disabled_status
  to set bank config instead of mocking get_config()
- test_link_expansion_retrieval.py: Changed fixture to use _get_raw_config()
- test_observations.py: Changed disable_observations fixture to use
  _get_raw_config()
- Regenerated OpenAPI spec and clients

All 39 previously failing tests now pass.

* fix: add missing config parameter to test calls of extract_facts_from_text()

Fixed 45 test failures where tests were calling extract_facts_from_text()
without the new required config parameter.

Changes:
- Added config=_get_raw_config() to all extract_facts_from_text() calls
- Fixed test_main_module.py to patch _get_raw_config instead of get_config
- Updated 6 test files with 37 function call sites

All tests should now pass.

* fix: add missing config parameter to test_skip_podcast_meta_commentary

One more test was missing the config parameter for extract_facts_from_text().
2026-02-12 13:14:57 +01:00

157 lines
5.3 KiB
Python

"""Tenant Extension for multi-tenancy and API key authentication."""
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any
from hindsight_api.extensions.base import Extension
from hindsight_api.models import RequestContext
class AuthenticationError(Exception):
"""Raised when authentication fails."""
def __init__(self, reason: str):
self.reason = reason
super().__init__(f"Authentication failed: {reason}")
@dataclass
class TenantContext:
"""
Tenant context returned by authentication.
Contains the PostgreSQL schema name for tenant isolation.
All database queries will use fully-qualified table names
with this schema (e.g., schema_name.memory_units).
"""
schema_name: str
@dataclass
class Tenant:
"""
Represents a tenant for worker discovery.
Used by list_tenants() to return tenant information including
the PostgreSQL schema name for database operations.
"""
schema: str
class TenantExtension(Extension, ABC):
"""
Extension for multi-tenancy and API key authentication.
This extension validates incoming requests and returns the tenant context
including the PostgreSQL schema to use for database operations.
Built-in implementation:
hindsight_api.extensions.builtin.tenant.ApiKeyTenantExtension
Enable via environment variable:
HINDSIGHT_API_TENANT_EXTENSION=hindsight_api.extensions.builtin.tenant:ApiKeyTenantExtension
HINDSIGHT_API_TENANT_API_KEY=your-secret-key
The returned schema_name is used for fully-qualified table names in queries,
enabling tenant isolation at the database level.
"""
@abstractmethod
async def authenticate(self, context: RequestContext) -> TenantContext:
"""
Authenticate the action context and return tenant context.
Args:
context: The action context containing API key and other auth data.
Returns:
TenantContext with the schema_name for database operations.
Raises:
AuthenticationError: If authentication fails.
"""
...
@abstractmethod
async def list_tenants(self) -> list[Tenant]:
"""
List all tenants that should be processed by workers.
This method is used by the worker to discover all tenants that need
task polling. Workers will poll for pending tasks in each tenant's schema.
Returns:
List of Tenant objects containing schema information.
For single-tenant setups, return [Tenant(schema="public")].
"""
...
async def get_tenant_config(self, context: RequestContext) -> dict[str, Any]:
"""
Get tenant-specific configuration overrides.
This method is called during hierarchical configuration resolution to get
tenant-level config overrides. The returned dict should contain Python field
names (lowercase snake_case) as keys, not environment variable names.
Example:
{"llm_model": "gpt-4", "retain_extraction_mode": "verbose"}
The default implementation returns an empty dict (no tenant-specific config).
Override this method in custom extensions to provide tenant-specific configuration.
Args:
context: The request context containing tenant information.
Returns:
Dict of config field names to values (only configurable fields).
Empty dict if no tenant-specific config.
"""
return {}
async def get_allowed_config_fields(self, context: RequestContext, bank_id: str) -> set[str] | None:
"""
Get set of config fields that this tenant/bank is allowed to modify.
This method controls which configurable fields can be modified via the bank config API.
It enables fine-grained permission control per tenant or per bank.
Examples:
- Return None: Allow all configurable fields (default)
- Return {"retain_chunk_size", "retain_custom_instructions"}: Allow only these fields
- Return set(): Allow no modifications (read-only)
The default implementation returns None (all configurable fields allowed).
Override this method in custom extensions to implement custom permission logic.
Args:
context: The request context containing tenant information.
bank_id: The bank identifier for per-bank permissions.
Returns:
Set of allowed field names, or None to allow all configurable fields.
Returned fields must be a subset of HindsightConfig.get_configurable_fields().
"""
return None
async def authenticate_mcp(self, context: RequestContext) -> TenantContext:
"""
Authenticate MCP requests.
By default, this calls authenticate(). Override this method to provide
different authentication behavior for MCP endpoints (e.g., to disable
auth for backwards compatibility with existing MCP servers).
Args:
context: The action context containing API key and other auth data.
Returns:
TenantContext with the schema_name for database operations.
Raises:
AuthenticationError: If authentication fails.
"""
return await self.authenticate(context)