fleet-memory/hindsight-control-plane/src/lib/api.ts
Nicolò Boschi 8d731f2e5f
feat: implement hierarchical configuration (system, tenant, bank) (#329)
* feat: implement hierarchical configuration (system, tenant, bank)

* feat: implement hierarchical configuration (system, tenant, bank)

* docs: add instructions for hierarchical config in CLAUDE.md

* feat: add ENABLE_BANK_CONFIG_API flag (disabled by default)

- Add HINDSIGHT_API_ENABLE_BANK_CONFIG_API env var (default: false)
- Return 403 Forbidden from bank config endpoints when disabled
- Update tests to enable the flag
- Update CLAUDE.md documentation

This provides security control over the bank configuration API,
ensuring it's only accessible when explicitly enabled.

* docs: add hierarchical configuration section

* feat(cli): add bank config commands (config, set-config, reset-config)

- Add 'hindsight bank config' to view bank configuration
- Add 'hindsight bank set-config' to update LLM settings per bank
- Add 'hindsight bank reset-config' to reset to defaults
- Implements client API calls to new bank config endpoints

* fix(cli): fix compilation errors in bank config commands

- Fix type signature: use ApiClient instead of api::Client
- Fix confirmation: use ui::prompt_confirmation instead of ui::confirm
- Fix error handling: use anyhow! macro instead of errors::Error
- Fix type conversion: convert HashMap to serde_json::Map for API call

* feat: implement type-safe hierarchical config with bank overrides

Implements a production-ready hierarchical configuration system that prevents
accidentally using global defaults when bank-specific overrides exist.

- Created StaticConfigProxy that wraps HindsightConfig
- get_config() now returns proxy that blocks access to bank-configurable fields
- Raises ConfigFieldAccessError with clear message when accessing configurable fields
- Added _get_raw_config() for internal use only
- Forces developers to use resolve_full_config(bank_id, context) for bank settings

- Added resolve_full_config() method that returns complete HindsightConfig
- Resolves hierarchy: Global (env) → Tenant → Bank
- No caching to support multi-server deployments (always fresh from DB)
- LLM provider pooling handles expensive operations separately

- Updated entire retain pipeline to pass resolved config through call chain
- memory_engine.py: Resolves config at top level where bank_id/context available
- orchestrator.py: Accepts and passes config to fact_extraction
- fact_extraction.py: Uses passed config instead of get_config()
- utils.py: Added optional config param for backward compatibility

- consolidator.py: Uses resolve_full_config() for enable_observations check
- memory_engine.py: Resolves config before triggering consolidation

- Renamed "Memory Bank" to "Bank Configuration" with tabs
- Combined Stats and Operations into "General" tab
- Consolidated Profile and Configuration into "Configuration" tab
- Moved Actions dropdown to page level (outside tabs)

- Created new component for managing bank-specific config
- Displays configurable fields: retain_chunk_size, retain_extraction_mode, etc.
- Edit via dialog with form validation
- Reset to defaults via AlertDialog confirmation
- Shows field IDs in monospace for clarity
- Visual separation with borders and hover effects

- Removed inline edit mode, switched to dialog-based editing
- Separate dialogs for Disposition and Mission editing
- Read-only display with clear edit buttons
- Removed duplicate stats cards and operations

- bank-stats-view.tsx: Overview statistics (memories, links, documents, pending ops)
- bank-operations-view.tsx: Background operations table with filtering

**Problem**: Consolidation always used global enable_observations, ignoring bank overrides
**Root Cause**: consolidator.py called get_config() instead of resolving bank-specific config
**Solution**: Pass resolved config through the entire pipeline

**Problem**: asyncpg returning JSONB as JSON string instead of parsed dict
**Solution**: Explicit JSON parsing in config_resolver.py with type checking

- All 19 API integration tests pass
- All 10 hierarchical config tests pass
- Retain operations work correctly with bank-specific config
- Consolidation respects bank-specific enable_observations setting

- Updated developer/configuration.md with type-safe config access pattern
- Added examples showing correct usage patterns
- Documented ConfigFieldAccessError and resolution methods

- get_config() now returns StaticConfigProxy (blocks configurable field access)
- Code accessing bank-configurable fields must use resolve_full_config()
- Clear migration path with helpful error messages

Fixes hierarchical configuration to be production-ready with proper type safety.

* refactor: remove LLM client pool and simplify config resolver

Since LLM config (provider, model, api_key) is now static and not
bank-configurable, the LLMClientPool is no longer needed.

Changes:
- Remove hindsight_api/llm_client_pool.py (no longer needed)
- Remove memory_engine._get_bank_llm_config() (dead code, never called)
- Simplify config_resolver.py by eliminating duplication between
  resolve_full_config() and get_bank_config()
- get_bank_config() now calls resolve_full_config() and filters results
- Remove outdated "LLM provider pooling" comments from docstrings

All tests pass (10 hierarchical config tests, 19 API integration tests)

* fix: update tests to use _get_raw_config() for configurable fields

Fixed test fixtures that were accessing configurable fields (like
enable_observations) from get_config(), which now raises
ConfigFieldAccessError due to type-safe config access.

Changes:
- test_consolidation.py: Changed enable_observations fixture to use
  _get_raw_config() instead of get_config()
- test_consolidation.py: Updated test_consolidation_returns_disabled_status
  to set bank config instead of mocking get_config()
- test_link_expansion_retrieval.py: Changed fixture to use _get_raw_config()
- test_observations.py: Changed disable_observations fixture to use
  _get_raw_config()
- Regenerated OpenAPI spec and clients

All 39 previously failing tests now pass.

* fix: add missing config parameter to test calls of extract_facts_from_text()

Fixed 45 test failures where tests were calling extract_facts_from_text()
without the new required config parameter.

Changes:
- Added config=_get_raw_config() to all extract_facts_from_text() calls
- Fixed test_main_module.py to patch _get_raw_config instead of get_config
- Updated 6 test files with 37 function call sites

All tests should now pass.

* fix: add missing config parameter to test_skip_podcast_meta_commentary

One more test was missing the config parameter for extract_facts_from_text().
2026-02-12 13:14:57 +01:00

732 lines
18 KiB
TypeScript

/**
* Client for calling Control Plane API routes (which proxy to the dataplane via SDK)
* This should be used in client components, not the SDK directly
*/
export interface MentalModel {
id: string;
bank_id: string;
name: string;
source_query: string;
content: string;
tags: string[];
max_tokens: number;
trigger: { refresh_after_consolidation: boolean };
last_refreshed_at: string;
created_at: string;
reflect_response?: any;
}
export class ControlPlaneClient {
private async fetchApi<T>(path: string, options?: RequestInit): Promise<T> {
const response = await fetch(path, {
...options,
headers: {
"Content-Type": "application/json",
...options?.headers,
},
});
if (!response.ok) {
const error = await response.text();
throw new Error(`API Error: ${response.status} - ${error}`);
}
return response.json();
}
/**
* List all banks
*/
async listBanks() {
return this.fetchApi<{ banks: any[] }>("/api/banks", { cache: "no-store" as RequestCache });
}
/**
* Create a new bank
*/
async createBank(bankId: string) {
return this.fetchApi<{ bank_id: string }>("/api/banks", {
method: "POST",
body: JSON.stringify({ bank_id: bankId }),
});
}
/**
* Recall memories
*/
async recall(params: {
query: string;
types?: string[];
bank_id: string;
budget?: string;
max_tokens?: number;
trace?: boolean;
include?: {
entities?: { max_tokens: number } | null;
chunks?: { max_tokens: number } | null;
observations?: { max_results?: number } | null;
};
query_timestamp?: string;
tags?: string[];
tags_match?: "any" | "all" | "any_strict" | "all_strict";
}) {
return this.fetchApi("/api/recall", {
method: "POST",
body: JSON.stringify(params),
});
}
/**
* Reflect and generate answer
*/
async reflect(params: {
query: string;
bank_id: string;
budget?: string;
max_tokens?: number;
include_facts?: boolean;
include_tool_calls?: boolean;
tags?: string[];
tags_match?: "any" | "all" | "any_strict" | "all_strict";
}) {
return this.fetchApi("/api/reflect", {
method: "POST",
body: JSON.stringify(params),
});
}
/**
* Retain memories (batch)
*/
async retain(params: {
bank_id: string;
items: Array<{
content: string;
timestamp?: string;
context?: string;
metadata?: Record<string, string>;
document_id?: string;
entities?: Array<{ text: string; type?: string }>;
}>;
document_id?: string;
async?: boolean;
}) {
const endpoint = params.async ? "/api/memories/retain_async" : "/api/memories/retain";
return this.fetchApi(endpoint, {
method: "POST",
body: JSON.stringify(params),
});
}
/**
* Get bank statistics
*/
async getBankStats(bankId: string) {
return this.fetchApi(`/api/stats/${bankId}`);
}
/**
* Get graph data
*/
async getGraph(params: { bank_id: string; type?: string; limit?: number }) {
const queryParams = new URLSearchParams();
queryParams.append("bank_id", params.bank_id);
if (params.type) queryParams.append("type", params.type);
if (params.limit) queryParams.append("limit", params.limit.toString());
return this.fetchApi(`/api/graph?${queryParams}`);
}
/**
* List operations with optional filtering and pagination
*/
async listOperations(
bankId: string,
options?: { status?: string; limit?: number; offset?: number }
) {
const params = new URLSearchParams();
if (options?.status) params.append("status", options.status);
if (options?.limit) params.append("limit", options.limit.toString());
if (options?.offset) params.append("offset", options.offset.toString());
const query = params.toString();
return this.fetchApi<{
bank_id: string;
total: number;
limit: number;
offset: number;
operations: Array<{
id: string;
task_type: string;
items_count: number;
document_id: string | null;
created_at: string;
status: string;
error_message: string | null;
}>;
}>(`/api/operations/${bankId}${query ? `?${query}` : ""}`);
}
/**
* Cancel a pending operation
*/
async cancelOperation(bankId: string, operationId: string) {
return this.fetchApi<{
success: boolean;
message: string;
operation_id: string;
}>(`/api/operations/${bankId}?operation_id=${operationId}`, {
method: "DELETE",
});
}
/**
* List entities
*/
async listEntities(params: { bank_id: string; limit?: number; offset?: number }) {
const queryParams = new URLSearchParams();
queryParams.append("bank_id", params.bank_id);
if (params.limit) queryParams.append("limit", params.limit.toString());
if (params.offset) queryParams.append("offset", params.offset.toString());
return this.fetchApi<{
items: any[];
total: number;
limit: number;
offset: number;
}>(`/api/entities?${queryParams}`);
}
/**
* Get entity details
*/
async getEntity(entityId: string, bankId: string) {
return this.fetchApi(`/api/entities/${entityId}?bank_id=${bankId}`);
}
/**
* Regenerate entity observations
*/
async regenerateEntityObservations(entityId: string, bankId: string) {
return this.fetchApi(`/api/entities/${entityId}/regenerate?bank_id=${bankId}`, {
method: "POST",
});
}
/**
* List documents
*/
async listDocuments(params: { bank_id: string; q?: string; limit?: number; offset?: number }) {
const queryParams = new URLSearchParams();
queryParams.append("bank_id", params.bank_id);
if (params.q) queryParams.append("q", params.q);
if (params.limit) queryParams.append("limit", params.limit.toString());
if (params.offset) queryParams.append("offset", params.offset.toString());
return this.fetchApi(`/api/documents?${queryParams}`);
}
/**
* Get document
*/
async getDocument(documentId: string, bankId: string) {
return this.fetchApi(`/api/documents/${documentId}?bank_id=${bankId}`);
}
/**
* Delete document and all its associated memory units
*/
async deleteDocument(documentId: string, bankId: string) {
return this.fetchApi<{
success: boolean;
message: string;
document_id: string;
memory_units_deleted: number;
}>(`/api/documents/${documentId}?bank_id=${bankId}`, {
method: "DELETE",
});
}
/**
* Delete an entire memory bank and all its data
*/
async deleteBank(bankId: string) {
return this.fetchApi<{
success: boolean;
message: string;
deleted_count: number;
}>(`/api/banks/${bankId}`, {
method: "DELETE",
});
}
/**
* Clear all observations for a bank
*/
async clearObservations(bankId: string) {
return this.fetchApi<{
success: boolean;
message: string;
deleted_count: number;
}>(`/api/banks/${bankId}/observations`, {
method: "DELETE",
});
}
/**
* Trigger consolidation for a bank
*/
async triggerConsolidation(bankId: string) {
return this.fetchApi<{
operation_id: string;
deduplicated: boolean;
}>(`/api/banks/${bankId}/consolidate`, {
method: "POST",
});
}
/**
* Get chunk
*/
async getChunk(chunkId: string) {
return this.fetchApi(`/api/chunks/${chunkId}`);
}
/**
* Get a single memory by ID
*/
async getMemory(memoryId: string, bankId: string) {
return this.fetchApi<{
id: string;
text: string;
context: string;
date: string;
type: string;
mentioned_at: string | null;
occurred_start: string | null;
occurred_end: string | null;
entities: string[];
document_id: string | null;
chunk_id: string | null;
tags: string[];
}>(`/api/memories/${memoryId}?bank_id=${bankId}`);
}
/**
* Get bank profile
*/
async getBankProfile(bankId: string) {
return this.fetchApi<{
bank_id: string;
name: string;
disposition: {
skepticism: number;
literalism: number;
empathy: number;
};
mission: string;
background?: string; // Deprecated, kept for backwards compatibility
}>(`/api/profile/${bankId}`);
}
/**
* Set bank mission
*/
async setBankMission(bankId: string, mission: string) {
return this.fetchApi(`/api/banks/${bankId}`, {
method: "PATCH",
body: JSON.stringify({ mission }),
});
}
/**
* List directives for a bank
*/
async listDirectives(bankId: string, tags?: string[], tagsMatch?: string) {
const params = new URLSearchParams();
if (tags && tags.length > 0) {
tags.forEach((t) => params.append("tags", t));
}
if (tagsMatch) {
params.append("tags_match", tagsMatch);
}
const query = params.toString();
return this.fetchApi<{
items: Array<{
id: string;
bank_id: string;
name: string;
content: string;
priority: number;
is_active: boolean;
tags: string[];
created_at: string;
updated_at: string;
}>;
}>(`/api/banks/${bankId}/directives${query ? `?${query}` : ""}`);
}
/**
* Create a directive
*/
async createDirective(
bankId: string,
params: {
name: string;
content: string;
priority?: number;
is_active?: boolean;
tags?: string[];
}
) {
return this.fetchApi<{
id: string;
bank_id: string;
name: string;
content: string;
priority: number;
is_active: boolean;
tags: string[];
created_at: string;
updated_at: string;
}>(`/api/banks/${bankId}/directives`, {
method: "POST",
body: JSON.stringify(params),
});
}
/**
* Get a directive
*/
async getDirective(bankId: string, directiveId: string) {
return this.fetchApi<{
id: string;
bank_id: string;
name: string;
content: string;
priority: number;
is_active: boolean;
tags: string[];
created_at: string;
updated_at: string;
}>(`/api/banks/${bankId}/directives/${directiveId}`);
}
/**
* Delete a directive
*/
async deleteDirective(bankId: string, directiveId: string) {
return this.fetchApi(`/api/banks/${bankId}/directives/${directiveId}`, {
method: "DELETE",
});
}
/**
* Update a directive
*/
async updateDirective(
bankId: string,
directiveId: string,
params: {
name?: string;
content?: string;
priority?: number;
is_active?: boolean;
tags?: string[];
}
) {
return this.fetchApi<{
id: string;
bank_id: string;
name: string;
content: string;
priority: number;
is_active: boolean;
tags: string[];
created_at: string;
updated_at: string;
}>(`/api/banks/${bankId}/directives/${directiveId}`, {
method: "PATCH",
body: JSON.stringify(params),
});
}
/**
* Get operation status
*/
async getOperationStatus(bankId: string, operationId: string) {
return this.fetchApi<{
operation_id: string;
status: "pending" | "completed" | "failed" | "not_found";
operation_type: string | null;
created_at: string | null;
updated_at: string | null;
completed_at: string | null;
error_message: string | null;
}>(`/api/banks/${bankId}/operations/${operationId}`);
}
/**
* Update bank profile
*/
async updateBankProfile(
bankId: string,
profile: {
name?: string;
disposition?: {
skepticism: number;
literalism: number;
empathy: number;
};
mission?: string;
}
) {
return this.fetchApi(`/api/profile/${bankId}`, {
method: "PUT",
body: JSON.stringify(profile),
});
}
// ============= OBSERVATIONS (auto-consolidated, read-only) =============
/**
* List observations for a bank (auto-consolidated knowledge)
*/
async listObservations(bankId: string, tags?: string[], tagsMatch?: string) {
const params = new URLSearchParams();
if (tags && tags.length > 0) {
tags.forEach((t) => params.append("tags", t));
}
if (tagsMatch) {
params.append("tags_match", tagsMatch);
}
const query = params.toString();
return this.fetchApi<{
items: Array<{
id: string;
bank_id: string;
text: string;
proof_count: number;
history: Array<{
previous_text: string;
changed_at: string;
reason: string;
}>;
tags: string[];
source_memory_ids: string[];
source_memories: Array<{
id: string;
text: string;
type: string;
context?: string;
occurred_start?: string;
mentioned_at?: string;
}>;
created_at: string;
updated_at: string;
}>;
}>(`/api/banks/${bankId}/observations${query ? `?${query}` : ""}`);
}
/**
* Get an observation with source memories
*/
async getObservation(bankId: string, observationId: string) {
return this.fetchApi<{
id: string;
bank_id: string;
text: string;
proof_count: number;
history: Array<{
previous_text: string;
changed_at: string;
reason: string;
}>;
tags: string[];
source_memory_ids: string[];
source_memories: Array<{
id: string;
text: string;
type: string;
context?: string;
occurred_start?: string;
mentioned_at?: string;
}>;
created_at: string;
updated_at: string;
}>(`/api/banks/${bankId}/observations/${observationId}`);
}
// ============= MENTAL MODELS (stored reflect responses) =============
/**
* List mental models for a bank
*/
async listMentalModels(bankId: string, tags?: string[], tagsMatch?: string) {
const params = new URLSearchParams();
if (tags && tags.length > 0) {
tags.forEach((t) => params.append("tags", t));
}
if (tagsMatch) {
params.append("tags_match", tagsMatch);
}
const query = params.toString();
return this.fetchApi<{
items: Array<{
id: string;
bank_id: string;
name: string;
source_query: string;
content: string;
tags: string[];
max_tokens: number;
trigger: { refresh_after_consolidation: boolean };
last_refreshed_at: string;
created_at: string;
reflect_response?: {
text: string;
based_on: Record<string, Array<{ id: string; text: string; type: string }>>;
};
}>;
}>(`/api/banks/${bankId}/mental-models${query ? `?${query}` : ""}`);
}
/**
* Create a mental model (async - content auto-generated in background)
* Returns operation_id to track progress
*/
async createMentalModel(
bankId: string,
params: {
id?: string;
name: string;
source_query: string;
tags?: string[];
max_tokens?: number;
trigger?: { refresh_after_consolidation: boolean };
}
) {
return this.fetchApi<{
operation_id: string;
}>(`/api/banks/${bankId}/mental-models`, {
method: "POST",
body: JSON.stringify(params),
});
}
/**
* Get a mental model
*/
async getMentalModel(bankId: string, mentalModelId: string): Promise<MentalModel> {
return this.fetchApi<MentalModel>(`/api/banks/${bankId}/mental-models/${mentalModelId}`);
}
/**
* Update a mental model
*/
async updateMentalModel(
bankId: string,
mentalModelId: string,
params: {
name?: string;
source_query?: string;
max_tokens?: number;
tags?: string[];
trigger?: { refresh_after_consolidation: boolean };
}
) {
return this.fetchApi<{
id: string;
bank_id: string;
name: string;
source_query: string;
content: string;
tags: string[];
max_tokens: number;
trigger: { refresh_after_consolidation: boolean };
last_refreshed_at: string;
created_at: string;
reflect_response?: {
text: string;
based_on: Record<string, Array<{ id: string; text: string; type: string }>>;
};
}>(`/api/banks/${bankId}/mental-models/${mentalModelId}`, {
method: "PATCH",
body: JSON.stringify(params),
});
}
/**
* Delete a mental model
*/
async deleteMentalModel(bankId: string, mentalModelId: string) {
return this.fetchApi(`/api/banks/${bankId}/mental-models/${mentalModelId}`, {
method: "DELETE",
});
}
/**
* Refresh a mental model (re-run source query) - async operation
*/
async refreshMentalModel(bankId: string, mentalModelId: string) {
return this.fetchApi<{
operation_id: string;
}>(`/api/banks/${bankId}/mental-models/${mentalModelId}/refresh`, {
method: "POST",
});
}
/**
* Get API version and feature flags
* Use this to check which capabilities are available in the dataplane
*/
async getVersion() {
return this.fetchApi<{
api_version: string;
features: {
observations: boolean;
mcp: boolean;
worker: boolean;
bank_config_api: boolean;
};
}>("/api/version");
}
/**
* Get bank configuration (resolved with hierarchy)
*/
async getBankConfig(bankId: string) {
return this.fetchApi<{
bank_id: string;
config: Record<string, any>;
overrides: Record<string, any>;
}>(`/api/banks/${bankId}/config`);
}
/**
* Update bank configuration overrides
*/
async updateBankConfig(bankId: string, updates: Record<string, any>) {
return this.fetchApi<{
bank_id: string;
config: Record<string, any>;
overrides: Record<string, any>;
}>(`/api/banks/${bankId}/config`, {
method: "PATCH",
body: JSON.stringify({ updates }),
});
}
/**
* Reset bank configuration to defaults
*/
async resetBankConfig(bankId: string) {
return this.fetchApi<{
bank_id: string;
config: Record<string, any>;
overrides: Record<string, any>;
}>(`/api/banks/${bankId}/config`, {
method: "DELETE",
});
}
}
// Export singleton instance
export const client = new ControlPlaneClient();