Commit graph

20 commits

Author SHA1 Message Date
Nicolò Boschi
30a319a6ab
feat: bank template import/export with Template Hub (#819)
* feat(api): add bank template import/export endpoints

Add POST /banks/{bank_id}/import and GET /banks/{bank_id}/export
endpoints for declarative bank setup via JSON manifests.

A template manifest (version 1) can include bank config overrides
and mental model definitions. Import creates or updates mental
models matched by id, applies config as per-bank overrides, and
returns async operation IDs for content generation.

Export dumps a bank's explicit overrides and mental models as a
manifest that can be re-imported into another bank.

Includes control plane UI: bank creation dialog now accepts an
optional template JSON to pre-configure the bank on creation.

* docs: add Template Gallery page and bank templates reference

- Template Gallery (/templates) with search, category filter, manifest
  preview modal with copy-to-clipboard
- 5 starter templates: Customer Support, Research Assistant, Personal
  Journal, Code Review Buddy, Meeting Notes
- Bank Templates API reference doc (developer/api/bank-templates)
- Sidebar entry under API section

* docs: add Template Gallery links to navbar and sidebar

- Top navbar: "Templates" link between Integrations and Changelog
- Sidebar: "Template Gallery" in Resources section

* fix(docs): remove emoji icons, autofocus search, fix placeholder in template gallery

* docs: rename to Bank Templates, move to Resources sidebar only

* docs: add Bank Templates to Resources navbar dropdown

* feat(api): add directives to bank template import/export

- Add BankTemplateDirective model with name, content, priority, is_active, tags
- Import creates/updates directives matched by name
- Export includes all directives (active and inactive)
- Validation: duplicate names rejected, empty name/content caught
- Tests: 24 tests covering directives create/update, existing vs new
  bank import, validation, export with directives, full round-trip

* docs: add directives to bank templates docs and sample templates

* feat(api): add JSON Schema endpoint for bank template validation

- GET /v1/default/bank-template-schema returns the JSON Schema
  auto-generated from the Pydantic BankTemplateManifest model
- Static schema file at docs/static/bank-template-schema.json
- Docs updated with schema endpoint, static file link, and
  validation examples (Python jsonschema, Node ajv-cli)

* feat(api): live schema validation on import, fix schema endpoint path

- Move schema endpoint to /v1/bank-template-schema (system-level, not per-bank)
- Import endpoint now accepts raw JSON and validates with Pydantic manually,
  returning clean 400 errors instead of raw 422s for all validation failures
- All validation (schema + semantic) returns consistent 400 with detailed messages

* docs: add interactive JSON Schema viewer to Bank Templates page

Renders the Pydantic-generated schema as a collapsible property tree
with types, required badges, defaults, and descriptions. The schema
is imported from the static bank-template-schema.json file.

* ui: add template toggle switch and browse link to bank creation dialog

- Replace always-visible textarea with a switch toggle ("Import from template")
- Textarea only shows when switch is on, keeping the dialog clean by default
- Add "Browse templates" link pointing to hindsight.vectorize.io/templates
- Reset template state when switch is toggled off or dialog is cancelled

* ui: add empty state with Add Document CTA to data view

When a bank has 0 memories, the data view (all tabs: constellation,
graph, table, timeline) shows a centered empty state with a CTA
button that opens the Add Document dialog.

* docs: replace templates with Conversation and Coding Agent

Remove generic placeholder templates. Add two practical templates
based on actual integration patterns:

- Conversation: for chat agents (LiteLLM, LangGraph, Pydantic AI,
  Vercel AI SDK). Tracks user preferences, open threads.
- Coding Agent: for Claude Code/Codex. Tracks technical decisions,
  project context, developer preferences. High literalism.

* docs: rename gallery to Bank Templates Hub, keep API doc as Bank Templates

* docs: register layout-template and file-json icons in navbar and sidebar

* docs: register layout-template icon in DefaultNavbarItem for dropdown items

* docs: show integration icons on template cards

Templates now have an optional `integrations` field referencing
integration IDs from integrations.json. Icons are resolved at render
time and shown in the card header next to the category badge.

* docs: add Personal Assistant template for OpenClaw, Hermes, NemoClaw

* feat: add Export Template to bank actions + map all integrations to templates

- Add "Export Template" to the bank Actions dropdown — exports config,
  mental models, and directives as JSON, copies to clipboard
- Add export API route and client method
- Map remaining integrations to templates: CrewAI, AG2, Agno, Strands,
  LlamaIndex, local-mcp, skills → Conversation; hindclaw → Personal Assistant

* feat: add --template flag to LoCoMo benchmark + remove schema from Hub

- LoCoMo benchmark accepts --template <path> to apply a bank template
  manifest (config, mental models, directives) before ingestion
- Template is applied per-bank in both single-phase and two-phase modes
- BenchmarkRunner.apply_template() reuses the same engine methods as
  the /import API endpoint
- Remove Manifest Schema section from Bank Templates Hub page
  (schema stays in the API reference doc)

* refactor: remove description field from bank template manifest

* docs: remove tags, fact_types, and directives from starter templates

* docs: remove reflect_mission and disposition fields from starter templates

* build: validate template manifests against JSON Schema during docs build

* cleanup: remove unused JsonSchemaViewer component

* docs: remove retain_extraction_mode from starter templates

* ui: enable word wrap in template manifest preview

* docs: add link to Bank Templates reference doc from Hub page

* docs: convert bank templates doc to mdx with multi-language code snippets

- Convert bank-templates.md to .mdx with Tabs/CodeSnippet components
- Add example files: bank-templates.py, .mjs, .sh, .go with doc markers
- Examples cover import, dry-run, export, round-trip, and schema
- Regenerate OpenAPI spec and all client SDKs (Python, TS, Rust, Go)

* fix: migration revision collision + use typed models in benchmark template

- Rename merge migration d6e7f8a9b0c1 -> d6e7f8a9b0c2 to resolve
  revision ID collision with case_insensitive_entities_trgm_index
- Update a4b5c6d7e8f9 down_revision to point to the renamed migration
- Fix f-string lint in case_insensitive migration
- BenchmarkRunner.apply_template() now validates manifest through
  BankTemplateManifest Pydantic model instead of raw dict access
- Remove redundant inline imports (json, Path already at module top)

* fix(docs): add missing Go tab to dry-run code snippet

* ci: retrigger

* fix: sync skills openapi.json + fix bankId null type error in export

- Copy updated openapi.json to skills/hindsight-docs/references/
- Add null guard for bankId in Export Template onClick handler

* fix: sync generated files (memory_engine formatting, docs skill references)

* cleanup: remove obsolete migration collision workaround
2026-04-02 12:21:53 +02:00
Nicolò Boschi
baf5447de2
refactor: replace LLMProvider classmethods with from_env() and document missing config fields (#816)
CI failures are unrelated to this PR:
- test_mental_models_dimension_change_empty_table: database OID error (infrastructure flake)
- test_reflect_searches_mental_models_when_available: LLM-dependent assertion (flaky)
2026-03-31 18:00:41 +02:00
Nicolò Boschi
0aa7c2b3a1
feat: batch observations consolidation (#430)
* feat: batch observations consolidation

* feat: batch observations consolidation

* docs: add CONSOLIDATION_LLM_BATCH_SIZE config flag documentation
2026-02-24 15:19:39 +01:00
Nicolò Boschi
5fddd9a79c
feat: add reflect mode to LoComo benchmark and improve reflect agent (#428)
* feat: add reflect mode to LoComo benchmark and improve reflect agent

- Replace think mode with reflect mode in LoComo benchmark using reflect_async with Budget.HIGH
- Add --question-index CLI flag to run a single question by its index
- Track and display original question index in logs and visualizer
- Update visualizer to show reflect mode results

Reflect agent improvements:
- tool_recall: always fetch chunks (max_chunk_tokens=1000 min, non-optional)
- tool_search_observations: use include_source_facts=True instead of separate DB query
- Use model_dump() throughout to avoid manual error-prone dict conversion
- Enforce minimum 1000 tokens for max_tokens and max_chunk_tokens in _execute_tool
- Fix NoneType error when LLM passes null for mental_model_ids/observation_ids arrays
- Add non-conversational constraint to system prompt to prevent follow-up questions
- Fix recall_fn Callable type hint to include max_chunk_tokens parameter
- Fix main.py missing reranker_zeroentropy fields in HindsightConfig constructor

* fix: update tests for reflect tool API changes

- source_memory_ids -> source_fact_ids in test_search_observations (MemoryFact.model_dump() field name)
- Remove proof_count check (not in MemoryFact, was ObservationResult-specific)
- Remove max_results param from tool_recall call (no longer supported)
- Fix recall_result["count"] -> len(recall_result["memories"])
2026-02-24 09:48:23 +01:00
Nicolò Boschi
1c9a7a0d5e
chore: cleanup benchmarks runner with old flags (#212)
* chore: cleanup benchmarks runner with old flags

* fix tests

* fix: observations rely on source_memory_ids, no link copying

Observations no longer copy any memory_links from their source facts.
Instead, retrieval uses source_memory_ids to traverse:
- Entity connections: observation → source_memory_ids → unit_entities
- Semantic similarity: observations have their own embeddings
- Temporal proximity: observations have their own temporal fields

This avoids data duplication and fixes bidirectionality issues with
entity links being copied to observations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* test: update consolidation test for source_memory_ids behavior

Updated test_consolidation_creates_memory_links to test_consolidation_uses_source_memory_ids
to reflect the new behavior where observations use source_memory_ids instead of memory_links
for traversal.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-28 13:22:48 +01:00
Nicolò Boschi
9db64ecda3
feat: revisit mental models, directives and reflections (#179)
* chore: run benchmarks with reflect mode

* chore: run benchmarks with reflect mode

* fixes

* new mm

* bunch of fixes

* initial commit

* fixes

* fixes

* fixes

* fix: sometimes memories gets extracted in the wrong language
2026-01-22 17:13:16 +01:00
Nicolò Boschi
32bca12c6f
fix: ollama structured support (#63)
* fix: ollama structured support

* fix: ollama structured support

* fix: ollama structured support
2025-12-22 16:35:24 +01:00
Nicolò Boschi
2a0c490c9e
feat: extensions (#54) 2025-12-22 11:05:23 +01:00
Nicolò Boschi
fa554b8980 brandind and misc fixes 2025-12-11 12:46:48 +01:00
Nicolò Boschi
eef43f59c2 improve retain performances, caching and tests 2025-12-08 18:21:56 +01:00
Nicolò Boschi
76cfa8f9c4 fix entity and migrate memory disposition 2025-12-08 16:14:49 +01:00
Nicolò Boschi
3bb0a58ded
Increase graph neighbor limit and benchmark improvements (#18)
* Improve LongMemEval benchmark with structured prompts and better options

- Add --context-format option with 'json' (original) and 'structured' modes
- Structured format groups facts with source chunks for better LLM comprehension
- Add detailed instructions for date calculations, relative time handling, and abstention
- Add --source-results flag to read failed questions from a different file
- Allow --category to be combined with --max-instances for sampling
- Fix Gemini structured output by passing response_schema parameter
- Add retry logic for empty Gemini responses with block reason logging
- Add judge prompt comparison documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix recall in benchmarks

* Improve LongMemEval prompt and Gemini error handling

- Add JSONDecodeError retry for Gemini truncated responses
- Increase max_tokens to 32768 for thinking models
- Add counting/disambiguation guidance to structured prompt
- Add "when in doubt, undercount" and overlap detection rules

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Add connection error retry and preference question guidance

- Add APIConnectionError retry for OpenAI client (server disconnects)
- Add recommendation/preference question guidance to structured prompt
- Instruct model to build on user's existing tools/experiences

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Make reasoning optional

* Seed for LLM through Groq

* fix entity and observations

* Increase graph retrieval neighbor limit for expanded entities

Doubled the neighbor limit multiplier from 10 to 20 in graph retrieval.
With expanded entity extraction (now including objects and concepts like
"kitchen"), facts share more common entities, causing the previous limit
to arbitrarily exclude relevant results. This fix ensures better recall
for questions about related items (e.g., kitchen items).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Expand entity extraction to include objects and concepts

Updated entity extraction prompt to include:
- Specific objects (coffee maker, toaster, car, laptop, kitchen)
- Abstract concepts/themes (friendship, career growth, loss, celebration)
- Places and organizations (IKEA, Goodwill, New York)

This enables better fact linking through shared entities. For example,
kitchen appliances now share a "kitchen" entity, allowing graph traversal
to find related facts like "replaced coffee maker" when querying about
"kitchen items".

Works in conjunction with the increased neighbor limit to improve recall.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Chris Bartholomew <chris.bartholomew@vectorize.io>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: andrew <andrew.neeser@me.com>
2025-12-08 15:24:13 +01:00
Nicolò Boschi
09fbcc2020 prepare for release 2025-12-03 11:52:25 +01:00
Nicolò Boschi
943f6e7844 control plane and api issues 2025-12-02 18:28:57 +01:00
Nicolò Boschi
f7cf33c610 fix regressions and bunch of issues 2025-12-01 18:44:49 +01:00
Nicolò Boschi
e75c483479 fix python client 2025-11-29 18:46:52 +01:00
Nicolò Boschi
3e72984cd2 chunks 2025-11-29 16:34:13 +01:00
Nicolò Boschi
d099ef870d rename to new names 2025-11-27 16:22:16 +01:00
Nicolò Boschi
a3ad76d165 many improvements 2025-11-26 18:22:25 +01:00
Nicolò Boschi
b1ff2e8823
rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
Renamed from memora-dev/benchmarks/common/benchmark_runner.py (Browse further)