fleet-memory/hindsight-api/tests
Nicolò Boschi 539190b69e
feat: support for codex and claude-code as llm (#276)
* feat: support for codex and claude-code as llm

* Remove refactoring plan file

* Consolidate Anthropic tests into main LLM provider test suite

- Add Anthropic models (Sonnet, Opus, Haiku) to MODEL_MATRIX
- Remove separate test_anthropic_provider.py file
- All Anthropic models now tested with standard memory operations

* Add provider-specific default models

Each LLM provider now has a sensible default model that's used when
HINDSIGHT_API_LLM_MODEL is not explicitly set. This simplifies
configuration - users can specify just the provider and API key.

Changes:
- Add PROVIDER_DEFAULT_MODELS mapping in config.py
- Update config logic to use provider defaults for both global and
  per-operation LLM configs
- Add comprehensive tests for provider default model selection
- Document provider defaults in models.md

Example usage:
  export HINDSIGHT_API_LLM_PROVIDER=anthropic
  export HINDSIGHT_API_LLM_API_KEY=sk-ant-xxx
  # Automatically uses claude-sonnet-4-20250514

Provider defaults:
  - openai: gpt-5-mini
  - anthropic: claude-sonnet-4-20250514
  - gemini: gemini-2.5-flash
  - groq: openai/gpt-oss-120b
  - ollama: gemma3:12b
  - lmstudio: local-model
  - vertexai: gemini-2.0-flash-001
  - openai-codex: o3-mini
  - claude-code: claude-sonnet-4-20250514
  - mock: mock-model

* Update provider default models

- openai: gpt-5-mini -> o3-mini
- anthropic: claude-sonnet-4-20250514 -> claude-haiku-4-5-20251001
- openai-codex: o3-mini -> gpt-5.2-codex
- claude-code: claude-sonnet-4-20250514 -> claude-sonnet-4-5-20250929

Updated tests and documentation to reflect new defaults.

* Move OpenAI Codex and Claude Code setup to models.md

Moved detailed setup instructions for OpenAI Codex and Claude Code from
configuration.md to models.md where they better fit with model-specific
documentation.

Changes:
- Move "OpenAI Codex Setup" section from configuration.md to models.md
- Move "Claude Code Setup" section from configuration.md to models.md
- Add cross-reference tip in configuration.md pointing to models.md
- Update default model in Claude Code example to claude-sonnet-4-5-20250929
- Keep basic provider examples in configuration.md for quick reference

This makes the configuration.md page more focused on environment
variables while models.md contains provider-specific setup details.
2026-02-02 12:54:44 +01:00
..
fixtures rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
__init__.py rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
conftest.py fix: improve pytorch model initialization to prevent meta tensor issues (#180) 2026-01-20 14:15:51 +01:00
test_admin_backup_restore.py feat: run db migrations offline (optionally) (#114) 2026-01-07 15:49:51 +01:00
test_agents_api.py feat: introduce mental models (#132) 2026-01-16 11:16:41 +01:00
test_batch_chunking.py feat: extensions (#54) 2025-12-22 11:05:23 +01:00
test_causal_relations.py fix: duplicated causal relationships and token optimization (#126) 2026-01-08 14:43:48 +01:00
test_causal_relationships.py fix: duplicated causal relationships and token optimization (#126) 2026-01-08 14:43:48 +01:00
test_chunking.py improve retain performances, caching and tests 2025-12-08 18:21:56 +01:00
test_combined_scoring.py feat: extensions (#54) 2025-12-22 11:05:23 +01:00
test_consolidation.py fix: deadlock in worker polling (#250) 2026-01-30 11:09:29 +01:00
test_custom_embedding_dimension.py fix: improve mental model consolidation (#197) 2026-01-26 09:54:25 +01:00
test_document_tracking.py feat: extensions (#54) 2025-12-22 11:05:23 +01:00
test_extensions.py fix: improve mental model consolidation (#197) 2026-01-26 09:54:25 +01:00
test_fact_extraction_analysis.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
test_fact_extraction_output_ratio.py fix: misc perf improvements (#133) 2026-01-08 22:49:04 +01:00
test_fact_extraction_quality.py feat: introduce mental models (#132) 2026-01-16 11:16:41 +01:00
test_fact_ordering.py feat: introduce mental models (#132) 2026-01-16 11:16:41 +01:00
test_http_api_integration.py fix: retain async with timestamp might fails (#253) 2026-01-30 14:54:32 +01:00
test_link_expansion_retrieval.py fix: misc fixes for observations and mental models (#209) 2026-01-27 15:37:57 +01:00
test_link_utils.py prepare for release 2025-12-03 11:52:25 +01:00
test_llm_provider.py feat: support for codex and claude-code as llm (#276) 2026-02-02 12:54:44 +01:00
test_llm_token_metrics.py feat: support for codex and claude-code as llm (#276) 2026-02-02 12:54:44 +01:00
test_llm_tools.py chore: internal renames (#204) 2026-01-27 09:53:28 +01:00
test_load_large_batch.py feat: new 'worker' service (#176) 2026-01-20 10:17:56 +01:00
test_main_module.py fix: improve mental model consolidation (#197) 2026-01-26 09:54:25 +01:00
test_mcp_local.py feat(mcp): add timestamp to retain (#190) 2026-01-23 16:00:43 +01:00
test_mcp_routing.py feat(mcp): add Bearer token authentication and tenant auth propagation (#241) 2026-01-30 09:08:18 +01:00
test_mcp_tools.py feat(mcp): add timestamp to retain (#190) 2026-01-23 16:00:43 +01:00
test_mental_model_hooks.py Add extension hooks for mental model operations (#260) 2026-01-31 09:30:53 -05:00
test_mental_models.py feat: revisit mental models, directives and reflections (#179) 2026-01-22 17:13:16 +01:00
test_metrics.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
test_mpfp_retrieval.py fix: improve mpfp retrieval (#146) 2026-01-12 18:58:05 +01:00
test_multilingual.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
test_observations.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
test_per_operation_llm_config.py feat: add more config options for llm retries (#234) 2026-01-29 17:50:43 +01:00
test_provider_default_models.py feat: support for codex and claude-code as llm (#276) 2026-02-02 12:54:44 +01:00
test_query_analyzer.py fix: ci and ui build (#9) 2025-12-03 15:47:53 +01:00
test_reflect_agent.py fix: graph endpoint not showing links for observations (#214) 2026-01-28 14:51:25 +01:00
test_reflections.py fix: misc fixes for observations and mental models (#209) 2026-01-27 15:37:57 +01:00
test_retain.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
test_schema_isolation.py fix: improve mental model consolidation (#197) 2026-01-26 09:54:25 +01:00
test_search_trace.py feat: extensions (#54) 2025-12-22 11:05:23 +01:00
test_server_module.py fix: improve mental model consolidation (#197) 2026-01-26 09:54:25 +01:00
test_sql_schema_safety.py chore: internal renames (#204) 2026-01-27 09:53:28 +01:00
test_tags_visibility.py chore: cleanup benchmarks runner with old flags (#212) 2026-01-28 13:22:48 +01:00
test_tei_cross_encoder.py fix: add defensive error handling to PyTorch device detection (#221) 2026-01-28 18:14:54 +01:00
test_temporal_ranges.py feat(mcp): add timestamp to retain (#190) 2026-01-23 16:00:43 +01:00
test_think.py chore: remove dead code (#245) 2026-01-30 09:16:32 +01:00
test_vertexai_provider.py Switch Vertex AI provider to native genai SDK (#242) 2026-01-30 08:35:59 +01:00
test_worker.py fix: worker doesn't pick up correct default schema (#259) 2026-01-31 09:15:59 +01:00