* feat(recall): add proof_count boost to combined scoring
Observations with more supporting evidence now rank slightly higher
in recall results. proof_count is threaded through the retrieval
pipeline and applied as a multiplicative boost in reranking:
- types.py: add proof_count field to RetrievalResult
- retrieval.py: include proof_count in SELECT columns
- reranking.py: add log1p-normalized proof_count boost (alpha=0.1)
The boost uses the same multiplicative pattern as recency and temporal
signals. proof_count=1 is neutral, proof_count=50 gives ~+5% boost.
Non-observation fact types are unaffected (neutral 0.5).
* fix(retrieval): Apply proof_count boost to graph and temporal retrieval, normalize scaling
* fix(retrieval): correct proof_norm math to zero-center at count 1
* fix(retrieval): Apply proof_count boost to link_expansion retrieval
* fix: remove BFS zombie, clamp proof_norm to [0,1], fix test comment (log1p->math.log)
Remove the BFS spreading activation and MPFP (Multi-Path Fact Propagation)
graph retrieval strategies, leaving link_expansion as the sole graph
retrieval algorithm. Rename MPFPTimings to GraphRetrievalTimings and
mpfp_timings field to graph_timings since the timing struct is used by
LinkExpansionRetriever.
Deleted:
- hindsight-api-slim/hindsight_api/engine/search/mpfp_retrieval.py
- hindsight-api-slim/tests/test_mpfp_retrieval.py
Removed config: HINDSIGHT_API_MPFP_TOP_K_NEIGHBORS
* fix: return metadata in recall responses (#674)
Metadata stored during retain was never retrieved during recall.
Add metadata to all SQL SELECT queries, the RetrievalResult dataclass,
ScoredResult.to_dict(), and MemoryFact construction in the recall pipeline.
* test: add metadata round-trip test for retain→recall
Replace placeholder metadata test with one that actually passes
metadata via retain_batch_async and asserts it is returned on recall.
* fix: parse metadata JSON string from database in MemoryFact
asyncpg may return JSONB columns as strings. Add a field_validator
to MemoryFact.metadata to handle JSON string deserialization.
* fix(recall): reject empty queries with 400 and fix SQL parameter gap causing IndeterminateDatatypeError
When query text contains only punctuation/symbols (no word characters after
normalization), the BM25 arms are skipped but the old code still placed `limit`
at \$3 in the params list. If tags or tag_groups were also set, their params
(\$4+) were referenced in the SQL while \$3 was a gap, causing PostgreSQL to
raise IndeterminateDatatypeError.
Fix the parameter layout so `limit` is only appended to params when tokens are
present (i.e. when BM25 arms actually use LIMIT \$3), and shift tags_param_idx
from 4 to 3 in the no-tokens path.
Also add a field_validator on RecallRequest.query that rejects empty-after-
normalization queries at the API layer with a 400 before they reach the DB.
* refactor: extract tokenize_query helper and reuse in RecallRequest validator
* feat: add compound tag filtering via tag_groups
Adds tag_groups to RecallRequest and ReflectRequest to express arbitrary
boolean tag predicates: leaf {tags, match}, and/or/not compounds.
Top-level groups are AND-ed. Existing tags/tags_match unchanged.
Examples:
Step filter AND user scope:
tag_groups: [{tags: ["step:5","step:8"], match: "any_strict"},
{tags: ["user:alice"], match: "all_strict"}]
Exclusion:
tag_groups: [{tags: ["user:alice"], match: "all_strict"},
{not: {tags: ["archived"], match: "any_strict"}}]
- Recursive SQL builder (build_tag_groups_where_clause) threads through
all 4 retrieval strategies (semantic/BM25, temporal, graph, MPFP)
- Python-side filter (filter_results_by_tag_groups) for post-traversal
- 22 new unit tests
- OpenAPI spec + all clients regenerated (Rust, Python, TypeScript, Go)
* fix: add tag_groups: None to Rust CLI struct initializers
* fix: add tag_groups: None to Rust client test RecallRequest initializer
* feat: reject tags+tag_groups together, add tag_groups integration tests
- Add model_validator to RecallRequest and ReflectRequest that returns 422
when both `tags` and `tag_groups` are set (mutually exclusive)
- Add 5 integration tests for tag_groups compound filtering:
* validation: 422 when both fields are set
* AND filter: two leaf groups (step scope AND user scope)
* OR compound: user:alice OR user:bob
* NOT compound: user:alice AND NOT archived
* Nested: user:alice AND (step:5 OR step:8)
* ci: trigger CI run
* feat: introduce hindsight-api-slim and hindsight-all-slim packages
Closes#552
- Move all source code from hindsight-api/ to new hindsight-api-slim/
- hindsight-api-slim has heavy ML deps (torch, sentence-transformers,
transformers, einops, flashrank, mlx, mlx-lm, safetensors) and
pg0-embedded as optional extras: [local-ml], [embedded-db], [all]
- hindsight-api becomes a zero-code meta-package depending on
hindsight-api-slim[all] for full backward compatibility
- Add hindsight-all-slim meta-package: hindsight-api-slim + client + embed
- hindsight-all updated to depend on hindsight-api-slim[all]
- pg0.py: lazy-import pg0 with clear ImportError pointing to [embedded-db]
- Dockerfile: replace sed hack with proper uv sync --extra flags
- Update release.yml, test.yml, lint.sh, release.sh, CLAUDE.md and
all path references throughout the repo
* refactor: rename hindsight/ directory to hindsight-all/
* docs: document hindsight-api-slim and hindsight-all-slim package variants
Add package variants table and extras explanation to installation.md
* docs: remove emojis from installation.md, use professional tone
* docs: link Docker slim variant to pip package variants section
* docs: consolidate Docker image variants into single table
* ci: fix working-directory paths after package restructure
- Replace all hindsight-api → hindsight-api-slim in test.yml
- Replace hindsight → hindsight-all in test.yml
- Add --extra embedded-db to test-embed API install step
* ci: add local-ml and embedded-db extras to API sync steps
These extras were previously implicit in the old hindsight-api package
(which bundled everything). Now that hindsight-api-slim uses optional
extras, we must explicitly request local-ml and embedded-db in CI.
* ci: add API install step with embedded-db to test-embed smoke test
The smoke test starts hindsight-api as a daemon, which requires pg0-embedded.
Add a dedicated install step for hindsight-api-slim with embedded-db extra
so the daemon can start successfully.
* ci: remove --no-install-project when using optional extras
When --no-install-project is combined with --extra, the optional deps
are not installed because extras require the project to be active.
Remove --no-install-project from steps that need local-ml or embedded-db.
* ci: fix ordering of uv sync steps to preserve optional extras
When uv sync runs for a different workspace member, it removes optional
extras installed for other members. Fix by always running extra-requiring
API sync last, after other workspace member syncs.
Also remove --no-install-project from embedded-db sync in test-embed,
as --no-install-project prevents optional extras from being active.
* ci: add local-ml extra to test-embed API install for smoke test
The smoke test starts the full API server which needs sentence-transformers
for local embeddings (default provider). Add local-ml extra to the install.
* ci: simplify extras with --all-extras and add slim pip smoke test
- Replace explicit --extra local-ml --extra embedded-db with --all-extras
for cleaner, more maintainable sync steps
- Add test-pip-slim job: tests hindsight-api-slim[embedded-db] without
local ML models, using Cohere for embeddings/reranking (mirrors Docker
slim smoke test approach)
* ci: simplify slim smoke test to health check only (mirrors Docker test)
2026-03-13 13:50:03 +01:00
Renamed from hindsight-api/hindsight_api/engine/search/retrieval.py (Browse further)
