fleet-memory/hindsight-docs
Nicolò Boschi 28dac7c7f8
fix: prevent silent memory loss on consolidation LLM failure (#601)
* fix: prevent silent memory loss on consolidation LLM failure

When all LLM retries are exhausted during consolidation, memories were
being marked consolidated_at unconditionally, permanently excluding them
from future consolidation runs without producing any observations.

Fix with two complementary mechanisms:
- Adaptive batch splitting: on LLM failure, the batch is halved and
  retried recursively down to batch_size=1, recovering most transient
  failures (rate limits, Pydantic validation on long prompts) without
  operator intervention
- consolidation_failed_at column: only single-memory batches that still
  fail after all retries are marked here instead of consolidated_at, so
  they remain visible and retryable
- New API endpoint POST /v1/default/banks/{bank_id}/consolidation/retry-failed
  resets these memories for the next consolidation run

* chore: regenerate OpenAPI spec

* fix: rename consolidation endpoint from /retry-failed to /recover

* fix: add consolidation_failed_at column, adaptive batch splitting, and recovery API

- Migration a3b4c5d6e7f8: add consolidation_failed_at TIMESTAMPTZ column to
  memory_units with an index for efficient failure queries; properly chains off
  g7h8i9j0k1l2 (backsweep_orphan_observations)
- Consolidator: filter pending memories with consolidation_failed_at IS NULL
  so failed memories are not re-fetched in an infinite loop
- Consolidator: adaptive batch splitting — when a batch exhausts all 3 LLM
  retries, halve it and retry sub-batches recursively; only single-memory
  batches that also exhaust all retries get consolidation_failed_at set
- New tests (9 total) covering: adaptive splitting recovers all memories,
  larger batch splitting, single-memory permanent failure, exclusion from
  next run, partial batch failure, recover resets columns, recover returns
  0 when none failed, recover-then-consolidate succeeds, HTTP endpoint

* chore: regenerate Go, Python, TypeScript clients with recover consolidation endpoint

* feat: add Recover Consolidation action to bank Actions dropdown

* style: apply ruff formatting to http.py and config.py

* fix: handle consolidation scope in large batch test mock LLM

The mock LLM was returning {"facts": ...} for ALL calls including consolidation.
Consolidation doesn't use skip_validation=True so it expects a _ConsolidationBatchResponse
instance, not a raw dict. Before this PR consolidation silently swallowed the AttributeError
(failed=False was returned); now failed=True triggers adaptive splitting and timeouts.

Fix: return _ConsolidationBatchResponse() when scope=="consolidation".

* fix: restrict claude-agent-sdk to macOS platform only (no Linux wheel available)

Also fix pre-existing type errors: use setattr for XLM-RoBERTa monkey-patch
and add missing reranker_local_fp16/bucket_batching/batch_size fields to main.py config constructor.

* fix: add UV_INDEX_STRATEGY=unsafe-best-match to fix markupsafe cp314 wheel conflict

PyTorch CPU index serves markupsafe==3.0.3 with only cp314 wheels.
uv's default first-index strategy stops at the first index with any version
even if no compatible wheel exists. unsafe-best-match searches all indices
for the best compatible wheel, falling back to PyPI for markupsafe.

* fix: use explicit pytorch index to prevent markupsafe wheel conflict

Configure the pytorch CPU index as explicit=true in pyproject.toml so it is
ONLY used for torch (via [tool.uv.sources]). All other packages (including
markupsafe) are resolved exclusively from PyPI, preventing the pytorch index
from serving incompatible cp314-only wheels for non-pytorch packages.

Remove UV_INDEX and UV_INDEX_STRATEGY from CI workflow (no longer needed
since the index is now configured in pyproject.toml).

* ci: trigger CI run

* ci: retry trigger

* ci: trigger after remote URL fix

* ci: add workflow_dispatch to unblock manual trigger

* fix: remove empty env blocks left after UV_INDEX removal

* fix: add type: ignore for optional claude_agent_sdk imports (macOS-only)

* fix: correct type: ignore rules for claude_agent_sdk and fix utcnow deprecation
2026-03-17 20:15:33 +01:00
..
blog blog: Streamlit chatbot with persistent memory (#602) 2026-03-17 14:45:54 -04:00
docs feat(retain): verbatim, chunks modes and named retain strategies (#593) 2026-03-17 18:08:25 +01:00
examples feat: change tags for a document (#517) 2026-03-07 09:00:13 +01:00
src feat: hindsight-hermes integration for Hermes Agent (#600) 2026-03-17 18:06:55 +01:00
static fix: prevent silent memory loss on consolidation LLM failure (#601) 2026-03-17 20:15:33 +01:00
versioned_docs Release v0.4.18 2026-03-13 15:21:03 +01:00
versioned_sidebars docs: add Best Practices unversioned page (#598) 2026-03-17 14:03:38 +01:00
.gitignore rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
docusaurus.config.ts blog: Streamlit chatbot with persistent memory (#602) 2026-03-17 14:45:54 -04:00
package.json docs: revamp sidebar with icon grids and language support (#563) 2026-03-13 14:12:42 +01:00
README.md rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
sidebars.ts docs: add Best Practices unversioned page (#598) 2026-03-17 14:03:38 +01:00
tsconfig.json rename to hindsight (#2) 2025-11-25 19:28:26 +01:00
versions.json doc: release notes for 0.4.0 (#217) 2026-01-28 16:54:05 +01:00

Website

This website is built using Docusaurus, a modern static website generator.

Installation

npm install

Local Development

npm start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

Build

npm run build

This command generates static content into the build directory and can be served using any static contents hosting service.

Deployment

Using SSH:

USE_SSH=true npm run deploy

Not using SSH:

GIT_USER=<Your GitHub username> npm run deploy

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.