* feat: add update_mode='append' for retain to concatenate content to existing documents
When retaining with update_mode='append' and a document_id that already exists,
the new content is appended to the existing document text and the full document
is reprocessed. Delta retain automatically skips unchanged chunks, so only the
new content triggers LLM extraction.
- Add update_mode field to MemoryItem (API), RetainContentDict (internal), MCP tools
- Validate that update_mode='append' requires a document_id
- Fetch existing document content and prepend before processing in orchestrator
- Update Python, TypeScript, Go generated clients and top-level client wrappers
- Add tests for append, multiple appends, no-existing-doc, validation, and default replace
* fix: add update_mode field to Rust CLI and client MemoryItem initializers
* chore: regenerate docs skill references for update_mode
* feat(retain): add verbatim extraction mode
Adds retain_extraction_mode="verbatim" that stores each chunk as-is
without LLM summarization. The LLM still runs to extract entities,
temporal info, and location for full indexability — only the fact text
is replaced with the original chunk content (one memory per chunk).
Useful for RAG-style indexing and benchmarks where original text
must be preserved in memory.
- Add "verbatim" to RETAIN_EXTRACTION_MODES in config.py
- Add VERBATIM_FACT_EXTRACTION_PROMPT with instructions to preserve text
- Add _collapse_to_verbatim() post-processing to enforce 1 fact/chunk
- Expose in bank config UI dropdown with updated description
- Update configuration.md docs with verbatim mode description
- Add unit test for _collapse_to_verbatim and integration test via LLM
- Fix pre-existing main.py CLI override missing new reranker fields
- Fix pre-existing cross_encoder.py ty type error via setattr
* refactor(retain): verbatim mode skips 'what' field entirely
Instead of asking the LLM to echo the chunk text back into 'what' and
then discarding it, verbatim mode now uses a dedicated schema
(VerbatimExtractedFact) that omits the 'what' field altogether.
The LLM only returns metadata (entities, temporal info, location, who),
saving output tokens and avoiding any risk of paraphrasing before the
backfill.
- Add VerbatimExtractedFact / VerbatimFactExtractionResponse models
- Verbatim mode skips causal-relations section (nothing to relate causally)
- _extract_facts_from_chunk: allow missing 'what' in verbatim mode,
set combined_text="" (backfilled by _collapse_to_verbatim)
- Update verbatim prompt to say DO NOT include 'what'
* feat(retain): add index_only extraction mode
Zero-LLM retain mode: chunks are stored as-is with no LLM call, no
entity extraction, and no temporal indexing. Embeddings still run for
semantic search. User-provided entities via RetainContent.entities
are the sole source of entity data.
Early return placed before the batch-API check so no LLM queue or
concurrency locks are acquired.
- Add "index_only" to RETAIN_EXTRACTION_MODES
- Add _extract_facts_index_only() with pure Python chunking path
- Add to UI dropdown and update description
- Update configuration.md with index_only docs and table entry
- Add unit test asserting zero token usage and exact text preservation
* feat(retain): add named retain strategies
Allows mixing extraction modes in a single bank via named strategies.
Each strategy is a set of hierarchical config overrides (extraction_mode,
chunk_size, entity_labels, entities_allow_free_form, etc.) applied on
top of the resolved bank config at retain time.
- retain_strategies: dict of strategy_name → config overrides (bank config)
- retain_default_strategy: default strategy when none specified (bank config)
- strategy field on /retain request: per-call override
- apply_strategy() in config_resolver applies overrides via dataclasses.replace()
- strategy propagates through retain_batch_async → _retain_batch_async_internal
and through the async worker task payload
- Any hierarchical field is overridable per strategy, including entity_labels
and entities_allow_free_form
- Docs updated with strategy configuration example and RRF fairness note
- Unit test for apply_strategy covering overrides, unknown strategy, and
non-hierarchical field filtering
* feat(retain): add per-item strategy and strategy tests
- Add `strategy` field to `MemoryItem` so individual items in a retain
request can override the request-level strategy
- Add `strategy` field to `FileRetainMetadata` for per-file strategy
override in file retain requests
- Group memory items by effective strategy in `api_retain`; each group
is processed as a separate batch, results are aggregated
- Thread strategy through `submit_async_file_retain` →
`_handle_file_convert_retain` → retain task payload
- Add `operation_ids` to `RetainResponse` for async requests with
mixed per-item strategies
- Add `test_strategy_overrides_extraction_mode_for_index_only`: unit
test verifying a named strategy with index_only bypasses the LLM
- Add `test_retain_request_per_item_strategy_field`: unit test for
per-item strategy grouping logic
* feat(ui): add retain strategies and default strategy to bank config UI
- Add StrategiesEditor component: per-strategy cards with name input and
JSON overrides textarea; supports add/remove; validates JSON inline
- Add Default Strategy text input (retain_default_strategy)
- Update RetainEdits type and retainSlice() to include both new fields
- Regenerate OpenAPI spec (retain_strategies, retain_default_strategy,
per-item strategy on MemoryItem/FileRetainMetadata, operation_ids on
RetainResponse)
* refactor(ui): move retain strategies into its own dedicated config section
* feat(ui): improve retain strategies UX and add strategy to document dialog
- Strategy form now includes entity section (free form toggle + entity labels editor)
- Default strategy selector moved outside tab panel, above strategy chips
- Strategy tabs redesigned with underline indicator style for clarity
- Remove strategy confirms with AlertDialog
- Fix tab re-render bug when typing strategy name (skipSyncRef)
- Add strategy field to Add New Document dialog (text + per-file for uploads)
- File upload collapsible uses same Document/Tags/Source tabbed layout
- API: validate empty strategy names in config_resolver
- api.ts: add strategy field to retain and uploadFiles types
* fix: forward strategy through HTTP layer and SDK; add integration test
- route.ts: extract and forward `strategy` from request body to retainBatch
- TypeScript SDK: accept and forward `strategy` in retainBatch options and per-item
- config_resolver.py: validate empty strategy name keys on update
- bank-config-view.tsx: merge entity fields into RetainStrategyForm, redesign strategy tabs with underline style, add confirmation dialog for removal, fix tab-reset-on-typing with skipSyncRef, move default strategy selector outside panel
- bank-selector.tsx: add strategy field to Add Document dialog (per-file in tabbed collapsible)
- test_retain.py: add end-to-end integration test verifying named strategy application (index_only = 0 LLM tokens)
* fix: regenerate TypeScript client with strategy field in RetainRequest/MemoryItem
- Regenerate OpenAPI spec to include strategy field in RetainRequest and MemoryItem
- Regenerate TypeScript client from updated spec
- Add strategy to MemoryItemInput interface
- Remove (item as any) cast now that strategy is properly typed
* rename: index_only extraction mode → chunks
* remove top-level strategy from RetainRequest; strategy is per-item only
* fix(clients): update Go and Python generated clients with strategy/operation_ids fields
* fix(ci): update hierarchical field count, add strategy to Rust MemoryItem initializers
* fix(go-client): minimal targeted YAML updates for strategy/operation_ids fields
* feat: support timestamp="unset" to retain content without a date
When callers retain timeless content (e.g. fictional documents, static
reference material), passing timestamp="unset" now skips the utcnow()
default so mentioned_at is stored as NULL instead of an artificial date.
- HTTP: validate_timestamp recognises "unset" sentinel and threads it
through api_retain as event_date=None (key present, value None), which
the orchestrator distinguishes from key-absent (still defaults to now)
- Orchestrator: new branching logic separates "key absent" → utcnow()
from "key present but None" → no date
- types.py: RetainContent.event_date and ProcessedFact.mentioned_at are
now datetime | None; removed the unused _now_utc factory
- fact_extraction.py: all event_date params accept datetime | None;
_build_user_message emits "Event Date: Unknown" when None; removed
mentioned_at from the Fact LLM response model (LLM never sets it)
- embedding_processing: skip date suffix when fact_date is None
- entity_resolver: COALESCE(event_date, now()) for first_seen/last_seen
so entities table NOT NULL constraint is preserved
- link_utils: skip temporal linking for units without event_date
- Migration aa2b3c4d5e6f: DROP NOT NULL on memory_units.event_date
- Tests: test_retain_no_timestamp and test_retain_omit_timestamp_defaults_to_now
- Docs + OpenAPI + TypeScript client updated
* refactor: replace _TIMESTAMP_UNKNOWN sentinel with plain string comparison
The sentinel object() was only needed to distinguish "unset" from None
at the boundary — but since the field type is datetime | str | None,
"unset" can pass through the validator unchanged and be compared directly.
* chore: regenerate OpenAPI spec and clients after timestamp type change
timestamp field is now datetime | str | None to accept the "unset" sentinel value.
* feat: observation_scopes field to drive observations granularity
* fix(migration): make a2b3c4d5e6f7 a no-op to fix CI on fresh DB
The z1u2v3w4x5y6 migration already creates observation_scopes directly,
so the rename migration fails on fresh installs where observation_tags
never existed.
* chore: remove no-op migration a2b3c4d5e6f7
* feat: regenerate clients with observation_scopes field
- Add observation_scopes to OpenAPI spec and all generated clients
- Fix Rust build.rs to handle anyOf with >2 variants containing null
(previously only handled 2-item anyOf, causing progenitor to panic
on the observation_scopes union type)
* fix(rust): add observation_scopes: None to MemoryItem struct literals
* fix(api): add title to observation_scopes Field for deterministic client generation
Adding title="ObservationScopes" makes the inline anyOf schema use
the explicit name instead of deriving it from the field name, which
was non-deterministic between arm64 (macOS) and amd64 (CI) Docker.
Also fixes description: "each entity" -> "each tag".
* fix(scripts): use linux/amd64 Docker for client generation to ensure reproducibility
Both Python and Go client generation now use --platform linux/amd64
Docker, ensuring identical output on macOS arm64 (local) and Linux
amd64 (CI). Also switches Go from JAR+Java to Docker to eliminate
Java version variability.
* chore: update generated clients to API v0.4.14
* fix(test): add retry logic to test_retain_chinese_content to handle non-deterministic LLM output
* fix(test): mark test_retain_chinese_content as xfail due to non-deterministic LLM translation
* ci: add Go client integration tests
Add test-go-client job to CI workflow following the same pattern as
Python, TypeScript, and Rust client tests. The job:
- Sets up Go 1.23 with dependency caching
- Starts the Hindsight API server
- Runs integration tests using the 'integration' build tag
- Displays server logs on failure
The integration tests (hindsight-clients/go/integration_test.go) cover
all core operations: retain, recall, reflect, bank management, and
end-to-end workflows.
* Move Go cookbook content to hindsight-cookbook repo
Removes Go-specific cookbook content that was added in PR #375:
- applications/go-memory-service.md
- recipes/go-quickstart.md
- recipes/go-concurrent-pipeline.md
These have been moved to the hindsight-cookbook repository where
cookbook content should live per project conventions.
* feat(go): add CI test for Go client and patch for ogen null handling
- Add test-go-client job to GitHub Actions CI workflow
- Create post-generation patch script (patch-ogen.sh) to fix ogen's
handling of null values in optional string fields
- Patch OptString.Decode() to check jx.Next() type before decoding,
properly handling explicit null in JSON responses
The patch ensures generated code persists across regenerations and
handles the Hindsight API's nullable optional fields correctly.
Fixes: Go client integration tests for retain and bank operations
Note: Some tests still fail for nullable arrays/objects - those
require additional patches for other Opt* types.
* feat: use official go generator for Go client
* feat: use official go generator for Go client
* ci fixes
* chore: sync Go client with latest OpenAPI spec
- Add model_child_operation_status.go (new model)
- Update model_operation_status_response.go with child operations
- Update go.mod/go.sum dependencies
- Update api/openapi.yaml
