224b7b74c1
* feat: accept pdf, images and office files * refactor: rename FileConverter to FileParser, simplify file retain API - Rename engine/converters/ → engine/parsers/, FileConverter → FileParser, ConverterRegistry → FileParserRegistry, MarkitdownConverter → MarkitdownParser - Rename env var HINDSIGHT_API_FILE_CONVERTER → HINDSIGHT_API_FILE_PARSER - Remove async/document_tags params from FileRetainRequest (always async now) - Add retain_files() to Python Hindsight client and retainFiles() to TypeScript client - Add sample.pdf to doc examples for working file upload demonstrations - Update test_file_retain.py to use new parser names and always-async behavior - Fix Go client missing os import in api_files.go - Simplify postgresql.py storage to minimal schema * fix: update rust CLI tests to use is_supported_file instead of is_text_file * fix: patch Go api_files.go to add missing 'os' import after generation * fix: insert 'os' import after 'net/url' in api_files.go patch for correct position * chore: regenerate OpenAPI spec and clients (converter→parser description update)
205 lines
7.6 KiB
Text
205 lines
7.6 KiB
Text
---
|
|
sidebar_position: 2
|
|
---
|
|
|
|
# Ingest Data
|
|
|
|
Store documents, conversations, and raw content into Hindsight to automatically extract and create memories.
|
|
|
|
When you **retain** content, Hindsight doesn't just store the raw text—it intelligently analyzes the content to extract meaningful facts, identify entities, and build a connected knowledge graph. This process transforms unstructured information into structured, queryable memories.
|
|
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
import CodeSnippet from '@site/src/components/CodeSnippet';
|
|
|
|
{/* Import raw source files */}
|
|
import retainPy from '!!raw-loader!@site/examples/api/retain.py';
|
|
import retainMjs from '!!raw-loader!@site/examples/api/retain.mjs';
|
|
import retainSh from '!!raw-loader!@site/examples/api/retain.sh';
|
|
|
|
:::info How Retain Works
|
|
Learn about fact extraction, entity resolution, and graph construction in the [Retain Architecture](/developer/retain) guide.
|
|
:::
|
|
|
|
:::tip Prerequisites
|
|
Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
|
|
:::
|
|
|
|
## Store a Single Memory
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-basic" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-basic" language="javascript" />
|
|
</TabItem>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-basic" language="bash" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
## The Importance of Context
|
|
|
|
The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.
|
|
|
|
**Why context matters:**
|
|
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
|
|
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
|
|
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)
|
|
|
|
## Store with Context and Date
|
|
|
|
Always provide context and event dates for optimal memory extraction:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-context" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-with-context" language="javascript" />
|
|
</TabItem>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-with-context" language="bash" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"
|
|
|
|
### Response Fields
|
|
|
|
The retain response includes:
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `success` | bool | Whether the operation succeeded |
|
|
| `bank_id` | string | The memory bank ID |
|
|
| `items_count` | int | Number of items processed |
|
|
| `async` | bool | Whether processed asynchronously |
|
|
| `usage` | TokenUsage | Token usage metrics for LLM calls (synchronous only) |
|
|
|
|
The `usage` field contains token metrics for cost tracking:
|
|
- `input_tokens`: Tokens consumed by prompts
|
|
- `output_tokens`: Tokens generated by the LLM
|
|
- `total_tokens`: Sum of input and output tokens
|
|
|
|
Note: `usage` is only present for synchronous operations. Async operations (`async: true`) do not return usage metrics.
|
|
|
|
## Batch Ingestion
|
|
|
|
Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-batch" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-batch" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
The `document_id` groups related memories for later management.
|
|
|
|
## Store from Files
|
|
|
|
Upload files directly — Hindsight automatically converts them to text and extracts memories. File processing always runs asynchronously and returns operation IDs for tracking.
|
|
|
|
**Supported formats:** PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, images (JPG, PNG, GIF, etc. — OCR), audio (MP3, WAV, FLAC, etc. — transcription), HTML, and plain text formats (TXT, MD, CSV, JSON, YAML, etc.)
|
|
|
|
<Tabs>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-files" language="bash" />
|
|
</TabItem>
|
|
<TabItem value="curl" label="HTTP">
|
|
<CodeSnippet code={retainSh} section="retain-files-curl" language="bash" />
|
|
</TabItem>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-files" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-files" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### File Retain Response
|
|
|
|
The file retain endpoint always returns asynchronously:
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `operation_ids` | string[] | One operation ID per uploaded file. Use `GET /v1/default/banks/{bank_id}/operations` to track progress. |
|
|
|
|
### Batch File Uploads
|
|
|
|
Upload up to 10 files per request (max 100 MB total). Each file becomes a separate document with optional per-file metadata:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-files-batch" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
:::info File Storage
|
|
Uploaded files are stored server-side (PostgreSQL by default, or S3/GCS/Azure for production). Configure storage via `HINDSIGHT_API_FILE_STORAGE_TYPE`. See [Configuration](../configuration#file-processing) for details.
|
|
:::
|
|
|
|
|
|
## Async Ingestion
|
|
|
|
For large batches, use async ingestion to avoid blocking:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-async" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-async" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
## Tagging Memories
|
|
|
|
Tags enable **visibility scoping**—useful when one memory bank serves multiple users but each should only see relevant memories. For example, an agent that chats with multiple users can tag memories by user ID and filter during recall.
|
|
|
|
### Tag Individual Items
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-tags" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### Apply Tags to All Items in a Batch
|
|
|
|
Use `document_tags` to apply the same tags to all items in a request:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-document-tags" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
When both `document_tags` and item-level `tags` are provided, they are merged together.
|
|
|
|
### Tag Naming Conventions
|
|
|
|
Use consistent naming patterns for tags:
|
|
|
|
| Pattern | Example | Use Case |
|
|
|---------|---------|----------|
|
|
| `user:<id>` | `user:alice` | Multi-user agent filtering |
|
|
| `session:<id>` | `session:123` | Session-based scoping |
|
|
| `room:<id>` | `room:general` | Chat room isolation |
|
|
| `topic:<name>` | `topic:feedback` | Topic categorization |
|
|
|
|
### Listing Tags
|
|
|
|
Use the list tags API to discover existing tags, useful for UI autocomplete or wildcard expansion:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-list-tags" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
See [Recall API](./recall#filter-by-tags) for filtering memories by tags during retrieval.
|