219 lines
8.3 KiB
Text
219 lines
8.3 KiB
Text
---
|
|
sidebar_position: 2
|
|
---
|
|
|
|
# Ingest Data
|
|
|
|
Store documents, conversations, and raw content into Hindsight to automatically extract and create memories.
|
|
|
|
When you **retain** content, Hindsight doesn't just store the raw text—it intelligently analyzes the content to extract meaningful facts, identify entities, and build a connected knowledge graph. This process transforms unstructured information into structured, queryable memories.
|
|
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
import CodeSnippet from '@site/src/components/CodeSnippet';
|
|
|
|
{/* Import raw source files */}
|
|
import retainPy from '!!raw-loader!@site/examples/api/retain.py';
|
|
import retainMjs from '!!raw-loader!@site/examples/api/retain.mjs';
|
|
import retainSh from '!!raw-loader!@site/examples/api/retain.sh';
|
|
|
|
:::info How Retain Works
|
|
Learn about fact extraction, entity resolution, and graph construction in the [Retain Architecture](/developer/retain) guide.
|
|
:::
|
|
|
|
:::tip Prerequisites
|
|
Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
|
|
:::
|
|
|
|
## Store a Single Memory
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-basic" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-basic" language="javascript" />
|
|
</TabItem>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-basic" language="bash" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
## The Importance of Context
|
|
|
|
The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.
|
|
|
|
**Why context matters:**
|
|
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
|
|
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
|
|
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)
|
|
|
|
## Store with Context and Date
|
|
|
|
Always provide context and event dates for optimal memory extraction:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-context" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-with-context" language="javascript" />
|
|
</TabItem>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-with-context" language="bash" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"
|
|
|
|
### Response Fields
|
|
|
|
The retain response includes:
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `success` | bool | Whether the operation succeeded |
|
|
| `bank_id` | string | The memory bank ID |
|
|
| `items_count` | int | Number of items processed |
|
|
| `async` | bool | Whether processed asynchronously |
|
|
| `usage` | TokenUsage | Token usage metrics for LLM calls (synchronous only) |
|
|
|
|
The `usage` field contains token metrics for cost tracking:
|
|
- `input_tokens`: Tokens consumed by prompts
|
|
- `output_tokens`: Tokens generated by the LLM
|
|
- `total_tokens`: Sum of input and output tokens
|
|
|
|
Note: `usage` is only present for synchronous operations. Async operations (`async: true`) do not return usage metrics.
|
|
|
|
## Batch Ingestion
|
|
|
|
Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-batch" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-batch" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
The `document_id` groups related memories for later management.
|
|
|
|
## Store from Files
|
|
|
|
Upload files directly — Hindsight automatically converts them to text and extracts memories. File processing always runs asynchronously and returns operation IDs for tracking.
|
|
|
|
**Supported formats:** PDF, DOCX, DOC, PPTX, PPT, XLSX, XLS, images (JPG, PNG, GIF, etc. — OCR), audio (MP3, WAV, FLAC, etc. — transcription), HTML, and plain text formats (TXT, MD, CSV, JSON, YAML, etc.)
|
|
|
|
<Tabs>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-files" language="bash" />
|
|
</TabItem>
|
|
<TabItem value="curl" label="HTTP">
|
|
<CodeSnippet code={retainSh} section="retain-files-curl" language="bash" />
|
|
</TabItem>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-files" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-files" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### File Retain Response
|
|
|
|
The file retain endpoint always returns asynchronously:
|
|
|
|
| Field | Type | Description |
|
|
|-------|------|-------------|
|
|
| `operation_ids` | string[] | One operation ID per uploaded file. Use `GET /v1/default/banks/{bank_id}/operations` to track progress. |
|
|
|
|
### Batch File Uploads
|
|
|
|
Upload up to 10 files per request (max 100 MB total). Each file becomes a separate document with optional per-file metadata:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-files-batch" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
:::info File Storage
|
|
Uploaded files are stored server-side (PostgreSQL by default, or S3/GCS/Azure for production). Configure storage via `HINDSIGHT_API_FILE_STORAGE_TYPE`. See [Configuration](../configuration#file-processing) for details.
|
|
:::
|
|
|
|
|
|
## Async Ingestion
|
|
|
|
For large batches, use async ingestion to avoid blocking:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-async" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-async" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### Cut Costs 50% with Provider Batch APIs
|
|
|
|
When using async retain, enable the provider Batch API to reduce LLM fact-extraction costs by 50%. OpenAI and Groq both offer this discount in exchange for a processing window of up to 24 hours — a trade-off that's typically invisible when retain already runs in the background.
|
|
|
|
```bash
|
|
export HINDSIGHT_API_RETAIN_BATCH_ENABLED=true
|
|
```
|
|
|
|
Hindsight submits fact extraction calls as a batch job to the provider, polls for completion, and processes results automatically. No changes to your API calls are needed.
|
|
|
|
:::note
|
|
Batch API cost savings require `async=true` in your retain request and a compatible provider (OpenAI or Groq).
|
|
:::
|
|
|
|
## Tagging Memories
|
|
|
|
Tags enable **visibility scoping**—useful when one memory bank serves multiple users but each should only see relevant memories. For example, an agent that chats with multiple users can tag memories by user ID and filter during recall.
|
|
|
|
### Tag Individual Items
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-tags" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
### Apply Tags to All Items in a Batch
|
|
|
|
Use `document_tags` to apply the same tags to all items in a request:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-document-tags" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
When both `document_tags` and item-level `tags` are provided, they are merged together.
|
|
|
|
### Tag Naming Conventions
|
|
|
|
Use consistent naming patterns for tags:
|
|
|
|
| Pattern | Example | Use Case |
|
|
|---------|---------|----------|
|
|
| `user:<id>` | `user:alice` | Multi-user agent filtering |
|
|
| `session:<id>` | `session:123` | Session-based scoping |
|
|
| `room:<id>` | `room:general` | Chat room isolation |
|
|
| `topic:<name>` | `topic:feedback` | Topic categorization |
|
|
|
|
### Listing Tags
|
|
|
|
Use the list tags API to discover existing tags, useful for UI autocomplete or wildcard expansion:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-list-tags" language="python" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
See [Recall API](./recall#filter-by-tags) for filtering memories by tags during retrieval.
|