---
sidebar_position: 2
---

# Ingest Data

Store documents, conversations, and raw content into Hindsight to automatically extract and create memories.

When you **retain** content, Hindsight doesn't just store the raw text—it intelligently analyzes the content to extract meaningful facts, identify entities, and build a connected knowledge graph. This process transforms unstructured information into structured, queryable memories.

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeSnippet from '@site/src/components/CodeSnippet';

{/* Import raw source files */}
import retainPy from '!!raw-loader!@site/examples/api/retain.py';
import retainMjs from '!!raw-loader!@site/examples/api/retain.mjs';
import retainSh from '!!raw-loader!@site/examples/api/retain.sh';

:::info How Retain Works
Learn about fact extraction, entity resolution, and graph construction in the [Retain Architecture](/developer/retain) guide.
:::

:::tip Prerequisites
Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
:::

## Store a Single Memory

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={retainPy} section="retain-basic" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={retainMjs} section="retain-basic" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={retainSh} section="retain-basic" language="bash" />
</TabItem>
</Tabs>

## The Importance of Context

The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.

**Why context matters:**
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)

## Store with Context and Date

Always provide context and event dates for optimal memory extraction:

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={retainPy} section="retain-with-context" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={retainMjs} section="retain-with-context" language="javascript" />
</TabItem>
<TabItem value="cli" label="CLI">
<CodeSnippet code={retainSh} section="retain-with-context" language="bash" />
</TabItem>
</Tabs>

The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"

### Response Fields

The retain response includes:

| Field | Type | Description |
|-------|------|-------------|
| `success` | bool | Whether the operation succeeded |
| `bank_id` | string | The memory bank ID |
| `items_count` | int | Number of items processed |
| `async` | bool | Whether processed asynchronously |
| `usage` | TokenUsage | Token usage metrics for LLM calls (synchronous only) |

The `usage` field contains token metrics for cost tracking:
- `input_tokens`: Tokens consumed by prompts
- `output_tokens`: Tokens generated by the LLM
- `total_tokens`: Sum of input and output tokens

Note: `usage` is only present for synchronous operations. Async operations (`async: true`) do not return usage metrics.

## Batch Ingestion

Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={retainPy} section="retain-batch" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={retainMjs} section="retain-batch" language="javascript" />
</TabItem>
</Tabs>

The `document_id` groups related memories for later management.

## Store from Files

<Tabs>
<TabItem value="cli" label="CLI">

```bash
# Single file
hindsight memory retain-files my-bank document.txt

# Directory (recursive by default)
hindsight memory retain-files my-bank ./documents/
```

</TabItem>
</Tabs>


## Async Ingestion

For large batches, use async ingestion to avoid blocking:

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={retainPy} section="retain-async" language="python" />
</TabItem>
<TabItem value="node" label="Node.js">
<CodeSnippet code={retainMjs} section="retain-async" language="javascript" />
</TabItem>
</Tabs>

## Tagging Memories

Tags enable **visibility scoping**—useful when one memory bank serves multiple users but each should only see relevant memories. For example, an agent that chats with multiple users can tag memories by user ID and filter during recall.

### Tag Individual Items

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={retainPy} section="retain-with-tags" language="python" />
</TabItem>
</Tabs>

### Apply Tags to All Items in a Batch

Use `document_tags` to apply the same tags to all items in a request:

<Tabs>
<TabItem value="python" label="Python">
<CodeSnippet code={retainPy} section="retain-with-document-tags" language="python" />
</TabItem>
</Tabs>

When both `document_tags` and item-level `tags` are provided, they are merged together.

### Tag Naming Conventions

Use consistent naming patterns for tags:

| Pattern | Example | Use Case |
|---------|---------|----------|
| `user:<id>` | `user:alice` | Multi-user agent filtering |
| `session:<id>` | `session:123` | Session-based scoping |
| `room:<id>` | `room:general` | Chat room isolation |
| `topic:<name>` | `topic:feedback` | Topic categorization |

### Listing Tags

Use the list tags API to discover existing tags, useful for UI autocomplete or wildcard expansion:

```python
# List all tags in a bank
tags = client.list_tags(bank_id="my-bank")
for tag in tags.items:
    print(f"{tag.tag}: {tag.count} memories")

# Search with wildcards (* matches any characters)
user_tags = client.list_tags(bank_id="my-bank", q="user:*")
admin_tags = client.list_tags(bank_id="my-bank", q="*-admin")
```

See [Recall API](./recall#filter-by-tags) for filtering memories by tags during retrieval.