d405b4feed
* Fix main-methods.py: entities is a dict, use .items() and .canonical_name * Migrate docs to use CodeSnippet components - Convert quickstart.md, retain.md, recall.md, reflect.md, memory-banks.md to .mdx - Use CodeSnippet to pull code from validated example scripts - Add missing 'name' parameter to create_bank calls - Fix main-methods.py entities iteration (dict not list) - Remove retain-new.mdx demo file * Migrate existing docs to match testing pattern with code snippet and add CLI tests to the CI * Fix doc-id issue + add main-method tests * CLI fixes * Update openAPI json * Fix rust build issues * increase sleep time for Hindsight to process the document * Added a polling sleep instead of fixed * Delete immediately fails, so create the doc a earlier in the test to get the doc ready * Add debug logs * Remove debug logs
112 lines
4 KiB
Text
112 lines
4 KiB
Text
---
|
|
sidebar_position: 2
|
|
---
|
|
|
|
# Ingest Data
|
|
|
|
Store documents, conversations, and raw content into Hindsight to automatically extract and create memories.
|
|
|
|
When you **retain** content, Hindsight doesn't just store the raw text—it intelligently analyzes the content to extract meaningful facts, identify entities, and build a connected knowledge graph. This process transforms unstructured information into structured, queryable memories.
|
|
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
import CodeSnippet from '@site/src/components/CodeSnippet';
|
|
|
|
{/* Import raw source files */}
|
|
import retainPy from '!!raw-loader!@site/examples/api/retain.py';
|
|
import retainMjs from '!!raw-loader!@site/examples/api/retain.mjs';
|
|
import retainSh from '!!raw-loader!@site/examples/api/retain.sh';
|
|
|
|
:::info How Retain Works
|
|
Learn about fact extraction, entity resolution, and graph construction in the [Retain Architecture](/developer/retain) guide.
|
|
:::
|
|
|
|
:::tip Prerequisites
|
|
Make sure you've completed the [Quick Start](./quickstart) to install the client and start the server.
|
|
:::
|
|
|
|
## Store a Single Memory
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-basic" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-basic" language="javascript" />
|
|
</TabItem>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-basic" language="bash" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
## The Importance of Context
|
|
|
|
The `context` parameter is crucial for guiding how Hindsight extracts memories from your content. Think of it as providing a lens through which the system interprets the information.
|
|
|
|
**Why context matters:**
|
|
- **Steers memory extraction**: Context tells the memory bank what type of information to focus on and how to interpret ambiguous content
|
|
- **Improves relevance**: Memories extracted with proper context are more accurately categorized and easier to retrieve
|
|
- **Disambiguates meaning**: The same sentence can have different implications depending on context (e.g., "the project was terminated" means different things in a career vs. product context)
|
|
|
|
## Store with Context and Date
|
|
|
|
Always provide context and event dates for optimal memory extraction:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-with-context" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-with-context" language="javascript" />
|
|
</TabItem>
|
|
<TabItem value="cli" label="CLI">
|
|
<CodeSnippet code={retainSh} section="retain-with-context" language="bash" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
The `timestamp` defaults to the current time if not specified. Providing explicit timestamps enables temporal queries like "What happened last spring?"
|
|
|
|
## Batch Ingestion
|
|
|
|
Store multiple items in a single request. **Batch ingestion is the recommended approach** as it significantly improves performance by reducing network overhead and allowing Hindsight to optimize the memory extraction process across related content.
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-batch" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-batch" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
The `document_id` groups related memories for later management.
|
|
|
|
## Store from Files
|
|
|
|
<Tabs>
|
|
<TabItem value="cli" label="CLI">
|
|
|
|
```bash
|
|
# Single file
|
|
hindsight memory retain-files my-bank document.txt
|
|
|
|
# Directory (recursive by default)
|
|
hindsight memory retain-files my-bank ./documents/
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
|
|
|
|
## Async Ingestion
|
|
|
|
For large batches, use async ingestion to avoid blocking:
|
|
|
|
<Tabs>
|
|
<TabItem value="python" label="Python">
|
|
<CodeSnippet code={retainPy} section="retain-async" language="python" />
|
|
</TabItem>
|
|
<TabItem value="node" label="Node.js">
|
|
<CodeSnippet code={retainMjs} section="retain-async" language="javascript" />
|
|
</TabItem>
|
|
</Tabs>
|