fleet-memory/hindsight-docs/blog/2026-03-26-llamaindex-agent-memory.md at 62c0992075c852517f6c4fcb20c6d62b2ba7f906

Ben 62c0992075

Llama index blog

2026-03-26 15:44:18 -04:00

16 KiB

Raw Blame History

title

description

authors

date

What Is LlamaIndex Agent Memory?

LlamaIndex has two generations of memory tooling. The original ChatMemoryBuffer stores raw conversation history in-session and resets on restart. The newer Memory class adds pluggable MemoryBlock modules -- including FactExtractionMemoryBlock -- backed by SQLite for persistence across restarts.

That's a real improvement. But even the newer system has hard limits: retrieval is vector similarity only, there's no knowledge graph or entity resolution, and it's tightly coupled to the LlamaIndex ecosystem. If you switch agent frameworks, your memory layer goes with it.

"Persistent" memory, in the fullest sense, means a system that extracts facts from conversations, builds a knowledge graph over time, and retrieves relevant context using multiple strategies in parallel -- not just the most recent embedding match. It should work regardless of which agent framework you're running.

hindsight-llamaindex provides this via three native LlamaIndex tools: retain_memory (store facts), recall_memory (search memory), and reflect_on_memory (synthesize a reasoned answer from everything stored).

The Problem with LlamaIndex's Built-In Memory

LlamaIndex's newer Memory API with SQLite persistence is a genuine step forward -- facts survive restarts, and FactExtractionMemoryBlock can pull structured information from conversations. For simple use cases, it's enough.

But it runs into hard architectural limits as soon as queries get complex:

Single retrieval strategy. LlamaIndex Memory uses vector similarity only. There's no BM25 for exact matches, no graph traversal for relationship queries, no temporal search for date-bounded recall. "What was I working on last Tuesday?" returns whatever is semantically close -- not what actually happened on Tuesday.
No knowledge graph. Facts are stored as embeddings, not as entities with relationships. The system can't connect "Priya owns the rate-limiting service" to "the rate-limiting blocker is resolved" without those facts appearing in the same conversation turn.
Framework coupling. LlamaIndex Memory is designed for LlamaIndex. If you migrate to a different agent framework, your memory layer doesn't come with you.
No synthesis. There's no equivalent of reflect_on_memory -- a query that reasons across everything stored and returns a synthesized answer, not just a ranked list of relevant chunks.

For agents that serve repeat users, you need all of this:

A coding assistant that remembers your stack, your team structure, and your open work items
A support agent that knows a user's history across dozens of sessions and can connect past issues to current ones
A research assistant that builds on findings from previous runs and understands what's already been explored

That's what Hindsight does. And hindsight-llamaindex wires it into LlamaIndex's standard tool system without replacing anything you already have.

How LlamaIndex Persistent Memory Works

LlamaIndex Agent (ReAct, FunctionCalling, etc.)
  └─ HindsightToolSpec (extends BaseToolSpec)
       ├─ retain_memory()     → Hindsight retain
       │                        (fact extraction, entity resolution, knowledge graph)
       ├─ recall_memory()     → Hindsight recall
       │                        (semantic + BM25 + graph + temporal retrieval)
       └─ reflect_on_memory() → Hindsight reflect
                                (synthesize a reasoned answer from all memories)

HindsightToolSpec extends LlamaIndex's BaseToolSpec. Call to_tool_list() and you get standard FunctionTool instances. No monkey-patching, no custom wrappers. Any LlamaIndex agent that accepts tools can use it.

Under the hood, Hindsight extracts structured facts, identifies entities, builds a knowledge graph, and runs four parallel retrieval strategies with cross-encoder reranking.

Step 1: Start Hindsight

pip install hindsight-all
export HINDSIGHT_API_LLM_API_KEY=YOUR_OPENAI_KEY
hindsight-api

Runs locally at http://localhost:8888 with embedded Postgres, embeddings, and reranking.

Or use Hindsight Cloud and skip self-hosting entirely.

Step 2: Install the Integration

pip install hindsight-llamaindex

Pulls in llama-index-core and hindsight-client.

Step 3: Create the Bank and Agent

Banks must exist before use. Since LlamaIndex agents are async, wrap everything in asyncio.run() for scripts (or use top-level await in notebooks):

import asyncio
from hindsight_client import Hindsight
from hindsight_llamaindex import HindsightToolSpec
from llama_index.llms.openai import OpenAI
from llama_index.core.agent import ReActAgent

async def main():
    client = Hindsight(base_url="http://localhost:8888")
    await client.acreate_bank("user-123", name="User 123 Memory")

    spec = HindsightToolSpec(
        client=client,
        bank_id="user-123",
        tags=["source:chat"],
        budget="mid",
    )
    tools = spec.to_tool_list()

    agent = ReActAgent(
        tools=tools,
        llm=OpenAI(model="gpt-4o-mini"),
        system_prompt=(
            "You are a helpful assistant with long-term memory. "
            "Use retain_memory to store important facts. "
            "Use recall_memory to search memory before answering."
        ),
    )

    # Session 1
    await agent.run(
        "I'm a data scientist. I use Python, SQL, and VS Code with dark mode."
    )

    # Session 2 (new agent instance, same bank_id)
    agent = ReActAgent(tools=tools, llm=OpenAI(model="gpt-4o-mini"))
    response = await agent.run("What IDE do I use?")
    # → "You use VS Code with dark mode."
    print(response)

asyncio.run(main())

Three tools, one bank. Memory persists across agent instances because it's stored in Hindsight, not in the agent object.

Convenience Factory

If you don't need full BaseToolSpec control:

from hindsight_llamaindex import create_hindsight_tools

tools = create_hindsight_tools(
    client=client,
    bank_id="user-123",
    include_reflect=False,  # only retain + recall
)

The factory wraps HindsightToolSpec and returns a filtered list. Use include_retain, include_recall, and include_reflect to control which tools are exposed.

Selective Tools and Per-User Memory Banks

LlamaIndex's BaseToolSpec natively supports selective export:

spec = HindsightToolSpec(client=client, bank_id="user-123")

# Read-only: only recall and reflect
tools = spec.to_tool_list(spec_functions=["recall_memory", "reflect_on_memory"])

This is useful when different agents need different permissions. A research agent gets all three tools. A reporting agent gets read-only access.

For multi-user applications, parameterize bank_id for per-user isolation:

def create_agent_for_user(user_id: str) -> ReActAgent:
    spec = HindsightToolSpec(
        client=client,
        bank_id=f"user-{user_id}",
    )
    return ReActAgent(
        tools=spec.to_tool_list(),
        llm=OpenAI(model="gpt-4o-mini"),
    )

Each bank is fully isolated. No cross-user data leakage.

Global Configuration

Set defaults once, override per-call:

from hindsight_llamaindex import configure

configure(
    hindsight_api_url="http://localhost:8888",
    api_key="YOUR_API_KEY",  # or HINDSIGHT_API_KEY env var
    budget="mid",
    tags=["source:llamaindex"],
)

# Now you can skip client= and url= arguments
tools = create_hindsight_tools(bank_id="user-123")

Per-call parameters override global config. Parameters you don't pass fall through to the global defaults.

Real-World Use Cases

Persistent memory is most valuable in agents designed for ongoing, multi-session relationships. A few patterns that work well:

Personal coding assistant. A developer uses the same agent across dozens of sessions. After the first session, the agent knows their stack (Python, gRPC, Postgres), their team structure, and their open work items. Every subsequent session starts with full context rather than blank state. The developer never re-explains their environment.

Customer support agent. A support bot serves a user who contacts it five times over two months. Without persistent memory, each interaction starts with identity verification and problem re-explanation. With Hindsight, the agent recalls the user's account tier, their previous issue history, and the resolution steps already tried. Resolution time drops because the agent skips the orientation phase.

Research assistant. A knowledge-gathering agent works through a multi-week research project across many sessions. Each session picks up where the last left off; the agent recalls which sources have been reviewed, which hypotheses have been tested, and which questions remain open. The agent accumulates expertise rather than starting fresh.

In each case, the value is the same: the agent's context grows over time rather than resetting. The longer the relationship, the more useful the agent becomes.

Common Pitfalls

Bank must exist first. Call client.create_bank(bank_id, name=...) before the agent starts. If the bank doesn't exist, retain/recall will fail with a 404-style error.

Async processing delay. After retain_memory, Hindsight processes content asynchronously. Extracting facts, entities, and embeddings takes 1-3 seconds. If you retain and immediately recall in the same session, the new memories may not be searchable yet.

Budget tuning. Default budget="mid" balances speed and thoroughness. Use "low" for latency-sensitive agents, "high" for deep analysis. Budget controls how many retrieval strategies run and how much reranking happens.

Reflect vs. recall. Use recall_memory for raw facts ("What IDE do I use?"). Use reflect_on_memory for synthesis ("Based on everything you know, what should I prioritize?"). Reflect is slower but produces reasoned answers that draw on the full knowledge graph.

LlamaIndex agents are async. HindsightToolSpec provides both sync and async tool methods. Async agents like ReActAgent use the async variants (aretain, arecall, areflect) directly.

When NOT to Use This

Persistent memory isn't always the right tool.

In-session context only. If your agent only needs to remember things within a single conversation, LlamaIndex's ChatMemoryBuffer is simpler and has zero latency overhead. Don't add Hindsight just because you can.
Document search (RAG). If you need vector search over a document corpus, use LlamaIndex's built-in VectorStoreIndex. Hindsight is a memory system for facts learned over time, not a document store.
Ephemeral agents. If each agent invocation is stateless by design -- batch processing, one-shot tasks -- persistent memory adds complexity without benefit.
Latency-critical hot paths. Each memory operation adds a network round-trip. If sub-100ms response time matters more than personalization, skip it.

How This Compares

	`ChatMemoryBuffer`	LlamaIndex `Memory` (SQLite)	Raw vector store	Hindsight
Scope	In-session only	Persistent	Persistent	Persistent
What it stores	Raw conversation history	Extracted facts (flat)	Embeddings of arbitrary content	Extracted facts + entities + knowledge graph
Retrieval	Sequential context window	Vector similarity only	Vector similarity only	Semantic + BM25 + graph + temporal, with reranking
Entity resolution	No	No	No	Yes
Synthesis	No	No	No	Yes (`reflect_on_memory`)
Framework coupling	LlamaIndex only	LlamaIndex only	None	None
Best for	Single-session chat	Simple cross-session recall within LlamaIndex	Document search (RAG)	Long-term user/agent memory across frameworks

vs. LangGraph/LangChain: If you're using LangGraph instead of LlamaIndex, see hindsight-langgraph which offers tools, graph nodes, and a BaseStore adapter.

FAQ

Why isn't recall_memory returning results I just stored? Hindsight processes retained content asynchronously. After calling retain_memory, wait 1-3 seconds before calling recall_memory. In production, architect your workflow so retain and recall happen in separate sessions rather than back-to-back in the same one.

What happens if I call recall_memory before the bank exists? The call will fail. Always call client.create_bank(bank_id) before initializing HindsightToolSpec. If you're not sure whether the bank exists, wrap the create call in a try/except and handle the conflict error.

My agent isn't calling retain_memory automatically. Why? The agent decides when to call tools based on its system prompt. If your system prompt doesn't instruct the agent to store important facts, it won't do so reliably. Include explicit instruction: "Use retain_memory to store any facts, preferences, or context the user shares." Then test with a few turns to confirm the behavior.

Can I use Hindsight with a non-ReAct LlamaIndex agent? Yes. HindsightToolSpec returns standard FunctionTool instances via to_tool_list(). Any LlamaIndex agent that accepts a tool list supports it, including FunctionCallingAgent and custom agent implementations that use LlamaIndex's tool abstractions.

Recap

Session memory resets. Long-term memory doesn't have to.

hindsight-llamaindex fits into any LlamaIndex agent in three steps: install the package, create a bank, pass the tools. From that point, every fact your agent learns is stored, indexed, and retrievable in the current session and in every session that follows. The agent that served a user in January knows what it learned in January when that user returns in March.

That's the practical difference between a session-scoped tool and an agent with genuine memory. The more it's used, the more useful it becomes.

Try it now: pip install hindsight-all hindsight-llamaindex and run the example above. Or start with Hindsight Cloud if you'd rather skip self-hosting.

Next Steps

Docs: LlamaIndex integration guide
Other integrations: LangGraph, Pydantic AI, CrewAI

16 KiB Raw Blame History