diff --git a/hindsight-docs/blog/2026-04-06-autogen-persistent-memory.md b/hindsight-docs/blog/2026-04-06-autogen-persistent-memory.md new file mode 100644 index 00000000..71386c71 --- /dev/null +++ b/hindsight-docs/blog/2026-04-06-autogen-persistent-memory.md @@ -0,0 +1,199 @@ +--- +title: "Persistent Memory for AutoGen Agents with Hindsight" +authors: [DK09876] +date: 2026-04-06 +tags: [autogen, integrations, agents, memory, python, microsoft] +description: "AutoGen agents lose all state when a session ends. hindsight-autogen adds three tools — retain, recall, reflect — that give your agents persistent memory across sessions." +image: /img/blog/autogen-persistent-memory.png +hide_table_of_contents: true +--- + +![Persistent Memory for AutoGen Agents with Hindsight](/img/blog/autogen-persistent-memory.png) + +AutoGen is Microsoft's open-source framework for building multi-agent systems: conversable agents, group chats, tool use, code execution. But when a session ends, every agent in the conversation forgets everything. `hindsight-autogen` fixes that by giving AutoGen agents persistent memory through three callable tools. + + + +## TL;DR + +- AutoGen agents have no built-in cross-session memory; state resets every run +- `hindsight-autogen` provides three `FunctionTool` instances for `AssistantAgent`: `hindsight_retain`, `hindsight_recall`, `hindsight_reflect` +- One pip install, pass `tools=[...]` to your agent, done +- Works with [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup) or self-hosted + +## The problem + +AutoGen gives you `AssistantAgent` with chat history within a session. That's a message list; it doesn't extract facts, doesn't build knowledge over time, and disappears when the process exits. + +For agents that serve repeat users or run across multiple sessions, you need more: + +- A coding assistant that remembers your stack, preferences, and past decisions +- A multi-agent team where a coordinator retains knowledge from previous group chats +- A support agent that knows your account history across dozens of conversations + +None of this works with in-session chat history. You need a system that extracts facts from conversations, builds knowledge over time, and retrieves relevant context semantically. + +That's what Hindsight does. And `hindsight-autogen` wires it into AutoGen's tool system. + +## Architecture + +``` +AutoGen AssistantAgent(tools=[...]) + └─ Hindsight FunctionTools (via create_hindsight_tools) + ├─ hindsight_retain → Hindsight retain + │ (fact extraction, entity resolution, knowledge graph) + ├─ hindsight_recall → Hindsight recall + │ (semantic + BM25 + graph + temporal retrieval) + └─ hindsight_reflect → Hindsight reflect + (synthesize a reasoned answer from all memories) +``` + +The tools are `FunctionTool` instances from `autogen_core.tools`, passed directly to `AssistantAgent(tools=[...])`. No subclassing, no custom agent types, just standard AutoGen tool use. + +Under the hood, Hindsight extracts structured facts, identifies entities, builds a knowledge graph, and runs four parallel retrieval strategies with cross-encoder reranking. + +## Step 1: Start Hindsight + +```bash +pip install hindsight-all +export HINDSIGHT_API_LLM_API_KEY=YOUR_OPENAI_KEY +hindsight-api +``` + +Runs locally at `http://localhost:8888` with embedded Postgres, embeddings, and reranking. + +Or use [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup) and skip self-hosting. + +## Step 2: Install the integration + +```bash +pip install hindsight-autogen autogen-agentchat "autogen-ext[openai]" +``` + +`hindsight-autogen` pulls in `autogen-core` and `hindsight-client`. You also need `autogen-agentchat` for `AssistantAgent` and `autogen-ext[openai]` for the model client. + +## Step 3: Create the bank and agent + +Banks must exist before use. AutoGen agents are async, so wrap everything in `asyncio.run()`: + +```python +import asyncio +from autogen_agentchat.agents import AssistantAgent +from autogen_ext.models.openai import OpenAIChatCompletionClient +from hindsight_client import Hindsight +from hindsight_autogen import create_hindsight_tools + +async def main(): + client = Hindsight(base_url="http://localhost:8888") + await client.acreate_bank("user-123", name="User 123 Memory") + + model_client = OpenAIChatCompletionClient(model="gpt-4o-mini") + tools = create_hindsight_tools( + client=client, + bank_id="user-123", + tags=["source:chat"], + budget="mid", + ) + + agent = AssistantAgent( + name="assistant", + model_client=model_client, + tools=tools, + reflect_on_tool_use=True, + system_message=( + "You are a helpful assistant with long-term memory. " + "Use hindsight_retain to store important facts the user shares. " + "Use hindsight_recall to search memory before answering questions." + ), + ) + + # Session 1: store preferences + result = await agent.run( + task="I'm a data scientist. I use Python, SQL, and VS Code with dark mode.", + ) + + # Wait for Hindsight to finish processing (fact extraction is async) + await asyncio.sleep(3) + + # Session 2: recall from memory (same bank, memory persists) + result = await agent.run( + task="What IDE do I use?", + ) + print(result.messages[-1].content) + # → "You use VS Code with dark mode." + + # Clean up + await client.aclose() + await model_client.close() + +asyncio.run(main()) +``` + +Three tools, one bank. Memory persists across conversations because it's stored in Hindsight, not in the agent. + +## Per-user memory banks + +Parameterize `bank_id` for per-user isolation: + +```python +def create_agent_for_user(user_id: str) -> AssistantAgent: + tools = create_hindsight_tools( + client=client, + bank_id=f"user-{user_id}", + ) + return AssistantAgent( + name="assistant", + model_client=OpenAIChatCompletionClient(model="gpt-4o-mini"), + tools=tools, + ) +``` + +Each bank is fully isolated; no cross-user data leakage. + +## When to use this + +- **Repeat-user agents** — Support bots, coding assistants, personal AI that should remember preferences and history across sessions +- **Multi-agent teams with shared memory** — A coordinator agent retains findings from group chats so future sessions start with context +- **Long-running workflows** — Agents that process data over days/weeks and need to accumulate knowledge incrementally +- **Personalization** — Any agent where "remembering the user" improves quality over time + +## When NOT to use this + +Be explicit: persistent memory isn't always the right tool. + +- **In-session context only** — If your agent only needs to remember things within a single conversation, AutoGen's built-in chat history is simpler and has zero latency overhead. Don't add Hindsight just because you can. +- **Document search (RAG)** — If you need vector search over a document corpus, use a dedicated vector store. Hindsight is a memory system for facts learned over time, not a document store. +- **Ephemeral agents** — If each agent invocation is stateless by design (batch processing, one-shot tasks), persistent memory adds complexity without benefit. +- **Latency-critical hot paths** — Each memory operation adds a network round-trip. If sub-100ms response time matters more than personalization, skip it. + +## Pitfalls and edge cases + +**Bank must exist first.** Call `await client.acreate_bank(bank_id, name=...)` before the agent starts. If the bank doesn't exist, retain/recall will fail. + +**Async processing delay.** After `hindsight_retain`, Hindsight processes content asynchronously, extracting facts, entities, embeddings. If you retain and immediately recall, the new memories may not be searchable yet. In practice, 1-3 seconds. + +**Budget tuning.** Default `budget="mid"` balances speed and thoroughness. Use `"low"` for latency-sensitive agents, `"high"` for deep analysis. Budget controls how many retrieval strategies run and how much reranking happens. + +**Reflect vs recall.** Use `hindsight_recall` for raw facts ("What IDE do I use?"). Use `hindsight_reflect` for synthesis ("Based on everything you know, what should I prioritize?"). Reflect is slower but produces reasoned answers that draw on the full knowledge graph. + +## How this compares + +**vs. AutoGen chat history:** Chat history stores raw messages in-session. It doesn't extract facts, doesn't generalize, and disappears when the conversation ends. Hindsight extracts structured facts, deduplicates, and retrieves only what's relevant — it compresses knowledge rather than accumulating tokens. + +**vs. raw vector stores (Pinecone, Weaviate, Chroma):** A vector store gives you embedding similarity search. Hindsight runs four parallel retrieval strategies (semantic, BM25, graph traversal, temporal) with cross-encoder reranking, plus it extracts entities, resolves coreferences, and builds a knowledge graph. It's a memory engine, not a database. For independent benchmark results on what that architecture achieves at scale, see [Hindsight on BEAM](https://hindsight.vectorize.io/blog/2026/04/02/beam-sota). + +**vs. other framework integrations:** If you're using LlamaIndex, LangGraph, CrewAI, or Pydantic AI instead of AutoGen, Hindsight has dedicated integrations for each: [LlamaIndex](/sdks/integrations/llamaindex), [LangGraph](/sdks/integrations/langgraph), [CrewAI](/sdks/integrations/crewai), [Pydantic AI](/sdks/integrations/pydantic-ai). + +## Recap + +- `hindsight-autogen` gives AutoGen agents persistent memory via `FunctionTool` instances passed to `AssistantAgent(tools=[...])` +- Three tools: `hindsight_retain` (store), `hindsight_recall` (search), `hindsight_reflect` (synthesize) +- Works with any AutoGen `AssistantAgent`, single agents or multi-agent teams +- Per-user banks for memory isolation, tags for scoping, budget for speed/depth tradeoff + +## Next steps + +- **Try it locally:** `pip install hindsight-all hindsight-autogen autogen-agentchat "autogen-ext[openai]"` and run the example above +- **Use Hindsight Cloud:** Skip self-hosting with a [free account](https://ui.hindsight.vectorize.io/signup) +- **Benchmark results:** [Why Hindsight leads on BEAM at 10M tokens](https://hindsight.vectorize.io/blog/2026/04/02/beam-sota) +- **Explore other integrations:** [LlamaIndex](/sdks/integrations/llamaindex), [LangGraph](/sdks/integrations/langgraph), [Pydantic AI](/sdks/integrations/pydantic-ai), [CrewAI](/sdks/integrations/crewai) diff --git a/hindsight-docs/static/img/blog/autogen-persistent-memory.png b/hindsight-docs/static/img/blog/autogen-persistent-memory.png new file mode 100644 index 00000000..0b4c49b4 Binary files /dev/null and b/hindsight-docs/static/img/blog/autogen-persistent-memory.png differ