blog: add CrewAI persistent memory post (#471)
* Add CrewAI persistent memory blog post * Update blog: add image, remove full example and alternatives sections * Add CrewAI blog hero image
This commit is contained in:
parent
1d70abfe85
commit
8138fa9002
2 changed files with 252 additions and 0 deletions
252
hindsight-docs/blog/2026-03-02-crewai.md
Normal file
252
hindsight-docs/blog/2026-03-02-crewai.md
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
---
|
||||
title: "Your CrewAI Agents Forget Everything Between Runs. Here's the Fix."
|
||||
authors: [hindsight]
|
||||
date: 2026-03-02
|
||||
tags: [crewai, agents, python, memory, tutorial]
|
||||
image: /img/blog/crewai-memory.png
|
||||
---
|
||||
|
||||
CrewAI agents lose all memory when a crew finishes. `hindsight-crewai` plugs into CrewAI's ExternalMemory to persist knowledge across runs -- three lines of setup, and your agents automatically store task outputs and recall relevant context.
|
||||
|
||||
<!-- truncate -->
|
||||
|
||||
---
|
||||
|
||||
## The Problem: Stateless Crews
|
||||
|
||||
CrewAI has a memory system. Short-term, long-term, entity memory. It works well within a single `kickoff()`.
|
||||
|
||||
Then the process exits.
|
||||
|
||||
Next run, the crew starts from zero. Every fact learned, every decision made, every entity discovered -- gone.
|
||||
|
||||
This matters when you build crews that run repeatedly:
|
||||
|
||||
- A research crew that deepens knowledge over time
|
||||
- A support crew that remembers customer history
|
||||
- A planning crew that tracks decisions across sprints
|
||||
|
||||
CrewAI's built-in memory backends (RAG storage, SQLite) are designed for single-run persistence. For cross-run, cross-session memory that actually compounds, you need something else.
|
||||
|
||||
That's what the `hindsight-crewai` package does. It implements CrewAI's `Storage` interface using Hindsight's memory engine, so your crews remember everything -- across runs, across days, across weeks.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
Here's how it fits together:
|
||||
|
||||
```
|
||||
CrewAI Crew
|
||||
└─ ExternalMemory
|
||||
└─ HindsightStorage (implements Storage interface)
|
||||
├─ save() → Hindsight retain (extract facts, entities, relationships)
|
||||
├─ search() → Hindsight recall (semantic + graph + temporal retrieval)
|
||||
└─ reset() → Hindsight delete_bank + recreate
|
||||
```
|
||||
|
||||
CrewAI calls `save()` after each task completes and `search()` before each task starts. You don't manage the lifecycle -- CrewAI drives it, Hindsight stores it.
|
||||
|
||||
Under the hood, Hindsight does more than store text. It extracts structured facts, identifies entities, builds a knowledge graph, and runs multi-strategy retrieval (semantic search, BM25, graph traversal, temporal ranking) with cross-encoder reranking.
|
||||
|
||||
Your crew gets a real memory system, not a vector dump.
|
||||
|
||||
---
|
||||
|
||||
## Step 1 -- Start Hindsight
|
||||
|
||||
Install and start the memory server:
|
||||
|
||||
```bash
|
||||
pip install hindsight-all
|
||||
```
|
||||
|
||||
```bash
|
||||
export HINDSIGHT_API_LLM_API_KEY=YOUR_OPENAI_KEY
|
||||
hindsight-api
|
||||
```
|
||||
|
||||
This runs locally at `http://localhost:8888` with embedded Postgres, embeddings, and reranking. No external infra needed.
|
||||
|
||||
> **Note:** You can also use [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup) and skip the self-hosted setup entirely.
|
||||
|
||||
---
|
||||
|
||||
## Step 2 -- Install the Integration
|
||||
|
||||
```bash
|
||||
pip install hindsight-crewai
|
||||
```
|
||||
|
||||
This pulls in `hindsight-client` and `crewai` as dependencies.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 -- Wire It Up
|
||||
|
||||
```python
|
||||
from hindsight_crewai import configure, HindsightStorage
|
||||
from crewai.memory.external.external_memory import ExternalMemory
|
||||
from crewai import Agent, Crew, Task
|
||||
|
||||
# Point at your Hindsight instance
|
||||
configure(hindsight_api_url="http://localhost:8888")
|
||||
|
||||
# Create agents
|
||||
researcher = Agent(
|
||||
role="Researcher",
|
||||
goal="Find accurate, detailed information on the given topic.",
|
||||
backstory="You are a thorough researcher who digs deep into topics.",
|
||||
llm="openai/gpt-4o-mini",
|
||||
)
|
||||
|
||||
writer = Agent(
|
||||
role="Writer",
|
||||
goal="Write clear, well-structured content based on research.",
|
||||
backstory="You are a technical writer who values clarity and precision.",
|
||||
llm="openai/gpt-4o-mini",
|
||||
)
|
||||
|
||||
# Create a task
|
||||
research_task = Task(
|
||||
description="Research the benefits of Rust for CLI tools.",
|
||||
expected_output="A detailed summary of Rust's strengths for CLI development.",
|
||||
agent=researcher,
|
||||
)
|
||||
|
||||
write_task = Task(
|
||||
description="Write a short article based on the research.",
|
||||
expected_output="A polished 3-paragraph article.",
|
||||
agent=writer,
|
||||
)
|
||||
|
||||
# Create the crew with persistent memory
|
||||
crew = Crew(
|
||||
agents=[researcher, writer],
|
||||
tasks=[research_task, write_task],
|
||||
external_memory=ExternalMemory(
|
||||
storage=HindsightStorage(
|
||||
bank_id="research-crew",
|
||||
mission="Track research findings, technical comparisons, and writing preferences.",
|
||||
)
|
||||
),
|
||||
)
|
||||
|
||||
crew.kickoff()
|
||||
```
|
||||
|
||||
That's it. After `kickoff()`, every task output is retained in Hindsight. Next time you run this crew, it recalls relevant prior work before starting each task.
|
||||
|
||||
---
|
||||
|
||||
## Step 4 -- Run It Again
|
||||
|
||||
Second run, different topic:
|
||||
|
||||
```python
|
||||
research_task = Task(
|
||||
description="Research how Go compares to Rust for CLI tools.",
|
||||
expected_output="A comparison of Go vs Rust for CLI development.",
|
||||
agent=researcher,
|
||||
)
|
||||
```
|
||||
|
||||
Now the researcher has context from the first run. It knows what it already found about Rust. The writer remembers the style and structure from the previous article.
|
||||
|
||||
Third run:
|
||||
|
||||
```python
|
||||
research_task = Task(
|
||||
description="Which language should I pick for a new CLI tool?",
|
||||
expected_output="A recommendation based on all prior research.",
|
||||
agent=researcher,
|
||||
)
|
||||
```
|
||||
|
||||
The crew now draws on two prior research sessions. Knowledge compounds.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 -- Add Reflect for Deeper Synthesis
|
||||
|
||||
CrewAI's Storage interface has `save` and `search`. But Hindsight also supports `reflect` -- a synthesis operation that reasons across all relevant memories instead of returning raw facts.
|
||||
|
||||
Since `reflect` doesn't map to the Storage interface, it's exposed as a CrewAI Tool:
|
||||
|
||||
```python
|
||||
from hindsight_crewai import HindsightReflectTool
|
||||
|
||||
reflect_tool = HindsightReflectTool(
|
||||
bank_id="research-crew",
|
||||
budget="mid",
|
||||
reflect_context="You are helping a development team evaluate programming languages.",
|
||||
)
|
||||
|
||||
researcher = Agent(
|
||||
role="Researcher",
|
||||
goal="Provide deep, synthesized analysis on technical topics.",
|
||||
backstory="You are a senior researcher. Use the hindsight_reflect tool to review what you already know before starting new research.",
|
||||
tools=[reflect_tool],
|
||||
llm="openai/gpt-4o-mini",
|
||||
)
|
||||
```
|
||||
|
||||
When the agent calls `hindsight_reflect`, it gets a synthesized, reasoned response that draws on the full knowledge graph -- not just the top-k vector matches.
|
||||
|
||||
---
|
||||
|
||||
## Per-Agent Memory Banks
|
||||
|
||||
By default, all agents share one bank. If you want each agent to have isolated memory:
|
||||
|
||||
```python
|
||||
storage = HindsightStorage(
|
||||
bank_id="research-crew",
|
||||
per_agent_banks=True,
|
||||
)
|
||||
```
|
||||
|
||||
The researcher writes to `research-crew-researcher`, the writer to `research-crew-writer`. Each agent builds its own knowledge base.
|
||||
|
||||
For full control, use a custom resolver:
|
||||
|
||||
```python
|
||||
storage = HindsightStorage(
|
||||
bank_id="research-crew",
|
||||
bank_resolver=lambda base, agent: f"{base}-{agent.lower()}" if agent else base,
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pitfalls and Edge Cases
|
||||
|
||||
**1. Bank ID collisions.** If multiple unrelated crews share a `bank_id`, their memories mix. Use unique bank IDs per crew or project.
|
||||
|
||||
**2. Large task outputs.** CrewAI passes the full task output to `save()`. If your tasks produce very long outputs, Hindsight handles the chunking, but retain latency increases. Set a reasonable `expected_output` length in your task definitions.
|
||||
|
||||
**3. Recall budget tuning.** The default `budget="mid"` balances speed and thoroughness. For latency-sensitive crews, use `"low"`. For deep analysis, use `"high"`. Budget affects how many retrieval strategies run and how much reranking happens.
|
||||
|
||||
**4. Async event loop conflicts.** CrewAI runs inside an async event loop. The integration handles this transparently via a dedicated thread pool, but if you're also doing async work in custom tools, avoid calling `hindsight-client` directly from the same event loop. Use the `HindsightStorage` and `HindsightReflectTool` abstractions instead.
|
||||
|
||||
---
|
||||
|
||||
## Recap
|
||||
|
||||
- `hindsight-crewai` gives CrewAI agents persistent, compounding memory
|
||||
- It implements CrewAI's `Storage` interface, so integration is three lines
|
||||
- Memories are automatically stored after tasks and recalled before tasks
|
||||
- `HindsightReflectTool` adds on-demand synthesis for deeper reasoning
|
||||
- Per-agent banks let you isolate or share knowledge as needed
|
||||
|
||||
The integration handles the hard parts: async compatibility, thread safety, fact extraction, multi-strategy retrieval. You just point it at a bank and let your crews learn.
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **Try it locally**: `pip install hindsight-all hindsight-crewai` and run the example above
|
||||
- **Use Hindsight Cloud**: Skip self-hosting with a [free account](https://ui.hindsight.vectorize.io/signup)
|
||||
- **Add tags for scoped memory**: Use `tags` on retain and `recall_tags` on search to partition memories by project, environment, or topic
|
||||
- **Inspect memories in the web UI**: Run `hindsight-control-plane` locally or use the cloud dashboard to browse facts, entities, and mental models
|
||||
- **Combine with per-agent banks**: Give specialized agents their own memory while sharing a common bank for cross-agent knowledge
|
||||
BIN
hindsight-docs/static/img/blog/crewai-memory.png
Normal file
BIN
hindsight-docs/static/img/blog/crewai-memory.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
Loading…
Reference in a new issue