diff --git a/hindsight-docs/blog/2026-03-02-crewai.md b/hindsight-docs/blog/2026-03-02-crewai.md new file mode 100644 index 00000000..cb43ff73 --- /dev/null +++ b/hindsight-docs/blog/2026-03-02-crewai.md @@ -0,0 +1,252 @@ +--- +title: "Your CrewAI Agents Forget Everything Between Runs. Here's the Fix." +authors: [hindsight] +date: 2026-03-02 +tags: [crewai, agents, python, memory, tutorial] +image: /img/blog/crewai-memory.png +--- + +CrewAI agents lose all memory when a crew finishes. `hindsight-crewai` plugs into CrewAI's ExternalMemory to persist knowledge across runs -- three lines of setup, and your agents automatically store task outputs and recall relevant context. + + + +--- + +## The Problem: Stateless Crews + +CrewAI has a memory system. Short-term, long-term, entity memory. It works well within a single `kickoff()`. + +Then the process exits. + +Next run, the crew starts from zero. Every fact learned, every decision made, every entity discovered -- gone. + +This matters when you build crews that run repeatedly: + +- A research crew that deepens knowledge over time +- A support crew that remembers customer history +- A planning crew that tracks decisions across sprints + +CrewAI's built-in memory backends (RAG storage, SQLite) are designed for single-run persistence. For cross-run, cross-session memory that actually compounds, you need something else. + +That's what the `hindsight-crewai` package does. It implements CrewAI's `Storage` interface using Hindsight's memory engine, so your crews remember everything -- across runs, across days, across weeks. + +--- + +## Architecture + +Here's how it fits together: + +``` +CrewAI Crew + └─ ExternalMemory + └─ HindsightStorage (implements Storage interface) + ├─ save() → Hindsight retain (extract facts, entities, relationships) + ├─ search() → Hindsight recall (semantic + graph + temporal retrieval) + └─ reset() → Hindsight delete_bank + recreate +``` + +CrewAI calls `save()` after each task completes and `search()` before each task starts. You don't manage the lifecycle -- CrewAI drives it, Hindsight stores it. + +Under the hood, Hindsight does more than store text. It extracts structured facts, identifies entities, builds a knowledge graph, and runs multi-strategy retrieval (semantic search, BM25, graph traversal, temporal ranking) with cross-encoder reranking. + +Your crew gets a real memory system, not a vector dump. + +--- + +## Step 1 -- Start Hindsight + +Install and start the memory server: + +```bash +pip install hindsight-all +``` + +```bash +export HINDSIGHT_API_LLM_API_KEY=YOUR_OPENAI_KEY +hindsight-api +``` + +This runs locally at `http://localhost:8888` with embedded Postgres, embeddings, and reranking. No external infra needed. + +> **Note:** You can also use [Hindsight Cloud](https://ui.hindsight.vectorize.io/signup) and skip the self-hosted setup entirely. + +--- + +## Step 2 -- Install the Integration + +```bash +pip install hindsight-crewai +``` + +This pulls in `hindsight-client` and `crewai` as dependencies. + +--- + +## Step 3 -- Wire It Up + +```python +from hindsight_crewai import configure, HindsightStorage +from crewai.memory.external.external_memory import ExternalMemory +from crewai import Agent, Crew, Task + +# Point at your Hindsight instance +configure(hindsight_api_url="http://localhost:8888") + +# Create agents +researcher = Agent( + role="Researcher", + goal="Find accurate, detailed information on the given topic.", + backstory="You are a thorough researcher who digs deep into topics.", + llm="openai/gpt-4o-mini", +) + +writer = Agent( + role="Writer", + goal="Write clear, well-structured content based on research.", + backstory="You are a technical writer who values clarity and precision.", + llm="openai/gpt-4o-mini", +) + +# Create a task +research_task = Task( + description="Research the benefits of Rust for CLI tools.", + expected_output="A detailed summary of Rust's strengths for CLI development.", + agent=researcher, +) + +write_task = Task( + description="Write a short article based on the research.", + expected_output="A polished 3-paragraph article.", + agent=writer, +) + +# Create the crew with persistent memory +crew = Crew( + agents=[researcher, writer], + tasks=[research_task, write_task], + external_memory=ExternalMemory( + storage=HindsightStorage( + bank_id="research-crew", + mission="Track research findings, technical comparisons, and writing preferences.", + ) + ), +) + +crew.kickoff() +``` + +That's it. After `kickoff()`, every task output is retained in Hindsight. Next time you run this crew, it recalls relevant prior work before starting each task. + +--- + +## Step 4 -- Run It Again + +Second run, different topic: + +```python +research_task = Task( + description="Research how Go compares to Rust for CLI tools.", + expected_output="A comparison of Go vs Rust for CLI development.", + agent=researcher, +) +``` + +Now the researcher has context from the first run. It knows what it already found about Rust. The writer remembers the style and structure from the previous article. + +Third run: + +```python +research_task = Task( + description="Which language should I pick for a new CLI tool?", + expected_output="A recommendation based on all prior research.", + agent=researcher, +) +``` + +The crew now draws on two prior research sessions. Knowledge compounds. + +--- + +## Step 5 -- Add Reflect for Deeper Synthesis + +CrewAI's Storage interface has `save` and `search`. But Hindsight also supports `reflect` -- a synthesis operation that reasons across all relevant memories instead of returning raw facts. + +Since `reflect` doesn't map to the Storage interface, it's exposed as a CrewAI Tool: + +```python +from hindsight_crewai import HindsightReflectTool + +reflect_tool = HindsightReflectTool( + bank_id="research-crew", + budget="mid", + reflect_context="You are helping a development team evaluate programming languages.", +) + +researcher = Agent( + role="Researcher", + goal="Provide deep, synthesized analysis on technical topics.", + backstory="You are a senior researcher. Use the hindsight_reflect tool to review what you already know before starting new research.", + tools=[reflect_tool], + llm="openai/gpt-4o-mini", +) +``` + +When the agent calls `hindsight_reflect`, it gets a synthesized, reasoned response that draws on the full knowledge graph -- not just the top-k vector matches. + +--- + +## Per-Agent Memory Banks + +By default, all agents share one bank. If you want each agent to have isolated memory: + +```python +storage = HindsightStorage( + bank_id="research-crew", + per_agent_banks=True, +) +``` + +The researcher writes to `research-crew-researcher`, the writer to `research-crew-writer`. Each agent builds its own knowledge base. + +For full control, use a custom resolver: + +```python +storage = HindsightStorage( + bank_id="research-crew", + bank_resolver=lambda base, agent: f"{base}-{agent.lower()}" if agent else base, +) +``` + +--- + +## Pitfalls and Edge Cases + +**1. Bank ID collisions.** If multiple unrelated crews share a `bank_id`, their memories mix. Use unique bank IDs per crew or project. + +**2. Large task outputs.** CrewAI passes the full task output to `save()`. If your tasks produce very long outputs, Hindsight handles the chunking, but retain latency increases. Set a reasonable `expected_output` length in your task definitions. + +**3. Recall budget tuning.** The default `budget="mid"` balances speed and thoroughness. For latency-sensitive crews, use `"low"`. For deep analysis, use `"high"`. Budget affects how many retrieval strategies run and how much reranking happens. + +**4. Async event loop conflicts.** CrewAI runs inside an async event loop. The integration handles this transparently via a dedicated thread pool, but if you're also doing async work in custom tools, avoid calling `hindsight-client` directly from the same event loop. Use the `HindsightStorage` and `HindsightReflectTool` abstractions instead. + +--- + +## Recap + +- `hindsight-crewai` gives CrewAI agents persistent, compounding memory +- It implements CrewAI's `Storage` interface, so integration is three lines +- Memories are automatically stored after tasks and recalled before tasks +- `HindsightReflectTool` adds on-demand synthesis for deeper reasoning +- Per-agent banks let you isolate or share knowledge as needed + +The integration handles the hard parts: async compatibility, thread safety, fact extraction, multi-strategy retrieval. You just point it at a bank and let your crews learn. + +--- + +## Next Steps + +- **Try it locally**: `pip install hindsight-all hindsight-crewai` and run the example above +- **Use Hindsight Cloud**: Skip self-hosting with a [free account](https://ui.hindsight.vectorize.io/signup) +- **Add tags for scoped memory**: Use `tags` on retain and `recall_tags` on search to partition memories by project, environment, or topic +- **Inspect memories in the web UI**: Run `hindsight-control-plane` locally or use the cloud dashboard to browse facts, entities, and mental models +- **Combine with per-agent banks**: Give specialized agents their own memory while sharing a common bank for cross-agent knowledge diff --git a/hindsight-docs/static/img/blog/crewai-memory.png b/hindsight-docs/static/img/blog/crewai-memory.png new file mode 100644 index 00000000..99c10c6c Binary files /dev/null and b/hindsight-docs/static/img/blog/crewai-memory.png differ