fleet-memory/hindsight/tests

Hindsight Integration Tests

This directory contains integration tests for the Hindsight all-in-one package.

Test Overview

test_server_integration.py

Comprehensive integration tests that verify the complete workflow:

1. test_server_context_manager_basic_workflow: Main integration test that:

   • Starts the Hindsight server using a context manager
   • Creates a memory bank with background information
   • Stores multiple memories (both single and batch operations)
   • Recalls memories based on different queries (programming preferences, ML topics)
   • Reflects (generates contextual answers) multiple times with different contexts
   • Automatically stops the server on context exit

2. test_server_manual_start_stop: Tests explicit server lifecycle management without context managers

3. test_server_with_client_context_manager: Tests nested context managers for both server and client

Running the Tests

Prerequisites

1. Install the hindsight package with test dependencies:

   cd hindsight
   uv pip install -e ".[test]"
   

2. Set up your LLM credentials in the .env file at the project root:

   HINDSIGHT_API_LLM_PROVIDER=groq
   HINDSIGHT_API_LLM_API_KEY=your-api-key
   HINDSIGHT_API_LLM_MODEL=openai/gpt-oss-20b
   

Run All Tests

cd hindsight
source ../.env
export HINDSIGHT_LLM_PROVIDER=$HINDSIGHT_API_LLM_PROVIDER
export HINDSIGHT_LLM_API_KEY=$HINDSIGHT_API_LLM_API_KEY
export HINDSIGHT_LLM_MODEL=$HINDSIGHT_API_LLM_MODEL
pytest tests/ -v

Note on Parallel Execution: These tests use embedded PostgreSQL (pg0), which is a singleton that cannot be shared across pytest-xdist worker processes. Therefore, these tests must run sequentially. Do not use pytest -n (parallel workers) with these tests.

Why Random bank_id Values?: Each test generates a unique bank_id using UUID. This provides several benefits:

• Clean test isolation: Tests don't interfere with each other's data
• Repeatable runs: Tests can be run multiple times without cleanup
• Debugging: Easy to identify which test created which data
• Future-ready: If you switch from pg0 to a real PostgreSQL instance, these tests could run in parallel

Run Specific Test

pytest tests/test_server_integration.py::test_server_context_manager_basic_workflow -v

Run with Verbose Output

pytest tests/ -v -s

The -s flag shows print statements, which is useful to see the test progress.

Run with Timeout

These tests involve LLM calls which can take time:

pytest tests/ --timeout=300

Test Configuration

The tests use:

• Embedded PostgreSQL (pg0) for the database - no external database required
• LLM provider configured via environment variables
• Automatic port allocation for the server (no port conflicts)

Expected Behavior

When tests run successfully, you should see:

1. Server starting on a random available port
2. Memory bank creation with background information
3. Multiple memories being stored (retain operations)
4. Memories being recalled based on different queries
5. Multiple reflection responses with contextual answers
6. Server automatically stopping (for context manager tests)

Sample Output

The main test workflow demonstrates:

• Step 1: Create memory bank
• Step 2: Store 5 memories (3 individual + 2 batch)
• Step 3: Recall memories about programming preferences
• Step 4: Recall memories about machine learning
• Step 5: Reflect on tool recommendations
• Step 6: Reflect with additional context about framework choices
• Step 7: Server auto-stops via context manager

Skipping Tests

Tests will automatically skip if:

• HINDSIGHT_LLM_API_KEY is not set

Troubleshooting

Test hangs or times out

• Increase timeout: pytest tests/ --timeout=600
• Check your LLM API key is valid
• Verify network connectivity to LLM provider

Database errors

• The tests use embedded PostgreSQL (pg0) which should handle cleanup automatically
• If you see "database system is shutting down" errors, wait a few seconds and retry
• Between test runs, the embedded database needs time to properly shut down

Port conflicts

• Tests automatically find free ports, so conflicts should be rare
• If you see port binding errors, check for other processes using high-numbered ports