fleet-memory/benchmarks/visualizer

Benchmark Visualizer

A standalone web service for visualizing benchmark results. Currently supports the LoComo benchmark with plans to add more benchmarks in the future.

Features

• Interactive web interface for viewing benchmark results
• Detailed breakdown by category (Multi-hop, Single-hop, Temporal, Open-domain)
• Filter options to view all, correct, or incorrect answers
• Expandable Q&A details with reasoning and retrieved memories
• Overall and per-item accuracy statistics

Running the Visualizer

Option 1: Using the serve script (recommended)

cd benchmarks/visualizer
./serve.sh

Option 2: Using uvicorn directly

cd benchmarks/visualizer
uv run uvicorn server:app --reload --host 0.0.0.0 --port 8001

Then open your browser to: http://localhost:8001

Usage

1. Select a benchmark from the dropdown:
   • LoComo (search): Traditional two-step approach (search → LLM answer generation)
   • LoComo (think): Integrated approach using think API (single call for retrieval + reasoning)
2. The visualization will automatically load and display:
   • Overall accuracy statistics
   • Category-wise performance breakdown
   • Detailed results for each conversation
3. Use the filter controls to show all answers, only incorrect, or only correct answers
4. Expand individual conversations to see Q&A details, reasoning, and retrieved memories

API Endpoints

• GET / - Main visualizer page
• GET /api/locomo?mode={search|think} - Returns LoComo benchmark results as JSON
  • mode=search (default): Returns results from benchmark_results.json
  • mode=think: Returns results from benchmark_results_think.json

Requirements

• FastAPI
• Uvicorn
• Python 3.11+

The visualizer reads benchmark results from:

• benchmarks/locomo/benchmark_results.json for search mode
• benchmarks/locomo/benchmark_results_think.json for think mode

Make sure to run the benchmark first to generate results:

• Search mode: cd benchmarks/locomo && uv run python run_benchmark.py
• Think mode: cd benchmarks/locomo && uv run python run_benchmark.py --use-think