62 lines
2.1 KiB
Markdown
62 lines
2.1 KiB
Markdown
# Benchmark Visualizer
|
|
|
|
A standalone web service for visualizing benchmark results. Currently supports the LoComo benchmark with plans to add more benchmarks in the future.
|
|
|
|
## Features
|
|
|
|
- Interactive web interface for viewing benchmark results
|
|
- Detailed breakdown by category (Multi-hop, Single-hop, Temporal, Open-domain)
|
|
- Filter options to view all, correct, or incorrect answers
|
|
- Expandable Q&A details with reasoning and retrieved memories
|
|
- Overall and per-item accuracy statistics
|
|
|
|
## Running the Visualizer
|
|
|
|
### Option 1: Using the serve script (recommended)
|
|
|
|
```bash
|
|
cd benchmarks/visualizer
|
|
./serve.sh
|
|
```
|
|
|
|
### Option 2: Using uvicorn directly
|
|
|
|
```bash
|
|
cd benchmarks/visualizer
|
|
uv run uvicorn server:app --reload --host 0.0.0.0 --port 8001
|
|
```
|
|
|
|
Then open your browser to: http://localhost:8001
|
|
|
|
## Usage
|
|
|
|
1. Select a benchmark from the dropdown:
|
|
- **LoComo (search)**: Traditional two-step approach (search → LLM answer generation)
|
|
- **LoComo (think)**: Integrated approach using think API (single call for retrieval + reasoning)
|
|
2. The visualization will automatically load and display:
|
|
- Overall accuracy statistics
|
|
- Category-wise performance breakdown
|
|
- Detailed results for each conversation
|
|
3. Use the filter controls to show all answers, only incorrect, or only correct answers
|
|
4. Expand individual conversations to see Q&A details, reasoning, and retrieved memories
|
|
|
|
## API Endpoints
|
|
|
|
- `GET /` - Main visualizer page
|
|
- `GET /api/locomo?mode={search|think}` - Returns LoComo benchmark results as JSON
|
|
- `mode=search` (default): Returns results from `benchmark_results.json`
|
|
- `mode=think`: Returns results from `benchmark_results_think.json`
|
|
|
|
## Requirements
|
|
|
|
- FastAPI
|
|
- Uvicorn
|
|
- Python 3.11+
|
|
|
|
The visualizer reads benchmark results from:
|
|
- `benchmarks/locomo/benchmark_results.json` for search mode
|
|
- `benchmarks/locomo/benchmark_results_think.json` for think mode
|
|
|
|
Make sure to run the benchmark first to generate results:
|
|
- Search mode: `cd benchmarks/locomo && uv run python run_benchmark.py`
|
|
- Think mode: `cd benchmarks/locomo && uv run python run_benchmark.py --use-think`
|