fleet-memory/hindsight-docs/docs/developer/metrics.md

Metrics

Hindsight exposes comprehensive metrics for monitoring performance, usage, and system health in production deployments.

Prometheus Metrics

Hindsight exposes metrics in Prometheus format at the /metrics endpoint.

Accessing Metrics

# View all metrics
curl http://localhost:8888/metrics

# Scrape with Prometheus
# Add to prometheus.yml:
scrape_configs:
  - job_name: 'hindsight'
    static_configs:
      - targets: ['localhost:8888']

Core Metrics

Request Metrics

Track API request performance and throughput.

hindsight_http_requests_total

Total number of HTTP requests by endpoint and status code.

Type: Counter Labels:

• method: HTTP method (GET, POST, etc.)
• endpoint: API endpoint path
• status_code: HTTP status code (200, 400, 500, etc.)

# Requests per second by endpoint
rate(hindsight_http_requests_total[5m])

# Error rate (4xx and 5xx)
rate(hindsight_http_requests_total{status_code=~"4..|5.."}[5m])

hindsight_http_request_duration_seconds

HTTP request latency distribution.

Type: Histogram Labels:

• method: HTTP method
• endpoint: API endpoint path

# P95 latency by endpoint
histogram_quantile(0.95,
  rate(hindsight_http_request_duration_seconds_bucket[5m])
)

# Average request duration
rate(hindsight_http_request_duration_seconds_sum[5m]) /
rate(hindsight_http_request_duration_seconds_count[5m])

Memory Operations

Track retain, recall, and reflect operations.

hindsight_retain_duration_seconds

Time spent in retain (ingestion) operations.

Type: Histogram Labels:

• bank_id: Memory bank identifier
• async: Whether operation was async (true/false)

# P99 retain latency
histogram_quantile(0.99,
  rate(hindsight_retain_duration_seconds_bucket[5m])
)

# Sync vs async retain performance
histogram_quantile(0.50,
  rate(hindsight_retain_duration_seconds_bucket{async="false"}[5m])
)
vs
histogram_quantile(0.50,
  rate(hindsight_retain_duration_seconds_bucket{async="true"}[5m])
)

hindsight_retain_items_total

Total number of memory items retained.

Type: Counter Labels:

• bank_id: Memory bank identifier

# Items retained per second
rate(hindsight_retain_items_total[5m])

# Total items retained per bank
sum by(bank_id) (hindsight_retain_items_total)

hindsight_recall_duration_seconds

Time spent in recall (search) operations.

Type: Histogram Labels:

• bank_id: Memory bank identifier
• budget: Thinking budget level (low, mid, high)

# P95 recall latency by budget
histogram_quantile(0.95,
  rate(hindsight_recall_duration_seconds_bucket[5m])
) by (budget)

# Recall operations per second
rate(hindsight_recall_duration_seconds_count[5m])

hindsight_recall_results_count

Number of results returned by recall operations.

Type: Histogram Labels:

• bank_id: Memory bank identifier

# Average number of results per recall
rate(hindsight_recall_results_count_sum[5m]) /
rate(hindsight_recall_results_count_count[5m])

hindsight_reflect_duration_seconds

Time spent in reflect (reasoning) operations.

Type: Histogram Labels:

• bank_id: Memory bank identifier
• budget: Thinking budget level

# P50, P95, P99 reflect latency
histogram_quantile(0.50, rate(hindsight_reflect_duration_seconds_bucket[5m]))
histogram_quantile(0.95, rate(hindsight_reflect_duration_seconds_bucket[5m]))
histogram_quantile(0.99, rate(hindsight_reflect_duration_seconds_bucket[5m]))

LLM Metrics

Track LLM provider usage and performance.

hindsight_llm_requests_total

Total number of LLM API requests.

Type: Counter Labels:

• provider: LLM provider (openai, groq, ollama, etc.)
• model: Model name
• operation: Operation type (fact_extraction, entity_resolution, reasoning, etc.)
• status: Request status (success, error, timeout)

# LLM requests per second by provider
rate(hindsight_llm_requests_total[5m]) by (provider)

# LLM error rate
rate(hindsight_llm_requests_total{status="error"}[5m]) /
rate(hindsight_llm_requests_total[5m])

hindsight_llm_request_duration_seconds

LLM API request latency.

Type: Histogram Labels:

• provider: LLM provider
• model: Model name
• operation: Operation type

# P95 LLM latency by provider
histogram_quantile(0.95,
  rate(hindsight_llm_request_duration_seconds_bucket[5m])
) by (provider)

hindsight_llm_tokens_total

Total tokens consumed (prompt + completion).

Type: Counter Labels:

• provider: LLM provider
• model: Model name
• token_type: Token type (prompt, completion)

# Tokens per second
rate(hindsight_llm_tokens_total[5m])

# Cost estimation (OpenAI GPT-4)
rate(hindsight_llm_tokens_total{provider="openai",model="gpt-4",token_type="prompt"}[5m]) * 0.00003 +
rate(hindsight_llm_tokens_total{provider="openai",model="gpt-4",token_type="completion"}[5m]) * 0.00006

Database Metrics

Track database connection pool and query performance.

hindsight_db_connections_active

Number of active database connections.

Type: Gauge

# Active connections
hindsight_db_connections_active

# Connection pool utilization %
(hindsight_db_connections_active / 20) * 100

hindsight_db_connections_idle

Number of idle database connections in the pool.

Type: Gauge

# Idle connections
hindsight_db_connections_idle

# Pool efficiency (lower is better)
hindsight_db_connections_idle /
(hindsight_db_connections_active + hindsight_db_connections_idle)

hindsight_db_query_duration_seconds

Database query latency distribution.

Type: Histogram Labels:

• query_type: Type of query (select, insert, update, vector_search)

# P95 vector search latency
histogram_quantile(0.95,
  rate(hindsight_db_query_duration_seconds_bucket{query_type="vector_search"}[5m])
)

# Slow queries (> 1s)
histogram_quantile(0.99,
  rate(hindsight_db_query_duration_seconds_bucket[5m])
)

Memory Bank Metrics

Track memory bank usage and statistics.

hindsight_bank_memory_units_total

Total number of memory units per bank.

Type: Gauge Labels:

• bank_id: Memory bank identifier
• fact_type: Fact type (world, agent, opinion)

# Total memories per bank
sum by(bank_id) (hindsight_bank_memory_units_total)

# Memory distribution by type
sum by(fact_type) (hindsight_bank_memory_units_total)

hindsight_bank_entities_total

Total number of entities per bank.

Type: Gauge Labels:

• bank_id: Memory bank identifier

# Entities per bank
hindsight_bank_entities_total

# Total entities across all banks
sum(hindsight_bank_entities_total)

System Metrics

Track system resource usage.

hindsight_embedding_model_memory_bytes

Memory used by embedding models.

Type: Gauge

# Model memory in GB
hindsight_embedding_model_memory_bytes / 1024 / 1024 / 1024

hindsight_process_cpu_seconds_total

Total CPU time used by the process.

Type: Counter

# CPU utilization %
rate(hindsight_process_cpu_seconds_total[5m]) * 100

hindsight_process_memory_bytes

Process memory usage.

Type: Gauge

# Memory usage in GB
hindsight_process_memory_bytes / 1024 / 1024 / 1024

Sample Queries

Performance Monitoring

# Request latency by endpoint (P95)
histogram_quantile(0.95,
  sum by(endpoint, le) (
    rate(hindsight_http_request_duration_seconds_bucket[5m])
  )
)

# Requests per second
sum(rate(hindsight_http_requests_total[5m]))

# Error rate %
sum(rate(hindsight_http_requests_total{status_code=~"5.."}[5m])) /
sum(rate(hindsight_http_requests_total[5m])) * 100

Capacity Planning

# Database connection pool saturation
hindsight_db_connections_active / 20 * 100

# LLM request rate trend
rate(hindsight_llm_requests_total[1h])

# Average items retained per operation
rate(hindsight_retain_items_total[5m]) /
rate(hindsight_retain_duration_seconds_count[5m])

Cost Analysis

# Estimated LLM cost per hour (OpenAI GPT-4)
(
  rate(hindsight_llm_tokens_total{provider="openai",model="gpt-4",token_type="prompt"}[1h]) * 0.00003 +
  rate(hindsight_llm_tokens_total{provider="openai",model="gpt-4",token_type="completion"}[1h]) * 0.00006
) * 3600

# Tokens per operation type
sum by(operation) (
  rate(hindsight_llm_tokens_total[5m])
)

Troubleshooting

# Slow recalls (> 1 second)
count(
  hindsight_recall_duration_seconds_bucket{le="1.0"} == 0
)

# LLM timeout rate
rate(hindsight_llm_requests_total{status="timeout"}[5m])

# Database connection exhaustion events
changes(hindsight_db_connections_active[5m])

Grafana Dashboard

Example Grafana dashboard configuration:

{
  "dashboard": {
    "title": "Hindsight Monitoring",
    "panels": [
      {
        "title": "Request Rate",
        "targets": [{
          "expr": "sum(rate(hindsight_http_requests_total[5m]))"
        }]
      },
      {
        "title": "P95 Latency by Endpoint",
        "targets": [{
          "expr": "histogram_quantile(0.95, rate(hindsight_http_request_duration_seconds_bucket[5m])) by (endpoint)"
        }]
      },
      {
        "title": "Error Rate",
        "targets": [{
          "expr": "sum(rate(hindsight_http_requests_total{status_code=~\"5..\"}[5m])) / sum(rate(hindsight_http_requests_total[5m])) * 100"
        }]
      },
      {
        "title": "LLM Requests by Provider",
        "targets": [{
          "expr": "sum by(provider) (rate(hindsight_llm_requests_total[5m]))"
        }]
      },
      {
        "title": "Database Connections",
        "targets": [
          {
            "expr": "hindsight_db_connections_active",
            "legendFormat": "Active"
          },
          {
            "expr": "hindsight_db_connections_idle",
            "legendFormat": "Idle"
          }
        ]
      }
    ]
  }
}

Alerting Rules

Example Prometheus alerting rules:

groups:
  - name: hindsight
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          sum(rate(hindsight_http_requests_total{status_code=~"5.."}[5m])) /
          sum(rate(hindsight_http_requests_total[5m])) > 0.05          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Slow recalls
      - alert: SlowRecalls
        expr: |
          histogram_quantile(0.95,
            rate(hindsight_recall_duration_seconds_bucket[5m])
          ) > 2.0          
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Recall operations are slow"
          description: "P95 recall latency is {{ $value }}s"

      # Database connection pool exhaustion
      - alert: DatabasePoolExhaustion
        expr: hindsight_db_connections_active / 20 > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Database connection pool nearly exhausted"
          description: "Pool utilization is {{ $value | humanizePercentage }}"

      # LLM API failures
      - alert: LLMAPIFailures
        expr: |
          rate(hindsight_llm_requests_total{status="error"}[5m]) > 0.1          
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High LLM API error rate"
          description: "LLM error rate: {{ $value }} errors/s"

      # High memory usage
      - alert: HighMemoryUsage
        expr: hindsight_process_memory_bytes / 1024 / 1024 / 1024 > 8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Process using {{ $value }}GB of memory"

Custom Metrics

Adding Custom Metrics

Extend Hindsight with custom metrics in your application code:

from prometheus_client import Counter, Histogram

# Define custom metric
custom_operations = Counter(
    'hindsight_custom_operations_total',
    'Total custom operations',
    ['operation_type']
)

# Use in code
custom_operations.labels(operation_type='batch_import').inc()

Trace Information

Enable detailed trace information in API responses:

result = client.recall_memories(
    bank_id="my-bank",
    query="test query",
    trace=True  # Enable trace
)

# Access trace data
if result.trace:
    print(f"Total time: {result.trace['total_time']}ms")
    print(f"Activations: {result.trace['activation_count']}")
    print(f"Vector search: {result.trace['vector_search_time']}ms")
    print(f"Reranking: {result.trace['rerank_time']}ms")

Logging Integration

Hindsight logs are structured and can be easily integrated with log aggregation systems:

JSON Structured Logs

export HINDSIGHT_API_LOG_FORMAT=json
export HINDSIGHT_API_LOG_LEVEL=info

{
  "timestamp": "2025-01-15T10:30:45.123Z",
  "level": "INFO",
  "logger": "hindsight.api",
  "message": "Recall completed",
  "bank_id": "my-bank",
  "query": "test query",
  "results_count": 15,
  "duration_ms": 423
}

Log Levels

• DEBUG: Detailed diagnostic information
• INFO: General informational messages
• WARNING: Warning messages for potentially harmful situations
• ERROR: Error messages for failures

Best Practices

1. Set up alerting for critical metrics (error rate, latency, connection pool)
2. Monitor costs by tracking LLM token usage
3. Track trends over time to identify capacity needs
4. Use dashboards to visualize key metrics
5. Set appropriate retention for metrics data (30-90 days)
6. Correlate metrics with logs for troubleshooting
7. Establish baselines for normal operation
8. Review metrics regularly to identify optimization opportunities

---

For metrics-related questions or issues, please open an issue on GitHub.