fleet-memory/hindsight-docs/docs/developer/temporal-reasoning.md

sidebar_position
4

Temporal Reasoning

Hindsight provides sophisticated temporal reasoning capabilities, allowing memory banks to understand and query memories based on when events occurred. This enables natural queries like "What did Alice do last spring?" or "What happened between March and May?"

Overview

Temporal reasoning in Hindsight operates at two levels:

1. Fact Storage: Every memory can have an event_date timestamp indicating when the event occurred
2. Query Analysis: Natural language temporal expressions are automatically detected and parsed
3. Temporal-Graph Retrieval: A specialized retrieval strategy that filters memories by time range while maintaining entity relationships

Storing Facts with Temporal Information

Event Date Assignment

When ingesting facts, Hindsight extracts temporal information in the following order:

1. Explicit event_date parameter: If provided, this is used directly
2. LLM-extracted date: The fact extraction LLM identifies temporal markers in the content
3. Default to storage time: If no date is found, uses the current timestamp

from hindsight_client import Hindsight
from datetime import datetime

client = Hindsight(base_url="http://localhost:8888")

# Explicit event date
client.put(
    agent_id="my-agent",
    content="Alice started working at Google",
    event_date=datetime(2023, 3, 15)
)

# LLM will extract the date from content
client.put(
    agent_id="my-agent",
    content="In June 2023, Alice moved to San Francisco"
)

# Falls back to current time
client.put(
    agent_id="my-agent",
    content="Alice likes Python"
)

LLM Temporal Extraction

During fact extraction, the LLM identifies temporal markers in natural language:

Input Content	Extracted Event Date
"In March 2024, Alice started a new job"	March 1, 2024 00:00:00
"Last spring, Bob learned Python"	March 1 (previous year)
"Yesterday I met with the team"	Yesterday's date
"Alice works at Google"	Current timestamp (no temporal marker)

The extracted event_date is stored as a timestamp with the fact:

CREATE TABLE memory_units (
    id UUID PRIMARY KEY,
    text TEXT NOT NULL,
    event_date TIMESTAMP DEFAULT NOW(),
    -- other fields...
);

Query-Time Temporal Analysis

Automatic Detection

When a search query is issued, Hindsight automatically analyzes it for temporal expressions:

# These queries activate temporal-graph retrieval
results = client.search(agent_id="my-agent", query="What did Alice do last spring?")
results = client.search(agent_id="my-agent", query="What happened in June?")
results = client.search(agent_id="my-agent", query="Events from last year")

# These do NOT activate temporal retrieval
results = client.search(agent_id="my-agent", query="What does Alice do?")
results = client.search(agent_id="my-agent", query="Tell me about Python")

T5-Based Parsing

Hindsight uses a T5 transformer model (google/flan-t5-small) to convert natural language temporal expressions into structured date ranges:

# Query: "What did Alice do last spring?"
#
# T5 Input Prompt:
# "Today is 2024-11-25. Convert temporal expressions to date ranges.
#  June 2024 = 2024-06-01 to 2024-06-30
#  last year = 2023-01-01 to 2023-12-31
#  What did Alice do last spring? ="
#
# T5 Generated Output:
# "2024-03-01 to 2024-05-31"
#
# Parsed Constraint:
# TemporalConstraint(
#     start_date=datetime(2024, 3, 1, 0, 0, 0),
#     end_date=datetime(2024, 5, 31, 23, 59, 59, 999999)
# )

Supported Temporal Expressions

Expression	Parsed Date Range	Notes
"last spring"	March 1 - May 31 (previous year)	Seasonal ranges
"in June"	June 1-30 (current/nearest year)	Specific month
"in June 2023"	June 1-30, 2023	Month with year
"last year"	Jan 1 - Dec 31 (previous year)	Relative year
"last week"	7 days ago - today	Relative week
"yesterday"	Yesterday's date (00:00 - 23:59)	Specific day
"between March and May"	March 1 - May 31 (current year)	Date range
"March 2023"	March 1-31, 2023	Month and year
"2023"	Jan 1 - Dec 31, 2023	Year only

No Temporal Expression

If no temporal expression is detected, the query analysis returns None and standard retrieval (without temporal filtering) is used:

# No temporal expression detected
query = "What does Alice do?"
temporal_constraint = extract_temporal_constraint(query)
# Returns: None

Temporal-Graph Retrieval Strategy

When a temporal constraint is detected, Hindsight activates the Temporal-Graph retrieval strategy as a 4th parallel search path.

How It Works

graph TD
    Q[Query: What did Alice do last spring?]
    Q --> T5[T5 Temporal Parser]
    T5 --> TC[TemporalConstraint<br/>2024-03-01 to 2024-05-31]
    TC --> VS[Vector Similarity<br/>Find semantic entry points]
    VS --> F[Filter by event_date]
    F --> GT[Graph Traversal<br/>Through time-filtered entities]
    GT --> R[Ranked Results]

Algorithm Steps

def retrieve_temporal(
    query_embedding,
    agent_id,
    start_date,
    end_date,
    budget,
    semantic_threshold=0.4
):
    """
    Temporal-Graph retrieval combines time filtering with entity relationships.

    Steps:
    1. Find semantic entry points (similarity >= threshold)
       WHERE event_date BETWEEN start_date AND end_date

    2. Initialize activation scores from similarity scores

    3. Graph traversal (spreading activation):
       - Follow entity links to related facts
       - ONLY traverse to facts within the time range
       - Propagate activation with decay (0.8x per hop)
       - Boost causal links (2x activation)

    4. Return facts sorted by activation score
    """

SQL Query Example

-- Step 1: Find temporal entry points
WITH temporal_candidates AS (
    SELECT
        mu.id,
        mu.text,
        mu.event_date,
        1 - (mu.embedding <=> $query_embedding) AS similarity
    FROM memory_units mu
    WHERE mu.agent_id = $agent_id
      AND mu.fact_type = ANY($fact_types)
      AND mu.event_date >= $start_date
      AND mu.event_date <= $end_date
      AND (1 - (mu.embedding <=> $query_embedding)) >= $semantic_threshold
    ORDER BY similarity DESC
    LIMIT 10
)
-- Step 2 & 3: Graph traversal happens in Python code
-- walking through memory_links while enforcing temporal bounds
SELECT * FROM temporal_candidates;

Key Differences from Standard Graph Traversal

Aspect	Standard Graph	Temporal-Graph
Entry Points	Top-K by similarity (any time)	Top-K by similarity within time range
Traversal	Follow all entity links	Only follow links to facts within time range
Use Case	"What does Alice do?"	"What did Alice do last spring?"
Activation Source	Pure semantic similarity	Time-filtered semantic similarity

Example Walkthrough

Query: "What did Alice do last spring?"

Step 1 - Temporal Parsing:

temporal_constraint = extract_temporal_constraint(query)
# Result: (2024-03-01 00:00:00, 2024-05-31 23:59:59)

Step 2 - Find Entry Points:

Semantic search for "What did Alice do"
WHERE event_date BETWEEN '2024-03-01' AND '2024-05-31'

Found:
- "Alice started learning Rust" (April 15, 2024, similarity: 0.78)
- "Alice attended ML conference" (May 3, 2024, similarity: 0.72)

Step 3 - Graph Traversal (time-filtered):

From "Alice started learning Rust":
  → Alice (entity)
    → "Alice joined Rust meetup" (April 20, 2024) ✓ within range
    → "Alice published Rust blog post" (April 28, 2024) ✓ within range
    → "Alice works at Google" (Jan 2023) ✗ outside range, skip

From "Alice attended ML conference":
  → ML Conference (entity)
    → "Conference keynote on LLMs" (May 3, 2024) ✓ within range
  → Alice (entity)
    → Same as above

Step 4 - Activation Scores:

1. "Alice started learning Rust" (0.78) - direct match
2. "Alice attended ML conference" (0.72) - direct match
3. "Alice published Rust blog post" (0.78 × 0.8 = 0.62) - 1 hop
4. "Alice joined Rust meetup" (0.78 × 0.8 = 0.62) - 1 hop
5. "Conference keynote on LLMs" (0.72 × 0.8 = 0.58) - 1 hop

Integration with Multi-Strategy Retrieval

Temporal-Graph runs in parallel with the other three strategies:

# When temporal constraint detected:
results = await asyncio.gather(
    retrieve_semantic(query_embedding, agent_id, fact_types),
    retrieve_bm25(query, agent_id, fact_types),
    retrieve_graph(query_embedding, agent_id, fact_types, budget),
    retrieve_temporal(query_embedding, agent_id, fact_types,
                     start_date, end_date, budget)  # 4th strategy
)

# Fusion via RRF
fused = reciprocal_rank_fusion(results)

# Rerank with temporal awareness
final = cross_encoder_rerank(query, fused, include_dates=True)

The 4th strategy (Temporal-Graph) contributes additional ranked results to the fusion process, ensuring that time-relevant facts are strongly represented in the final results.

Cross-Encoder Temporal Awareness

The cross-encoder reranker receives temporal context for more accurate relevance scoring:

# Standard input:
cross_encoder.predict([
    (query, memory_text)
])

# With temporal awareness:
cross_encoder.predict([
    (query, f"[Date: {date_readable}] {memory_text}")
])

# Example:
# Query: "What did Alice do last spring?"
# Input to cross-encoder:
# ("What did Alice do last spring?",
#  "[Date: April 15, 2024] Alice started learning Rust")

This allows the reranker to boost facts that align with the query's temporal intent.

Performance Considerations

Latency

Component	Typical Latency	Notes
T5 temporal parsing	~30-80ms (CPU)	Cached for repeated queries
Temporal entry point search	~25-40ms	PostgreSQL query with time+vector filter
Time-filtered graph traversal	~35-60ms	Similar to standard graph, fewer candidates
Total temporal overhead	~50-120ms	Only when temporal expressions detected

Optimization Tips

1. Event Date Indexing: Ensure event_date column is indexed:

CREATE INDEX idx_memory_units_event_date
ON memory_units (event_date)
WHERE event_date IS NOT NULL;

2. Composite Index: For frequent temporal queries:

CREATE INDEX idx_memory_units_agent_temporal
ON memory_units (agent_id, event_date, fact_type);

3. T5 Model Caching: The model is loaded once and reused across queries

API Reference

Python Client

from hindsight_client import Hindsight
from datetime import datetime

client = Hindsight(base_url="http://localhost:8888")

# Store with explicit event date
client.put(
    agent_id="my-agent",
    content="Event content",
    event_date=datetime(2024, 3, 15)
)

# Search with natural language temporal expression
results = client.search(
    agent_id="my-agent",
    query="What happened last spring?"
)

# Results include event_date
for r in results:
    print(f"{r['text']} (date: {r['event_date']})")

REST API

# Store with event date
curl -X POST http://localhost:8888/api/v1/put \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "my-agent",
    "content": "Alice started new job",
    "event_date": "2024-03-15T10:00:00Z"
  }'

# Search (temporal parsing automatic)
curl -X POST http://localhost:8888/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{
    "agent_id": "my-agent",
    "query": "What did Alice do last spring?"
  }'

Advanced: Temporal Ranges in Memory bank Profiles

Memory banks can have temporal ranges that represent their active periods or episodic boundaries:

# Create agent with temporal range
client.create_agent(
    agent_id="my-agent",
    name="Alice",
    temporal_ranges=[
        {
            "start_date": "2023-01-01T00:00:00Z",
            "end_date": "2023-12-31T23:59:59Z",
            "description": "2023 academic year"
        },
        {
            "start_date": "2024-01-01T00:00:00Z",
            "end_date": "2024-12-31T23:59:59Z",
            "description": "2024 academic year"
        }
    ]
)

These ranges can be used to:

• Segment memories into distinct time periods
• Filter retrieval to specific life phases
• Support episodic memory queries

Limitations and Future Work

Current Limitations

1. Relative Dates: "yesterday", "last week" are resolved relative to query time, not event time
2. Duration Expressions: "for 3 months" or "during the conference" not fully supported
3. Fuzzy Temporal: "around June" or "early spring" treated as exact boundaries
4. Recurring Events: "every Monday" or "annual conference" not handled specially

Planned Enhancements

• Duration-based queries ("events lasting more than a week")
• Temporal relationship extraction ("after Alice joined Google", "before the conference")
• Recurrence pattern recognition
• Relative temporal reasoning ("what happened next")

Summary

Hindsight's temporal reasoning capabilities enable natural, time-aware memory queries:

✅ Automatic fact timestamping - LLM extracts dates from content ✅ Natural language parsing - T5 converts "last spring" to date ranges ✅ Temporal-Graph retrieval - 4th parallel strategy for time-filtered search ✅ Cross-encoder awareness - Reranker considers temporal alignment ✅ Low overhead - Only activates when temporal expressions detected

This allows memory banks to answer complex temporal queries like:

• "What did I learn last quarter?"
• "What was Alice working on in March?"
• "Events between the conference and the project launch"