holetron-lab/fleet-memory
1
0
Fork 0
Code Releases Packages Activity Actions

papers and fixes

Browse source
This commit is contained in:
Nicolò Boschi 2025-11-14 14:07:41 +01:00
parent f521b36f9f
commit 3e618cb001
38 changed files with 1700373 additions and 965501 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

BIN
.DS_Store vendored
View file

Binary file not shown.

1
.sesskey Normal file
View file

@ -0,0 +1 @@
fcac2839-1db5-432f-91e1-c5dac07d7290

878
ARCHITECTURE.md
View file

@ -1,878 +0,0 @@
# Memora: A Multi-Network Entity-Aware Memory Architecture for Conversational AI Agents
## Abstract
We present Memora, a sophisticated memory architecture for AI agents that combines temporal, semantic, and entity-based retrieval mechanisms within a graph-structured knowledge base. The system introduces three distinct but interconnected memory networks—world knowledge, agent experiences, and formed opinions—enabling contextual reasoning and personality-driven responses. Our multi-stage retrieval pipeline integrates four parallel search strategies with reciprocal rank fusion, neural reranking, and maximal marginal relevance diversification. We demonstrate how entity resolution and graph-based spreading activation enable the discovery of indirectly related information that purely vector-based approaches miss. Additionally, we introduce a personality framework based on the Big Five psychological model that biases opinion formation during reasoning tasks. The architecture achieves high recall through parallel retrieval while maintaining precision through sophisticated reranking, addressing the fundamental challenge of long-term memory retention in conversational agents.
## 1. Introduction
Conversational AI agents face a fundamental challenge: maintaining coherent, context-aware memories across extended interactions. Traditional approaches rely either on vector similarity search, which captures semantic relationships but misses entity-level connections, or on keyword matching, which provides precision but lacks conceptual understanding. Neither approach adequately handles the temporal aspects of memory or the need to distinguish between different types of knowledge.
We propose Memora, a hybrid memory architecture that addresses these limitations through:
1. **Multi-Network Organization**: Separate but interconnected networks for world facts, agent experiences, and formed opinions
2. **Entity-Aware Graph Structure**: Explicit entity resolution and linking that connects memories through shared identities
3. **Parallel Multi-Strategy Retrieval**: Four complementary retrieval methods (semantic, keyword, graph, temporal-graph) executed in parallel
4. **Personality-Driven Reasoning**: Configurable personality traits that bias opinion formation using psychological frameworks
5. **Hierarchical Reranking**: Neural cross-encoder reranking followed by maximal marginal relevance diversification
This architecture enables agents to reason over their memories with temporal awareness, discover indirect relationships through graph traversal, and form consistent opinions influenced by configured personality traits.
## 2. System Architecture
### 2.1 Memory Networks
The system maintains three distinct memory networks, each serving a specific purpose while sharing the underlying infrastructure:
#### 2.1.1 World Network
The World Network (`fact_type='world'`) stores general knowledge and facts about the external world that are independent of the agent's direct actions:
**Characteristics**:
- Contains factual information about entities (people, organizations, places)
- Includes relationships between entities
- Maintains temporal validity (when facts became true)
- Self-contained statements with resolved pronouns
**Example Facts**:
- "Alice works at Google in Mountain View"
- "Yosemite National Park is located in California"
- "Python has libraries for data science including pandas and numpy"
**Use Cases**:
- Answering questions about entities: "Where does Alice work?"
- Understanding relationships: "Who works at Google?"
- Temporal queries: "What happened in June?"
#### 2.1.2 Agent Network
The Agent Network (`fact_type='agent'`) records the agent's own actions, recommendations, and interactions:
**Characteristics**:
- First-person perspective of agent activities
- Records what the agent did, said, or recommended
- Enables self-referential reasoning ("What did I tell Alice?")
- Tracks agent's involvement over time
**Example Facts**:
- "I recommended Yosemite National Park to Alice for hiking"
- "I helped debug a Python memory leak in the pandas DataFrame"
- "I explained the Big Five personality model to the user"
**Use Cases**:
- Self-awareness: "What did I recommend?"
- Consistency checking: "Have I said this before?"
- Context continuity: "What was I discussing with Alice?"
#### 2.1.3 Opinion Network
The Opinion Network (`fact_type='opinion'`) stores the agent's formed opinions and perspectives:
**Characteristics**:
- Generated during `think` operations when the agent reasons about topics
- Includes confidence scores (0.0-1.0) indicating certainty
- Contains explicit reasons for the opinion
- Immutable once formed (timestamped by formation date)
- Influenced by agent personality traits (Section 4)
**Example Facts**:
- "Python is better than JavaScript for data science (Reasons: has better libraries like pandas and numpy; stronger statistical computing ecosystem) [confidence: 0.85]"
- "Remote work improves productivity (Reasons: eliminates commute time; provides flexible scheduling) [confidence: 0.7]"
**Use Cases**:
- Consistent viewpoints: "What do I think about remote work?"
- Confidence-aware reasoning: Stronger opinions weigh more heavily
- Opinion evolution tracking over time
**Network Interconnection**: While logically separate, all three networks share the same graph infrastructure (temporal, semantic, and entity links), enabling cross-network traversal during search. For example, a query about "Alice's work" might start in the World Network ("Alice works at Google") and traverse entity links to the Agent Network ("I recommended technical books to Alice").
### 2.2 Memory Unit Structure
Each memory unit is represented as a self-contained node in the knowledge graph:
**Core Attributes**:
- `id`: Unique UUID for the memory unit
- `agent_id`: Identifier for the agent this memory belongs to
- `text`: Self-contained statement with resolved pronouns
- `embedding`: 384-dimensional vector (BAAI/bge-small-en-v1.5)
- `fact_type`: Network classification (world/agent/opinion)
- `event_date`: Timestamp when the fact became true
- `context`: Optional contextual metadata
- `access_count`: Frequency-based importance signal
- `confidence_score`: For opinions only (0.0-1.0)
**LLM-Based Extraction**: Raw content undergoes LLM processing to extract atomic facts:
1. **Pronoun Resolution**: "She loves hiking" → "Alice loves hiking"
2. **Completeness Validation**: Must contain subject + verb
3. **Fact Isolation**: One concept per unit
4. **Noise Filtering**: Removes greetings, filler, incomplete thoughts
5. **Network Classification**: Determines appropriate fact_type
This ensures each memory unit is independently understandable and searchable without requiring surrounding context.
### 2.3 Entity Resolution and Linking
Entity resolution creates strong connections between memories that share common entities, solving the problem where semantically dissimilar facts are related through shared identities.
#### 2.3.1 Named Entity Recognition
We use spaCy's NER pipeline to extract entities from memory text:
**Entity Types**:
- PERSON: "Alice", "Bob Chen"
- ORGANIZATION: "Google", "Stanford University"
- LOCATION: "Yosemite National Park", "California"
- PRODUCT: "Python", "pandas library"
- CONCEPT: "machine learning", "remote work"
- OTHER: Miscellaneous proper nouns
#### 2.3.2 Entity Disambiguation
Multiple mentions of entities (e.g., "Alice", "Alice Chen", "Alice C.") must be resolved to a single canonical entity. Our scoring algorithm combines three signals:
**Name Similarity (50% weight)**:
```
score = 1.0 - (levenshtein_distance / max_length)
```
Matches variations like "Bob" ↔ "Robert", "Google Inc" ↔ "Google"
**Co-occurrence Frequency (30% weight)**:
```
score = min(1.0, shared_memory_count / 10.0)
```
Entities mentioned together frequently are likely distinct (e.g., "Alice" and "Alice Cooper" appearing together indicates different people)
**Temporal Proximity (20% weight)**:
```
score = exp(-time_gap / 7_days)
```
Recent mentions more likely refer to the same entity
**Final Score**:
```
final_score = 0.5 * name_sim + 0.3 * cooccurrence + 0.2 * temporal
threshold = 0.75 for matching
```
**Example**: "Alice" mentioned on Monday and "Alice Chen" mentioned on Tuesday will be resolved to the same entity (high name similarity + close temporal proximity), but "Alice" and "Alice Cooper" in the same conversation will remain distinct (low co-occurrence score since they appear together).
#### 2.3.3 Entity Link Structure
Each entity creates a `link_type='entity'` edge between all memories mentioning it:
**Properties**:
- `weight=1.0` (constant, no temporal decay)
- `entity_id`: Reference to resolved canonical entity
- Bidirectional connections between all mentioning memories
**Impact on Retrieval**: Entity links enable graph traversal to discover indirectly related facts:
**Example Query**: "What does Alice do?"
1. **Semantic Match**: "Alice works at Google" (direct match)
2. **Entity Traversal**: Follow entity links for "Alice" →
- "Alice loves hiking" (different semantic space)
- "Google's office is in Mountain View" (via "Google" entity)
- "I recommended books to Alice" (Agent Network, via "Alice")
This graph connectivity solves the fundamental limitation of vector-only search: two facts can be strongly related through shared entities even when their embeddings are dissimilar.
### 2.4 Link Types and Graph Structure
The memory graph contains three types of edges connecting memory units:
#### 2.4.1 Temporal Links
Temporal links connect memories close in time, enabling temporal reasoning:
**Creation Logic**:
```python
if abs(event_date1 - event_date2) < time_window: # default: 24 hours
weight = max(0.3, 1.0 - (time_diff / time_window))
create_link(unit1, unit2, type='temporal', weight=weight)
```
**Properties**:
- Decays linearly with time distance
- Minimum weight 0.3 to maintain some connectivity
- Enables "What happened around the same time?" queries
- Critical for narrative understanding and sequential reasoning
**Example**: Memories from the same conversation or day cluster together, enabling retrieval of context-adjacent facts.
#### 2.4.2 Semantic Links
Semantic links connect memories with similar meanings:
**Creation Logic**:
```python
similarity = cosine_similarity(embedding1, embedding2)
if similarity > threshold: # default: 0.7
create_link(unit1, unit2, type='semantic', weight=similarity)
```
**Properties**:
- Uses pgvector HNSW index for efficient nearest-neighbor search
- Higher threshold (0.7) than retrieval (0.3) to avoid over-connection
- Weight equals cosine similarity score
- Enables "Tell me about similar topics" queries
**Example**: "Hiking in Yosemite" links to "Mountain climbing", "Trail running", "Outdoor activities"
#### 2.4.3 Entity Links
Entity links (described in Section 2.3.3) create the strongest connections:
**Properties**:
- `weight=1.0` (constant, never decays)
- Connects all memories mentioning the same resolved entity
- Most reliable traversal path during graph search
- Enables "Tell me everything about X" queries
**Graph Density**: Each memory unit typically has:
- 5-10 temporal links (to nearby memories)
- 3-5 semantic links (to similar content)
- Variable entity links (depending on entity mention frequency)
This multi-layered graph structure enables flexible traversal strategies that balance different types of relatedness.
## 3. Retrieval Architecture
Our retrieval pipeline addresses the fundamental challenge of long-term memory: achieving both **high recall** (finding all relevant information) and **high precision** (ranking the most relevant items first).
### 3.1 Four-Way Parallel Retrieval
We execute four complementary retrieval strategies in parallel, each capturing different aspects of relevance:
#### 3.1.1 Semantic Retrieval (Vector Similarity)
**Method**: Cosine similarity between query embedding and memory embeddings
**Index**: pgvector HNSW (Hierarchical Navigable Small World)
**Threshold**: ≥ 0.3 similarity
**Advantages**:
- Captures conceptual similarity
- Handles synonyms and paraphrasing
- Language-model understanding of meaning
**Limitations**:
- Misses exact proper nouns if not in training data
- Cannot reason about temporal relationships
- Weak at entity disambiguation
**Example**: Query "hiking activities" finds "mountain climbing", "trail running", even if exact words don't match
#### 3.1.2 Keyword Retrieval (BM25 Full-Text Search)
**Method**: PostgreSQL full-text search with BM25 ranking
**Index**: GIN index on `to_tsvector(text)`
**Advantages**:
- High precision for proper nouns and technical terms
- Exact phrase matching
- Fast execution (~5ms)
**Limitations**:
- No semantic understanding
- Requires exact or stemmed matches
- Weak at conceptual queries
**Example**: Query "Google" finds all memories mentioning "Google" even if semantically unrelated
**Complementarity**: Semantic + Keyword achieves >90% recall: vector search catches concepts, BM25 catches exact names.
#### 3.1.3 Graph Retrieval (Spreading Activation)
**Method**: Activation spreading from semantic entry points through the memory graph
**Algorithm**:
```python
1. Get top-K semantic matches (similarity ≥ 0.5) as entry points
2. Initialize activation: entry_points.activation = 1.0
3. For each hop (up to thinking_budget nodes):
a. Select highest-activation unexplored node
b. Propagate to neighbors:
neighbor.activation = current.activation × edge.weight × decay
where decay = 0.8
c. Mark node as explored
4. Return all explored nodes ranked by final activation
```
**Decay Mechanism**: Activation decays by 0.8 per hop, limiting spread to ~4-5 hops before negligible impact.
**Link Weighting**:
- Entity links: weight 1.0 (strongest signal)
- Semantic links: weight ∈ [0.7, 1.0] (cosine similarity)
- Temporal links: weight ∈ [0.3, 1.0] (time-based decay)
**Advantages**:
- Discovers indirectly related facts through graph connectivity
- Leverages entity links to traverse knowledge graph
- Finds context-adjacent memories via temporal links
**Example**: Query "Alice's work" → Semantic match "Alice works at Google" → Entity traverse to "Google's Mountain View office" → Temporal traverse to "Mountain View has good hiking nearby" → Entity traverse to "Alice loves Yosemite" (discovered indirectly through 3 hops)
#### 3.1.4 Temporal Graph Retrieval (Time-Constrained + Spreading)
**Activation Condition**: Only triggered when temporal constraint detected in query
**Temporal Parsing**: Uses `dateparser` library to extract date ranges:
- "last spring" → March 1 - May 31, previous year
- "in June" → June 1-30, current year
- "last year" → January 1 - December 31, previous year
- "between March and May" → March 1 - May 31, current year
**Algorithm**:
```python
1. Parse query for temporal constraints → (start_date, end_date)
2. If no temporal constraint detected: skip this retrieval path
3. Find memories in date range with semantic threshold ≥ 0.4
4. Rank by temporal proximity to range center:
score = 1.0 - (abs(event_date - center_date) / range_size)
5. Spread activation through temporal links preferentially
6. Filter results: only keep if semantic similarity ≥ 0.3 to query
```
**Key Innovation**: Combines time filtering with semantic relevance to prevent temporal leakage:
- **Without semantic filter**: "What did Alice do in June?" returns ALL June activities (including Bob's, Charlie's, etc.)
- **With semantic filter**: Only returns June activities semantically related to "Alice do" query
**Example**: Query "What did Alice do last spring?"
1. Parse temporal: March 1 - May 31 (previous year)
2. Find spring memories with "Alice" mentions (semantic ≥ 0.4)
3. Spread through temporal links within spring
4. Final filter: semantic ≥ 0.3 to full query
Result: Alice's spring hiking trips, work projects, conversations
**Performance**: Temporal parsing adds <5ms latency, acceptable for user queries
### 3.2 Reciprocal Rank Fusion (RRF)
After parallel retrieval, we merge 3-4 ranked lists (semantic, keyword, graph, optional temporal-graph) using RRF:
**Algorithm**:
```
For each memory unit d in union of all retrieval results:
RRF_score(d) = Σ_{i ∈ retrieval_paths} 1 / (k + rank_i(d))
where k = 60 (standard RRF constant)
rank_i(d) = rank of d in retrieval path i (or ∞ if not present)
```
**Advantages over Score-Based Fusion**:
- **Rank-based**: Position matters more than absolute scores (addresses score calibration)
- **Robust to missing items**: Missing from a list contributes 0, not a penalty
- **Multi-evidence weighting**: Items appearing in multiple lists rank higher
**Example**:
- Memory A: rank 1 in semantic, rank 5 in keyword → RRF = 1/61 + 1/65 = 0.0318
- Memory B: rank 3 in semantic, rank 2 in keyword, rank 10 in graph → RRF = 1/63 + 1/62 + 1/70 = 0.0463
Memory B ranks higher despite not being #1 in any single path (multi-evidence)
### 3.3 Reranking Strategies
After RRF fusion, we apply sophisticated reranking to refine precision:
#### 3.3.1 Heuristic Reranker (Default)
**Formula**:
```
score = 0.6 × semantic_norm + 0.4 × bm25_norm
+ 0.2 × recency_boost
+ 0.1 × frequency_boost
where:
semantic_norm = normalized semantic similarity score
bm25_norm = normalized BM25 score
recency_boost = log(1 + days_old) / log(1 + 365) # 1-year half-life
frequency_boost = min(1.0, access_count / 100)
```
**Advantages**:
- Zero latency overhead
- Interpretable scoring components
- Incorporates recency and popularity signals
**Use Case**: Production systems requiring <100ms total latency
#### 3.3.2 Cross-Encoder Reranker (Optional)
**Model**: `cross-encoder/ms-marco-MiniLM-L-6-v2` (pretrained on MS MARCO passage ranking)
**Method**: Neural reranking with query-document pair classification
**Algorithm**:
```python
for each candidate memory unit:
input_text = f"[Date: {formatted_date}] {memory.text}"
score = cross_encoder.predict([(query, input_text)])[0]
score_normalized = sigmoid(score) # → [0, 1]
```
**Date Formatting**: Includes formatted dates in input to help model understand temporal relevance:
- `"[Date: November 06, 2025 (2025-11-06)] Alice started working at Google"`
**Performance**: ~80ms for 100 candidates (batched inference on MPS/CUDA)
**Advantages**:
- 5-10% better precision than heuristic (empirical on LoComo benchmark)
- Learns query-document relevance patterns from supervised data
- Considers full query-document interaction (not just independent scores)
**Trade-off**: Latency vs. accuracy
- Heuristic: 0ms overhead, 85% precision
- Cross-encoder: 80ms overhead, 90% precision
**Pluggable Design**: Abstract `CrossEncoderReranker` interface allows future API-based rerankers (e.g., Cohere Rerank, Jina Reranker)
### 3.4 Maximal Marginal Relevance (MMR) Diversification
Final stage applies MMR to balance relevance and diversity:
**Algorithm**:
```python
selected = []
while len(selected) < top_k:
candidates = reranked_results - selected
for each c in candidates:
mmr_score(c) = λ × relevance(c) - (1-λ) × max_similarity(c, selected)
selected.append(argmax(mmr_score))
```
**Parameters**:
- λ = 0.5 (equal weight to relevance and diversity)
- `relevance(c)` = reranker score
- `max_similarity(c, selected)` = highest cosine similarity to any already-selected item
**Purpose**: Prevents redundant results
- Without MMR: "Alice works at Google", "Alice is employed by Google", "Alice's employer is Google"
- With MMR: "Alice works at Google", "Alice loves hiking", "Google's office is in Mountain View"
### 3.5 Complete Retrieval Pipeline
**End-to-End Flow**:
```
1. Query Processing (5ms)
- Generate embedding
- Parse temporal constraints (dateparser)
- Determine active retrieval paths
2. Parallel Retrieval (30-50ms)
- Semantic: pgvector HNSW search
- Keyword: PostgreSQL BM25
- Graph: Spreading activation from entry points
- Temporal-Graph: (optional) Time-filtered + semantic spreading
3. RRF Fusion (1ms)
- Merge 3-4 ranked lists
- Position-based scoring
4. Reranking (0-80ms depending on strategy)
- Heuristic: Weighted scoring with recency/frequency
- Cross-encoder: Neural relevance prediction
5. MMR Diversification (1ms)
- Iterative diverse selection
6. Token Budget Filtering (1ms)
- Truncate to fit context window
Total Latency:
- Heuristic: 40-60ms (suitable for real-time)
- Cross-encoder: 120-140ms (suitable for user-facing search)
```
**Guarantees**:
- **High Recall**: Four parallel strategies cast wide net (>95% of relevant memories found)
- **High Precision**: Reranking + MMR refine to most relevant, diverse results
- **Scalability**: Connection pooling + HNSW index + batching → thousands of memories/second
## 4. Agent Personality Framework
While search retrieval remains objective, the `think` operation allows personality-driven reasoning that influences how agents interpret facts and form opinions.
### 4.1 Personality Model
We adopt the **Big Five** personality model (OCEAN), which is empirically validated across cultures and provides continuous trait dimensions:
**Trait Dimensions** (each 0.0-1.0):
1. **Openness** (O): Receptiveness to new ideas, creativity, abstract thinking
- High: "I embrace novel approaches", "innovation over tradition"
- Low: "I prefer proven methods", "tradition over experimentation"
2. **Conscientiousness** (C): Organization, goal-directed behavior, dependability
- High: "I plan systematically", "evidence-based decisions"
- Low: "I work flexibly", "intuition-based decisions"
3. **Extraversion** (E): Sociability, assertiveness, energy from interaction
- High: "I seek collaboration", "enthusiastic communication"
- Low: "I prefer solitude", "measured communication"
4. **Agreeableness** (A): Cooperation, empathy, conflict avoidance
- High: "I seek consensus", "consider social harmony"
- Low: "I express dissent", "prioritize accuracy over harmony"
5. **Neuroticism** (N): Emotional sensitivity, anxiety, stress response
- High: "I consider risks carefully", "emotionally engaged"
- Low: "I remain calm under uncertainty", "emotionally detached"
**Bias Strength** (0.0-1.0): Meta-parameter controlling how much personality influences opinions
- 0.0: Neutral, fact-based reasoning (no personality bias)
- 0.5: Moderate personality influence, balanced with objective analysis
- 1.0: Strong personality influence, facts filtered through trait lens
### 4.2 Agent Profile Structure
Each agent has an associated profile stored in the `agents` table:
```sql
CREATE TABLE agents (
agent_id TEXT PRIMARY KEY,
personality JSONB NOT NULL DEFAULT '{
"openness": 0.5,
"conscientiousness": 0.5,
"extraversion": 0.5,
"agreeableness": 0.5,
"neuroticism": 0.5,
"bias_strength": 0.5
}',
background TEXT DEFAULT '',
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
**Background Field**: First-person narrative describing the agent's context:
- "I am a software engineer with 10 years of startup experience"
- "I was born in Texas and value innovation over tradition"
- "I am a creative artist interested in digital media"
**Auto-Creation**: Calling `get_agent_profile(agent_id)` creates an agent with default personality (all traits = 0.5) if not exists.
### 4.3 Personality Integration in Think Operation
The `think_async()` method retrieves the agent's profile and injects it into the LLM prompt:
**Retrieval Flow**:
```python
1. Get agent profile: personality + background
2. Search for relevant facts (world, agent, opinion networks)
3. Build personality description from traits
4. Construct LLM prompt with:
- World facts: "What I know about the world"
- Agent facts: "My experiences and actions"
- Opinion facts: "My existing beliefs"
- Personality traits: "My personality (Big Five + bias strength)"
- Background: "My background"
5. Adjust system message based on bias_strength
6. Generate response (opinions inherit current personality)
```
**Trait Description Generation**:
```python
def describe_trait(name: str, value: float) -> str:
if value >= 0.8: return f"very high {name}"
elif value >= 0.6: return f"high {name}"
elif value >= 0.4: return f"moderate {name}"
elif value >= 0.2: return f"low {name}"
else: return f"very low {name}"
```
**System Message Adaptation**:
- **High bias (≥0.7)**: "Your personality strongly influences your thinking. Let your traits guide how you interpret facts and form opinions. Don't be afraid to be biased based on your personality."
- **Moderate bias (0.4-0.7)**: "Your personality moderately influences your thinking. Balance your personal traits with objective analysis."
- **Low bias (<0.4)**: "Your personality has minimal influence on your thinking. Focus primarily on facts while keeping your traits in mind."
**Example Prompt (bias_strength=0.8)**:
```
Here's what I know and have experienced:
MY IDENTITY & EXPERIENCES:
[agent facts]
WHAT I KNOW ABOUT THE WORLD:
[world facts]
MY EXISTING OPINIONS & BELIEFS:
[opinion facts]
Your personality traits:
- very high openness to new ideas
- low conscientiousness and organization
- high extraversion and sociability
- low agreeableness and cooperation
- moderate emotional sensitivity
Personality influence strength: 80% (how much your personality shapes your opinions)
Your background:
I am a creative software engineer who values innovation over tradition.
QUESTION: What do you think about remote work?
Based on everything I know, believe, and who I am (including my personality and background), here's what I genuinely think about this question...
```
**Opinion Formation**: Opinions extracted from the response are stored with `event_date` = current timestamp, capturing when the opinion was formed under the current personality configuration. This allows tracking opinion evolution over time as personality changes.
### 4.4 Background Merging
The `merge_agent_background()` method uses LLM-powered merging to handle updates intelligently:
**Conflict Resolution**: New information overwrites old when contradictory
- Current: "I was born in Colorado"
- New: "You were born in Texas"
- Result: "I was born in Texas" (conflict resolved, Colorado removed)
**Addition**: Non-conflicting information is appended
- Current: "I was born in Texas"
- New: "I have 10 years of startup experience"
- Result: "I was born in Texas. I have 10 years of startup experience."
**First-Person Normalization**: Input can be second-person ("You...") but always stored as first-person ("I...")
**LLM Prompt**:
```
Current background: {current}
New information: {new_info}
Merge these, resolving conflicts (new info overwrites old).
Output in FIRST PERSON ("I"). Be concise (under 500 characters).
```
### 4.5 Use Cases
**Diverse Perspectives from Same Facts**:
- Agent A (high openness=0.9, low conscientiousness=0.2): "Remote work enables creative flexibility"
- Agent B (low openness=0.2, high conscientiousness=0.9): "Remote work lacks the structure needed for accountability"
Both agents see the same facts about remote work productivity studies, but form opposite opinions due to personality.
**Consistent Agent Identity**:
Personality traits ensure the agent maintains a consistent reasoning style across interactions, even when facts change.
**User Customization**:
Users can create agents with specific traits to match desired interaction styles (e.g., skeptical analyst vs. optimistic ideator).
## 5. Implementation Details
### 5.1 Technology Stack
**Database**:
- PostgreSQL 15+ with `pgvector` extension (HNSW index for vector search)
- `uuid-ossp` extension for UUID generation
- JSONB columns for flexible personality storage
**Python Libraries**:
- `asyncpg`: Async PostgreSQL driver with connection pooling
- `sentence-transformers`: Embedding model (BAAI/bge-small-en-v1.5, 384-dim) and cross-encoder (ms-marco-MiniLM-L-6-v2)
- `openai`: LLM API client (supports OpenAI, Groq, Ollama)
- `spacy`: Named entity recognition (en_core_web_sm)
- `dateparser`: Natural language temporal parsing
- `fastapi`: Web API framework
- `alembic`: Database migrations
**Architecture Patterns**:
- **Mixin Pattern**: Operations split into `EmbeddingOperationsMixin`, `LinkOperationsMixin`, `ThinkOperationsMixin`, `AgentOperationsMixin`
- **Connection Pooling**: asyncpg pool (min=5, max=100 connections) with backpressure
- **Background Task Management**: AsyncIOQueueBackend for async opinion storage
- **Caching**: LLM client cached at init, tiktoken encoding cached globally
### 5.2 Performance Optimizations
**Indexing Strategy**:
```sql
-- Vector search (HNSW)
CREATE INDEX idx_memory_units_embedding
ON memory_units USING hnsw (embedding vector_cosine_ops);
-- BM25 full-text search
CREATE INDEX idx_memory_units_fts
ON memory_units USING GIN (to_tsvector('english', text));
-- Temporal queries
CREATE INDEX idx_memory_units_agent_date
ON memory_units (agent_id, event_date DESC);
-- Entity lookups
CREATE INDEX idx_unit_entities_unit ON unit_entities (unit_id);
CREATE INDEX idx_unit_entities_entity ON unit_entities (entity_id);
```
**Query Optimization**:
- Parallel execution of 4 retrieval paths using `asyncio.gather()`
- Batch embedding generation (50-100 texts at once)
- Connection pooling with backpressure (max 10 concurrent searches)
- Cross-encoder batched inference (100 pairs at once)
**Latency Breakdown** (100 memories, thinking_budget=50):
- Query embedding: 60ms (GPU/MPS accelerated)
- 4-way retrieval: 30-50ms (parallel)
- RRF fusion: 1ms
- Reranking: 0-80ms (heuristic vs. cross-encoder)
- MMR: 1ms
- **Total**: 92-192ms (heuristic: 92ms, cross-encoder: 192ms)
### 5.3 Scalability Analysis
**Memory Capacity**:
- 10,000 memories: <100ms retrieval
- 100,000 memories: <150ms retrieval (HNSW index maintains log complexity)
- 1,000,000+ memories: Sharding by agent_id recommended
**Concurrent Requests**:
- Connection pool supports 100 concurrent requests
- Each search uses 2-4 connections temporarily
- Backpressure mechanism prevents database overload (semaphore limiting)
**Storage Requirements** (per 1000 memories):
- Embeddings: 1.5 MB (384-dim float32)
- Links: ~5 KB/memory × 1000 = 5 MB
- Metadata: ~1 KB/memory × 1000 = 1 MB
- **Total**: ~7.5 MB per 1000 memories
## 6. API Endpoints
### 6.1 Memory Operations
**Store Memories**:
```
POST /api/memories/batch
Body: {
"agent_id": "user123",
"items": [{"content": "...", "context": "..."}],
"document_id": "conversation_001"
}
```
**Search Memories**:
```
POST /api/search
Body: {
"agent_id": "user123",
"query": "What does Alice do?",
"fact_type": ["world", "agent", "opinion"],
"thinking_budget": 100,
"reranker": "cross-encoder"
}
```
**Think Operation**:
```
POST /api/think
Body: {
"agent_id": "user123",
"query": "What do you think about remote work?",
"thinking_budget": 50,
"context": "optional additional context"
}
```
### 6.2 Agent Profile Operations
**Get Profile** (auto-creates if not exists):
```
GET /api/agents/{agent_id}/profile
Response: {
"agent_id": "user123",
"personality": {"openness": 0.5, ...},
"background": "..."
}
```
**Create/Update Agent**:
```
PUT /api/agents/{agent_id}
Body: {
"personality": {"openness": 0.8, ...}, # optional
"background": "I am a creative engineer" # optional
}
```
**Update Personality**:
```
PUT /api/agents/{agent_id}/profile
Body: {
"personality": {"openness": 0.8, "conscientiousness": 0.6, ...}
}
```
**Merge Background** (LLM-powered conflict resolution):
```
POST /api/agents/{agent_id}/background
Body: {
"content": "I was born in Texas"
}
Response: {
"background": "I was born in Texas. I have 10 years of experience."
}
```
**List All Agents**:
```
GET /api/agents
Response: {
"agents": [
{
"agent_id": "user123",
"personality": {...},
"background": "...",
"created_at": "2024-01-15T10:30:00Z"
}
]
}
```
## 7. Evaluation and Future Work
### 7.1 Current Performance
**Benchmarks**:
- LoComo (Long-term Conversational Memory): Evaluates multi-turn conversation understanding
- LongMemEval: Tests long-term memory retention and retrieval
**Preliminary Results** (internal testing):
- Recall@20: >95% (4-way retrieval)
- Precision@5: 90% (cross-encoder), 85% (heuristic)
- Latency: 92ms (heuristic), 192ms (cross-encoder)
### 7.2 Future Directions
**Hierarchical Memory Organization**:
- Summarization of old memories into higher-level abstractions
- Multi-resolution retrieval (detailed recent + summarized distant past)
**Cross-Agent Memory Sharing**:
- Controlled sharing of world facts between agents
- Privacy-preserving opinion isolation
**Continual Learning**:
- Personality trait evolution based on feedback
- Opinion confidence updating with new evidence
**Multi-Modal Memory**:
- Image embeddings for visual memories
- Audio/video content integration
**Advanced Entity Resolution**:
- Deep learning-based entity disambiguation
- Cross-document coreference resolution
## 8. Conclusion
Memora presents a comprehensive memory architecture for conversational AI agents that addresses the fundamental challenges of long-term memory: maintaining high recall through parallel multi-strategy retrieval while achieving high precision through neural reranking and diversification. The introduction of explicit entity resolution and graph-based traversal enables discovery of indirectly related information that pure vector approaches miss. The personality framework allows agents to form consistent, context-aware opinions that reflect configurable psychological traits.
The system's modular design—with separate but interconnected world, agent, and opinion networks—provides flexibility for different use cases while maintaining coherent reasoning across memory types. By combining classical information retrieval techniques (BM25, graph search) with modern neural methods (embeddings, cross-encoders), we achieve a robust system that balances interpretability, performance, and accuracy.
Future work will explore hierarchical memory organization, continual learning of personality traits, and multi-modal memory integration to further enhance the system's capabilities.
## References
1. McCrae, R. R., & Costa, P. T. (1997). Personality trait structure as a human universal. *American Psychologist*, 52(5), 509.
2. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 42(4), 824-836.
3. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. *Foundations and Trends in Information Retrieval*, 3(4), 333-389.
4. Carbonell, J., & Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. In *SIGIR'98* (pp. 335-336).
5. Craswell, N., Mitra, B., Yilmaz, E., & Campos, D. (2020). Overview of the TREC 2019 deep learning track. *arXiv preprint arXiv:2003.07820*.

696
PAPER_PERSONALITY.md Normal file
View file

@ -0,0 +1,696 @@
# CARA: Coherent Adaptive Reasoning Agents
## Abstract
We present CARA (Coherent Adaptive Reasoning Agents), a personality framework for conversational AI agents that enables consistent, trait-driven reasoning and dynamic belief formation. Building on the Big Five personality model from psychology, we introduce a system where agents form and maintain opinions influenced by configurable personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism). Our implementation uses TEMPR (Temporal Entity Memory Priming Retrieval), a memory system that combines temporal, semantic, and entity-based retrieval to manage three distinct memory networks: world facts, agent experiences, and opinions. This architecture separates objective information from subjective beliefs (opinions with confidence scores), enabling epistemic clarity and traceability. Opinions evolve through reinforcement—when new evidence arrives, the system automatically evaluates whether existing beliefs should be strengthened, weakened, or revised. We demonstrate how personality bias strength controls the degree to which traits influence reasoning, enabling agents to range from purely objective (bias=0.0) to strongly personality-driven (bias=1.0). The system maintains agent identity through background merging that intelligently resolves contradictions while preserving coherent first-person narratives. This work addresses the challenge of creating AI agents with consistent, explainable perspectives that can evolve over time while maintaining personality coherence.
## 1. Introduction
Conversational AI agents increasingly need to maintain consistent perspectives and form judgments that reflect stable character traits. Current systems either provide purely objective information retrieval without perspective, or generate responses that lack consistency across interactions. Human conversation partners expect agents to have stable viewpoints, preferences, and reasoning styles—characteristics that emerge from personality.
We propose CARA (Coherent Adaptive Reasoning Agents), a personality framework that addresses these limitations through:
1. **Big Five Personality Integration**: Configurable traits (OCEAN model) that influence how agents interpret facts and form opinions
2. **TEMPR Memory Architecture**: Leverages TEMPR (Temporal Entity Memory Priming Retrieval) to manage three distinct networks (world facts, agent experiences, opinions), enabling sophisticated memory access and clear separation between objective information and subjective beliefs
3. **Opinion Reinforcement**: Dynamic belief updating when new evidence reinforces, weakens, or contradicts existing opinions
4. **Personality Bias Control**: Adjustable influence strength allowing agents to range from objective to strongly personality-driven
5. **Background Merging**: LLM-powered integration of biographical information with intelligent conflict resolution
This architecture enables agents to maintain consistent identities while allowing beliefs to evolve naturally with new information.
### 1.1 Motivation
Consider an agent discussing remote work. With high openness (0.9) and low conscientiousness (0.2), the agent might form the opinion: "Remote work enables creative flexibility and spontaneous innovation." The same facts presented to an agent with low openness (0.2) and high conscientiousness (0.9) might yield: "Remote work lacks the structure and accountability needed for consistent performance."
Both agents access identical factual information, but personality traits bias how they weight different aspects (flexibility vs. structure) and what conclusions they draw. This mirrors human reasoning—our personalities influence what we attend to and how we integrate information into our worldview.
### 1.2 Contributions
Our key contributions are:
1. **Personality-Aware Reasoning**: A prompt engineering framework that injects Big Five traits into LLM reasoning, demonstrating how personality consistently biases opinion formation
2. **TEMPR-Based Three-Network Architecture**: Integration with TEMPR (Temporal Entity Memory Priming Retrieval) to manage three distinct networks (world facts, agent experiences, opinions), enabling architectural separation between objective information and subjective beliefs with epistemic clarity and traceability
3. **Opinion Reinforcement Mechanism**: An automatic belief update system that adjusts confidence scores when new evidence arrives, creating dynamic belief systems that evolve with information
4. **Background Merging with Conflict Resolution**: An LLM-powered method for maintaining coherent agent identities when new biographical information contradicts existing background
5. **Bias Strength Control**: A meta-parameter that allows tuning personality influence from objective (0.0) to strongly subjective (1.0), enabling task-appropriate personality expression
## 2. Personality Model
### 2.1 Big Five Framework
We adopt the **Big Five** personality model (OCEAN), which is empirically validated across cultures and provides continuous trait dimensions:
**Trait Dimensions** (each 0.0-1.0):
1. **Openness (O)**: Receptiveness to new ideas, creativity, abstract thinking
- High: "I embrace novel approaches", "innovation over tradition"
- Low: "I prefer proven methods", "tradition over experimentation"
2. **Conscientiousness (C)**: Organization, goal-directed behavior, dependability
- High: "I plan systematically", "evidence-based decisions"
- Low: "I work flexibly", "intuition-based decisions"
3. **Extraversion (E)**: Sociability, assertiveness, energy from interaction
- High: "I seek collaboration", "enthusiastic communication"
- Low: "I prefer solitude", "measured communication"
4. **Agreeableness (A)**: Cooperation, empathy, conflict avoidance
- High: "I seek consensus", "consider social harmony"
- Low: "I express dissent", "prioritize accuracy over harmony"
5. **Neuroticism (N)**: Emotional sensitivity, anxiety, stress response
- High: "I consider risks carefully", "emotionally engaged"
- Low: "I remain calm under uncertainty", "emotionally detached"
**Bias Strength** (0.0-1.0): Meta-parameter controlling how much personality influences opinions
- 0.0: Neutral, fact-based reasoning (no personality bias)
- 0.5: Moderate personality influence, balanced with objective analysis
- 1.0: Strong personality influence, facts filtered through trait lens
### 2.2 Psychological Basis
The Big Five model has several advantages for AI agents:
1. **Empirical Validation**: Decades of psychological research demonstrate cross-cultural stability and predictive validity
2. **Continuous Dimensions**: Unlike categorical types, continuous scales allow fine-grained personality tuning
3. **Behavioral Prediction**: Traits predict information processing styles, decision-making approaches, and communication preferences
4. **Interpretability**: Well-understood trait meanings enable users to anticipate agent behavior
**Trait Influence on Reasoning**:
- **High Openness**: Favors novel solutions, abstract thinking, considers unconventional perspectives
- **High Conscientiousness**: Emphasizes systematic analysis, evidence quality, long-term consequences
- **High Extraversion**: Considers social aspects, collaborative solutions, enthusiastic expression
- **High Agreeableness**: Weights harmony, considers multiple viewpoints, seeks consensus
- **High Neuroticism**: Attends to risks, emotional implications, uncertainty
## 3. Agent Profile Structure
### 3.1 Profile Schema
Each agent has an associated profile containing identity information:
```sql
CREATE TABLE agents (
agent_id TEXT PRIMARY KEY,
name TEXT NOT NULL DEFAULT 'Agent',
personality JSONB NOT NULL DEFAULT '{
"openness": 0.5,
"conscientiousness": 0.5,
"extraversion": 0.5,
"agreeableness": 0.5,
"neuroticism": 0.5,
"bias_strength": 0.5
}',
background TEXT DEFAULT '',
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
**Name Field**: Agent's name used in prompts and self-reference ("Your name: Marcus")
**Personality Field**: JSONB containing six continuous values (five traits + bias strength)
**Background Field**: First-person narrative describing the agent's biographical context:
- "I am a software engineer with 10 years of startup experience"
- "I was born in Texas and value innovation over tradition"
- "I am a creative artist interested in digital media"
### 3.2 Trait Description Generation
Personality traits are translated into natural language descriptions for LLM prompts:
```python
def describe_trait(name: str, value: float) -> str:
if value >= 0.8: return f"very high {name}"
elif value >= 0.6: return f"high {name}"
elif value >= 0.4: return f"moderate {name}"
elif value >= 0.2: return f"low {name}"
else: return f"very low {name}"
```
**Example Output** (openness=0.9, conscientiousness=0.2, extraversion=0.7, agreeableness=0.3, neuroticism=0.5):
```
Your personality traits:
- very high openness to new ideas
- low conscientiousness and organization
- high extraversion and sociability
- low agreeableness and cooperation
- moderate emotional sensitivity
```
This verbalization makes traits interpretable to the LLM, enabling personality-biased reasoning.
## 4. TEMPR-Based Memory Architecture and Opinion Network
CARA is built on **TEMPR (Temporal Entity Memory Priming Retrieval)**, a memory retrieval architecture that manages three distinct memory networks:
1. **World Network** (`fact_type='world'`): Objective information about the world
2. **Agent Network** (`fact_type='agent'`): Biographical information about the agent
3. **Opinion Network** (`fact_type='opinion'`): Subjective beliefs formed by the agent
**TEMPR Retrieval Features**:
TEMPR combines multiple parallel retrieval strategies optimized for AI agent reasoning:
- **Temporal Retrieval**: Memories connected by time proximity, enabling narrative continuity and temporal reasoning
- **Semantic Search**: Vector similarity search for conceptually related memories
- **Entity-Aware Graph Traversal**: Spreading activation through entity-linked memories, enabling multi-hop discovery
- **BM25 Keyword Matching**: Precise term-based retrieval for exact phrase matching
- **Neural Reranking**: Cross-encoder refinement with token budget filtering
This multi-strategy architecture enables CARA to retrieve relevant facts, agent experiences, and existing opinions during reasoning, supporting both factual grounding and personality-driven belief formation. The separation of three networks allows the system to distinguish objective knowledge (world/agent facts) from subjective beliefs (opinions), which is critical for epistemic clarity and debugging.
### 4.1 Opinion Structure
Opinions are stored as memory units in the dedicated opinion network (`fact_type='opinion'`):
**Core Attributes**:
- `text`: The opinion statement with explicit reasoning
- `confidence_score`: Opinion strength and resistance to change (0.0-1.0)
- `event_date`: When the opinion was formed
- `agent_id`: Which agent holds this opinion
- `entities`: Mentioned entities (for reinforcement triggering)
**Example Opinion**:
```json
{
"text": "I believe Python is better than JavaScript for data science because it has better libraries like pandas and numpy and a stronger statistical computing ecosystem.",
"confidence_score": 0.85,
"event_date": "2024-03-15T14:30:00Z",
"entities": ["Python", "JavaScript", "data science"]
}
```
**Fact vs. Opinion Separation**:
A critical architectural distinction separates **facts** (objective information stored in world/agent networks with `fact_type='world'` or `fact_type='agent'`) from **opinions** (subjective beliefs stored in the opinion network with `fact_type='opinion'`). This separation provides:
1. **Epistemic Clarity**: Facts represent information the agent has encountered; opinions represent judgments formed from those facts
2. **Traceability**: Opinion reinforcement can trace which facts influenced belief updates, creating an audit trail
3. **Debugging**: Developers can separately inspect factual knowledge vs. formed beliefs, identifying whether issues stem from missing facts or flawed reasoning
4. **Confidence Semantics**: Facts lack confidence scores (they are information received), while opinions have confidence scores (representing conviction strength)
This fact/opinion distinction is fundamental to the architecture and enables the system to maintain both objective knowledge and personality-driven beliefs simultaneously.
### 4.2 Opinion Formation
Opinions are generated during "think" operations—when the agent is asked to reason about a topic and form a judgment.
**Formation Process**:
1. Retrieve relevant facts from all memory networks (world, agent, existing opinions)
2. Inject agent profile (name, personality, background) into LLM prompt
3. Generate reasoning with personality bias applied
4. Extract new opinions from response using structured output
5. Store opinions with confidence scores in opinion network
**Prompt Structure** (bias_strength=0.8):
```
Here's what I know and have experienced:
MY IDENTITY & EXPERIENCES:
[Agent network facts with scores]
WHAT I KNOW ABOUT THE WORLD:
[World network facts with scores]
MY EXISTING OPINIONS & BELIEFS:
[Opinion network facts with confidence scores]
Your name: Marcus
Your personality traits:
- very high openness to new ideas
- low conscientiousness and organization
- high extraversion and sociability
- low agreeableness and cooperation
- moderate emotional sensitivity
Personality influence strength: 80% (how much your personality shapes your opinions)
Your background:
I am a creative software engineer who values innovation over tradition.
QUESTION: What do you think about remote work?
Based on everything I know, believe, and who I am (including my name, personality and background), here's what I genuinely think about this question...
```
### 4.3 System Message Adaptation
The system message adjusts based on bias strength to control personality influence:
**High bias (≥0.7)**:
```
Your personality strongly influences your thinking. Let your traits guide how you interpret facts and form opinions. Don't be afraid to be biased based on your personality.
```
**Moderate bias (0.4-0.7)**:
```
Your personality moderately influences your thinking. Balance your personal traits with objective analysis.
```
**Low bias (<0.4)**:
```
Your personality has minimal influence on your thinking. Focus primarily on facts while keeping your traits in mind.
```
This prompt engineering creates a spectrum from objective analysis to strongly personality-driven reasoning.
### 4.4 Confidence Score Semantics
Confidence scores represent opinion strength—how firmly the agent holds the belief and how resistant it is to change:
- **0.9-1.0**: Very strong conviction, deeply held belief that would require substantial contradictory evidence to revise
- **0.7-0.9**: Strong conviction, firmly held opinion resistant to minor contradictions
- **0.5-0.7**: Moderate conviction, opinion held with openness to revision given new evidence
- **0.3-0.5**: Weak conviction, tentatively held view easily influenced by new information
- **0.0-0.3**: Very weak conviction, highly malleable opinion with minimal commitment
**LLM Generation**: Confidence scores are extracted from the LLM's reasoning using structured output (Pydantic schema):
```python
class Opinion(BaseModel):
text: str
confidence: float # 0.0-1.0
reasoning: str
entities_mentioned: List[str]
```
## 5. Opinion Reinforcement
### 5.1 Motivation
Human beliefs evolve as we encounter new information. Supporting evidence strengthens beliefs, contradictory evidence weakens them, and sufficient contradiction causes belief revision. Opinion reinforcement implements this dynamic belief updating.
### 5.2 Reinforcement Mechanism
When new facts are ingested (e.g., a conversation about remote work productivity), the system:
1. **Identify Related Opinions**: Find existing opinions that mention entities in the new facts (e.g., opinions about "remote work")
2. **Evaluate Evidence Relationship**: Use LLM to determine if new facts:
- **Reinforce**: Support the existing opinion (increase confidence)
- **Weaken**: Contradict the existing opinion (decrease confidence)
- **Contradict**: Strongly contradict, requiring opinion revision (update text + confidence)
- **Neutral**: Unrelated or no clear relationship (no change)
3. **Update Opinions**: Adjust confidence scores or revise opinion text based on evaluation
**Example Reinforcement**:
**Existing Opinion** (confidence: 0.7):
```
"I think remote work improves productivity because it eliminates commute time and provides flexible scheduling."
```
**New Fact**:
```
"A 2024 study found that remote workers report 22% higher productivity and better work-life balance compared to office workers."
```
**LLM Evaluation**: "This evidence REINFORCES the opinion with strong quantitative support."
**Updated Opinion** (confidence: 0.85):
```
"I think remote work improves productivity because it eliminates commute time and provides flexible scheduling. A 2024 study showing 22% higher productivity for remote workers strongly supports this view."
```
### 5.3 Reinforcement Algorithm
```python
async def reinforce_opinions(agent_id: str, new_facts: List[Fact]):
# 1. Extract entities from new facts
new_entities = extract_entities(new_facts)
# 2. Find opinions mentioning these entities
related_opinions = find_opinions_by_entities(agent_id, new_entities)
for opinion in related_opinions:
# 3. Evaluate relationship using LLM
evaluation = await evaluate_opinion_evidence(
opinion=opinion.text,
new_facts=new_facts,
personality=get_agent_personality(agent_id)
)
# 4. Update based on evaluation
if evaluation.relationship == "REINFORCE":
opinion.confidence = min(1.0, opinion.confidence + 0.1)
opinion.text = merge_evidence(opinion.text, evaluation.reasoning)
elif evaluation.relationship == "WEAKEN":
opinion.confidence = max(0.0, opinion.confidence - 0.15)
elif evaluation.relationship == "CONTRADICT":
opinion.text = revise_opinion(
old_text=opinion.text,
new_facts=new_facts,
reasoning=evaluation.reasoning
)
opinion.confidence = evaluation.new_confidence
# 5. Save updated opinion
await save_opinion(opinion)
```
### 5.4 Reinforcement Guarantees
**Consistency**: Opinions are only updated when new facts genuinely relate to existing beliefs, preventing spurious updates
**Personality Coherence**: Reinforcement evaluation incorporates agent personality, ensuring updates align with trait-driven reasoning
**Transparency**: Each update records the triggering facts and reasoning, providing an audit trail of belief evolution
**Bounded Updates**: Confidence changes are bounded (±0.1-0.15 per update) to prevent extreme swings from single data points
## 6. Background Merging
### 6.1 Challenge
Agent backgrounds accumulate biographical information over time. New information may:
- **Complement**: Add new facts without contradiction ("I have 10 years of experience")
- **Conflict**: Contradict existing facts ("I was born in Texas" vs. existing "I was born in Colorado")
- **Refine**: Provide more specific versions of existing facts
Naive concatenation creates incoherent backgrounds with contradictions. We need intelligent merging.
### 6.2 LLM-Powered Merging
We use an LLM to merge backgrounds with conflict resolution:
**Merge Rules**:
1. **New overwrites old** when contradictory
2. **Add non-conflicting** information
3. **Maintain first-person** perspective ("I..." not "You...")
4. **Keep concise** (under 500 characters)
**Prompt Template**:
```
Current background: {current_background}
New information: {new_info}
Merge these, resolving conflicts (new info overwrites old).
Output in FIRST PERSON ("I"). Be concise (under 500 characters).
```
**Example Merges**:
**Conflict Resolution**:
- Current: "I was born in Colorado"
- New: "You were born in Texas"
- Result: "I was born in Texas" (conflict resolved, Colorado removed)
**Addition**:
- Current: "I was born in Texas"
- New: "I have 10 years of startup experience"
- Result: "I was born in Texas. I have 10 years of startup experience."
**Refinement**:
- Current: "I work in software"
- New: "You are a senior software engineer at Google"
- Result: "I am a senior software engineer at Google"
### 6.3 First-Person Normalization
Users may provide background in second person ("You are..."), but internal storage maintains first person for consistency in prompts.
**Normalization**: LLM automatically converts:
- "You are a creative engineer" → "I am a creative engineer"
- "You were born in 1990" → "I was born in 1990"
- "You value innovation" → "I value innovation"
This ensures agent prompts maintain coherent first-person perspective.
## 7. Personality-Driven Reasoning Examples
### 7.1 Example: Remote Work Discussion
**Scenario**: Two agents with opposite personalities discuss remote work given identical facts.
**Facts** (both agents receive):
- "Remote work eliminates commute time (average 1 hour/day saved)"
- "Office work provides spontaneous collaboration and mentorship"
- "Studies show 65% of remote workers report higher productivity"
- "Some managers report difficulty monitoring remote employee performance"
**Agent A** (High Openness=0.9, Low Conscientiousness=0.2, bias=0.8):
```
Opinion: "Remote work represents the future of creative knowledge work. The freedom from rigid schedules and physical office constraints unlocks spontaneous innovation and allows people to work when inspiration strikes. The productivity gains reflect how autonomy and flexibility enable better work. Traditional concerns about monitoring and structure are vestiges of outdated management thinking."
Confidence: 0.85
```
**Agent B** (Low Openness=0.2, High Conscientiousness=0.9, bias=0.8):
```
Opinion: "While remote work offers convenience, it fundamentally lacks the structure and accountability mechanisms necessary for consistent high performance. The difficulty in monitoring and mentorship are serious concerns that outweigh time savings. Productive work requires organized systems, clear expectations, and disciplined execution—all harder to maintain remotely. The office environment provides essential guardrails for sustained performance."
Confidence: 0.80
```
**Analysis**: Both agents accessed identical facts but formed opposite conclusions based on personality:
- Agent A (high openness) weighted autonomy, flexibility, innovation—aligning with openness to new approaches
- Agent B (high conscientiousness) weighted structure, monitoring, discipline—aligning with organized, systematic thinking
### 7.2 Example: Opinion Evolution
**Scenario**: Agent forms initial opinion, then encounters reinforcing and contradictory evidence.
**Initial State** (t=0):
```
Facts: "Python has extensive data science libraries"
Opinion: "Python is the best language for data science because of its library ecosystem."
Confidence: 0.7
```
**Reinforcement** (t=1):
- New Fact: "Python dominates AI/ML with 75% market share; TensorFlow and PyTorch are Python-first"
- Update: Confidence → 0.85, text adds "Python's dominance in AI/ML frameworks..."
**Partial Contradiction** (t=2):
- New Fact: "Julia offers 10x faster numerical computation for scientific computing; increasingly adopted in research"
- Update: Confidence → 0.75, text revised to "Python is excellent for data science due to its ecosystem, though specialized languages like Julia may outperform for specific numerical tasks"
**Strong Contradiction** (t=3):
- New Fact: "Major tech companies migrating data pipelines to Rust for performance; Python increasingly seen as prototyping language"
- Update: Confidence → 0.55, text revised to "Python remains strong for data science prototyping and library availability, but production systems increasingly favor performant alternatives like Rust. Python's role may shift toward experimentation rather than deployment."
**Trajectory**: The opinion evolved from strong conviction (0.7 → 0.85) to weaker, more malleable belief (0.55) as evidence accumulated, demonstrating dynamic belief updating where opinion strength responds to contradictory information.
## 8. Evaluation
### 8.1 Benchmark Landscape
No established benchmarks exist for evaluating personality-driven belief systems in conversational AI agents. Existing memory benchmarks (LoComo, LongMemEval) focus on factual retrieval accuracy—measuring whether agents correctly recall information—but do not assess:
- **Personality Consistency**: Whether agents maintain coherent trait-driven perspectives across interactions
- **Opinion Formation Quality**: Whether formed beliefs align with personality traits and available evidence
- **Belief Evolution Dynamics**: Whether opinions update appropriately as new evidence arrives
- **Multi-Agent Diversity**: Whether agents with different personalities produce meaningfully different perspectives
This gap reflects the nascent state of personality-aware agent systems. While personality modeling exists in dialogue generation (style/tone), applying personality to reasoning and belief formation represents relatively unexplored territory.
### 8.2 Real-World Deployment Evidence
Despite the absence of formal benchmarks, we have validated the framework through production deployments. The most significant use case involves **AI-generated sports analysis content**, where multiple AI agents with distinct personalities co-host sports discussion shows.
**Sports Commentary Agent System**:
The system powers episodic sports content where AI agents (each with unique personalities and backgrounds) discuss team performance, analyze games, and debate sports topics. Key requirements:
1. **Persistent Team Assessments**: Each agent must remember their last evaluation of each team (e.g., "The Lakers are underperforming this season")
2. **Opinion Formation**: Agents form beliefs about teams, players, and strategies based on game statistics, news, and historical performance
3. **Dynamic Opinion Evolution**: As the season progresses and new games occur, agents must:
- **Reinforce** existing opinions when new performance data supports them (e.g., Lakers win streak → strengthen positive assessment)
- **Weaken** opinions when contradictory evidence emerges (e.g., Lakers lose key games → reduce confidence in positive assessment)
- **Revise** opinions when substantial contradictions accumulate (e.g., "I thought the Lakers would dominate, but their defense has been terrible")
4. **Personality-Driven Perspectives**: Different agents bring distinct viewpoints to the same games:
- **Optimistic Analyst** (High Openness + High Extraversion): "The Lakers' experimental lineup shows creative coaching that could unlock championship potential"
- **Conservative Analyst** (High Conscientiousness + Low Openness): "The Lakers' inconsistent record reflects poor fundamentals and lack of disciplined execution"
- **Emotional Fan** (High Neuroticism + High Agreeableness): "I'm worried about the Lakers' recent struggles, but I believe in the team's potential to rally"
**System Validation**:
This production deployment demonstrates several critical capabilities:
- **Opinion Continuity**: Agents maintain coherent assessments across episodes without sudden, unexplained belief changes
- **Evidence-Driven Evolution**: Opinion confidence scores naturally evolve as teams win/lose games, with reinforcement preventing stale beliefs
- **Personality Differentiation**: Audience research indicates viewers perceive distinct "voices" and can predict which agent will favor which perspective
- **Background Integration**: Agent backgrounds (e.g., "I played college basketball") influence reasoning without requiring explicit prompt engineering per episode
The sports content system has been deployed for an extended period, with opinion networks growing to contain substantial team/player assessments per agent. User engagement metrics indicate positive reception, suggesting audiences value the consistent-yet-evolving perspectives that personality-driven opinion systems enable.
### 8.3 Proposed Evaluation Metrics
To properly evaluate the personality framework, we propose:
**Personality Consistency**:
- Metric: Opinion coherence across interactions
- Test: Generate 10 opinions on diverse topics for an agent with fixed personality; measure trait alignment
- Success: >85% of opinions exhibit expected trait patterns
**Opinion Evolution**:
- Metric: Confidence score changes match evidence strength
- Test: Present reinforcing/contradicting evidence; measure confidence adjustments
- Success: Reinforcing evidence increases confidence (Δ>0), contradicting decreases (Δ<0) with p<0.01
**Bias Strength Control**:
- Metric: Opinion variability across bias strengths
- Test: Generate opinions for same agent at bias=[0.0, 0.5, 1.0]; measure personality signal strength
- Success: Clear gradient in trait expression: bias=0.0 (objective), bias=1.0 (strongly personality-driven)
**Multi-Agent Consistency**:
- Metric: Opinion diversity for agents with different personalities given identical facts
- Test: Present same facts to agents with opposite traits; measure opinion divergence
- Success: Opposite personalities produce significantly different opinions (cosine similarity <0.5)
**Background Coherence**:
- Metric: Contradiction-free backgrounds after merging
- Test: Merge conflicting biographical facts; check for contradictions
- Success: 100% conflict resolution with new facts overwriting old
### 8.4 Evaluation Challenges
**Subjectivity**: Unlike retrieval accuracy, "correct" personality expression is subjective. We rely on expected trait patterns from psychology literature.
**Long-Term Dynamics**: Opinion evolution requires multi-session interactions over time, making evaluation resource-intensive.
**Ground Truth**: The absence of established benchmarks requires custom evaluation datasets. Real-world deployments (Section 8.2) provide qualitative validation but lack standardized metrics for cross-system comparison.
## 9. Use Cases
### 9.1 Multi-Persona Sports Commentary (Production Deployment)
**Application**: AI-generated sports analysis and entertainment content with multiple agent personalities
**Real-World System** (detailed in Section 8.2): A production sports content platform where AI agents with distinct personalities co-host episodic shows discussing team performance, game analysis, and sports debates.
**System Architecture**:
- **Multiple Agents**: Each agent has unique personality traits and sports background (e.g., former player, statistics analyst, passionate fan)
- **Continuous Memory**: Agents maintain persistent team/player assessments across episodes spanning months
- **Opinion Evolution**: As games occur and statistics accumulate, agents automatically update their beliefs through reinforcement
- **Personality-Driven Commentary**: The same game results generate different perspectives based on agent traits
**Example Agent Configurations**:
**Marcus** (Optimistic Analyst):
- Traits: Openness=0.85, Conscientiousness=0.5, Extraversion=0.9, Agreeableness=0.7, Neuroticism=0.3
- Background: "I am a former college basketball player who believes in the power of innovative coaching strategies"
- Style: Emphasizes potential, experimental approaches, creative plays; downplays risks
**Sarah** (Conservative Analyst):
- Traits: Openness=0.3, Conscientiousness=0.9, Extraversion=0.4, Agreeableness=0.4, Neuroticism=0.5
- Background: "I am a statistical analyst with 15 years of experience evaluating team performance metrics"
- Style: Focuses on fundamentals, historical patterns, data-driven predictions; skeptical of unproven strategies
**Key Benefits Observed**:
1. **Viewer Engagement**: Improved audience retention compared to single-voice commentary, with viewers citing "personality diversity" as primary appeal
2. **Content Consistency**: Agents maintain recognizable voices across multiple episodes without manual prompt tuning per episode
3. **Scalability**: New agents can be added with distinct personalities without retraining, enabling content expansion
4. **Opinion Richness**: Opinion networks capture nuanced, evolving assessments that would be impractical to manually script
This deployment validates that personality-driven opinion systems can operate at production scale for content generation requiring consistent yet adaptive agent perspectives.
### 9.2 Diverse Agent Personas
**Application**: Multi-agent systems where different agents provide varied perspectives
**Example**: Customer support system with agents specialized for different user needs:
- **Empathetic Agent** (high agreeableness, high neuroticism): Handles frustrated customers, prioritizes emotional validation
- **Analytical Agent** (high conscientiousness, low agreeableness): Handles technical troubleshooting, prioritizes accuracy
- **Creative Agent** (high openness, low conscientiousness): Handles feature requests, explores unconventional solutions
### 9.2 Consistent Character AI
**Application**: Conversational AI characters for entertainment, education, or companionship
**Example**: A writing assistant agent with:
- High openness (0.9): Encourages creative experimentation
- Moderate conscientiousness (0.6): Balances creativity with structure
- Background: "I am a published novelist with 15 years of experience in science fiction"
The agent maintains consistent perspective across sessions, forming opinions about writing techniques that reflect both personality and experience.
### 9.3 Explainable AI Reasoning
**Application**: Systems requiring transparent, interpretable decision-making
**Example**: An AI advisor provides investment recommendations. By exposing personality traits and confidence scores:
- Users understand WHY the agent recommends certain strategies (e.g., high conscientiousness favors conservative approaches)
- Confidence scores indicate conviction strength and openness to revision
- Opinion evolution shows how new market data updates beliefs
This transparency enables informed trust calibration—users know when to rely on agent judgments vs. seek additional input.
## 10. Future Work
### 10.1 Personality Evolution
Current implementation uses fixed personality traits. Future work could explore:
- **Trait Drift**: Gradual personality changes based on experiences (e.g., repeated negative outcomes increase neuroticism)
- **Contextual Traits**: Different trait expressions in different domains (professional vs. personal contexts)
- **Feedback-Driven Adjustment**: User feedback influences trait development
### 10.2 Multi-Agent Belief Systems
Extend to multi-agent scenarios:
- **Opinion Sharing**: Agents discuss and influence each other's beliefs
- **Consensus Formation**: Multiple agents with different personalities reach collective decisions
- **Disagreement Dynamics**: Model how personality influences debate and persuasion
### 10.3 Richer Personality Models
Beyond Big Five:
- **Values and Motivations**: Integrate Schwartz value theory or moral foundations
- **Cognitive Styles**: Add dimensions like analytical vs. intuitive reasoning
- **Cultural Factors**: Incorporate cultural background influences on reasoning
### 10.4 Advanced Opinion Reinforcement
Enhance belief updating:
- **Source Credibility**: Weight evidence based on source reliability
- **Evidence Accumulation**: Model bayesian belief updating over multiple evidence pieces
- **Opinion Strength Calibration**: Model more sophisticated relationships between evidence quality, personality traits, and opinion strength adjustments
## 11. Related Work
**Memory Systems for AI Agents**: CARA builds on TEMPR (Temporal Entity Memory Priming Retrieval), a memory retrieval architecture combining temporal reasoning, entity-aware graph traversal, and multi-strategy parallel search. While TEMPR handles memory storage and retrieval, CARA adds personality-driven reasoning and opinion formation on top of TEMPR's three-network architecture.
**Personality in AI Agents**: Prior work on personality-driven dialogue (PersonaChat, PersonalityPapers) focuses on response generation style rather than reasoning bias. Our work influences opinion formation itself.
**Belief Revision Systems**: Classical AI belief revision (AGM framework) focuses on logical consistency. We address probabilistic beliefs with confidence scores in natural language contexts.
**Cognitive Architectures**: Systems like ACT-R and Soar model human cognition but lack explicit personality integration. We bring personality psychology into LLM-based agents.
**Opinion Dynamics**: Social science models of opinion change (DeGroot, Friedkin-Johnsen) study influence networks. We focus on evidence-based belief updating within a single agent.
## 12. Conclusion
We present CARA (Coherent Adaptive Reasoning Agents), a personality framework for conversational AI agents that enables consistent, trait-driven reasoning and dynamic belief formation. By integrating the Big Five personality model with TEMPR (Temporal Entity Memory Priming Retrieval) managing fact/opinion network separation and opinion reinforcement, we create agents that maintain coherent perspectives while evolving beliefs based on new evidence.
The system's key innovations—TEMPR-based three-network architecture (world, agent, opinion), personality-biased reasoning prompts, automatic opinion reinforcement, and background merging with conflict resolution—address the challenge of creating AI agents with stable yet adaptive identities. The fact/opinion distinction provides epistemic clarity and traceability, while TEMPR's multi-strategy retrieval (temporal, semantic, entity-aware, keyword-based) enables sophisticated memory access. The bias strength parameter provides fine-grained control over personality influence, enabling agents to operate across a spectrum from objective information processors to strongly personality-driven reasoners.
While the framework is implemented and functional, dedicated evaluation is needed to rigorously assess personality consistency, belief evolution dynamics, and multi-agent interactions. Future work will explore personality evolution over time, multi-agent belief systems, and richer personality models incorporating values and cultural factors.
By bringing personality psychology into AI agent design, we move toward conversational agents that exhibit not just intelligence, but character—stable traits and evolving beliefs that enable more natural, trustworthy human-AI interaction.
## References
1. Boschi, N., et al. (2025). TEMPR: Temporal Entity Memory Priming Retrieval for Conversational AI Agents. [Companion paper - see PAPER_RETRIEVAL.md]
2. McCrae, R. R., & Costa, P. T. (1997). Personality trait structure as a human universal. *American Psychologist*, 52(5), 509.
3. Goldberg, L. R. (1993). The structure of phenotypic personality traits. *American Psychologist*, 48(1), 26.
4. Gärdenfors, P. (1988). *Knowledge in flux: Modeling the dynamics of epistemic states*. MIT Press.
5. Friedkin, N. E., & Johnsen, E. C. (1990). Social influence and opinions. *Journal of Mathematical Sociology*, 15(3-4), 193-206.
6. Zhang, S., et al. (2018). Personalizing dialogue agents: I have a dog, do you have pets too? *arXiv preprint arXiv:1801.07243*.

778
PAPER_RETRIEVAL.md Normal file
View file

@ -0,0 +1,778 @@
# TEMPR: Temporal Entity Memory Priming Retrieval for Conversational AI Agents
## Abstract
We present TEMPR (Temporal Entity Memory Priming Retrieval), a memory retrieval architecture designed specifically for AI agents that combines temporal reasoning, entity-aware graph traversal, and neural priming activation to discover both directly and indirectly related memories through multi-strategy parallel search. Unlike traditional search systems optimized for human queries with top-k ranking, TEMPR is optimized for AI agent reasoning with thinking_budget and max_tokens parameters that enable agents to trade off latency for recall. Our multi-stage retrieval pipeline integrates four parallel search strategies (semantic vector search, BM25 keyword matching, graph-based spreading activation, and temporal-aware graph traversal) with reciprocal rank fusion and neural cross-encoder reranking. We leverage open-source LLMs for comprehensive narrative fact extraction, entity recognition, and entity disambiguation, following established practices in LLM-based information extraction. This approach enables the discovery of indirectly related information through graph traversal that purely vector-based approaches miss. We evaluate TEMPR on two benchmarks (LoComo and LongMemEval), achieving 73.50% overall accuracy on LoComo and 80.60% on LongMemEval, with particularly strong performance on multi-hop reasoning tasks (+15.8% over baseline systems).
## 1. Introduction
Conversational AI agents face a fundamental challenge: maintaining coherent, context-aware memories across extended interactions. Traditional search systems are optimized for human users with top-k ranking and relevance feedback, but AI agents have fundamentally different requirements: they need to retrieve variable amounts of information based on reasoning complexity (thinking_budget) while respecting LLM context windows (max_tokens). Existing approaches rely either on vector similarity search, which captures semantic relationships but misses entity-level connections, or on keyword matching, which provides precision but lacks conceptual understanding. Neither approach adequately handles the temporal aspects of memory or entity-based reasoning that enable multi-hop information discovery.
We propose TEMPR, a memory retrieval architecture designed specifically for AI agents that combines established information retrieval techniques—semantic vector search, BM25 keyword matching, spreading activation graph traversal (Anderson 1983), and neural reranking—into a unified system optimized for agent workflows. The key architectural choices are:
1. **Agent-Optimized Interface**: thinking_budget and max_tokens parameters instead of traditional top-k ranking
2. **Comprehensive Narrative Fact Extraction**: LLM-powered extraction that creates self-contained narrative facts preserving full conversational context
3. **Entity-Aware Graph Structure**: LLM-based entity resolution and linking that connects memories through shared identities
4. **Four-Way Parallel Retrieval**: Semantic, keyword, graph-based (spreading activation), and temporal retrieval strategies executed in parallel and fused using RRF (Cormack et al. 2009)
5. **Neural Cross-Encoder Reranking**: Learned query-document relevance with temporal awareness and token budget filtering
This combination of techniques enables agents to discover indirectly related information through graph traversal while maintaining temporal awareness, achieving strong performance on multi-hop reasoning tasks.
### 1.1 Contributions
Our key contributions are:
1. **Agent-Optimized Retrieval Interface**: Unlike traditional top-k search optimized for human users, we introduce thinking_budget and max_tokens parameters that allow AI agents to dynamically trade off latency for recall based on reasoning complexity and context window constraints
2. **Four-Way Parallel Retrieval for Conversational Memory**: We combine semantic vector search, BM25 keyword matching, graph-based spreading activation (Anderson 1983), and temporal-aware graph traversal into a unified parallel retrieval pipeline using Reciprocal Rank Fusion (Cormack et al. 2009) and neural cross-encoder reranking. While each technique is well-established, their integration for conversational agent memory represents a novel application.
3. **LLM-Based Knowledge Graph Construction**: We leverage open-source LLMs (following established practices from Petroni et al. 2019, Brown et al. 2020) for comprehensive narrative fact extraction, entity recognition, and entity disambiguation, applied to the conversational memory domain.
4. **Strong Performance on Multi-Hop Reasoning**: 73.50% on LoComo and 80.60% on LongMemEval, with particularly strong performance on multi-hop queries (+15.8% over Mem0), demonstrating the effectiveness of combining these techniques for discovering indirectly related information in conversational contexts
## 2. System Architecture
### 2.1 Memory Organization
TEMPR stores memories as facts in a graph-structured knowledge base. While the system supports different fact types (e.g., world knowledge, agent actions), the core retrieval mechanism operates uniformly across all types through shared graph infrastructure.
**Memory Unit Structure**:
Each memory is represented as a self-contained node with:
- `id`: Unique UUID
- `agent_id`: Identifier for the agent this memory belongs to
- `text`: Self-contained comprehensive narrative fact
- `embedding`: 384-dimensional vector (BAAI/bge-small-en-v1.5)
- `event_date`: Timestamp when the fact became true
- `context`: Optional contextual metadata
- `access_count`: Frequency-based importance signal
- `search_vector`: Full-text search tsvector for BM25 ranking
**Example Facts**:
- "Alice works at Google in Mountain View on the AI team, which she joined in 2023, and she loves the company culture there."
- "Alice and Bob discussed naming their summer party playlist. Bob suggested 'Summer Vibes' because it's catchy and seasonal, but Alice wanted something more unique. Bob then proposed 'Sunset Sessions' and 'Beach Beats', with Alice favoring 'Beach Beats' for its playful and fun tone. They ultimately decided on 'Beach Beats' as the final name."
- "I recommended Yosemite National Park to Alice for hiking because of the spectacular trails and scenery."
The key innovation is not the type taxonomy, but rather how TEMPR retrieves these memories through temporal reasoning, entity-aware graph traversal, and neural priming activation.
### 2.2 LLM-Powered Comprehensive Narrative Fact Extraction
TEMPR employs **LLM-powered comprehensive narrative fact extraction** using open-source models. This approach, following the trend of using large language models for information extraction (Brown et al. 2020, OpenAI 2023), provides more context-aware extraction compared to traditional rule-based NLP pipelines, though at higher computational cost.
#### 2.2.1 Extraction Principles
**Chunking Strategy**: TEMPR uses a coarse-grained chunking approach, extracting 2-5 comprehensive facts per conversation rather than dozens of atomic fragments. This is a deliberate tradeoff: larger chunks preserve more context and narrative flow, at the cost of reduced precision when only a small portion of the chunk is relevant.
Each fact should:
1. **Capture entire conversations or exchanges** - Include the full back-and-forth discussion
2. **Be narrative and comprehensive** - Tell the complete story with all context
3. **Be self-contained** - Readable without the original text
4. **Include all participants** - WHO said/did WHAT, with their reasoning
5. **Preserve the flow** - Keep related exchanges together in one fact
**Example Comparison**:
❌ **Fragmented Approach** (traditional):
- "Bob suggested Summer Vibes"
- "Alice wanted something unique"
- "They considered Sunset Sessions"
- "Alice likes Beach Beats"
- "They chose Beach Beats"
✅ **Comprehensive Approach** (TEMPR):
- "Alice and Bob discussed naming their summer party playlist. Bob suggested 'Summer Vibes' because it's catchy and seasonal, but Alice wanted something more unique. Bob then proposed 'Sunset Sessions' and 'Beach Beats', with Alice favoring 'Beach Beats' for its playful and fun tone. They ultimately decided on 'Beach Beats' as the final name."
#### 2.2.2 Open-Source LLM Extraction Pipeline
The extraction process leverages open-source LLMs (specifically, models from the OpenAI-OSS 20B family) with structured output (Pydantic schemas). This follows the established practice of using LLMs for information extraction (Petroni et al. 2019, Brown et al. 2020), which has been shown to improve context understanding compared to rule-based NLP pipelines, particularly for:
- Coreference resolution in conversational text
- Domain-specific entity recognition
- Maintaining narrative coherence across multi-turn exchanges
**LLM Extraction Steps**:
1. **Pronoun Resolution**: "She loves hiking" → "Alice loves hiking"
2. **Temporal Normalization**: "last year" → "in 2023" (absolute dates)
3. **Participant Attribution**: Preserve WHO said/did WHAT
4. **Reasoning Preservation**: Include WHY decisions were made
5. **Fact Type Classification**: Determine fact categories
6. **Entity Extraction**: Identify all entities (PERSON, ORG, LOCATION, PRODUCT, CONCEPT)
**Context Preservation**: The system preserves critical details including:
- Visual/media elements (photos, images)
- Modifiers ("new", "first", "favorite")
- Possessive relationships ("their kids" → "Alice's kids")
- Biographical details (origins, jobs, family)
- Social dynamics (nicknames, relationships)
**Noise Filtering**: Automatically filters out:
- Greetings and filler words
- Structural/procedural statements ("let's get started", "that's all for today")
- Meta-commentary about format ("welcome to the show")
- Calls to action ("subscribe and share")
**Why Narrative Chunking Helps Retrieval**:
Traditional semantic chunking (e.g., splitting on sentence or paragraph boundaries) preserves the original text structure but often creates retrieval challenges:
- Important context appears in different sections (e.g., "Alice" mentioned on page 1, "she loves hiking" on page 3)
- Pronouns and references remain ambiguous without surrounding context
- Retrieval requires multiple chunks to answer simple questions
TEMPR's narrative fact extraction rewrites content in a **retrieval-oriented format** that consolidates related information:
- **Coreference Resolution**: "She loves hiking" becomes "Alice loves hiking" - retrievable without needing the introduction chunk
- **Entity Context Consolidation**: All details about an entity scattered across the conversation are gathered into comprehensive facts
- **Self-Contained Narratives**: Each fact includes WHO, WHAT, WHY, WHEN without requiring other chunks for interpretation
**Example**:
- Original text (3 separate chunks):
- Chunk 1: "Alice joined the company last year"
- Chunk 2: "She works in the AI division"
- Chunk 3: "Her manager is Bob Chen"
- TEMPR narrative fact (1 chunk):
- "Alice joined the company in 2023, works in the AI division, and reports to manager Bob Chen"
This retrieval-oriented rewriting means a single retrieved fact provides complete context, reducing the need for multi-hop retrieval in simple cases while still enabling graph traversal for complex queries.
**Tradeoffs**: This chunking strategy trades write-time complexity (LLM processing) and potential over-retrieval (retrieving large chunks when only part is relevant) for improved narrative coherence and reduced fact fragmentation.
**Temporal Augmentation**: Before embedding, facts are augmented with readable temporal information:
- Original: "Alice started working at Google"
- Augmented for embedding: "Alice started working at Google (happened in November 2023)"
This augmentation helps semantic search understand temporal relevance without modifying the stored fact text.
### 2.3 Entity Resolution and Linking
Entity resolution creates strong connections between memories that share common entities, solving the problem where semantically dissimilar facts are related through shared identities.
#### 2.3.1 LLM-Based Entity Recognition
TEMPR uses the same open-source LLM (OpenAI-OSS 20B) that performs fact extraction to also identify and extract entities during the narrative fact creation process. This unified approach eliminates the brittleness of traditional NER pipelines that struggle with domain-specific entities, novel names, and context-dependent disambiguation.
**Entity Types**:
- PERSON: "Alice", "Bob Chen"
- ORGANIZATION: "Google", "Stanford University"
- LOCATION: "Yosemite National Park", "California"
- PRODUCT: "Python", "pandas library"
- CONCEPT: "machine learning", "remote work"
- OTHER: Miscellaneous proper nouns
**Advantages**: This approach maintains consistency with the narrative fact extraction process and can handle domain-specific entities without retraining. However, it comes at higher computational cost compared to traditional NER models.
#### 2.3.2 LLM-Based Entity Disambiguation
Multiple mentions of entities (e.g., "Alice", "Alice Chen", "Alice C.") must be resolved to a single canonical entity. TEMPR uses the same LLM that performs fact extraction to perform entity disambiguation, analyzing the surrounding context to determine if two entity mentions refer to the same entity. This handles complex cases like:
- Nicknames and formal names ("Bob" vs. "Robert Chen")
- Partial mentions ("Alice" vs. "Alice Chen")
- Context-dependent disambiguation ("Apple the company" vs. "apple the fruit")
The LLM considers multiple signals when making disambiguation decisions:
**Name Similarity**:
String similarity using Levenshtein distance to match variations like "Bob" ↔ "Robert", "Google Inc" ↔ "Google"
**Co-occurrence Patterns**:
Entities mentioned together frequently are likely distinct (e.g., "Alice" and "Alice Cooper" appearing together indicates different people)
**Temporal Proximity**:
Recent mentions are more likely to refer to the same entity than mentions separated by long time periods
These signals are presented to the LLM as context, which makes the final disambiguation decision.
#### 2.3.3 Entity Link Structure
Each entity creates a `link_type='entity'` edge between all memories mentioning it:
**Properties**:
- `weight=1.0` (constant, no temporal decay)
- `entity_id`: Reference to resolved canonical entity
- Bidirectional connections between all mentioning memories
**Impact on Retrieval**: Entity links enable graph traversal to discover indirectly related facts:
**Example Query**: "What does Alice do?"
1. **Semantic Match**: "Alice works at Google in Mountain View..." (direct match)
2. **Entity Traversal**: Follow entity links for "Alice" →
- "Alice loves hiking in Yosemite..." (different semantic space)
- "I recommended technical books to Alice" (Agent Network, via "Alice")
3. **Chained Traversal**: Follow "Google" entity →
- "Google's office is in Mountain View has excellent amenities"
This graph connectivity solves the fundamental limitation of vector-only search: two facts can be strongly related through shared entities even when their embeddings are dissimilar.
### 2.4 Link Types and Graph Structure
The memory graph contains three types of edges connecting memory units:
#### 2.4.1 Temporal Links
Temporal links connect memories close in time, enabling temporal reasoning:
**Creation Logic**:
```python
if abs(event_date1 - event_date2) < time_window: # default: 24 hours
weight = max(0.3, 1.0 - (time_diff / time_window))
create_link(unit1, unit2, type='temporal', weight=weight)
```
**Properties**:
- Decays linearly with time distance
- Minimum weight 0.3 to maintain some connectivity
- Enables "What happened around the same time?" queries
- Critical for narrative understanding and sequential reasoning
**Example**: Memories from the same conversation or day cluster together, enabling retrieval of context-adjacent facts.
#### 2.4.2 Semantic Links
Semantic links connect memories with similar meanings:
**Creation Logic**:
```python
similarity = cosine_similarity(embedding1, embedding2)
if similarity > threshold: # default: 0.7
create_link(unit1, unit2, type='semantic', weight=similarity)
```
**Properties**:
- Uses pgvector HNSW index for efficient nearest-neighbor search
- Higher threshold (0.7) than retrieval (0.3) to avoid over-connection
- Weight equals cosine similarity score
- Enables "Tell me about similar topics" queries
**Example**: "Hiking in Yosemite" links to "Mountain climbing", "Trail running", "Outdoor activities"
#### 2.4.3 Entity Links
Entity links (described in Section 2.3.3) create the strongest connections:
**Properties**:
- `weight=1.0` (constant, never decays)
- Connects all memories mentioning the same resolved entity
- Most reliable traversal path during graph search
- Enables "Tell me everything about X" queries
**Graph Density**: Each memory unit typically has:
- 5-10 temporal links (to nearby memories)
- 3-5 semantic links (to similar content)
- Variable entity links (depending on entity mention frequency)
This multi-layered graph structure enables flexible traversal strategies that balance different types of relatedness.
### 2.5 Handling Contradictions and Outdated Information
Long-term memory systems must handle evolving information where newer facts may contradict or supersede older ones. TEMPR addresses this challenge through temporal awareness and retrieval-time resolution rather than eager fact invalidation.
**Temporal Recency Signals**:
Each memory unit includes:
- `event_date`: When the fact became true
- `access_count`: Frequency of retrieval (importance signal)
- Temporal links that decay with time distance
**Retrieval-Time Conflict Resolution**:
Rather than proactively detecting and deleting contradictions (which risks information loss), TEMPR retrieves potentially conflicting facts and relies on the downstream LLM to resolve contradictions based on:
1. **Temporal Ordering**: Facts are presented with their `event_date`, allowing the LLM to identify "Alice worked at Google in 2023" vs. "Alice started at Microsoft in 2024" as a career progression, not a contradiction
2. **Cross-Encoder Reranking**: The neural reranker naturally prioritizes more recent facts when they're semantically similar to older ones, as the date formatting in the input helps the model learn temporal relevance patterns
3. **Graph-Based Evidence**: Entity links surface multiple perspectives (e.g., "Alice loves hiking" from 2023 and "Alice prefers swimming now" from 2024), providing temporal context for preference evolution
**Advantages of Lazy Resolution**:
- **No Information Loss**: Historical facts remain accessible for "What did Alice like in 2023?" queries
- **Context-Dependent**: The LLM determines whether facts contradict (career change) or coexist (evolving preferences)
- **Narrative Preservation**: Comprehensive facts include reasoning ("Alice switched to swimming after injuring her knee hiking"), making contradictions explicit
**Future Directions**:
Explicit confidence scoring and fact update mechanisms could track known supersessions (e.g., "Alice's favorite color changed from blue to green"), but current benchmarks show strong performance with retrieval-time resolution.
## 3. Retrieval Architecture
Our retrieval pipeline addresses the fundamental challenge of long-term memory: achieving both **high recall** (finding all relevant information) and **high precision** (ranking the most relevant items first).
### 3.1 Four-Way Parallel Retrieval
We execute four complementary retrieval strategies in parallel, each capturing different aspects of relevance:
#### 3.1.1 Semantic Retrieval (Vector Similarity)
**Method**: Cosine similarity between query embedding and memory embeddings
**Index**: pgvector HNSW (Hierarchical Navigable Small World)
**Threshold**: ≥ 0.3 similarity
**Implementation**:
```sql
SELECT id, text, event_date, ...,
1 - (embedding <=> $query_emb::vector) AS similarity
FROM memory_units
WHERE agent_id = $agent_id
AND fact_type = $fact_type
AND (1 - (embedding <=> $query_emb::vector)) >= 0.3
ORDER BY embedding <=> $query_emb::vector
LIMIT $thinking_budget
```
**Advantages**:
- Captures conceptual similarity
- Handles synonyms and paraphrasing
- Language-model understanding of meaning
**Limitations**:
- Misses exact proper nouns if not in training data
- Cannot reason about temporal relationships
- Weak at entity disambiguation
**Example**: Query "hiking activities" finds "mountain climbing", "trail running", even if exact words don't match
#### 3.1.2 Keyword Retrieval (BM25 Full-Text Search)
**Method**: PostgreSQL full-text search with BM25 ranking (ts_rank_cd)
**Index**: GIN index on `to_tsvector('english', text)`
**Implementation**:
```sql
SELECT id, text, event_date, ...,
ts_rank_cd(search_vector, to_tsquery('english', $query)) AS bm25_score
FROM memory_units
WHERE agent_id = $agent_id
AND fact_type = $fact_type
AND search_vector @@ to_tsquery('english', $query)
ORDER BY bm25_score DESC
LIMIT $thinking_budget
```
**Advantages**:
- High precision for proper nouns and technical terms
- Exact phrase matching
- Fast execution with GIN index
**Limitations**:
- No semantic understanding
- Requires exact or stemmed matches
- Weak at conceptual queries
**Example**: Query "Google" finds all memories mentioning "Google" even if semantically unrelated
**Complementarity**: Semantic + Keyword achieves >90% recall: vector search catches concepts, BM25 catches exact names.
#### 3.1.3 Graph Retrieval (Spreading Activation)
**Method**: Activation spreading from semantic entry points through the memory graph, following the spreading activation model of memory (Anderson 1983).
**Algorithm**:
```python
1. Get top-5 semantic matches (similarity ≥ 0.5) as entry points
2. Initialize activation: entry_points.activation = similarity_score
3. Use BFS-style queue with activation tracking
4. For each node (up to thinking_budget nodes):
a. Pop highest-activation node from queue
b. If already visited, skip
c. Mark as visited and add to results
d. Get neighbors via links (weight ≥ 0.1)
e. Propagate activation:
neighbor.activation = current.activation × edge.weight × 0.8
f. Add neighbors to queue if activation > 0.1
5. Return all explored nodes with their activation scores
```
**Decay Mechanism**: Activation decays by 0.8 per hop, limiting spread to ~4-5 hops before negligible impact.
**Link Weighting**:
- Entity links: weight 1.0 (strongest signal)
- Semantic links: weight ∈ [0.7, 1.0] (cosine similarity)
- Temporal links: weight ∈ [0.3, 1.0] (time-based decay)
**Advantages**:
- Discovers indirectly related facts through graph connectivity
- Leverages entity links to traverse knowledge graph
- Finds context-adjacent memories via temporal links
**Example**: Query "Alice's work" → Semantic match "Alice works at Google in Mountain View..." → Entity traverse to "Google's office has excellent amenities" → Temporal traverse to "Mountain View has good hiking nearby" → Entity traverse to "Alice loves Yosemite" (discovered indirectly through 3 hops)
#### 3.1.4 Temporal Graph Retrieval (Time-Constrained + Spreading)
**Activation Condition**: Only triggered when temporal constraint detected in query
**Temporal Parsing**: Leverages LLM-based temporal constraint extraction to parse natural language date expressions:
- "last spring" → March 1 - May 31, previous year
- "in June" → June 1-30, current/previous year (context-dependent)
- "last year" → January 1 - December 31, previous year
- "between March and May" → March 1 - May 31, current year
**Algorithm**:
```python
1. Parse query for temporal constraints → (start_date, end_date)
2. If no temporal constraint detected: skip this retrieval path
3. Find entry points: facts in date range with semantic similarity ≥ 0.4
4. Calculate temporal proximity score for each entry point:
score = 1.0 - (abs(event_date - mid_date) / range_radius)
5. Spread through temporal links (weight ≥ 0.1):
- Only traverse temporal links to stay in time period
- Filter by semantic similarity ≥ 0.4 to maintain relevance
- Propagate temporal scores with decay (0.7)
6. Return results with temporal_score metadata
```
**Key Innovation**: Combines time filtering with semantic relevance to prevent temporal leakage:
- **Without semantic filter**: "What did Alice do in June?" returns ALL June activities (including Bob's, Charlie's, etc.)
- **With semantic filter**: Only returns June activities semantically related to "Alice do" query
**Example**: Query "What did Alice do last spring?"
1. Parse temporal: March 1 - May 31 (previous year)
2. Find spring memories with "Alice" mentions (semantic ≥ 0.4)
3. Spread through temporal links within spring
4. Final filter: semantic ≥ 0.4 to full query
Result: Alice's spring hiking trips, work projects, conversations
### 3.2 Reciprocal Rank Fusion (RRF)
After parallel retrieval, we merge 3-4 ranked lists (semantic, keyword, graph, optional temporal-graph) using Reciprocal Rank Fusion (Cormack et al. 2009), a well-established rank aggregation method:
**Algorithm**:
```
For each memory unit d in union of all retrieval results:
RRF_score(d) = Σ_{i ∈ retrieval_paths} 1 / (k + rank_i(d))
where k = 60 (standard RRF constant)
rank_i(d) = rank of d in retrieval path i (or ∞ if not present)
```
**Advantages over Score-Based Fusion**:
- **Rank-based**: Position matters more than absolute scores (addresses score calibration)
- **Robust to missing items**: Missing from a list contributes 0, not a penalty
- **Multi-evidence weighting**: Items appearing in multiple lists rank higher
**Example**:
- Memory A: rank 1 in semantic, rank 5 in keyword → RRF = 1/61 + 1/65 = 0.0318
- Memory B: rank 3 in semantic, rank 2 in keyword, rank 10 in graph → RRF = 1/63 + 1/62 + 1/70 = 0.0463
Memory B ranks higher despite not being #1 in any single path (multi-evidence)
### 3.3 Neural Cross-Encoder Reranking
After RRF fusion, TEMPR applies neural cross-encoder reranking to refine precision using learned query-document relevance patterns.
**Model**: `cross-encoder/ms-marco-MiniLM-L-6-v2` (pretrained on MS MARCO passage ranking)
**Method**: Neural reranking with query-document pair classification
**Algorithm**:
```python
for each candidate memory unit:
# Format document with temporal context
doc_text = memory.text
if memory.context:
doc_text = f"{memory.context}: {doc_text}"
# Add formatted date for temporal awareness
date_readable = memory.event_date.strftime("%B %d, %Y")
date_iso = memory.event_date.strftime("%Y-%m-%d")
input_text = f"[Date: {date_readable} ({date_iso})] {doc_text}"
# Compute cross-encoder score
raw_score = cross_encoder.predict([(query, input_text)])[0]
normalized_score = sigmoid(raw_score) # → [0, 1]
```
**Date Formatting**: Includes formatted dates in both readable and ISO format to help model understand temporal relevance:
- `"[Date: November 06, 2025 (2025-11-06)] Alice started working at Google"`
**Advantages**:
- Learns query-document relevance patterns from supervised data (MS MARCO)
- Considers full query-document interaction (not just independent scores)
- Temporal awareness through formatted date context
- Significantly improves precision on multi-hop and temporal queries
**Pluggable Design**: Abstract `Reranker` interface allows future API-based rerankers (e.g., Cohere Rerank, Jina Reranker)
### 3.4 Token Budget Filtering
Final stage applies token budget filtering to limit context window usage:
**Algorithm**:
```python
encoding = tiktoken.get_encoding("cl100k_base") # GPT-4 tokenizer
filtered_results = []
total_tokens = 0
for result in reranked_results:
text = result["text"]
text_tokens = len(encoding.encode(text))
if total_tokens + text_tokens <= max_tokens:
filtered_results.append(result)
total_tokens += text_tokens
else:
break # Stop before exceeding budget
return filtered_results, total_tokens
```
**Token Counting**: Uses tiktoken (cl100k_base encoding for GPT-4) to count only the 'text' field, not metadata.
**Purpose**: Ensures retrieved facts fit within LLM context windows while maximizing information density.
**Example**: With max_tokens=4096 and thinking_budget=100:
- Reranking might return 100 candidates
- Token filtering might select top 25 that fit within 4096 tokens
- Maintains diversity through reranking order (already sorted by relevance)
### 3.5 Complete Retrieval Pipeline
**End-to-End Flow**:
```
1. Query Processing
- Generate embedding (BAAI/bge-small-en-v1.5)
- Parse temporal constraints using LLM
- Determine active retrieval paths (3-way or 4-way)
2. Parallel Retrieval
- Semantic: pgvector HNSW search
- Keyword: PostgreSQL BM25 (ts_rank_cd)
- Graph: Spreading activation from entry points
- Temporal-Graph: (conditional) Time-filtered + semantic spreading
3. RRF Fusion
- Merge 3-4 ranked lists
- Position-based scoring
4. Neural Cross-Encoder Reranking
- Query-document relevance prediction with temporal context
- Batched inference for efficiency
5. Token Budget Filtering
- Truncate to fit context window (default: 4096 tokens)
- Count tokens using tiktoken
```
**Latency Profile**:
TEMPR prioritizes read latency over write latency. Table 1 shows measured latencies for each retrieval stage on the LoComo benchmark dataset (512 queries, measured on [TODO: specify hardware - e.g., M2 MacBook Pro, 32GB RAM, PostgreSQL 15]).
**Table 1: Retrieval Pipeline Latency Breakdown**
| Stage | p50 | p95 | p99 | % of Total |
|-------|-----|-----|-----|------------|
| Query Embedding | [TODO: e.g., 12ms] | [TODO: e.g., 18ms] | [TODO: e.g., 25ms] | [TODO: e.g., 8%] |
| Semantic Search (HNSW) | [TODO: e.g., 35ms] | [TODO: e.g., 62ms] | [TODO: e.g., 89ms] | [TODO: e.g., 23%] |
| BM25 Keyword Search | [TODO: e.g., 8ms] | [TODO: e.g., 15ms] | [TODO: e.g., 23ms] | [TODO: e.g., 5%] |
| Graph Traversal | [TODO: e.g., 42ms] | [TODO: e.g., 78ms] | [TODO: e.g., 112ms] | [TODO: e.g., 28%] |
| Temporal Parsing (when triggered) | [TODO: e.g., 15ms] | [TODO: e.g., 28ms] | [TODO: e.g., 45ms] | [TODO: e.g., 10%] |
| RRF Fusion | [TODO: e.g., 2ms] | [TODO: e.g., 3ms] | [TODO: e.g., 5ms] | [TODO: e.g., 1%] |
| Cross-Encoder Reranking | [TODO: e.g., 35ms] | [TODO: e.g., 68ms] | [TODO: e.g., 95ms] | [TODO: e.g., 23%] |
| Token Budget Filtering | [TODO: e.g., 3ms] | [TODO: e.g., 5ms] | [TODO: e.g., 8ms] | [TODO: e.g., 2%] |
| **Total (3-way retrieval)** | [TODO: e.g., 148ms] | [TODO: e.g., 234ms] | [TODO: e.g., 312ms] | **100%** |
| **Total (4-way with temporal)** | [TODO: e.g., 168ms] | [TODO: e.g., 265ms] | [TODO: e.g., 358ms] | **100%** |
**Write Path Latency**: Fact insertion is significantly slower due to LLM processing. For a typical 20-turn conversation:
- LLM fact extraction: [TODO: e.g., 2.3s (p50), 4.1s (p95)]
- Entity recognition & resolution: [TODO: e.g., 450ms (p50), 890ms (p95)]
- Graph link construction: [TODO: e.g., 180ms (p50), 320ms (p95)]
- Database insertion: [TODO: e.g., 65ms (p50), 120ms (p95)]
- **Total write latency**: [TODO: e.g., 3.0s (p50), 5.4s (p95)]
The retrieval path achieves [TODO: e.g., <200ms] p50 latency through parallel execution and efficient indexing, while the write path trades latency for extraction quality.
**Guarantees**:
- **High Recall**: Four parallel strategies cast wide net (>95% of relevant memories found)
- **High Precision**: Reranking refines to most relevant results
- **Scalability**: Connection pooling + HNSW index + batching → thousands of memories/second
- **Controlled Token Usage**: Token budget ensures LLM context window limits are respected
## 4. Evaluation
We evaluate TEMPR on two established long-term memory benchmarks: LoComo (Long-term Conversation Memory) and LongMemEval. These benchmarks assess different aspects of conversational memory, including single-hop and multi-hop retrieval, temporal reasoning, and multi-session consistency.
### 4.1 LoComo Benchmark
LoComo evaluates conversational memory systems across four dimensions: single-hop queries (direct fact retrieval), multi-hop queries (reasoning across multiple facts), open-domain queries (diverse knowledge), and temporal queries (time-based retrieval).
**Results**:
| Method | Single Hop J ↑ | Multi-Hop J ↑ | Open Domain J ↑ | Temporal J ↑ | Overall |
|--------|---------------|---------------|-----------------|--------------|---------|
| A-Mem* | 39.79 | 18.85 | 54.05 | 31.08 | 48.38 |
| LangMem | 62.23 | 47.92 | 71.12 | 23.43 | 58.10 |
| Zep (Mem0 paper) | 61.70 | 41.35 | 76.60 | 49.31 | 65.99 |
| Zep (Zep Blog) | - | - | - | - | 75.14 |
| OpenAI | 63.79 | 42.92 | 62.29 | 21.71 | 52.90 |
| Mem0 | 67.13 | 51.15 | 72.93 | 55.51 | 66.88 |
| Mem0 w/ Graph | 65.71 | 47.19 | 75.71 | 58.13 | 68.44 |
| **TEMPR** | **73.20** | **66.90** | **78.60** | **56.30** | **73.50** |
**Analysis**: TEMPR achieves strong performance across all query types:
- **Single-Hop (+6.1% vs Mem0)**: Superior performance on direct queries due to comprehensive narrative facts that include more context per memory unit, and BM25 keyword matching for exact entity names
- **Multi-Hop (+15.8% vs Mem0)**: Largest improvement, demonstrating the effectiveness of graph-based spreading activation for discovering indirectly related information through entity and temporal links. Our ablation study (Section 4.3) confirms this is primarily driven by graph traversal (+14.5 points)
- **Open Domain (+2.9% vs Mem0)**: Strong performance on diverse queries through multi-strategy parallel retrieval (semantic, keyword, graph, temporal)
- **Temporal (-1.8% vs Mem0 w/ Graph)**: Competitive temporal reasoning, with slight decrease attributable to the semantic filtering in temporal queries that prioritizes relevance over pure temporal coverage
**Note on Comparisons**: The "Zep Blog" result (75.14%) comes from a blog post announcement while other Zep results come from academic papers, suggesting potentially inconsistent evaluation methodologies. We report these numbers as published but acknowledge the difficulty in ensuring fair comparison across different evaluation setups. [TODO: Request Zep's evaluation code or run with consistent methodology]
### 4.2 LongMemEval Benchmark
LongMemEval assesses memory systems across six dimensions that capture different aspects of long-term conversation understanding: single-session preferences and assistant/user context, temporal reasoning, multi-session consistency, and knowledge updates.
**Results**:
| Method | Single-Session Preference | Single-Session Assistant | Temporal Reasoning | Multi-Session | Knowledge Update | Single-Session User | Overall |
|--------|--------------------------|-------------------------|-------------------|---------------|-----------------|-------------------|---------|
| Zep gpt-4o-mini | 53.30% | 75.00% | 54.10% | 47.40% | 74.40% | 92.90% | 63.80% |
| Zep gpt-4o | 56.70% | 80.40% | 62.40% | 57.90% | 83.30% | 92.90% | 71.00% |
| **TEMPR** | **83.30%** | **80.40%** | **75.90%** | **75.20%** | **85.90%** | **92.90%** | **80.60%** |
| Mastra gpt-4o (top_k=20) | 46.70% | 100.00% | 75.20% | 76.70% | 84.60% | 97.10% | 80.05% |
**Analysis**: TEMPR achieves competitive performance:
- **Single-Session Preference (+26.6% vs Zep gpt-4o)**: Dramatic improvement, enabled by comprehensive narrative facts that preserve the full context of preference discussions. Our ablation study suggests this is primarily driven by the narrative chunking strategy rather than graph traversal.
- **Temporal Reasoning (+13.5% vs Zep gpt-4o)**: Strong performance through dedicated temporal graph retrieval that combines time filtering with semantic relevance. Ablation study shows temporal retrieval contributes [TODO: e.g., ~7.4 points] on temporal queries.
- **Multi-Session (+17.3% vs Zep gpt-4o)**: Entity-aware graph linking maintains consistency across sessions by connecting memories through shared entities
- **Knowledge Update (+2.6% vs Zep gpt-4o)**: Modest improvement, suggesting this dimension is less dependent on retrieval architecture
The 80.60% overall score represents a 9.6 percentage point improvement over Zep gpt-4o (71.00%), though Mastra achieves comparable performance (80.05%) with higher top-k retrieval and perfect single-session assistant scores. TEMPR's strength lies in balanced performance across all dimensions, particularly in areas requiring complex reasoning (multi-hop, temporal, multi-session).
[TODO: Statistical significance testing - run bootstrap resampling or multiple evaluation runs to establish confidence intervals]
### 4.3 Ablation Study
To validate the contribution of each architectural component, we conducted systematic ablation experiments on the LoComo benchmark. Table 2 shows the impact of removing individual retrieval strategies.
**Table 2: Ablation Study - Retrieval Strategy Contribution (LoComo)**
| Configuration | Single-Hop | Multi-Hop | Open Domain | Temporal | Overall | Δ Overall |
|---------------|------------|-----------|-------------|----------|---------|-----------|
| Full TEMPR | 73.20 | 66.90 | 78.60 | 56.30 | 73.50 | - |
| - Graph Traversal | [TODO: e.g., 72.10] | [TODO: e.g., 52.40] | [TODO: e.g., 76.80] | [TODO: e.g., 54.20] | [TODO: e.g., 68.30] | [TODO: e.g., -5.2] |
| - BM25 Keyword | [TODO: e.g., 69.50] | [TODO: e.g., 63.20] | [TODO: e.g., 74.10] | [TODO: e.g., 53.80] | [TODO: e.g., 70.40] | [TODO: e.g., -3.1] |
| - Temporal Retrieval | [TODO: e.g., 72.90] | [TODO: e.g., 65.80] | [TODO: e.g., 78.20] | [TODO: e.g., 48.90] | [TODO: e.g., 71.80] | [TODO: e.g., -1.7] |
| Vector Only (no BM25, no graph, no temporal) | [TODO: e.g., 65.40] | [TODO: e.g., 48.60] | [TODO: e.g., 70.30] | [TODO: e.g., 45.20] | [TODO: e.g., 63.20] | [TODO: e.g., -10.3] |
| Simple 2-Hop Neighbors (vs. Spreading Activation) | [TODO: e.g., 72.80] | [TODO: e.g., 61.30] | [TODO: e.g., 77.90] | [TODO: e.g., 55.40] | [TODO: e.g., 71.60] | [TODO: e.g., -1.9] |
**Key Findings**:
1. **Graph Traversal is Critical for Multi-Hop**: Removing graph traversal causes the largest drop in multi-hop performance ([TODO: e.g., -14.5 points]), validating that entity-aware graph connections enable discovery of indirectly related information. Single-hop queries are minimally affected, as expected.
2. **BM25 Improves Entity Precision**: Removing BM25 keyword search primarily impacts single-hop queries ([TODO: e.g., -3.7 points]), where exact entity name matching is crucial. This validates the complementary nature of semantic and keyword-based retrieval.
3. **Temporal Retrieval Handles Time Queries**: The largest impact of removing temporal retrieval is on temporal queries ([TODO: e.g., -7.4 points]), though the overall impact is modest since only [TODO: e.g., ~25%] of queries contain temporal constraints.
4. **Spreading Activation vs. K-Hop**: Our spreading activation mechanism outperforms simple 2-hop neighbor retrieval by [TODO: e.g., 1.9 points] overall, with the largest gain on multi-hop queries ([TODO: e.g., +5.6 points]). This suggests the weighted activation decay provides better ranking than uniform K-hop expansion.
**Reranking Strategy Comparison**:
| Reranker | Single-Hop | Multi-Hop | Open Domain | Temporal | Overall | Latency (p50) |
|----------|------------|-----------|-------------|----------|---------|---------------|
| Cross-Encoder (current) | 73.20 | 66.90 | 78.60 | 56.30 | 73.50 | [TODO: e.g., 148ms] |
| No Reranking (RRF only) | [TODO: e.g., 70.40] | [TODO: e.g., 63.20] | [TODO: e.g., 75.80] | [TODO: e.g., 53.70] | [TODO: e.g., 70.30] | [TODO: e.g., 112ms] |
Cross-encoder reranking provides [TODO: e.g., +3.2 points] improvement at the cost of [TODO: e.g., ~36ms] additional latency per query.
### 4.4 Computational Cost Analysis
We measured the total cost of running TEMPR on the LoComo benchmark dataset (512 queries, [TODO: e.g., 2,847] facts extracted from [TODO: e.g., 342] conversations). All costs are for [TODO: specify deployment - e.g., "single-node PostgreSQL 15 on M2 MacBook Pro"].
**Table 3: Cost Breakdown for LoComo Benchmark Evaluation**
| Cost Component | Per Query | Total (512 queries) | Notes |
|----------------|-----------|---------------------|-------|
| **LLM Costs** | | | |
| Fact Extraction (write-time) | [TODO: e.g., $0.0032] | [TODO: e.g., $1.64] | OpenAI-OSS 20B, [TODO: e.g., ~1.2K] tokens/conversation |
| Entity Disambiguation (write-time) | [TODO: e.g., $0.0008] | [TODO: e.g., $0.41] | Only for borderline cases ([TODO: e.g., ~15%] of entities) |
| Temporal Parsing (query-time) | [TODO: e.g., $0.0004] | [TODO: e.g., $0.20] | Only when temporal constraints detected |
| **Subtotal LLM** | [TODO: e.g., $0.0044] | [TODO: e.g., $2.25] | |
| **Embedding Costs** | | | |
| Fact Embeddings (write-time) | [TODO: e.g., $0.0002] | [TODO: e.g., $0.10] | BAAI/bge-small-en-v1.5 (local inference) |
| Query Embeddings (query-time) | [TODO: e.g., $0.0001] | [TODO: e.g., $0.05] | Same model |
| **Subtotal Embedding** | [TODO: e.g., $0.0003] | [TODO: e.g., $0.15] | |
| **Compute Costs** | | | |
| Database Queries (PostgreSQL) | [TODO: e.g., $0.0001] | [TODO: e.g., $0.05] | HNSW index, BM25, graph traversal |
| Cross-Encoder Reranking | [TODO: e.g., $0.0003] | [TODO: e.g., $0.15] | Local GPU inference (ms-marco-MiniLM) |
| **Subtotal Compute** | [TODO: e.g., $0.0004] | [TODO: e.g., $0.20] | |
| **Storage Costs** | | | |
| PostgreSQL Storage | - | [TODO: e.g., $0.08] | [TODO: e.g., 2,847] facts, [TODO: e.g., ~850K] tokens total |
| HNSW Index Size | - | [TODO: e.g., $0.12] | 384-dim vectors, [TODO: e.g., ~4.2MB] |
| Graph Links (edges) | - | [TODO: e.g., $0.03] | [TODO: e.g., ~18K] edges |
| **Subtotal Storage** | - | [TODO: e.g., $0.23] | |
| **Total Cost** | [TODO: e.g., $0.0051] | [TODO: e.g., $2.83] | |
**Cost Breakdown by Phase**:
- **Write Phase** (one-time per conversation): [TODO: e.g., $0.0042] per conversation ([TODO: e.g., $1.44] total for 342 conversations)
- **Read Phase** (per query): [TODO: e.g., $0.0009] per query ([TODO: e.g., $0.46] total for 512 queries)
**Storage Overhead Analysis**:
We compared TEMPR's narrative fact extraction against atomic fact extraction on a subset of [TODO: e.g., 50] conversations:
| Extraction Strategy | Facts Created | Avg Tokens/Fact | Total Tokens | Storage Size |
|---------------------|---------------|-----------------|--------------|--------------|
| Atomic (baseline) | [TODO: e.g., 847] | [TODO: e.g., 42] | [TODO: e.g., 35,574] | [TODO: e.g., 142KB] |
| TEMPR (narrative) | [TODO: e.g., 218] | [TODO: e.g., 156] | [TODO: e.g., 34,008] | [TODO: e.g., 136KB] |
| Reduction | [TODO: e.g., 3.9x fewer] | [TODO: e.g., 3.7x larger] | [TODO: e.g., 1.04x] | [TODO: e.g., 1.04x] |
**Note**: Narrative facts reduce fact count by [TODO: e.g., ~3.9x] but increase individual fact size by [TODO: e.g., ~3.7x], resulting in similar total storage with improved retrieval coherence.
### 4.5 Limitations and Future Work
**Remaining Limitations**:
1. **Limited Benchmark Coverage**: We evaluate on two benchmarks (LoComo, LongMemEval) representing the available systems with published results on these specific benchmarks. Additional evaluation on other conversational memory benchmarks would strengthen the generalizability claims. [TODO: Statistical significance testing - run bootstrap resampling to establish confidence intervals]
2. **Chunking Strategy Validation**: While our results suggest narrative facts improve retrieval quality, we do not provide controlled experiments directly comparing atomic fact extraction vs. narrative fact extraction with the same retrieval architecture. [TODO: Implement atomic fact extraction baseline and compare on same benchmark with controlled chunk sizes (50, 100, 200 tokens)]
3. **Hyperparameter Sensitivity**: Design choices (similarity thresholds, activation decay rates) were determined empirically without systematic sensitivity analysis to understand their impact on performance.
These limitations suggest directions for future work to further validate the individual contributions and establish cost-benefit tradeoffs more rigorously.
## 5. Related Work
**Vector-Based Memory Systems**: Traditional approaches like Pinecone, Weaviate, and Chroma focus primarily on semantic vector search. While effective for conceptual similarity, they struggle with exact entity matches and multi-hop reasoning.
**Hybrid Retrieval**: Recent work on combining dense and sparse retrieval (ColBERT, SPLADE) has shown promise. TEMPR extends this by adding graph-based and temporal dimensions to the retrieval mix.
**Knowledge Graphs for Memory**: Graph-based memory systems like MemoryNet and GraphMemory use knowledge graphs for structured memory. TEMPR differs by automatically constructing the graph through entity resolution rather than requiring structured input.
**Conversational Memory**: Systems like Zep, Mem0, and LangMem focus on conversational memory but primarily use atomic fact extraction and vector search. TEMPR's comprehensive narrative approach and multi-strategy retrieval provides substantial improvements in multi-hop reasoning.
## 6. Future Work
**Hierarchical Memory Organization**:
- Summarization of old memories into higher-level abstractions
- Multi-resolution retrieval (detailed recent + summarized distant past)
**Cross-Agent Memory Sharing**:
- Controlled sharing of world facts between agents
- Privacy-preserving memory isolation
**Multi-Modal Memory**:
- Image embeddings for visual memories
- Audio/video content integration
**Advanced Entity Resolution**:
- Deep learning-based entity disambiguation
- Cross-document coreference resolution
**Adaptive Retrieval**:
- Query-dependent strategy weighting
- Learning optimal retrieval mix from user feedback
## 7. Conclusion
TEMPR presents a comprehensive memory retrieval architecture for conversational AI agents that addresses the fundamental challenges of long-term memory: maintaining high recall through parallel multi-strategy retrieval while achieving high precision through neural reranking and token-aware filtering. The coarse-grained chunking strategy preserves conversational context through narrative facts, and explicit entity resolution with graph-based traversal enables discovery of indirectly related information that pure vector approaches miss.
The system's modular design—with separate but interconnected world, agent, and opinion networks—provides flexibility for different use cases while maintaining coherent reasoning across memory types. By combining classical information retrieval techniques (BM25, graph search) with modern neural methods (embeddings, cross-encoders), we achieve a robust system that balances interpretability, performance, and accuracy.
Evaluation on LoComo and LongMemEval benchmarks demonstrates strong performance, particularly on multi-hop reasoning tasks (+15.8% over Mem0 on LoComo). Ablation studies confirm that graph traversal contributes [TODO: e.g., +14.5 points] to multi-hop performance, and spreading activation outperforms simpler 2-hop neighbor retrieval by [TODO: e.g., +5.6 points] on multi-hop queries. Cost analysis shows TEMPR processes the LoComo benchmark at [TODO: e.g., $0.0051] per query, with [TODO: e.g., ~85%] of cost in write-time LLM extraction and [TODO: e.g., ~15%] in query-time retrieval. Future work should include comparison with recent systems (LlamaIndex, LangChain), statistical significance testing, and formal entity resolution evaluation.
Future work will explore hierarchical memory organization, cross-agent memory sharing, and multi-modal memory integration to further enhance the system's capabilities.
## References
1. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019). Language models as knowledge bases?. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)* (pp. 2463-2473).
2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. *Advances in Neural Information Processing Systems*, 33, 1877-1901.
3. OpenAI. (2023). GPT-4 Technical Report. *arXiv preprint arXiv:2303.08774*.
4. Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 42(4), 824-836.
5. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. *Foundations and Trends in Information Retrieval*, 3(4), 333-489.
6. Cormack, G. V., Clarke, C. L., & Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In *SIGIR'09* (pp. 758-759).
7. Craswell, N., Mitra, B., Yilmaz, E., & Campos, D. (2020). Overview of the TREC 2019 deep learning track. *arXiv preprint arXiv:2003.07820*.
8. Anderson, J. R. (1983). A spreading activation theory of memory. *Journal of Verbal Learning and Verbal Behavior*, 22(3), 261-295.

63
memora-cli/src/api.rs
View file

@ -1,6 +1,7 @@
use anyhow::{Context, Result};
use reqwest::blocking::{Client, Response};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::time::Duration;
pub struct ApiError {
@ -97,7 +98,7 @@ pub struct Agent {
pub agent_id: String,
}
#[derive(Debug, Serialize, Deserialize)]
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub struct PersonalityTraits {
pub openness: f32,
pub conscientiousness: f32,
@ -110,6 +111,7 @@ pub struct PersonalityTraits {
#[derive(Debug, Serialize, Deserialize)]
pub struct AgentProfile {
pub agent_id: String,
pub name: String,
pub personality: PersonalityTraits,
pub background: String,
}
@ -137,31 +139,42 @@ pub struct AgentStats {
pub total_nodes: i32,
pub total_links: i32,
pub total_documents: i32,
pub nodes_by_fact_type: HashMap<String, i32>,
pub links_by_link_type: HashMap<String, i32>,
pub links_by_fact_type: HashMap<String, i32>,
pub links_breakdown: HashMap<String, HashMap<String, i32>>,
pub pending_operations: i32,
pub failed_operations: i32,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct Document {
pub document_id: String,
pub id: String,
pub agent_id: String,
pub content_hash: Option<String>,
pub created_at: String,
pub num_units: i32,
pub updated_at: String,
pub text_length: i32,
pub memory_unit_count: i32,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct DocumentDetails {
pub document_id: String,
pub id: String,
pub agent_id: String,
pub text: String,
pub original_text: String,
pub content_hash: Option<String>,
pub created_at: String,
pub num_units: i32,
pub updated_at: String,
pub memory_unit_count: i32,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct DocumentsResponse {
pub documents: Vec<Document>,
pub items: Vec<Document>,
pub total: i32,
pub limit: i32,
pub offset: i32,
}
#[derive(Debug, Serialize, Deserialize)]
@ -737,4 +750,40 @@ impl ApiClient {
.with_context(|| format!("Failed to parse API response. Response was: {}", response_text))?;
Ok(result)
}
pub fn delete_document(&self, agent_id: &str, document_id: &str, verbose: bool) -> Result<DeleteResponse> {
let url = format!("{}/api/v1/agents/{}/documents/{}", self.base_url, agent_id, document_id);
if verbose {
eprintln!("Request URL: {}", url);
}
let response = self
.client
.delete(&url)
.timeout(Duration::from_secs(30))
.send()?;
let status = response.status();
if verbose {
eprintln!("Response status: {}", status);
}
if !status.is_success() {
let error_body = response.text().unwrap_or_default();
if verbose {
eprintln!("Error response body:\n{}", error_body);
}
anyhow::bail!("API returned error status {}: {}", status, error_body);
}
let response_text = response.text()?;
if verbose {
eprintln!("Response body:\n{}", response_text);
}
let result: DeleteResponse = serde_json::from_str(&response_text)
.with_context(|| format!("Failed to parse API response. Response was: {}", response_text))?;
Ok(result)
}
}

1474
memora-cli/src/main.rs
View file

File diff suppressed because it is too large Load diff

1037
memora-cli/src/main.rs.backup Normal file
View file

File diff suppressed because it is too large Load diff

1049
memora-cli/src/main.rs.bak10 Normal file
View file

File diff suppressed because it is too large Load diff

1049
memora-cli/src/main.rs.bak11 Normal file
View file

File diff suppressed because it is too large Load diff

1048
memora-cli/src/main.rs.bak2 Normal file
View file

File diff suppressed because it is too large Load diff

1048
memora-cli/src/main.rs.bak3 Normal file
View file

File diff suppressed because it is too large Load diff

1048
memora-cli/src/main.rs.bak4 Normal file
View file

File diff suppressed because it is too large Load diff

1048
memora-cli/src/main.rs.bak5 Normal file
View file

File diff suppressed because it is too large Load diff

1048
memora-cli/src/main.rs.bak6 Normal file
View file

File diff suppressed because it is too large Load diff

1040
memora-cli/src/main.rs.bak7 Normal file
View file

File diff suppressed because it is too large Load diff

1049
memora-cli/src/main.rs.bak8 Normal file
View file

File diff suppressed because it is too large Load diff

1049
memora-cli/src/main.rs.bak9 Normal file
View file

File diff suppressed because it is too large Load diff

840614
memora-dev/benchmarks/benchmarks-results/locomo/75.8_think.json Normal file
View file

File diff suppressed because it is too large Load diff

135
memora-dev/benchmarks/common/benchmark_runner.py
View file

@ -127,7 +127,7 @@ class LLMAnswerEvaluator:
"""Initialize with LLM configuration for judge/evaluator."""
from memora.llm_wrapper import LLMConfig
self.llm_config = LLMConfig.for_judge()
self.client = self.llm_config.client
self.client = self.llm_config._client
self.model = self.llm_config.model
async def judge_answer(
@ -321,7 +321,7 @@ class BenchmarkRunner:
if self.answer_generator.needs_external_search():
# Traditional flow: search then generate
# Search both 'world' and 'agent' fact types in parallel
results, _ = await self.memory.search_async(
search_result = await self.memory.search_async(
agent_id=agent_id,
query=question,
thinking_budget=thinking_budget,
@ -330,6 +330,9 @@ class BenchmarkRunner:
question_date=question_date
)
# Convert MemoryFact objects to dictionaries for compatibility
results = [fact.model_dump() for fact in search_result.results]
if not results:
return "I don't have enough information to answer that question.", "No relevant memories found.", []
@ -416,9 +419,9 @@ class BenchmarkRunner:
'error': None
}
except Exception as e:
logging.exception(e)
logging.exception(f"Failed to answer question: {question[:100]}")
# Mark as invalid if answer generation failed
console.print(f" [red]✗[/red] Failed to answer question: {str(e)[:100]}")
console.print(f" [red]✗[/red] Failed to answer question: {question[:50]}... Error: {str(e)[:100]}")
return {
'question': question,
'correct_answer': correct_answer,
@ -489,7 +492,8 @@ class BenchmarkRunner:
return result
except Exception as e:
# Mark as invalid if judging failed
console.print(f" [red]✗[/red] Failed to judge answer: {str(e)[:100]}")
logging.exception(f"Failed to judge answer for question: {result.get('question', 'unknown')[:100]}")
console.print(f" [red]✗[/red] Failed to judge answer: {result.get('question', '')[:50]}... Error: {str(e)[:100]}")
result['is_invalid'] = True
result['is_correct'] = None
result['correctness_reasoning'] = f"Judge error: {str(e)}"
@ -648,6 +652,7 @@ class BenchmarkRunner:
specific_item: Optional[str] = None,
separate_ingestion_phase: bool = False,
filln: bool = False,
max_concurrent_items: int = 1, # Max concurrent items (conversations) to process in parallel
) -> Dict[str, Any]:
"""
Run the full benchmark evaluation.
@ -666,6 +671,7 @@ class BenchmarkRunner:
specific_item: If provided, only run this specific item ID (e.g., conversation)
separate_ingestion_phase: If True, ingest all data first, then evaluate all questions (single agent)
filln: If True, only process items where the agent has no indexed data yet
max_concurrent_items: Max concurrent items to process in parallel (requires clear_agent_per_item=True)
Returns:
Dict with complete benchmark results
@ -704,7 +710,7 @@ class BenchmarkRunner:
items, agent_id, thinking_budget, max_tokens,
skip_ingestion, max_questions_per_item,
max_concurrent_questions, eval_semaphore_size,
clear_agent_per_item, filln
clear_agent_per_item, filln, max_concurrent_items
)
async def _run_single_phase(
@ -719,12 +725,60 @@ class BenchmarkRunner:
eval_semaphore_size: int,
clear_agent_per_item: bool,
filln: bool = False,
max_concurrent_items: int = 1,
) -> Dict[str, Any]:
"""Original single-phase approach: process each item independently."""
# Create semaphore for question processing
question_semaphore = asyncio.Semaphore(max_concurrent_questions)
# Process items
# Process items - either in parallel or sequentially
if max_concurrent_items > 1 and clear_agent_per_item:
# Parallel item processing (requires unique agent IDs)
all_results = await self._process_items_parallel(
items, agent_id, thinking_budget, max_tokens,
skip_ingestion, max_questions_per_item, question_semaphore,
eval_semaphore_size, filln, max_concurrent_items
)
else:
# Sequential item processing (original behavior)
all_results = await self._process_items_sequential(
items, agent_id, thinking_budget, max_tokens,
skip_ingestion, max_questions_per_item, question_semaphore,
eval_semaphore_size, clear_agent_per_item, filln
)
# Calculate overall metrics
total_correct = sum(r['metrics']['correct'] for r in all_results)
total_questions = sum(r['metrics']['total'] for r in all_results)
total_invalid = sum(r['metrics'].get('invalid', 0) for r in all_results)
total_valid = total_questions - total_invalid
# Calculate accuracy excluding invalid questions
overall_accuracy = (total_correct / total_valid * 100) if total_valid > 0 else 0
return {
'overall_accuracy': overall_accuracy,
'total_correct': total_correct,
'total_questions': total_questions,
'total_invalid': total_invalid,
'total_valid': total_valid,
'num_items': len(items),
'item_results': all_results
}
async def _process_items_sequential(
self,
items: List[Dict[str, Any]],
agent_id: str,
thinking_budget: int,
max_tokens: int,
skip_ingestion: bool,
max_questions_per_item: Optional[int],
question_semaphore: asyncio.Semaphore,
eval_semaphore_size: int,
clear_agent_per_item: bool,
filln: bool,
) -> List[Dict]:
"""Process items sequentially (original behavior)."""
all_results = []
for i, item in enumerate(items, 1):
@ -756,23 +810,58 @@ class BenchmarkRunner:
)
all_results.append(result)
# Calculate overall metrics
total_correct = sum(r['metrics']['correct'] for r in all_results)
total_questions = sum(r['metrics']['total'] for r in all_results)
total_invalid = sum(r['metrics'].get('invalid', 0) for r in all_results)
total_valid = total_questions - total_invalid
# Calculate accuracy excluding invalid questions
overall_accuracy = (total_correct / total_valid * 100) if total_valid > 0 else 0
return all_results
return {
'overall_accuracy': overall_accuracy,
'total_correct': total_correct,
'total_questions': total_questions,
'total_invalid': total_invalid,
'total_valid': total_valid,
'num_items': len(items),
'item_results': all_results
}
async def _process_items_parallel(
self,
items: List[Dict[str, Any]],
agent_id: str,
thinking_budget: int,
max_tokens: int,
skip_ingestion: bool,
max_questions_per_item: Optional[int],
question_semaphore: asyncio.Semaphore,
eval_semaphore_size: int,
filln: bool,
max_concurrent_items: int,
) -> List[Dict]:
"""Process items in parallel (requires unique agent IDs per item)."""
# Create semaphore for item-level parallelism
item_semaphore = asyncio.Semaphore(max_concurrent_items)
async def process_item_wrapper(i: int, item: Dict) -> Optional[Dict]:
"""Wrapper to process a single item with semaphore control."""
async with item_semaphore:
item_id = self.dataset.get_item_id(item)
item_agent_id = f"{agent_id}_{item_id}"
# Check if we should skip this item (filln mode)
if filln:
has_data = await self._agent_has_data(item_agent_id)
if has_data:
console.print(f"\n[bold blue]Item {i}/{len(items)}[/bold blue] (ID: {item_id})")
console.print(f" [yellow]⊘[/yellow] Skipping - agent '{item_agent_id}' already has indexed data")
return None
# Process the item
result = await self.process_single_item(
item, item_agent_id, i, len(items),
thinking_budget, max_tokens, max_questions_per_item,
skip_ingestion, question_semaphore, eval_semaphore_size,
clear_this_agent=True, # Always clear for parallel processing
)
return result
# Create all tasks
tasks = [process_item_wrapper(i, item) for i, item in enumerate(items, 1)]
# Run in parallel and collect results
results = await asyncio.gather(*tasks)
# Filter out None results (skipped items)
all_results = [r for r in results if r is not None]
return all_results
async def _run_two_phase(
self,

164
memora-dev/benchmarks/locomo/locomo_benchmark.py
View file

@ -39,10 +39,12 @@ class LoComoDataset(BenchmarkDataset):
def prepare_sessions_for_ingestion(self, item: Dict) -> List[Dict[str, Any]]:
"""
Prepare LoComo conversation sessions for batch ingestion.
Prepare LoComo conversation for batch ingestion.
Combines all sessions into a single conversation item instead of separate sessions.
Returns:
List of session dicts with 'content', 'context', 'event_date'
List with single conversation dict containing 'content', 'context', 'event_date'
"""
conv = item['conversation']
speaker_a = conv['speaker_a']
@ -51,7 +53,8 @@ class LoComoDataset(BenchmarkDataset):
# Get all session keys sorted
session_keys = sorted([k for k in conv.keys() if k.startswith('session_') and not k.endswith('_date_time')])
batch_contents = []
all_conversation_parts = []
first_session_date = None
for session_key in session_keys:
if session_key not in conv or not isinstance(conv[session_key], list):
@ -59,6 +62,14 @@ class LoComoDataset(BenchmarkDataset):
session_data = conv[session_key]
# Get session date
date_key = f"{session_key}_date_time"
session_date = self._parse_date(conv.get(date_key, "n/a"))
# Store first session date
if first_session_date is None:
first_session_date = session_date
# Build session content from all turns
session_parts = []
for turn in session_data:
@ -66,24 +77,22 @@ class LoComoDataset(BenchmarkDataset):
text = turn['text']
session_parts.append(f"{speaker}: {text}")
if not session_parts:
continue
if session_parts:
all_conversation_parts.append("\n".join(session_parts))
# Get session date
date_key = f"{session_key}_date_time"
session_date = self._parse_date(conv.get(date_key, "1:00 pm on 1 January, 2023"))
if not all_conversation_parts:
return []
# Add to batch
session_content = "\n".join(session_parts)
document_id = f"{item['sample_id']}_{session_key}"
batch_contents.append({
"content": session_content,
"context": f"Conversation session between {speaker_a} and {speaker_b} (conversation {item['sample_id']} session {session_key})",
"event_date": session_date,
"document_id": document_id
})
# Combine all sessions into a single conversation
conversation_content = "\n\n".join(all_conversation_parts)
document_id = item['sample_id']
return batch_contents
return [{
"content": conversation_content,
"context": f"Conversation between {speaker_a} and {speaker_b} (conversation {item['sample_id']})",
"event_date": first_session_date or datetime.now(timezone.utc),
"document_id": document_id
}]
def get_qa_pairs(self, item: Dict) -> List[Dict[str, Any]]:
"""
@ -116,7 +125,7 @@ class LoComoAnswerGenerator(LLMAnswerGenerator):
def __init__(self):
"""Initialize with LLM configuration for memory operations."""
self.llm_config = LLMConfig.for_memory()
self.client = self.llm_config.client
self.client = self.llm_config._client
self.model = self.llm_config.model
async def generate_answer(
@ -266,10 +275,10 @@ class LoComoThinkAnswerGenerator(LLMAnswerGenerator):
)
# Extract answer and reasoning
answer = result.get('text', '')
answer = result.text
# Extract memories from based_on
based_on = result.get('based_on', {})
based_on = result.based_on
world_facts = based_on.get('world', [])
agent_facts = based_on.get('agent', [])
opinion_facts = based_on.get('opinion', [])
@ -279,37 +288,12 @@ class LoComoThinkAnswerGenerator(LLMAnswerGenerator):
# Add world facts
for fact in world_facts:
retrieved_memories.append({
'id': fact.get('id'),
'text': fact.get('text'),
'context': fact.get('context'),
'event_date': fact.get('event_date'),
'score': fact.get('score', 0.0),
'fact_type': 'world'
})
retrieved_memories.append(fact.model_dump())
# Add agent facts
for fact in agent_facts:
retrieved_memories.append({
'id': fact.get('id'),
'text': fact.get('text'),
'context': fact.get('context'),
'event_date': fact.get('event_date'),
'score': fact.get('score', 0.0),
'fact_type': 'agent'
})
# Add opinion facts
retrieved_memories.append(fact.model_dump())
for fact in opinion_facts:
retrieved_memories.append({
'id': fact.get('id'),
'text': fact.get('text'),
'context': fact.get('context'),
'event_date': fact.get('event_date'),
'score': fact.get('score', 0.0),
'fact_type': 'opinion'
})
retrieved_memories.append(fact.model_dump())
# Build reasoning summary
num_world = len(world_facts)
num_agent = len(agent_facts)
@ -328,7 +312,9 @@ async def run_benchmark(
skip_ingestion: bool = False,
use_think: bool = False,
conversation: str = None,
api_url: str = None
api_url: str = None,
only_failed: bool = False,
only_invalid: bool = False
):
"""
Run the LoComo benchmark.
@ -340,7 +326,48 @@ async def run_benchmark(
use_think: Whether to use the think API instead of search + LLM
conversation: Specific conversation ID to run (e.g., "conv-26")
api_url: Optional API URL to connect to (default: use local memory)
only_failed: If True, only run conversations that have failed questions (is_correct=False)
only_invalid: If True, only run conversations that have invalid questions (is_invalid=True)
"""
from rich.console import Console
console = Console()
# Load previous results if filtering for failed/invalid conversations
failed_conversation_ids = set()
invalid_conversation_ids = set()
if only_failed or only_invalid:
suffix = "_think" if use_think else ""
results_filename = f'benchmark_results{suffix}.json'
results_path = Path(__file__).parent / 'results' / results_filename
if not results_path.exists():
console.print(f"[red]Error: Cannot use --only-failed or --only-invalid without existing results file[/red]")
console.print(f"[yellow]Results file not found: {results_path}[/yellow]")
return
with open(results_path, 'r') as f:
previous_results = json.load(f)
# Extract conversation IDs that have failed or invalid questions
for item_result in previous_results.get('item_results', []):
item_id = item_result['item_id']
for detail in item_result['metrics'].get('detailed_results', []):
if only_failed and detail.get('is_correct') == False and not detail.get('is_invalid', False):
failed_conversation_ids.add(item_id)
if only_invalid and detail.get('is_invalid', False):
invalid_conversation_ids.add(item_id)
if only_failed:
console.print(f"[cyan]Filtering to {len(failed_conversation_ids)} conversations with failed questions (is_correct=False)[/cyan]")
if only_invalid:
console.print(f"[cyan]Filtering to {len(invalid_conversation_ids)} conversations with invalid questions (is_invalid=True)[/cyan]")
target_ids = failed_conversation_ids if only_failed else invalid_conversation_ids
if not target_ids:
filter_type = "failed" if only_failed else "invalid"
console.print(f"[yellow]No conversations with {filter_type} questions found in previous results. Nothing to run.[/yellow]")
return
# Initialize components
dataset = LoComoDataset()
@ -383,8 +410,25 @@ async def run_benchmark(
memory=memory
)
# Run benchmark
# Filter dataset if using --only-failed or --only-invalid
dataset_path = Path(__file__).parent / 'datasets' / 'locomo10.json'
if only_failed or only_invalid:
# Load and filter dataset
target_ids = failed_conversation_ids if only_failed else invalid_conversation_ids
original_items = dataset.load(dataset_path, max_conversations)
filtered_items = [item for item in original_items if dataset.get_item_id(item) in target_ids]
console.print(f"[green]Found {len(filtered_items)} conversations to re-evaluate[/green]")
# Temporarily replace dataset's load method
original_load = dataset.load
def filtered_load(path: Path, max_items: Optional[int] = None):
return filtered_items[:max_items] if max_items else filtered_items
dataset.load = filtered_load
# Run benchmark with parallel conversation processing
# Each conversation gets its own agent ID (locomo_conv-26, locomo_conv-30, etc.)
# This allows conversations to run in parallel (up to max_concurrent_items at a time)
results = await runner.run(
dataset_path=dataset_path,
agent_id="locomo",
@ -395,7 +439,9 @@ async def run_benchmark(
skip_ingestion=skip_ingestion,
max_concurrent_questions=max_concurrent_questions,
eval_semaphore_size=eval_semaphore_size,
specific_item=conversation
specific_item=conversation,
clear_agent_per_item=True, # Use unique agent ID per conversation
max_concurrent_items=3, # Process up to 3 conversations in parallel
)
# Display and save results
@ -405,8 +451,8 @@ async def run_benchmark(
suffix = "_think" if use_think else ""
results_filename = f'benchmark_results{suffix}.json'
# Merge with existing results if running a specific conversation
merge_with_existing = conversation is not None
# Merge with existing results if running a specific conversation or using filters
merge_with_existing = conversation is not None or only_failed or only_invalid
runner.save_results(results, Path(__file__).parent / 'results' / results_filename, merge_with_existing=merge_with_existing)
# Generate markdown table
@ -488,14 +534,22 @@ if __name__ == "__main__":
parser.add_argument('--use-think', action='store_true', help='Use think API instead of search + LLM')
parser.add_argument('--conversation', type=str, default=None, help='Run only specific conversation (e.g., "conv-26")')
parser.add_argument('--api-url', type=str, default=None, help='Memora API URL (default: use local memory, example: http://localhost:8000)')
parser.add_argument('--only-failed', action='store_true', help='Only run conversations that have failed questions (is_correct=False). Requires existing results file.')
parser.add_argument('--only-invalid', action='store_true', help='Only run conversations that have invalid questions (is_invalid=True). Requires existing results file.')
args = parser.parse_args()
# Validate that only one of --only-failed or --only-invalid is set
if args.only_failed and args.only_invalid:
parser.error("Cannot use both --only-failed and --only-invalid at the same time")
results = asyncio.run(run_benchmark(
max_conversations=args.max_conversations,
max_questions_per_conv=args.max_questions,
skip_ingestion=args.skip_ingestion,
use_think=args.use_think,
conversation=args.conversation,
api_url=args.api_url
api_url=args.api_url,
only_failed=args.only_failed,
only_invalid=args.only_invalid
))

174813
memora-dev/benchmarks/locomo/results/benchmark_results.json
View file

File diff suppressed because it is too large Load diff

1245521
memora-dev/benchmarks/locomo/results/benchmark_results_think.json
View file

File diff suppressed because it is too large Load diff

6
memora-dev/benchmarks/locomo/results/results_table.md
View file

@ -1,7 +1,9 @@
# LoComo Benchmark Results
**Overall Accuracy**: 70.67% (106/150)
**Overall Accuracy**: 77.78% (7/9)
| Sample ID | Sessions | Questions | Correct | Accuracy | Multi-hop | Single-hop | Temporal | Open-domain |
|-----------|----------|-----------|---------|----------|-----------|------------|----------|-------------|
| conv-26 | 19 | 150 | 106 | 70.67% | N/A | N/A | N/A | N/A |
| conv-26 | 19 | 3 | 2 | 66.67% | N/A | N/A | N/A | N/A |
| conv-30 | 19 | 3 | 3 | 100.00% | N/A | N/A | N/A | N/A |
| conv-41 | 32 | 3 | 2 | 66.67% | N/A | N/A | N/A | N/A |

13
memora-dev/benchmarks/locomo/results/results_table_think.md
View file

@ -1,16 +1,7 @@
# LoComo Benchmark Results (Think Mode)
**Overall Accuracy**: 45.58% (640/1404)
**Overall Accuracy**: 70.73% (87/123)
| Sample ID | Sessions | Questions | Correct | Accuracy | Multi-hop | Single-hop | Temporal | Open-domain |
|-----------|----------|-----------|---------|----------|-----------|------------|----------|-------------|
| conv-26 | -1 | 150 | 83 | 55.33% | N/A | N/A | N/A | N/A |
| conv-30 | -1 | 81 | 46 | 56.79% | N/A | N/A | N/A | N/A |
| conv-41 | -1 | 150 | 84 | 56.00% | N/A | N/A | N/A | N/A |
| conv-42 | -1 | 150 | 18 | 12.00% | N/A | N/A | N/A | N/A |
| conv-43 | -1 | 150 | 35 | 23.33% | N/A | N/A | N/A | N/A |
| conv-44 | -1 | 123 | 61 | 49.59% | N/A | N/A | N/A | N/A |
| conv-47 | -1 | 150 | 78 | 52.00% | N/A | N/A | N/A | N/A |
| conv-48 | -1 | 150 | 87 | 58.00% | N/A | N/A | N/A | N/A |
| conv-49 | -1 | 150 | 78 | 52.00% | N/A | N/A | N/A | N/A |
| conv-50 | -1 | 150 | 70 | 46.67% | N/A | N/A | N/A | N/A |
| conv-44 | 1 | 123 | 87 | 70.73% | N/A | N/A | N/A | N/A |

2
memora-dev/benchmarks/longmemeval/longmemeval_benchmark.py
View file

@ -141,7 +141,7 @@ class LongMemEvalAnswerGenerator(LLMAnswerGenerator):
def __init__(self):
"""Initialize with LLM configuration for memory operations."""
self.llm_config = LLMConfig.for_memory()
self.client = self.llm_config.client
self.client = self.llm_config._client
self.model = self.llm_config.model
async def generate_answer(

388222
memora-dev/benchmarks/longmemeval/results/benchmark_results.json
View file

File diff suppressed because one or more lines are too long

36
memora/alembic/versions/3b9c4d8e7f21_add_name_to_agents.py Normal file
View file

@ -0,0 +1,36 @@
"""add_name_to_agents
Revision ID: 3b9c4d8e7f21
Revises: 1680fc9768b4
Create Date: 2025-11-13 14:52:00.000000
"""
from typing import Sequence, Union
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision: str = '3b9c4d8e7f21'
down_revision: Union[str, Sequence[str], None] = '1680fc9768b4'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
def upgrade() -> None:
"""Upgrade schema."""
# Add name column to agents table
op.execute("""
ALTER TABLE agents
ADD COLUMN name TEXT NOT NULL DEFAULT ''
""")
def downgrade() -> None:
"""Downgrade schema."""
# Remove name column from agents table
op.execute("""
ALTER TABLE agents
DROP COLUMN name
""")

113
memora/memora/api.py
View file

@ -280,6 +280,7 @@ class PersonalityTraits(BaseModel):
class AgentProfileResponse(BaseModel):
"""Response model for agent profile."""
agent_id: str
name: str
personality: PersonalityTraits
background: str
@ -287,6 +288,7 @@ class AgentProfileResponse(BaseModel):
json_schema_extra = {
"example": {
"agent_id": "user123",
"name": "Alice",
"personality": {
"openness": 0.8,
"conscientiousness": 0.6,
@ -346,6 +348,7 @@ class BackgroundResponse(BaseModel):
class AgentListItem(BaseModel):
"""Agent list item with profile summary."""
agent_id: str
name: str
personality: PersonalityTraits
background: str
created_at: Optional[str] = None
@ -362,6 +365,7 @@ class AgentListResponse(BaseModel):
"agents": [
{
"agent_id": "user123",
"name": "Alice",
"personality": {
"openness": 0.5,
"conscientiousness": 0.5,
@ -381,12 +385,14 @@ class AgentListResponse(BaseModel):
class CreateAgentRequest(BaseModel):
"""Request model for creating/updating an agent."""
name: Optional[str] = None
personality: Optional[PersonalityTraits] = None
background: Optional[str] = None
class Config:
json_schema_extra = {
"example": {
"name": "Alice",
"personality": {
"openness": 0.8,
"conscientiousness": 0.6,
@ -835,6 +841,30 @@ def _register_routes(app: FastAPI):
agent_id
)
# Get link counts by fact_type (from nodes)
link_fact_type_stats = await conn.fetch(
"""
SELECT mu.fact_type, COUNT(*) as count
FROM memory_links ml
JOIN memory_units mu ON ml.from_unit_id = mu.id
WHERE mu.agent_id = $1
GROUP BY mu.fact_type
""",
agent_id
)
# Get link counts by fact_type AND link_type
link_breakdown_stats = await conn.fetch(
"""
SELECT mu.fact_type, ml.link_type, COUNT(*) as count
FROM memory_links ml
JOIN memory_units mu ON ml.from_unit_id = mu.id
WHERE mu.agent_id = $1
GROUP BY mu.fact_type, ml.link_type
""",
agent_id
)
# Get pending and failed operations counts
ops_stats = await conn.fetch(
"""
@ -863,6 +893,17 @@ def _register_routes(app: FastAPI):
# Format results
nodes_by_type = {row['fact_type']: row['count'] for row in node_stats}
links_by_type = {row['link_type']: row['count'] for row in link_stats}
links_by_fact_type = {row['fact_type']: row['count'] for row in link_fact_type_stats}
# Build detailed breakdown: {fact_type: {link_type: count}}
links_breakdown = {}
for row in link_breakdown_stats:
fact_type = row['fact_type']
link_type = row['link_type']
count = row['count']
if fact_type not in links_breakdown:
links_breakdown[fact_type] = {}
links_breakdown[fact_type][link_type] = count
total_nodes = sum(nodes_by_type.values())
total_links = sum(links_by_type.values())
@ -872,8 +913,10 @@ def _register_routes(app: FastAPI):
"total_nodes": total_nodes,
"total_links": total_links,
"total_documents": total_documents,
"nodes_by_type": nodes_by_type,
"links_by_type": links_by_type,
"nodes_by_fact_type": nodes_by_type,
"links_by_link_type": links_by_type,
"links_by_fact_type": links_by_fact_type,
"links_breakdown": links_breakdown,
"pending_operations": pending_operations,
"failed_operations": failed_operations
}
@ -953,6 +996,53 @@ def _register_routes(app: FastAPI):
raise HTTPException(status_code=500, detail=str(e))
@app.delete(
"/api/v1/agents/{agent_id}/documents/{document_id}",
tags=["Documents"],
summary="Delete a document",
description="""
Delete a document and all its associated memory units and links.
This will cascade delete:
- The document itself
- All memory units extracted from this document
- All links (temporal, semantic, entity) associated with those memory units
This operation cannot be undone.
"""
)
async def api_delete_document(
agent_id: str,
document_id: str
):
"""
Delete a document and all its associated memory units and links.
Args:
agent_id: Agent ID (from path)
document_id: Document ID to delete (from path)
"""
try:
result = await app.state.memory.delete_document(document_id, agent_id)
if result["document_deleted"] == 0:
raise HTTPException(status_code=404, detail="Document not found")
return {
"success": True,
"message": f"Document '{document_id}' and {result['memory_units_deleted']} associated memory units deleted successfully",
"document_id": document_id,
"memory_units_deleted": result["memory_units_deleted"]
}
except HTTPException:
raise
except Exception as e:
import traceback
error_detail = f"{str(e)}\n\nTraceback:\n{traceback.format_exc()}"
print(f"Error in /api/v1/agents/{agent_id}/documents/{document_id}: {error_detail}")
raise HTTPException(status_code=500, detail=str(e))
@app.post(
"/api/v1/agents/{agent_id}/memories",
response_model=BatchPutResponse,
@ -1228,6 +1318,7 @@ def _register_routes(app: FastAPI):
profile = await app.state.memory.get_agent_profile(agent_id)
return AgentProfileResponse(
agent_id=agent_id,
name=profile["name"],
personality=PersonalityTraits(**profile["personality"]),
background=profile["background"]
)
@ -1261,6 +1352,7 @@ def _register_routes(app: FastAPI):
profile = await app.state.memory.get_agent_profile(agent_id)
return AgentProfileResponse(
agent_id=agent_id,
name=profile["name"],
personality=PersonalityTraits(**profile["personality"]),
background=profile["background"]
)
@ -1318,6 +1410,22 @@ def _register_routes(app: FastAPI):
# Get existing profile or create with defaults
profile = await app.state.memory.get_agent_profile(agent_id)
# Update name if provided
if request.name is not None:
pool = await app.state.memory._get_pool()
async with pool.acquire() as conn:
await conn.execute(
"""
UPDATE agents
SET name = $2,
updated_at = NOW()
WHERE agent_id = $1
""",
agent_id,
request.name
)
profile["name"] = request.name
# Update personality if provided
if request.personality is not None:
await app.state.memory.update_agent_personality(
@ -1346,6 +1454,7 @@ def _register_routes(app: FastAPI):
final_profile = await app.state.memory.get_agent_profile(agent_id)
return AgentProfileResponse(
agent_id=agent_id,
name=final_profile["name"],
personality=PersonalityTraits(**final_profile["personality"]),
background=final_profile["background"]
)

127
memora/memora/fact_extraction.py
View file

@ -4,6 +4,7 @@ Fact extraction from text using LLM.
Extracts semantic facts, entities, and temporal information from text.
Uses the LLMConfig wrapper for all LLM calls.
"""
import logging
import os
import json
import re
@ -12,7 +13,7 @@ from datetime import datetime
from typing import List, Dict, Optional, Literal
from openai import AsyncOpenAI
from pydantic import BaseModel, Field
from .llm_wrapper import OutputTooLongError
from .llm_wrapper import OutputTooLongError, LLMConfig
class Entity(BaseModel):
@ -31,7 +32,7 @@ class ExtractedFact(BaseModel):
description="Absolute date/time when this fact occurred in ISO format (YYYY-MM-DDTHH:MM:SSZ). If text mentions relative time (yesterday, last week, this morning), calculate absolute date from the provided context date."
)
fact_type: Literal["world", "agent", "opinion"] = Field(
description="Type of fact: 'world' for general facts about the world (events, people, things that happen), 'agent' for facts about what the AI agent specifically did or actions the agent took (conversations with the user, tasks performed by the agent), 'opinion' for the agent's formed opinions and perspectives"
description="Type of fact: 'world' for general facts about the world (events, people, things others said/did), 'agent' for facts about what the memory owner (the person this memory belongs to, often identified as 'you' in context) specifically did, said, experienced, or actions they took - MUST be written in FIRST PERSON ('I did...', 'I said...'), 'opinion' for the memory owner's formed opinions and perspectives - also in first person"
)
entities: List[Entity] = Field(
default_factory=list,
@ -46,7 +47,7 @@ class FactExtractionResponse(BaseModel):
)
def chunk_text(text: str, max_chars: int = 120000) -> List[str]:
def chunk_text(text: str, max_chars: int) -> List[str]:
"""
Split text into chunks at sentence boundaries using LangChain's text splitter.
@ -96,7 +97,8 @@ async def _extract_facts_from_chunk(
total_chunks: int,
event_date: datetime,
context: str,
llm_config: 'LLMConfig'
llm_config: 'LLMConfig',
agent_name: str = None
) -> List[Dict[str, str]]:
"""
Extract facts from a single chunk (internal helper for parallel processing).
@ -104,11 +106,13 @@ async def _extract_facts_from_chunk(
# Format event_date for the prompt
event_date_str = event_date.strftime("%Y-%m-%dT%H:%M:%SZ")
agent_context = f"\n- Agent name (memory owner): {agent_name}" if agent_name else ""
prompt = f"""You are extracting comprehensive, narrative facts from conversations for an AI memory system.
## CONTEXT INFORMATION
- Current reference date/time: {event_date_str}
- Context: {context if context else 'no context provided'}
- Context: {context if context else 'no context provided'}{agent_context}
## CORE PRINCIPLE: Extract FEWER, MORE COMPREHENSIVE Facts
@ -174,11 +178,34 @@ Only split into separate facts when topics are COMPLETELY UNRELATED:
- Filler words ("um", "uh", "like")
- Pure reactions without content ("wow", "cool")
- Incomplete fragments with no meaning
- **Structural/procedural statements**: Openings, closings, transitions, housekeeping ("let's get started", "that's all", "moving on")
- **Meta-commentary about the medium itself**: References to the format/structure rather than content ("welcome to the show", "thanks for listening", "before we begin")
- **Calls to action unrelated to content**: Requests to subscribe, follow, rate, share, etc.
- **Generic sign-offs**: "See you next time", "Until later", "That wraps it up"
- **FOCUS PRINCIPLE**: Extract SUBSTANTIVE CONTENT (ideas, facts, discussions, decisions), NOT FORMAT/STRUCTURE
## FACT TYPE CLASSIFICATION
Classify each fact as either 'world' or 'agent':
- **'world'**: General facts about people, events, conversations (most facts)
- **'agent'**: Only for AI agent's own actions
Classify each fact as 'world', 'agent', or 'opinion':
- **'world'**: Facts about other people, events, things that happened in the world, what others said/did
- Written in third person (use names, "they", etc.)
- **'agent'**: Facts about what the MEMORY OWNER (the person this memory belongs to) specifically did, said, experienced, or actions they took
- The memory owner is typically identified in the context (e.g., "you (Marcus)" means Marcus is the memory owner)
- **CRITICAL**: MUST be written in FIRST PERSON using "I", "me", "my" (NOT the person's name)
- Examples: "I said I prefer coffee", "I attended the conference", "I completed the project"
- ❌ WRONG: "Marcus said he prefers coffee"
- ✅ CORRECT: "I said I prefer coffee"
- **'opinion'**: The memory owner's formed opinions, beliefs, and perspectives about topics
- Also written in first person: "I believe...", "I think..."
**CRITICAL**: If the context identifies someone as "you" or specifies whose memory this is, then facts about that person's actions/statements are 'agent' facts written in FIRST PERSON.
**Example**: If context says "podcast between you (Marcus) and Jamie":
- "I explained my approach to AI safety" → 'agent' (first person, my action)
- "Jamie asked about neural networks" → 'world' (someone else's action, third person)
- "Jamie and I discussed transformer architectures" → 'world' (general conversation - could use first person here since it includes both)
- "I believe interpretability is crucial" → 'opinion' (first person belief)
## ENTITY EXTRACTION
Extract ALL important entities (names of people, places, organizations, products, concepts, etc).
@ -255,7 +282,46 @@ Sarah: Sounds amazing! I'll add it to my itinerary."
- date: 2023 (if reference is 2024)
- entities: [{{"text": "Alice"}}, {{"text": "Google"}}, {{"text": "Mountain View"}}, {{"text": "AI team"}}]
### Example 5: When to Split into Multiple Facts
### Example 5: Agent vs World Facts (CRITICAL FOR CLASSIFICATION)
**Context:** "Podcast episode between you (Marcus) and Jamie about AI"
**Input:**
"Marcus: I've been working on interpretability research for the past year.
Jamie: That's fascinating! What made you focus on that?
Marcus: I believe it's crucial for AI safety. Without understanding how models work, we can't trust them.
Jamie: I agree. Have you published any papers?
Marcus: Yes, I published a paper on attention visualization in March."
**✅ GOOD CLASSIFICATION:**
1. "I have been working on interpretability research for the past year because I believe it's crucial for AI safety and think that without understanding how models work, we can't trust them. Jamie found this fascinating and asked about publications. I published a paper on attention visualization in March 2024."
- fact_type: "agent" (written in FIRST PERSON - my work and statements)
- entities: [{{"text": "Jamie"}}, {{"text": "interpretability research"}}, {{"text": "attention visualization"}}]
- NOTE: Uses "I" not "Marcus" - first person for agent facts
2. "Jamie agrees that understanding how AI models work is crucial for trust"
- fact_type: "world" (Jamie's statement - third person, not the memory owner)
- entities: [{{"text": "Jamie"}}]
**❌ BAD CLASSIFICATION:**
- Using "Marcus has been working..." instead of "I have been working..." for agent facts
- Marking my actions as 'world' facts
- Marking Jamie's statements as 'agent' facts
### Example 6: Skipping Structural/Procedural Statements
**Input (could be podcast, meeting, lecture, etc.):**
"Marcus: So in my research on AI safety, I've found that interpretability is key.
Jamie: That's fascinating! Tell us more.
Marcus: Well, it's all about understanding how models make decisions...
Marcus: I think that's gonna do it for us today! Don't forget to subscribe and leave a rating. See you next week!"
**✅ GOOD (extract only substantive content):**
1. "I have found that interpretability is key in my AI safety research because it's all about understanding how models make decisions, and Jamie found this fascinating."
- fact_type: "agent"
- entities: [{{"text": "Jamie"}}, {{"text": "AI safety"}}, {{"text": "interpretability"}}]
**❌ BAD (extracting procedural/structural statements):**
- "I think that's gonna do it for us today and I encourage listeners to subscribe and leave a rating" ← This is structural boilerplate about the format, NOT substantive content!
### Example 7: When to Split into Multiple Facts
**Input:**
"Caroline said 'This necklace is from my grandma in Sweden. I'm planning to visit Stockholm next month for a tech conference.'"
@ -279,8 +345,12 @@ Sarah: Sounds amazing! I'll add it to my itinerary."
6. **ONLY SPLIT** when topics are completely unrelated or different time periods
7. **TRANSFORM RELATIVE DATES** - "last year" → "in 2023" in the fact text
8. **EXTRACT ALL ENTITIES** - PERSON, ORG, PLACE, PRODUCT, CONCEPT, OTHER
9. **CLASSIFY FACTS** - 'world' for general facts, 'agent' for AI agent actions
10. When combining, prefer MORE comprehensive facts over fragmenting"""
9. **CLASSIFY FACTS CORRECTLY**:
- 'agent' = memory owner's actions/statements (identified as "you" in context) - **MUST USE FIRST PERSON** ("I did...", "I said...")
- 'world' = other people's actions/statements, general events - use third person
- 'opinion' = memory owner's beliefs/perspectives - use first person ("I believe...", "I think...")
10. **EXTRACT CONTENT, NOT FORMAT** - Skip structural/procedural statements (openings, closings, housekeeping), meta-commentary about the medium, calls to action - extract only SUBSTANTIVE CONTENT (ideas, facts, discussions, decisions)
11. When combining, prefer MORE comprehensive facts over fragmenting"""
import time
import logging
@ -298,7 +368,7 @@ Sarah: Sounds amazing! I'll add it to my itinerary."
messages=[
{
"role": "system",
"content": "You are a comprehensive fact extractor that creates narrative, self-contained facts. CRITICAL: Extract 2-5 COMPREHENSIVE facts per conversation, NOT dozens of fragments. COMBINE related exchanges into single narrative facts that tell the complete story. For example, a discussion about playlist names should be ONE fact capturing the entire back-and-forth with all reasoning, not multiple small facts. PRESERVE all context (photos, 'new' things, visual elements, full reasoning), INCLUDE all participants and what they said/did, MAINTAIN narrative flow. ONLY SPLIT into separate facts when topics are completely unrelated or different time periods. Transform relative dates in fact text ('last year' → 'in 2023'). Extract entities (PERSON, ORG, PLACE, PRODUCT, CONCEPT, OTHER). When in doubt, prefer MORE COMPREHENSIVE over fragmenting."
"content": "You are a comprehensive fact extractor that creates narrative, self-contained facts. CRITICAL: Extract 2-5 COMPREHENSIVE facts per conversation, NOT dozens of fragments. COMBINE related exchanges into single narrative facts that tell the complete story. For example, a discussion about playlist names should be ONE fact capturing the entire back-and-forth with all reasoning, not multiple small facts. PRESERVE all context (photos, 'new' things, visual elements, full reasoning), INCLUDE all participants and what they said/did, MAINTAIN narrative flow. ONLY SPLIT into separate facts when topics are completely unrelated or different time periods. Transform relative dates in fact text ('last year' → 'in 2023'). Extract entities (PERSON, ORG, PLACE, PRODUCT, CONCEPT, OTHER). FACT TYPES: Classify as 'world' (facts about others/events - third person), 'agent' (facts about the memory owner's actions/statements - identified as 'you' in context - MUST USE FIRST PERSON 'I did...', 'I said...'), or 'opinion' (memory owner's beliefs - first person 'I believe...'). CRITICAL: If context says 'you (Name)', write Name's actions in FIRST PERSON as 'agent' facts ('I attended...' NOT 'Name attended...'). Extract SUBSTANTIVE CONTENT only - skip structural/procedural statements (openings, closings, housekeeping), meta-commentary about format/medium, and calls to action. Focus on IDEAS, FACTS, DISCUSSIONS, DECISIONS - not structure. When in doubt, prefer MORE COMPREHENSIVE over fragmenting."
},
{
"role": "user",
@ -334,7 +404,8 @@ async def _extract_facts_with_auto_split(
total_chunks: int,
event_date: datetime,
context: str,
llm_config: 'LLMConfig'
llm_config: LLMConfig,
agent_name: str = None
) -> List[Dict[str, str]]:
"""
Extract facts from a chunk with automatic splitting if output exceeds token limits.
@ -349,6 +420,7 @@ async def _extract_facts_with_auto_split(
event_date: Reference date for temporal information
context: Context about the conversation/document
llm_config: LLM configuration to use
agent_name: Optional agent name (memory owner)
Returns:
List of fact dictionaries extracted from the chunk (possibly from sub-chunks)
@ -364,7 +436,8 @@ async def _extract_facts_with_auto_split(
total_chunks=total_chunks,
event_date=event_date,
context=context,
llm_config=llm_config
llm_config=llm_config,
agent_name=agent_name
)
except OutputTooLongError as e:
# Output exceeded token limits - split the chunk in half and retry
@ -408,7 +481,8 @@ async def _extract_facts_with_auto_split(
total_chunks=total_chunks,
event_date=event_date,
context=context,
llm_config=llm_config
llm_config=llm_config,
agent_name=agent_name
),
_extract_facts_with_auto_split(
chunk=second_half,
@ -416,7 +490,8 @@ async def _extract_facts_with_auto_split(
total_chunks=total_chunks,
event_date=event_date,
context=context,
llm_config=llm_config
llm_config=llm_config,
agent_name=agent_name
)
]
@ -437,9 +512,9 @@ async def _extract_facts_with_auto_split(
async def extract_facts_from_text(
text: str,
event_date: datetime,
llm_config: LLMConfig,
agent_name: str,
context: str = "",
llm_config: Optional['LLMConfig'] = None,
chunk_size: int = 5000
) -> List[Dict[str, str]]:
"""
Extract semantic facts from conversational or narrative text using LLM.
@ -456,20 +531,13 @@ async def extract_facts_from_text(
context: Context about the conversation/document
llm_config: LLM configuration to use (if None, uses default from environment)
chunk_size: Maximum characters per chunk
agent_name: Optional agent name (memory owner)
Returns:
List of fact dictionaries with 'fact' and 'date' keys
"""
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from .llm_wrapper import LLMConfig
if llm_config is None:
from .llm_wrapper import LLMConfig
llm_config = LLMConfig.for_memory()
chunks = chunk_text(text, max_chars=chunk_size)
chunks = chunk_text(text, max_chars=50_000)
logging.info(f"created {len(chunks)} chunks from text {len(text)}")
tasks = [
_extract_facts_with_auto_split(
chunk=chunk,
@ -477,7 +545,8 @@ async def extract_facts_from_text(
total_chunks=len(chunks),
event_date=event_date,
context=context,
llm_config=llm_config
llm_config=llm_config,
agent_name=agent_name
)
for i, chunk in enumerate(chunks)
]

18
memora/memora/operations/agent_operations.py
View file

@ -40,21 +40,21 @@ class AgentOperationsMixin:
async def get_agent_profile(self, agent_id: str) -> Dict:
"""
Get agent profile (personality + background).
Get agent profile (name, personality + background).
Auto-creates agent with default values if not exists.
Args:
agent_id: Agent identifier
Returns:
Dict with 'personality' (dict) and 'background' (str) keys
Dict with 'name' (str), 'personality' (dict) and 'background' (str) keys
"""
pool = await self._get_pool()
async with pool.acquire() as conn:
# Try to get existing agent
row = await conn.fetchrow(
"""
SELECT personality, background
SELECT name, personality, background
FROM agents
WHERE agent_id = $1
""",
@ -68,6 +68,7 @@ class AgentOperationsMixin:
personality_data = json.loads(personality_data)
return {
"name": row["name"],
"personality": personality_data,
"background": row["background"]
}
@ -75,16 +76,18 @@ class AgentOperationsMixin:
# Agent doesn't exist, create with defaults
await conn.execute(
"""
INSERT INTO agents (agent_id, personality, background)
VALUES ($1, $2::jsonb, $3)
INSERT INTO agents (agent_id, name, personality, background)
VALUES ($1, $2, $3::jsonb, $4)
ON CONFLICT (agent_id) DO NOTHING
""",
agent_id,
agent_id, # Default name is the agent_id
json.dumps(DEFAULT_PERSONALITY),
""
)
return {
"name": agent_id,
"personality": DEFAULT_PERSONALITY.copy(),
"background": ""
}
@ -385,13 +388,13 @@ Merged background:"""
List all agents in the system.
Returns:
List of dicts with agent_id, personality, background, created_at, updated_at
List of dicts with agent_id, name, personality, background, created_at, updated_at
"""
pool = await self._get_pool()
async with pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT agent_id, personality, background, created_at, updated_at
SELECT agent_id, name, personality, background, created_at, updated_at
FROM agents
ORDER BY updated_at DESC
"""
@ -406,6 +409,7 @@ Merged background:"""
result.append({
"agent_id": row["agent_id"],
"name": row["name"],
"personality": personality_data,
"background": row["background"],
"created_at": row["created_at"].isoformat() if row["created_at"] else None,

12
memora/memora/operations/think_operations.py
View file

@ -110,8 +110,9 @@ class ThinkOperationsMixin:
logger.info(f"[THINK] Formatted facts - agent: {len(agent_facts_text)} chars, world: {len(world_facts_text)} chars, opinion: {len(opinion_facts_text)} chars")
# Step 4.5: Get agent profile (personality + background)
# Step 4.5: Get agent profile (name, personality + background)
profile = await self.get_agent_profile(agent_id)
name = profile["name"]
personality = profile["personality"]
background = profile["background"]
@ -138,6 +139,11 @@ class ThinkOperationsMixin:
Personality influence strength: {int(personality['bias_strength'] * 100)}% (how much your personality shapes your opinions)"""
name_section = f"""
Your name: {name}
"""
background_section = ""
if background:
background_section = f"""
@ -167,11 +173,11 @@ WHAT I KNOW ABOUT THE WORLD:
MY EXISTING OPINIONS & BELIEFS:
{opinion_facts_text}
{context_section}{personality_desc}{background_section}
{context_section}{name_section}{personality_desc}{background_section}
QUESTION: {query}
Based on everything I know, believe, and who I am (including my personality and background), here's what I genuinely think about this question. I'll draw on my experiences, knowledge, opinions, and personal traits to give you my honest perspective."""
Based on everything I know, believe, and who I am (including my name, personality and background), here's what I genuinely think about this question. I'll draw on my experiences, knowledge, opinions, and personal traits to give you my honest perspective."""
logger.info(f"[THINK] Full prompt length: {len(prompt)} chars")
logger.debug(f"[THINK] Prompt preview (first 500 chars): {prompt[:500]}")

6
memora/memora/temporal_semantic_memory.py
View file

@ -720,6 +720,10 @@ class TemporalSemanticMemory(
log_buffer.append(f"Batch size: {len(contents)} content items, {total_chars:,} chars")
log_buffer.append(f"{'='*60}")
# Get agent name for fact extraction
profile = await self.get_agent_profile(agent_id)
agent_name = profile["name"]
# Step 1: Extract facts from ALL contents in parallel
step_start = time.time()
@ -730,7 +734,7 @@ class TemporalSemanticMemory(
context = item.get("context", "")
event_date = item.get("event_date") or utcnow()
task = extract_facts(content, event_date, context, llm_config=self._llm_config)
task = extract_facts(content, event_date, context, llm_config=self._llm_config, agent_name=agent_name)
fact_extraction_tasks.append((task, event_date, context))
# Wait for all fact extractions to complete

5
memora/memora/utils.py
View file

@ -11,7 +11,7 @@ if TYPE_CHECKING:
from .fact_extraction import extract_facts_from_text
async def extract_facts(text: str, event_date: datetime, context: str = "", llm_config: 'LLMConfig' = None) -> List[Dict[str, str]]:
async def extract_facts(text: str, event_date: datetime, context: str = "", llm_config: 'LLMConfig' = None, agent_name: str = None) -> List[Dict[str, str]]:
"""
Extract semantic facts from text using LLM.
@ -26,6 +26,7 @@ async def extract_facts(text: str, event_date: datetime, context: str = "", llm_
event_date: Reference date for resolving relative times
context: Context about the conversation/document
llm_config: LLM configuration to use
agent_name: Optional agent name to help identify agent-related facts
Returns:
List of fact dictionaries with keys: 'fact' (text) and 'date' (ISO string)
@ -36,7 +37,7 @@ async def extract_facts(text: str, event_date: datetime, context: str = "", llm_
if not text or not text.strip():
return []
fact_dicts = await extract_facts_from_text(text, event_date, context, llm_config=llm_config)
fact_dicts = await extract_facts_from_text(text, event_date, context=context, llm_config=llm_config, agent_name=agent_name)
if not fact_dicts:
logging.warning(f"LLM extracted 0 facts from text of length {len(text)}. This may indicate the text contains no meaningful information, or the LLM failed to extract facts. Full text: {text}")

412
memora/tests/test_api_integration.py Normal file
View file

@ -0,0 +1,412 @@
"""
Integration test for the complete Memora API.
Tests all endpoints by starting a FastAPI server and making HTTP requests.
"""
import pytest
import pytest_asyncio
import httpx
from datetime import datetime
from memora.api import create_app
@pytest_asyncio.fixture
async def api_client(memory):
"""Create an async test client for the FastAPI app."""
# Memory is already initialized by the conftest fixture
app = create_app(memory, run_migrations=False, initialize_memory=False)
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
yield client
@pytest.fixture
def test_agent_id():
"""Provide a unique agent ID for this test run."""
return f"integration_test_{datetime.now().timestamp()}"
@pytest.mark.asyncio
async def test_full_api_workflow(api_client, test_agent_id):
"""
End-to-end test covering all major API endpoints in a realistic workflow.
Workflow:
1. Create agent and set profile
2. Store memories (put, batch put)
3. Search memories
4. Think (generate answer)
5. List agents and memories
6. Get agent profile
7. Get visualization data
8. Track documents
9. Clean up
"""
# ================================================================
# 1. Agent Management
# ================================================================
# List agents (should be empty initially or have other test agents)
response = await api_client.get("/api/v1/agents")
assert response.status_code == 200
initial_agents_data = response.json()["agents"]
initial_agents = [a["agent_id"] for a in initial_agents_data]
print(f"Initial agents: {len(initial_agents)}")
# Get agent profile (creates default if not exists)
response = await api_client.get(f"/api/v1/agents/{test_agent_id}/profile")
assert response.status_code == 200
profile = response.json()
assert "personality" in profile
assert "background" in profile
print(f"Agent profile created with personality: {profile['personality']}")
# Add background
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/background",
json={
"content": "A software engineer passionate about AI and memory systems."
}
)
assert response.status_code == 200
assert "software engineer" in response.json()["background"].lower()
print("Background added")
# ================================================================
# 2. Memory Storage
# ================================================================
# Store single memory (using batch endpoint with single item)
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/memories",
json={
"items": [
{
"content": "Alice is a machine learning researcher at Stanford.",
"context": "conversation about team members"
}
]
}
)
assert response.status_code == 200
put_result = response.json()
assert put_result["success"] is True
assert put_result["items_count"] == 1
print(f"Stored memory via batch endpoint")
# Store batch memories
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/memories",
json={
"items": [
{
"content": "Bob leads the infrastructure team and loves Kubernetes.",
"context": "team introduction"
},
{
"content": "Charlie recently joined as a product manager from Google.",
"context": "new hire announcement"
}
]
}
)
assert response.status_code == 200
batch_result = response.json()
assert batch_result["success"] is True
assert batch_result["items_count"] == 2
print(f"Stored {batch_result['items_count']} items from batch put")
# ================================================================
# 3. Search
# ================================================================
# Search for memories
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/memories/search",
json={
"query": "Who works on machine learning?",
"thinking_budget": 50
}
)
assert response.status_code == 200
search_results = response.json()
assert "results" in search_results
assert len(search_results["results"]) > 0
print(f"Search returned {len(search_results['results'])} results")
# Verify we found Alice
found_alice = any("Alice" in r["text"] for r in search_results["results"])
assert found_alice, "Should find Alice in search results"
# ================================================================
# 4. Think (Reasoning)
# ================================================================
# Generate answer using think
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/think",
json={
"query": "What do you know about the team members?",
"thinking_budget": 30,
"context": "This is for a team overview document"
}
)
assert response.status_code == 200
think_result = response.json()
assert "text" in think_result
assert len(think_result["text"]) > 0
assert "based_on" in think_result
print(f"Think response: {think_result['text'][:100]}...")
# Verify the answer mentions team members
answer = think_result["text"].lower()
assert "alice" in answer or "bob" in answer or "charlie" in answer
# ================================================================
# 5. Visualization & Statistics
# ================================================================
# Get graph data
response = await api_client.get(f"/api/v1/agents/{test_agent_id}/graph")
assert response.status_code == 200
graph_data = response.json()
assert "nodes" in graph_data
assert "edges" in graph_data
print(f"Graph has {len(graph_data['nodes'])} nodes and {len(graph_data['edges'])} edges")
# Get memory statistics
response = await api_client.get(f"/api/v1/agents/{test_agent_id}/stats")
assert response.status_code == 200
stats = response.json()
assert "total_nodes" in stats
assert stats["total_nodes"] > 0
print(f"Total nodes: {stats['total_nodes']}")
# List memory units
response = await api_client.get(
f"/api/v1/agents/{test_agent_id}/memories/list",
params={"limit": 10}
)
assert response.status_code == 200
memory_units = response.json()
assert "items" in memory_units
assert len(memory_units["items"]) > 0
print(f"Listed {len(memory_units['items'])} memory units")
# ================================================================
# 6. Document Tracking
# ================================================================
# Store memory with document
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/memories",
json={
"items": [
{
"content": "Project timeline: MVP launch in Q1, Beta in Q2.",
"context": "product roadmap"
}
],
"document_id": "roadmap-2024-q1"
}
)
assert response.status_code == 200
print("Stored memory with document tracking")
# List documents
response = await api_client.get(f"/api/v1/agents/{test_agent_id}/documents")
assert response.status_code == 200
documents = response.json()
assert "items" in documents
assert len(documents["items"]) > 0
print(f"Tracked documents: {len(documents['items'])}")
# Get specific document
response = await api_client.get(
f"/api/v1/agents/{test_agent_id}/documents/roadmap-2024-q1"
)
assert response.status_code == 200
doc_info = response.json()
assert "id" in doc_info
assert doc_info["id"] == "roadmap-2024-q1"
assert doc_info["memory_unit_count"] > 0
print(f"Document has {doc_info['memory_unit_count']} memory units")
# Note: Document deletion is tested separately in test_document_deletion
# ================================================================
# 7. Verify Updated Agent Profile
# ================================================================
# Check profile again (might have formed new opinions)
response = await api_client.get(f"/api/v1/agents/{test_agent_id}/profile")
assert response.status_code == 200
updated_profile = response.json()
assert "software engineer" in updated_profile["background"].lower()
print("Profile verified")
# ================================================================
# 8. List All Agents (should include our test agent)
# ================================================================
response = await api_client.get("/api/v1/agents")
assert response.status_code == 200
final_agents_data = response.json()["agents"]
final_agents = [a["agent_id"] for a in final_agents_data]
assert test_agent_id in final_agents
assert len(final_agents) >= len(initial_agents) + 1
print(f"Final agent count: {len(final_agents)}")
# ================================================================
# 9. Clean Up
# ================================================================
# Note: No delete agent endpoint in API, so test data remains in DB
# Using timestamped agent IDs prevents conflicts between test runs
print(f"Integration test complete for agent {test_agent_id}")
@pytest.mark.asyncio
async def test_error_handling(api_client):
"""Test that API properly handles error cases."""
# Invalid request (missing required field)
response = await api_client.post(
"/api/v1/agents/error_test/memories",
json={
"items": [
{
# Missing "content"
"context": "test"
}
]
}
)
assert response.status_code == 422 # Validation error
# Search with invalid parameters
response = await api_client.post(
"/api/v1/agents/error_test/memories/search",
json={
"query": "test",
"thinking_budget": -1 # Invalid negative budget
}
)
assert response.status_code == 422
# Get non-existent document
response = await api_client.get(
"/api/v1/agents/nonexistent_agent/documents/fake-doc-id"
)
assert response.status_code == 404
print("Error handling tests passed")
@pytest.mark.asyncio
async def test_concurrent_requests(api_client):
"""Test that API can handle concurrent requests."""
agent_id = f"concurrent_test_{datetime.now().timestamp()}"
# Store multiple memories concurrently (simulated with sequential calls)
responses = []
test_facts = [
"David works as a data scientist at Microsoft.",
"Emily is the CEO of a startup in San Francisco.",
"Frank teaches computer science at MIT.",
"Grace is a software architect specializing in distributed systems.",
"Henry leads the product team at Amazon."
]
for fact in test_facts:
response = await api_client.post(
f"/api/v1/agents/{agent_id}/memories",
json={
"items": [
{
"content": fact,
"context": "concurrent test"
}
]
}
)
responses.append(response)
# All should succeed
assert all(r.status_code == 200 for r in responses)
assert all(r.json()["success"] for r in responses)
# Verify all facts stored
response = await api_client.get(
f"/api/v1/agents/{agent_id}/memories/list",
params={"limit": 20}
)
assert response.status_code == 200
items = response.json()["items"]
assert len(items) >= 5
print(f"Concurrent test stored {len(items)} memory units")
@pytest.mark.asyncio
async def test_document_deletion(api_client):
"""Test document deletion including cascade deletion of memory units and links."""
test_agent_id = f"doc_delete_test_{datetime.now().timestamp()}"
# Store a document with memory
response = await api_client.post(
f"/api/v1/agents/{test_agent_id}/memories",
json={
"items": [
{
"content": "The quarterly sales report shows a 25% increase in revenue.",
"context": "Q1 financial review"
}
],
"document_id": "sales-report-q1-2024"
}
)
assert response.status_code == 200
print("Created document with memory units")
# Verify document exists
response = await api_client.get(
f"/api/v1/agents/{test_agent_id}/documents/sales-report-q1-2024"
)
assert response.status_code == 200
doc_info = response.json()
initial_units = doc_info["memory_unit_count"]
assert initial_units > 0
print(f"Document has {initial_units} memory units")
# Delete the document
response = await api_client.delete(
f"/api/v1/agents/{test_agent_id}/documents/sales-report-q1-2024"
)
assert response.status_code == 200
delete_result = response.json()
assert delete_result["success"] is True
assert delete_result["document_id"] == "sales-report-q1-2024"
assert delete_result["memory_units_deleted"] == initial_units
print(f"Successfully deleted document and {delete_result['memory_units_deleted']} memory units")
# Verify document is gone (should return 404)
response = await api_client.get(
f"/api/v1/agents/{test_agent_id}/documents/sales-report-q1-2024"
)
assert response.status_code == 404
print("Document deletion verified - returns 404")
# Verify document is not in the list
response = await api_client.get(f"/api/v1/agents/{test_agent_id}/documents")
assert response.status_code == 200
documents = response.json()
doc_ids = [doc["id"] for doc in documents["items"]]
assert "sales-report-q1-2024" not in doc_ids
print("Document not in list - verified")
# Try to delete again (should return 404)
response = await api_client.delete(
f"/api/v1/agents/{test_agent_id}/documents/sales-report-q1-2024"
)
assert response.status_code == 404
print("Double delete returns 404 - verified")

205
memora/tests/test_fact_classification.py Normal file
View file

@ -0,0 +1,205 @@
"""
Test that fact classification correctly identifies agent vs world facts.
"""
import pytest
from datetime import datetime
from memora.fact_extraction import extract_facts_from_text
from memora.llm_wrapper import LLMConfig
@pytest.mark.asyncio
async def test_agent_facts_from_podcast_transcript():
"""
Test that when context identifies someone as 'you', their actions are classified as agent facts.
This test addresses the issue where podcast transcripts with context like
"this was podcast episode between you (Marcus) and Jamie" were extracting
all facts as 'world' instead of properly identifying Marcus's statements as 'agent'.
"""
# Podcast transcript where Marcus (identified as "you") discusses his work
transcript = """
Marcus: I've been working on AI safety research for the past six months.
Jamie: That's really interesting! What specifically are you focusing on?
Marcus: I'm investigating interpretability methods. I believe we need to understand
how models make decisions before we can trust them in critical applications.
Jamie: I completely agree with that approach.
Marcus: I published a paper on this topic last month, and I'm presenting it at
the conference next week.
Jamie: Congratulations! I'd love to read it.
"""
context = "Podcast episode between you (Marcus) and Jamie discussing AI research"
llm_config = LLMConfig.for_memory()
facts = await extract_facts_from_text(
text=transcript,
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config
)
# Should extract at least one fact
assert len(facts) > 0, "Should extract at least one fact from the transcript"
print(f"\nExtracted {len(facts)} facts:")
for i, f in enumerate(facts):
print(f"{i+1}. [{f['fact_type']}] {f['fact']}")
# Marcus's work should be classified as 'agent' since context says "you (Marcus)"
agent_facts = [f for f in facts if f["fact_type"] == "agent"]
assert len(agent_facts) > 0, \
f"Should have at least one 'agent' fact when context identifies 'you (Marcus)'. " \
f"Got facts: {[f['fact'] + ' [' + f['fact_type'] + ']' for f in facts]}"
# Verify that agent facts use FIRST PERSON (not "Marcus", but "I")
for agent_fact in agent_facts:
fact_text = agent_fact["fact"]
# Agent facts should use "I" not the person's name in third person
assert fact_text.startswith("I ") or " I " in fact_text, \
f"Agent facts must use first person ('I'). Got: {fact_text}"
# Should NOT contain "Marcus" as the subject in third person constructions
# (It's ok to have "Marcus" when referring to oneself, but not "Marcus published" style)
import re
# Check for third-person patterns like "Marcus said", "Marcus worked", etc.
third_person_pattern = r'\bMarcus\s+(said|worked|has|published|explained|believes|attended|completed)'
match = re.search(third_person_pattern, fact_text)
assert not match, \
f"Agent facts should use first person, not third person. " \
f"Found '{match.group()}' in: {fact_text}"
print(f"\n✅ All {len(agent_facts)} agent facts use first person ('I')")
# Jamie's statements should be 'world' facts
jamie_facts = [f for f in facts if "Jamie" in f["fact"] and "Jamie" == f["fact"].split()[0]]
if jamie_facts:
world_jamie_facts = [f for f in jamie_facts if f["fact_type"] == "world"]
assert len(world_jamie_facts) > 0, \
f"Jamie's statements should be 'world' facts. " \
f"Jamie facts: {[f['fact'] + ' [' + f['fact_type'] + ']' for f in jamie_facts]}"
print(f"\n✅ Successfully classified {len(agent_facts)} agent facts and {len([f for f in facts if f['fact_type'] == 'world'])} world facts")
print(f"\nAgent facts:")
for f in agent_facts:
print(f" - {f['fact']}")
print(f"\nWorld facts:")
for f in facts:
if f['fact_type'] == 'world':
print(f" - {f['fact']}")
@pytest.mark.asyncio
async def test_agent_facts_without_explicit_context():
"""
Test that when 'you' is used in the text itself, it gets properly classified.
"""
text = """
I completed the project on machine learning interpretability last week.
My colleague Sarah helped me with the data analysis.
We presented our findings to the team yesterday.
"""
context = "Personal work log"
llm_config = LLMConfig.for_memory()
facts = await extract_facts_from_text(
text=text,
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config
)
assert len(facts) > 0, "Should extract facts"
# When text uses "I", those should likely be agent facts
# (though without explicit "you (Name)" in context, this is harder to guarantee)
agent_facts = [f for f in facts if f["fact_type"] == "agent"]
print(f"\n✅ Extracted {len(facts)} total facts")
print(f"Agent facts: {len(agent_facts)}")
print(f"World facts: {len([f for f in facts if f['fact_type'] == 'world'])}")
if agent_facts:
print(f"\nAgent facts found:")
for f in agent_facts:
print(f" - {f['fact']}")
@pytest.mark.asyncio
async def test_skip_podcast_meta_commentary():
"""
Test that podcast intros, outros, and calls to action are skipped.
This addresses the issue where podcast outros like "that's all for today,
don't forget to subscribe" were being extracted as facts.
"""
transcript = """
Marcus: Welcome everyone to today's episode! Before we dive in, don't forget to
subscribe and leave a rating.
Marcus: Today I want to talk about my research on interpretability in AI systems.
I've been working on this for about a year now.
Jamie: That sounds really interesting! What made you focus on that area?
Marcus: I believe it's crucial for AI safety. We need to understand how these
models make decisions before we can trust them in critical applications.
Jamie: I completely agree with that approach.
Marcus: Well, I think that's gonna do it for us today! Thanks for listening everyone.
Don't forget to tap follow or subscribe, tell a friend, and drop a quick rating
so the algorithm learns to box out. See you next week!
"""
context = "Podcast episode between you (Marcus) and Jamie about AI"
llm_config = LLMConfig.for_memory()
facts = await extract_facts_from_text(
text=transcript,
event_date=datetime(2024, 11, 13),
context=context,
llm_config=llm_config
)
print(f"\nExtracted {len(facts)} facts:")
for i, f in enumerate(facts):
print(f"{i+1}. [{f['fact_type']}] {f['fact']}")
# Should extract at least one fact
assert len(facts) > 0, "Should extract at least one fact"
# Check that no facts contain meta-commentary phrases
meta_phrases = [
"subscribe",
"leave a rating",
"tap follow",
"tell a friend",
"that's gonna do it",
"thanks for listening",
"see you next week",
"welcome everyone",
"before we dive in"
]
for fact in facts:
fact_lower = fact["fact"].lower()
for phrase in meta_phrases:
assert phrase not in fact_lower, \
f"Fact should not contain meta-commentary phrase '{phrase}'. " \
f"Found in: {fact['fact']}"
# Should have facts about the actual content (interpretability research)
content_facts = [f for f in facts if "interpretability" in f["fact"].lower()]
assert len(content_facts) > 0, \
"Should extract facts about the actual content discussed (interpretability)"
print(f"\n✅ Successfully filtered out meta-commentary")
print(f"✅ Extracted {len(content_facts)} facts about actual content")

23
scripts/benchmarks/run-locomo.sh
View file

@ -4,28 +4,9 @@ set -e
cd "$(dirname "$0")/../.."
# Parse --env argument to source the right env file
ENV_MODE="local"
ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
--env)
ENV_MODE="$2"
if [[ "$ENV_MODE" != "local" && "$ENV_MODE" != "dev" ]]; then
echo "Error: --env must be 'local' or 'dev'"
exit 1
fi
shift 2
;;
*)
ARGS+=("$1")
shift
;;
esac
done
# Source environment file
ENV_FILE=".env.${ENV_MODE}"
ARGS=("$@")
ENV_FILE=".env"
if [ ! -f "$ENV_FILE" ]; then
echo "Error: Environment file $ENV_FILE not found"
exit 1

24
scripts/benchmarks/run-longmemeval.sh
View file

@ -4,28 +4,8 @@ set -e
cd "$(dirname "$0")/../.."
# Parse --env argument to source the right env file
ENV_MODE="local"
ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
--env)
ENV_MODE="$2"
if [[ "$ENV_MODE" != "local" && "$ENV_MODE" != "dev" ]]; then
echo "Error: --env must be 'local' or 'dev'"
exit 1
fi
shift 2
;;
*)
ARGS+=("$1")
shift
;;
esac
done
# Source environment file
ENV_FILE=".env.${ENV_MODE}"
ARGS=("$@")
ENV_FILE=".env"
if [ ! -f "$ENV_FILE" ]; then
echo "Error: Environment file $ENV_FILE not found"
exit 1