holetron-lab/fleet-memory
1
0
Fork 0
Code Releases Packages Activity Actions

t5

Browse source
This commit is contained in:
Nicolò Boschi 2025-11-17 18:39:23 +01:00
parent b894e818fd
commit 5ab7769f0e
3 changed files with 1538735 additions and 797186 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2335902
memora-dev/benchmarks/locomo/results/benchmark_results.json
View file

File diff suppressed because it is too large Load diff

13
memora-dev/benchmarks/locomo/results/results_table.md
View file

@ -1,7 +1,16 @@
# LoComo Benchmark Results
**Overall Accuracy**: 69.11% (132/191)
**Overall Accuracy**: 68.42% (1055/1542)
| Sample ID | Sessions | Questions | Correct | Accuracy | Multi-hop | Single-hop | Temporal | Open-domain |
|-----------|----------|-----------|---------|----------|-----------|------------|----------|-------------|
| conv-48 | 30 | 191 | 132 | 69.11% | N/A | N/A | N/A | N/A |
| conv-26 | 19 | 154 | 97 | 62.99% | N/A | N/A | N/A | N/A |
| conv-30 | 19 | 81 | 56 | 69.14% | N/A | N/A | N/A | N/A |
| conv-41 | 32 | 152 | 111 | 73.03% | N/A | N/A | N/A | N/A |
| conv-42 | 29 | 199 | 133 | 66.83% | N/A | N/A | N/A | N/A |
| conv-43 | 29 | 178 | 124 | 69.66% | N/A | N/A | N/A | N/A |
| conv-44 | 28 | 123 | 85 | 69.11% | N/A | N/A | N/A | N/A |
| conv-47 | 31 | 150 | 99 | 66.00% | N/A | N/A | N/A | N/A |
| conv-48 | 30 | 191 | 125 | 65.45% | N/A | N/A | N/A | N/A |
| conv-49 | 25 | 156 | 110 | 70.51% | N/A | N/A | N/A | N/A |
| conv-50 | 30 | 158 | 115 | 72.78% | N/A | N/A | N/A | N/A |

6
memora/memora/query_analyzer.py
View file

@ -105,12 +105,12 @@ class TransformerQueryAnalyzer(QueryAnalyzer):
"Install it with: pip install transformers"
)
logger.info(f"Loading T5 model: {self.model_name}...")
logger.debug(f"Loading T5 model: {self.model_name}...")
self._tokenizer = AutoTokenizer.from_pretrained(self.model_name)
self._model = AutoModelForSeq2SeqLM.from_pretrained(self.model_name)
self._model.to(self.device)
self._model.eval()
logger.info(f"Model loaded on {self.device}")
logger.debug(f"Model loaded on {self.device}")
def analyze(
self, query: str, reference_date: Optional[datetime] = None
@ -158,7 +158,7 @@ what is the weather = none
)
result = self._tokenizer.decode(outputs[0], skip_special_tokens=True).strip()
logger.info(f"T5 generated: '{result}'")
logger.debug(f"T5 generated: '{result}'")
# Parse the generated output
temporal = self._parse_generated_output(result, reference_date)