Evaluation: Observed Fixture Metrics
Local verification on a synthetic corpus — exact numbers, formulas, and how to rerun. No implied general ranking quality.
Fixture and Environment
Tested on Python 3.14 (2026-10-25) with seed=7 and k=5. The corpus is a synthetic set of 16 entries; the evaluation uses exactly 5 queries. Two independent runs produced identical aggregate metric rows, but the full JSON output differed in event IDs and timestamps — the cause is unverified. We do not claim bit‑reproducibility, Python 3.9 compatibility, or any external benchmark performance.
Observed Aggregate Metrics
| Retriever | mean_recall@k | mean_precision@k | MRR |
|---|---|---|---|
| BM25 | 0.90 | 0.28 | 1.00 |
| Tag scorer | 1.00 | 0.32 | 1.00 |
Values are exact from the local run. The evaluation artifact is at /evaluation/eval-results.json.
Metric Formulas
- recall@k = |relevant ∩ retrieved| / |relevant|
- precision@k = |relevant ∩ retrieved| / k
- MRR = mean over queries of 1 / rank of first relevant result
Bounds: recall@k ∈ [0,1], precision@k ∈ [0,1], MRR ∈ [0,1].
How to Rerun
Clone the source and run the evaluation module:
python eval.py --seed 7 --k 5
The evaluation code is MIT‑licensed and lives at /code/agentmemory/eval.py. See the license for terms.
Why MRR = 1.0 Despite Imperfect Recall
MRR measures the rank of the first relevant result. With only 5 queries and a small synthetic corpus, even when one query misses a relevant item (recall 0.9), the first retrieved result can still be relevant for all queries — giving a perfect MRR. This is a fixture‑specific observation, not evidence of general ranking quality. The corpus is tiny and synthetic; results should not be extrapolated to real‑world retrieval.
Reproducibility Caveat
Aggregate metrics are stable across runs, but the full JSON is not bit‑identical. Event IDs and timestamps differ; we have not diagnosed the cause. We only claim observed metric stability, not deterministic output. Do not rely on exact JSON equality.
The reference archive contains the runnable package, examples and evaluation artifact. Source files are browsable at /code/agentmemory/*.py and examples at /code/examples/quickstart.py and /code/examples/provenance.py.
