--- title: "Benchmarks" description: "BEAM accuracy at 100K, 500K, 1M and 10M tokens against published results from memory providers, LoCoMo on corrected keys, and the evaluation method." canonical: https://past.dev/benchmarks --- # Benchmark results and methodology > past.dev ranks first on BEAM at every size tested, on complete splits from 100K to 10M tokens. Current public results report answer accuracy on BEAM and LoCoMo, and task accuracy on DolphinBench. The evaluation method treats retrieval sufficiency, cost, and latency as separate metrics. Source: https://past.dev/benchmarks ## BEAM accuracy by history size past.dev figures from the run of 29 September 2026. Every other figure is that vendor's own published number, linked in the table. - 92.08% at 100K tokens, 5.9 points above Hindsight, the next published result at this size. - 89.63% at 500K tokens, 9.5 points above Hindsight, the next published result at this size. - 90.65% at 1M tokens, 9.9 points above Graphon, the next published result at this size. - 85.03% at 10M tokens, 12.0 points above Graphon, the next published result at this size. | System | 100K | 500K | 1M | 10M | Source | | --- | --- | --- | --- | --- | --- | | past.dev | 92.08% | 89.63% | 90.65% | 85.03% | [past.dev](/benchmarks) | | Graphon (no published method) | 83.3% | not published | 80.7% | 73.0% | [graphonlabs.ai](https://www.graphonlabs.ai/#benchmarks) | | Exabase M-1 (harness not published) | 76.9% | not published | 75.0% | 68.0% | [exabase.io](https://exabase.io/blog/exabase-m1-achieves-state-of-the-art-on-beam-benchmark) | | Hindsight | 86.2% | 80.1% | 79.1% | 64.1% | [agentmemorybenchmark.ai](https://agentmemorybenchmark.ai/dataset/beam) | | Honcho | 63.0% | 64.9% | 63.1% | 40.6% | [plasticlabs.ai](https://plasticlabs.ai/blog/research/Benchmarking-Honcho) | | mem0 | not published | not published | 64.1% | 48.6% | [mem0.ai](https://mem0.ai/blog/ai-memory-benchmarks-in-2026) | Every competitor figure is the highest score that vendor has published for its own system on a full BEAM split, linked in the table. Hindsight, Honcho and mem0 publish the code and the per-question results of their runs. Exabase M-1 states its models and links its results, and has not published the harness it ran, a fork of Hindsight's. Graphon publishes every answer and its grade for the three sizes it ran, and no harness, answer model or settings. Graphon's figures were read on November 13, 2026, the others as of October 2, 2026. These are separately published runs rather than one harness, so models, judges and configurations differ between rows, and the past.dev figures are complete splits at every size. past.dev's scores use the same publicly available prompts as other top published scores, unchanged, including their guidance from the dataset's question annotations. One model, openai/gpt-5.6-luna, both answers and judges in this run. [The BEAM paper, published at ICLR 2026](https://arxiv.org/abs/2510.27246) [Reproduce the BEAM run, on GitHub](https://github.com/pastdotdev/benchmarks/tree/main/beam) [The BEAM leaderboard, ranked at each size](https://past.dev/benchmarks/beam) [The DolphinBench leaderboard, agent memory scored on actions](https://past.dev/benchmarks/dolphinbench) ## DolphinBench DolphinBench scores agent memory on the action an agent takes. A task passes when the agent calls the right app with the right arguments, or writes a document its judge accepts. With past.dev as its memory, Hermes Agent passed 490 of 600 tasks (81.7%) on GPT 5.6 Luna, against 424 for Mem0's own memory, the best official result on that configuration. With our own agent in place of Hermes, the same memory passed 520 (86.7%). past.dev's runs are self-administered. DolphinBench has not run or verified them and they are absent from their leaderboard. Tasks passed of 600. Both past.dev rows are our own runs on GPT 5.6 Luna, one with our own agent and one with Hermes Agent. The other five are Mem0's runs on Hermes Agent. | Memory | Tasks passed | Share | | --- | --- | --- | | past.dev (own agent) | 520 / 600 | 86.7% | | past.dev (Hermes) | 490 / 600 | 81.7% | | Mem0 | 424 / 600 | 70.7% | | Hindsight | 417 / 600 | 69.5% | | Honcho | 411 / 600 | 68.5% | | Hermes built-in memory | 394 / 600 | 65.7% | | Supermemory | 355 / 600 | 59.2% | [The DolphinBench leaderboard, every published result](https://past.dev/benchmarks/dolphinbench) [How the run went, task by task](https://past.dev/blog/dolphinbench-results) ## LoCoMo 93.12% over 1,540 questions across 10 conversations, using the community-corrected answer keys. The macro average across the four categories is 89.18%. Run of 16 September 2026. | Category | Accuracy | n | | --- | --- | --- | | Single-hop | 95.01% | 841 | | Temporal | 93.46% | 321 | | Multi-hop | 93.26% | 282 | | Open-domain | 75.00% | 96 | ## Evaluation scope ### Datasets - **BEAM**: Complete splits at 100K, 500K, 1M and 10M tokens, graded across ten abilities. The results above. - **LoCoMo**: Run with the community-corrected answer keys. An audit found errors in roughly 6% of the original answers, and using those keys counts a label error as a memory error. ### How a result is reported - **Baselines**: Full context and plain RAG, both run alongside. - **Judges**: Each judge configuration is recorded with its false-accept rate. - **Metrics**: Accuracy, cost and latency reported separately rather than as one score. [How to reproduce this](https://past.dev/benchmarks/methodology) Get API key: https://sso.past.dev/sign-up