✨ TL;DR
MathNet is a large-scale, multilingual dataset of 30,676 Olympiad-level math problems from 47 countries spanning two decades, designed to benchmark both mathematical reasoning in generative models and mathematical retrieval in embedding systems. The benchmark reveals that even state-of-the-art models struggle with these problems, with top models achieving only 78.4% accuracy, and that retrieval quality significantly impacts retrieval-augmented generation performance.
Existing mathematical reasoning benchmarks suffer from significant limitations in scale, language diversity, and task coverage. Current datasets are too small to adequately test modern large language models, focus predominantly on English, and fail to evaluate critical capabilities like mathematical retrieval—the ability to find semantically or structurally similar problems. This gap is particularly problematic as mathematical problem solving represents a fundamental test of reasoning ability, and real-world mathematical applications often require both solving problems and retrieving relevant prior work or similar examples.
The authors constructed MathNet by collecting 30,676 expert-authored Olympiad-level mathematics problems with solutions from 47 countries across 17 languages, spanning two decades of competitions. They designed a comprehensive benchmark supporting three distinct tasks: (i) Problem Solving, where models generate solutions to problems; (ii) Math-Aware Retrieval, where embedding models must retrieve mathematically equivalent or structurally similar problems from a corpus; and (iii) Retrieval-Augmented Problem Solving, which combines retrieval with generation. For the retrieval benchmark, human experts curated pairs of mathematically equivalent and structurally similar problems to enable rigorous evaluation of mathematical understanding in embedding models.
What the paper shows.
The benchmark evaluation revealed that top-performing models achieve 78.4% accuracy (Gemini-3.1-Pro) and 69.3% accuracy (GPT-5) on the problem-solving task, demonstrating that even state-of-the-art systems are challenged by Olympiad-level mathematics. Embedding models showed poor performance on the mathematical retrieval task, indicating fundamental limitations in capturing mathematical equivalence and structural similarity. In the retrieval-augmented setting, DeepSeek-V3.2-Speciale achieved the highest scores on the benchmark with gains of up to 12% when provided with retrieved examples, though performance was highly dependent on retrieval quality. These results establish strong baselines while highlighting significant gaps in both mathematical reasoning and retrieval capabilities.
While the paper presents a comprehensive benchmark, potential limitations include the focus on Olympiad-level problems which, though challenging, represent a specific subset of mathematical reasoning and may not capture all aspects of mathematical problem-solving encountered in research or applied settings. The retrieval benchmark relies on human-curated pairs of equivalent and similar problems, which may introduce subjective biases in what constitutes mathematical equivalence or structural similarity. The dataset, despite being the largest of its kind, is still limited to competition mathematics from specific countries and time periods, potentially missing mathematical traditions or problem types from underrepresented regions. Additionally, the evaluation focuses primarily on final answer accuracy, which may not fully capture the quality of mathematical reasoning or the pedagogical value of solution approaches.
✨ Generated by Claude · Apr 21, 2026 · Read the PDF for authoritative content.