WHERE DO SMALL OPEN LANGUAGE MODELS BREAK ON OLYMPIAD MATHEMATICS? DOMAIN, DIFFICULTY AND MODEL SIZE ON OMNI-MATH

Abstract

Small language models that can run on a laptop are increasingly used by students to check solutions, yet little is known about where on olympiad mathematics they fail. This study tested three open models of the same family, Qwen2.5-Instruct with 1.5, 3 and 7 billion parameters, on 287 problems from the Omni-MATH benchmark, sampled across four domains (algebra, geometry, number theory and discrete mathematics) and three difficulty bands. Before any model was run, the author predicted that a larger model would gain most in algebra, where known methods carry a solution, and least in geometry, where the right configuration or construction has to be noticed. Overall accuracy was 15.3%, 13.2% and 26.8% for the 1.5B, 3B and 7B models. The 7B model solved 50.7% of algebra problems but only 17.3% of geometry problems, 25.3% of number theory and 11.3% of discrete mathematics. From the 3B to the 7B model, algebra rose by 28 percentage points and geometry by 8. On middle-difficulty problems the 7B model solved 40% in algebra and 4% in geometry. Nine geometry problems could not be solved from the text alone because they referred to a missing figure; removing them raised 7B geometry accuracy only to 19.7%. The results support the prediction: for small models, size helps algebra much more than geometry, and the geometry gap is not explained by missing figures.

Keywords: language models, olympiad mathematics, geometry, model size, Omni-MATH

Список литературы

1. Hendrycks D., Burns C., Kadavath S., Arora A., Basart S., Tang E., Song D., Steinhardt J. Measuring mathematical problem solving with the MATH dataset // arXiv. 2021. arXiv:2103.03874. https://arxiv.org/abs/2103.03874
2. Gao B., Song F., Yang Z., Cai Z., et al. Omni-MATH: A universal Olympiad level mathematic benchmark for large language models // arXiv. 2024. arXiv:2410.07985. https://arxiv.org/abs/2410.07985
3. Qwen Team (Yang A., Yang B., et al.). Qwen2.5 technical report // arXiv. 2024. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
4. Yang A., Zhang B., Hui B., Gao B., et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement // arXiv. 2024. arXiv:2409.12122. https://arxiv.org/abs/2409.12122
5. Kaplan J., McCandlish S., Henighan T., Brown T.B., et al. Scaling laws for neural language models // arXiv. 2020. arXiv:2001.08361. https://arxiv.org/abs/2001.08361

Как цитировать

Idrissov Amirzhan WHERE DO SMALL OPEN LANGUAGE MODELS BREAK ON OLYMPIAD MATHEMATICS? DOMAIN, DIFFICULTY AND MODEL SIZE ON OMNI-MATH // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 1. – Pp. 296–300.