Central Asian
Scientific
Journal

ПӘНДЕР АРАЛЫҚ ҒЫЛЫМДАР ‒ МЕЖДИСЦИПЛИНАРНЫЕ НАУКИ ‒ INTERDISCIPLINARY SCIENCES · ӘОЖ 004.8:811.512.122 · English

How much worse do language models solve the same math problems in Kazakh than in Russian? A re-analysis of published gsm8k results

  • Idrissov Amirzhan Kokshetau Bilim-Innovation Lyceum, Kokshetau, Kazakhstan
Шығарылым
№4(32), Т. 2, 2026
Беттер
485–493
Жарияланды
17.11.2026
PDF жүктеу

Abstract

This paper asks how much less often a language model solves the same mathematics problem in Kazakh than in Russian, and whether the difference shrinks for larger or stronger models. Instead of running models, the study re-analyses the accuracies that three papers published for the same 1,319 GSM8K word problems in Kazakh and Russian machine translation and compares them with the author's forecast, made before he was shown any GSM8K result, that Kazakh accuracy would be about 10% lower. All 10 general-purpose settings evaluated with worked examples scored lower in Kazakh, by 5.3 to 40.2 percentage points (median 14.5): the three strongest in Russian lost 5.3, 9.0 and 11.7 points, close to the forecast, while Qwen2.5-Omni-3B solved 11.4% of the Kazakh problems and 51.6% of the Russian ones. The gap was larger for models with lower Russian accuracy (Spearman ρ = −0.82), smaller for the 7- than for the 3-billion-parameter model of one line (29.4 against 40.2 points) but not steadily smaller with size in a Kazakh-adapted family, and smaller with reasoning in all five comparisons of a reasoning and a non-reasoning variant. For 9 of the 10 general settings the mathematics gap was numerically smaller than the gap in reading comprehension, which is consistent with the author's reasoning that the difficulty lies in understanding the Kazakh text, although the two benchmarks also differ in format and translation.

Keywords: Kazakh language, Russian language, language models, mathematical reasoning, GSM8K, low-resource languages

Әдебиеттер тізімі

  1. Idrissov A. Where do small open language models break on olympiad mathematics? Domain, difficulty and model size on Omni-MATH // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 1. – Pp. 410–414. – DOI: 10.5281/zenodo.23758386.
  2. Togmanov M., Mukhituly N., Turmakhan D. et al. KazMMLU: Evaluating language models on Kazakh, Russian, and regional knowledge of Kazakhstan // Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – 2025. – Pp. 14403–14416. – DOI: 10.18653/v1/2025.acl-long.701.
  3. Mussa A., Tuimebayev Z., Mansurova M. et al. When does fine-tuning matter for a low-resource language? Evaluating open-weight LLMs on Kazakh benchmarks // Information. – 2026. – Vol. 17, No. 9. – Art. 921. – DOI: 10.3390/info17090921.
  4. Zhumagali Y. Does a bigger model close the gap? Kazakh-language accuracy on Kazakhstan-specific versus universal school subjects in small open-weight language models // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 1. – Pp. 593–596. – DOI: 10.5281/zenodo.23758467.
  5. Institute of Smart Systems and Artificial Intelligence (ISSAI). GSM8k_Kazakh_Russian: dataset card [Electronic resource]. – URL: https://huggingface.co/datasets/issai/GSM8k_Kazakh_Russian (accessed: 07.11.2026).
  6. Cobbe K., Kosaraju V., Bavarian M. et al. Training verifiers to solve math word problems [Electronic resource] // arXiv preprint. – 2021. – arXiv:2110.14168. – URL: https://arxiv.org/abs/2110.14168 (accessed: 15.11.2026).
  7. Arystanbekov B., Nurimanov A., Maxutov A. et al. Qolda: A small vision-language model for the Kazakh language // IEEE Access. – 2026. – Vol. 14. – Pp. 46392–46414. – DOI: 10.1109/ACCESS.2026.3676919.
  8. Arystanbekov B., Maxutov A., Nurimanov A. et al. Extending a vision-language model with audio understanding: Introducing Qolda-AVL for the Kazakh language // Big Data and Cognitive Computing. – 2026. – Vol. 10, No. 6. – Art. 192. – DOI: 10.3390/bdcc10060192.
  9. Institute of Smart Systems and Artificial Intelligence (ISSAI). Qolda-AVL-5B: model card [Electronic resource]. – URL: https://huggingface.co/issai/Qolda-AVL-5B (accessed: 07.11.2026).
  10. Bandarkar L., Liang D., Muller B. et al. The Belebele benchmark: A parallel reading comprehension dataset in 122 language variants // Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – 2024. – Pp. 749–775. – DOI: 10.18653/v1/2024.acl-long.44.

Дәйексөз келтіру үлгісі

Idrissov Amirzhan How much worse do language models solve the same math problems in Kazakh than in Russian? A re-analysis of published gsm8k results // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 2. – Pp. 485–493.

CC BY 4.0 Мақала CC BY 4.0 лицензиясы бойынша ашық қолжетімді: оны авторлары мен мақалаға сілтеме көрсетіле отырып еркін оқуға, көшіруге және таратуға болады. Лицензия туралы толығырақ

Мақала туралы пікірлер

Пікір қалдыру

Мақаланы бағалап, пікіріңізбен бөлісіңіз. Пікір модератор тексергеннен кейін шығады.

Баға