DOES A BIGGER MODEL CLOSE THE GAP? KAZAKH-LANGUAGE ACCURACY ON KAZAKHSTAN-SPECIFIC VERSUS UNIVERSAL SCHOOL SUBJECTS IN SMALL OPEN-WEIGHT LANGUAGE MODELS

Abstract

Open-weight language models are usually less accurate in Kazakh than in Russian, but a single Kazakh-versus-Russian score cannot tell whether the model struggles with the language itself or with knowledge about Kazakhstan. We separate the two with a difference-in-differences design on KazMMLU, a bilingual school-level benchmark. Kazakh History, a Kazakhstan-specific subject available in both languages, is compared with mathematics, physics and chemistry, which are Kazakhstan-neutral. We evaluate three sizes of the same open model family (Qwen2.5-Instruct 1.5B, 3B and 7B) with deterministic multiple-choice scoring. On universal subjects the gap between Kazakh and Russian stays within about 1.5 percentage points at every size. On Kazakh History, Kazakh is 13 to 18 points behind Russian, and the gap does not shrink as the model grows: the extra Kazakh penalty on Kazakhstan content is −12.6, −12.9 and −16.7 points (95% CI excludes zero for all three). Scaling within this family improves knowledge about Kazakhstan mainly through Russian. We discuss training-data imbalance as the likely cause and the limits of a single-family, single-subject design.

Keywords: Kazakh language, language models, KazMMLU, low-resource languages, regional knowledge, model scaling

Әдебиеттер тізімі

1. Togmanov M., Mukhituly N., Turmakhan D. [et al.] KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan // Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. Vol. 1. – 2025. – P. 14403–14416. – URL: https://arxiv.org/abs/2502.12829 (accessed: 05.10.2026).
2. Mussa A., Tuimebayev Z., Mansurova M., Shihab A.H. When Does Fine-Tuning Matter for a Low-Resource Language? Evaluating Open-Weight LLMs on Kazakh Benchmarks // Information. – 2026. – Vol. 17, No. 9. – Art. 921. – URL: https://www.mdpi.com/2078-2489/17/9/921 (accessed: 05.10.2026).
3. Goloburda M., Laiyk N., Turmakhan D. [et al.] Qorgau: Evaluating LLM Safety in Kazakh-Russian Bilingual Contexts [Electronic resource]. – 2025. – arXiv:2502.13640. – URL: https://arxiv.org/abs/2502.13640 (accessed: 05.10.2026).
4. Qwen Team. Qwen2.5 Technical Report [Electronic resource]. – 2024. – arXiv:2412.15115. – URL: https://arxiv.org/abs/2412.15115 (accessed: 05.10.2026).

Дәйексөз келтіру үлгісі

Zhumagali Y. DOES A BIGGER MODEL CLOSE THE GAP? KAZAKH-LANGUAGE ACCURACY ON KAZAKHSTAN-SPECIFIC VERSUS UNIVERSAL SCHOOL SUBJECTS IN SMALL OPEN-WEIGHT LANGUAGE MODELS // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 1. – Pp. 304–307.