A hybrid model for automatic news message classification

Abstract

This paper presents a hybrid model for automatic news classification that combines a pre-trained BERT encoder with a parallel convolutional stream of 2-, 3- and 4-gram filters operating over a shared trainable embedding. The model was evaluated on the AG News benchmark, six further public datasets and a purpose-built noisy corpus of 15,340 articles collected from BBC News, Reuters and Al Jazeera RSS feeds. On AG News the hybrid model reached 95.2% accuracy and a macro F1 of 0.951, 1.7 pp above the BERT-only baseline; on the noisy corpus, together with weighted cross-entropy loss, label smoothing and back-translation, the gain grew to 7.8 pp. The paper also reports a systematic ablation study, an error analysis of the Science/Technology–Business boundary, post-hoc temperature scaling and a boundary-aware re-ranker that further improves macro F1 by 0.7 pp.

Keywords: news classification, BERT, convolutional neural network, hybrid model, text classification, noisy data, natural language processing

Список литературы

1. McCallum A., Nigam K. A comparison of event models for Naive Bayes text classification // Proceedings of the AAAI-98 Workshop on Learning for Text Categorization. – 1998. – P. 41–48.
2. Salton G., Buckley C. Term-weighting approaches in automatic text retrieval // Information Processing and Management. – 1988. – Vol. 24, No. 5. – P. 513–523.
3. Joachims T. Text categorization with support vector machines: learning with many relevant features // Proceedings of ECML-98. – 1998. – P. 137–142.
4. Chen T., Guestrin C. XGBoost: a scalable tree boosting system // Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining. – 2016. – P. 785–794.
5. Distributed representations of words and phrases and their compositionality / T. Mikolov, I. Sutskever, K. Chen [et al.] // Advances in Neural Information Processing Systems. – 2013. – Vol. 26.
6. Pennington J., Socher R., Manning C. D. GloVe: global vectors for word representation // Proceedings of EMNLP 2014. – 2014. – P. 1532–1543.
7. Kim Y. Convolutional neural networks for sentence classification // Proceedings of EMNLP 2014. – 2014. – P. 1746–1751.
8. Hochreiter S., Schmidhuber J. Long short-term memory // Neural Computation. – 1997. – Vol. 9, No. 8. – P. 1735–1780.
9. Attention is all you need / A. Vaswani, N. Shazeer, N. Parmar [et al.] // Advances in Neural Information Processing Systems. – 2017. – Vol. 30.
10. BERT: pre-training of deep bidirectional transformers for language understanding / J. Devlin, M.-W. Chang, K. Lee, K. Toutanova // Proceedings of NAACL-HLT 2019. – 2019. – P. 4171–4186.
11. How to fine-tune BERT for text classification? / C. Sun, X. Qiu, Y. Xu, X. Huang // Proceedings of CCL 2019. – 2019. – P. 194–206.
12. RoBERTa: a robustly optimized BERT pretraining approach / Y. Liu, M. Ott, N. Goyal [et al.] // arXiv preprint. – 2019. – arXiv:1907.11692.
13. XLNet: generalized autoregressive pretraining for language understanding / Z. Yang, Z. Dai, Y. Yang [et al.] // Advances in Neural Information Processing Systems. – 2019. – Vol. 32.
14. ERNIE: enhanced representation through knowledge integration / Y. Sun, S. Wang, Y. Li [et al.] // arXiv preprint. – 2019. – arXiv:1904.09223.
15. Zhang X., Zhao J., LeCun Y. Character-level convolutional networks for text classification // Advances in Neural Information Processing Systems. – 2015. – Vol. 28.
16. Safaya A., Abdullatif M., Yuret D. KUISAIL at SemEval-2020 Task 12: BERT-CNN for offensive speech identification in social media // Proceedings of SemEval-2020. – 2020. – P. 2054–2059.
17. A BERT-based hybrid short text classification model incorporating CNN and attention-based BiGRU / T. Bao, N. Ren, R. Luo [et al.] // Journal of Organizational and End User Computing. – 2021. – Vol. 33, No. 6. – P. 1–21.
18. Alghamdi J., Lin Y., Luo S. Modeling fake news detection using BERT-CNN-BiLSTM architecture // Proceedings of IEEE MIPR 2022. – 2022. – P. 354–357.
19. Chen X., Cong P., Lv S. A long-text classification method of Chinese news based on BERT and CNN // IEEE Access. – 2022. – Vol. 10. – P. 34059–34070.
20. Qwen TextCNN and BERT models for enhanced multilabel news classification in mobile apps / D. Yuan, G. Liang, B. Liu [et al.] // Scientific Reports. – 2025. – Vol. 15. – Art. 43787.
21. Comparative analysis of machine learning algorithms for biomedical text document classification / E. Kucuk, I. B. Cicek, Z. Kucukakcali, C. Yetis // Med. Sci. – 2024. – Vol. 13, No. 1.
22. Fine-tuning BERT for automated news classification / M. I. Salih, S. M. Mohammed, A. K. Ibrahim [et al.] // Engineering, Technology & Applied Science Research. – 2025. – Vol. 15, No. 3. – P. 22953–22959.
23. What does BERT learn about the structure of language? / B. van Aken, B. Winter, A. Löser, F. A. Gers // arXiv preprint. – 2019. – arXiv:1906.04341.
24. Learning with imbalanced noisy data by preventing bias in sample selection / H. Liu, M. Sheng, Z. Sun [et al.] // arXiv preprint. – 2024. – arXiv:2402.11242.
25. Singh N., Jain A. Dual stream graph augmented transformer model integrating BERT and GNNs for context aware fake news detection // Scientific Reports. – 2025. – Vol. 15.
26. Bag of tricks for efficient text classification / A. Joulin, E. Grave, P. Bojanowski, T. Mikolov // Proceedings of EACL 2017. – 2017. – P. 427–431.
27. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter / V. Sanh, L. Debut, J. Chaumond, T. Wolf // arXiv preprint. – 2019. – arXiv:1910.01108.
28. ELECTRA: pre-training text encoders as discriminators rather than generators / K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning // Proceedings of ICLR 2020. – 2020.
29. DeBERTa: decoding-enhanced BERT with disentangled attention / P. He, X. Liu, J. Gao, W. Chen // Proceedings of ICLR 2021. – 2021.
30. He P., Gao J., Chen W. DeBERTaV3: improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing // Proceedings of ICLR 2023. – 2023.
31. ALBERT: a lite BERT for self-supervised learning of language representations / Z. Lan, M. Chen, S. Goodman [et al.] // Proceedings of ICLR 2020. – 2020.

Как цитировать

Astana IT University, Astana, Kazakhstan A hybrid model for automatic news message classification // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 2. – Pp. 56–78. – DOI: 10.5281/zenodo.23796452.