Central Asian
Scientific
Journal

ТЕХНИКАЛЫҚ ҒЫЛЫМДАР - ТЕХНИЧЕСКИЕ НАУКИ - TECHNICAL SCIENCE · UDC 004.8:004.415.53 · English

How much do admission tests catch? A mutation study of gates for LLM-generated tools from two models

  • Kaisar Turysbek International Information Technology University, Almaty, Kazakhstan
Issue
No. 4(32), Vol. 2, 2026
Pages
424–434
Published
Nov 17, 2026
Download PDF

Abstract

This study measures how many defective tools two admission tests let into the tool library of an agent built on a large language model, using as test material the tools that language models actually wrote. From 90 tools written by one model (study V7) and 134 written by a second model (study V13) we generated 2,650 and 4,096 token-level mutants (one token or one adjacent token pair changed), ran seeded samples of 480 and 448 in a sandbox through a basic gate (shared execution checks plus a visibly valid result on the tool's original task) and a stronger gate (the basic gate plus five authored examples), and judged every mutant on nine authored test cases that no gate sees. In V7, 106 mutants passed the shared checks but failed a test case that their unmodified tool passed; the basic gate admitted 88 of them and the stronger gate 10, at the cost of 1,050 extra sandbox calls and 4 rejections among 123 mutants with no observed defect. In V13, among the mutants of tools that themselves passed both gates, the same counts were 92, 68 and 8. On these injected defects the stronger gate detected far more defective tools than the basic gate, and we consider its extra cost worth paying.

Keywords: large language model agents, generated tools, admission testing, mutation testing, test adequacy, sandboxed execution

References

  1. Cai T., Wang X., Ma T. et al. Large Language Models as Tool Makers [Electronic resource] // arXiv preprint arXiv:2305.17126v2. – 2023 (rev. 2024). – URL: https://arxiv.org/abs/2305.17126v2 (accessed: 17.11.2026).
  2. Yuan L., Chen Y., Wang X. et al. CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [Electronic resource] // arXiv preprint arXiv:2309.17428v2. – 2023 (rev. 2024). – URL: https://arxiv.org/abs/2309.17428v2 (accessed: 17.11.2026).
  3. Qian C., Han C., Fung Y. R. et al. CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [Electronic resource] // arXiv preprint arXiv:2305.14318v3. – 2023 (rev. 2024). – URL: https://arxiv.org/abs/2305.14318v3 (accessed: 17.11.2026).
  4. Wang Z., Neubig G., Fried D. TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks // Proceedings of Machine Learning Research. – 2024. – Vol. 235. – P. 51177–51191. – URL: https://proceedings.mlr.press/v235/wang24az.html (accessed: 17.11.2026).
  5. Wang G., Xie Y., Jiang Y. et al. Voyager: An Open-Ended Embodied Agent with Large Language Models [Electronic resource] // arXiv preprint arXiv:2305.16291v2. – 2023. – URL: https://arxiv.org/abs/2305.16291v2 (accessed: 17.11.2026).
  6. Berlot-Attwell I., Rudzicz F., Si X. Library Learning Doesn't: The Curious Case of the Single-Use "Library" [Electronic resource] // 4th MATH-AI Workshop at NeurIPS 2024; arXiv preprint arXiv:2410.20274. – 2024. – URL: https://arxiv.org/abs/2410.20274 (accessed: 17.11.2026).
  7. Liu J., Xia C. S., Wang Y., Zhang L. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation [Electronic resource] // arXiv preprint arXiv:2305.01210v3. – 2023. – URL: https://arxiv.org/abs/2305.01210v3 (accessed: 17.11.2026).
  8. Wang Y., Pradel M., Liu Z. Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study [Electronic resource] // arXiv preprint arXiv:2503.15223v2. – 2025. – URL: https://arxiv.org/abs/2503.15223v2 (accessed: 17.11.2026).
  9. Turysbek K. Validating before reuse: a controlled study of admission gates for LLM-generated tool libraries // Студенческий вестник: электрон. научн. журн. – 2026. – № 42(422). – URL: https://www.internauka.org/journal/stud/herald/422 (accessed: 17.11.2026).
  10. DeMillo R. A., Lipton R. J., Sayward F. G. Hints on Test Data Selection: Help for the Practicing Programmer // Computer. – 1978. – Vol. 11, No. 4. – P. 34–41. – DOI: 10.1109/C-M.1978.218136.
  11. Jia Y., Harman M. An Analysis and Survey of the Development of Mutation Testing // IEEE Transactions on Software Engineering. – 2011. – Vol. 37, No. 5. – P. 649–678. – DOI: 10.1109/TSE.2010.62.
  12. Just R., Jalali D., Inozemtseva L. et al. Are Mutants a Valid Substitute for Real Faults in Software Testing? // Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). – 2014. – P. 654–665. – DOI: 10.1145/2635868.2635929.
  13. Claessen K., Hughes J. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs // Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP '00). – 2000. – P. 268–279. – DOI: 10.1145/351240.351266.
  14. Chen X., Lin M., Schärli N., Zhou D. Teaching Large Language Models to Self-Debug [Electronic resource] // arXiv preprint arXiv:2304.05128v2. – 2023. – URL: https://arxiv.org/abs/2304.05128v2 (accessed: 17.11.2026).
  15. Bubblewrap: Low-Level Unprivileged Sandboxing Tool [Electronic resource] // GitHub, containers project. – URL: https://github.com/containers/bubblewrap (accessed: 08.11.2026).
  16. Seccomp BPF (SECure COMPuting with Filters) [Electronic resource] // The Linux Kernel Documentation. – URL: https://www.kernel.org/doc/html/latest/userspace-api/seccomp_filter.html (accessed: 08.11.2026).
  17. Kapoor S., Stroebl B., Siegel Z. S. et al. AI Agents That Matter [Electronic resource] // arXiv preprint arXiv:2407.01502v1. – 2024. – URL: https://arxiv.org/abs/2407.01502v1 (accessed: 17.11.2026).

How to cite

Kaisar Turysbek How much do admission tests catch? A mutation study of gates for LLM-generated tools from two models // Central Asian Scientific Journal. – 2026. – No. 4(32), Vol. 2. – Pp. 424–434.

CC BY 4.0 Open access under the CC BY 4.0 license: you may freely read, copy and share this article, giving credit to the authors and a link to it. About the license

Reviews of this article

Leave a review

Rate the article and share your opinion. Your review appears after moderation.

Rating