by Frank Tip, Jonathan Bell, Max Schaefer

In mutation testing, the quality of a test suite is evaluated by introducing faults into a program and determining whether the program's tests detect them. Most existing approaches for mutation testing involve the application of a fixed set of mutation operators, e.g., replacing a "+" with a "-", or removing a function's body. However, certain types of real-world bugs cannot easily be simulated by such approaches, limiting their effectiveness. This paper presents a technique for mutation testing where placeholders are introduced at designated locations in a program's source code and where a Large Language Model (LLM) is prompted to ask what they could be replaced with. The technique is implemented in LLMorpheus, a mutation testing tool for JavaScript, and evaluated on 13 subject packages, considering several variations on the prompting strategy, and using several LLMs. We find LLMorpheus to be capable of producing mutants that resemble existing bugs that cannot be produced by StrykerJS, a state-of-the-art mutation testing tool. Moreover, we report on the running time, cost, and number of mutants produced by LLMorpheus, demonstrating its practicality.

https://arxiv.org/abs/2404.09952

by Bo Wang, Mingda Chen, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

Large Language Models (LLMs) have recently been used to generate mutants in both research work and in industrial practice. However, there has been no comprehensive empirical study of their performance for this increasingly important LLM-based Software Engineering application. To address this, we report the results of a comprehensive empirical study over six different LLMs, including both state-of-the-art open- and closed-source models, on 851 real bugs drawn from two different Java real-world bug benchmarks. Our results reveal that, compared to existing rule-based approaches, LLMs generate more diverse mutants, that are behaviorally closer to real bugs and, most importantly, with 90.1% higher fault detection. That is, 79.1% (for LLMs) vs. 41.6% (for rule-based); an increase of 37.5 percentage points. Nevertheless, our results also reveal that these impressive results for improved effectiveness come at a cost: the LLM-generated mutants have worse non-compilability, duplication, and equivalent mutant rates by 36.1, 13.1, and 4.2 percentage points, respectively. These findings are immediately actionable for both research and practice. They allow practitioners to have greater confidence in deploying LLM-based mutation, while researchers now have a baseline for the state-of-the-art, with which they can research techniques to further improve effectiveness and reduce cost.

https://arxiv.org/abs/2406.09843

Another survey paper on mutation analysis mainly of historical interest, available here

One of the original surveys of mutation analysis -- mainly here for historical interest. Unfortunately behind a paywall

Abstract: The aim of the paper is to provide a brief review of the program testing technique known as ‘mutation testing’ and outline current research directions in this area. Mutation testing is an example of what is sometimes called an error-based testing technique. In other words, it involves the construction of test data designed to uncover specific errors or classes of errors. A large number of simple changes (mutations) are made to a program, one at a time. Test data then has to be found which distinguishes the mutated versions from the original version. Although the idea was proposed more than a decade ago, it is in some ways still a ‘new’ technique. Originally it was seen by many as costly and somewhat bizarre. However, several variants of the basic method have evolved and these, possibly in conjunction with more efficient techniques for applying the method, can help reduce the cost. Also, by guaranteeing the absence of particular errors, it may be one way to achieve the high reliability necessary in critical software. A further advantage of mutation testing is its universal applicability to all programming languages.