State of mutation testing at google Petrovic, Goran; Ivankovic, Marko
2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP),
05/2018
Conference Proceeding
Odprti dostop
Mutation testing assesses test suite efficacy by inserting small faults into programs and measuring the ability of the test suite to detect them. It is widely considered the strongest test criterion ...in terms of finding the most faults and it subsumes a number of other coverage criteria. Traditional mutation analysis is computationally prohibitive which hinders its adoption as an industry standard. In order to alleviate the computational issues, we present a diff-based probabilistic approach to mutation analysis that drastically reduces the number of mutants by omitting lines of code without statement coverage and lines that are determined to be uninteresting - we dub these arid lines. Furthermore, by reducing the number of mutants and carefully selecting only the most interesting ones we make it easier for humans to understand and evaluate the result of mutation analysis. We propose a heuristic for judging whether a node is arid or not, conditioned on the programming language. We focus on a code-review based approach and consider the effects of surfacing mutation results on developer attention. The described system is used by 6,000 engineers in Google on all code changes they author or review, affecting in total more than 13,000 code authors as part of the mandatory code review process. The system processes about 30% of all diffs across Google that have statement coverage calculated. About 15% of coverage statement calculations fail across Google.
Does Mutation Testing Improve Testing Practices? Petrovic, Goran; Ivankovic, Marko; Fraser, Gordon ...
2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)
Conference Proceeding
Various proxy metrics for test quality have been defined in order to guide developers when writing tests. Code coverage is particularly well established in practice, even though the question of how ...coverage relates to test quality is a matter of ongoing debate. Mutation testing offers a promising alternative: Artificial defects can identify holes in a test suite, and thus provide concrete suggestions for additional tests. Despite the obvious advantages of mutation testing, it is not yet well established in practice. Until recently, mutation testing tools and techniques simply did not scale to complex systems. Although they now do scale, a remaining obstacle is lack of evidence that writing tests for mutants actually improves test quality. In this paper we aim to fill this gap: By analyzing a large dataset of almost 15 million mutants, we investigate how these mutants influenced developers over time, and how these mutants relate to real faults. Our analyses suggest that developers using mutation testing write more tests, and actively improve their test suites with high quality tests such that fewer mutants remain. By analyzing a dataset of past fixes of real high-priority faults, our analyses further provide evidence that mutants are indeed coupled with real faults. In other words, had mutation testing been used for the changes introducing the faults, it would have reported a live mutant that could have prevented the bug.
Mutation testing has been demonstrated to motivate developers to write more tests when presented with undetected, actionable mutants. To facilitate this effect, modern mutation systems aim to ...generate and surface only actionable mutants-few in numbers but highly valuable to the developer. This requires a deeper understanding of the extent to which developers resolve surfaced mutants and how: If they decide not to resolve an undetected mutant, why not? On the other hand, if they do resolve a mutant, do they simply add a test that detects it, or do they also improve the code?In order to answer these questions we compiled and analyzed a dataset of 1,538 merge requests with corresponding mutants surfaced during the code review phase. Our analysis reveals that determining whether a mutant is indeed resolved during code review is actually a non-trivial problem: for 64% of mutants, the mutated code changes as the merge request evolves, requiring dedicated techniques to precisely resurface the same mutants and to discover which of them remain unresolved after a code change. Overall, our analysis demonstrates that 38% of all surfaced mutants are resolved via code changes or test additions. Out of all mutants that are endorsed by a reviewer, 60% are resolved and result in additional tests, code refactorings, and improved documentation. Unresolved, yet endorsed, mutants stem from developers questioning the value of adding tests for surfaced mutants, later resolving mutants in deferred code changes (atomicity of merge requests), and false positives (mutants being resolved by tests not considered when creating the mutants, e.g., in integration test suites).
The immortelle (Helichrysum italicum /Roth/ G. Don), native to the Mediterranean region grew up and is prevalent on the rockery and rocky grasslands in Bosnia and Herzegovina. The long tradition of ...growing and collecting the medicinal plants in Herzegovina is the reason of its usage in folk medicine according to its antibacterial, antifungal and antioxidant properties. In the last decade we witnessed increased interest of the scientific community and the market demand for this ornamental herb, and the present study evaluated two different rows spacing (0.9 x 0.4 m and 0.8 x 0.4 m) to reveal the economic performance of organically grown immortelle. Results of the research for the both organic immortelle planting spaces revealed that they are economically profitable for producers. Sensitivity analysis with 10% lower yields and with the same retail price revealed that investment is economically efficient.
There are severale Salmonid species, found in the river Neretva basin, among which S. trutta and S. obtusirostris. Also, natural hybrids such as S. obtusirostris x S. trutta have been observed. In ...one fish farm on the river Neretva, S. trutta and S. obtusirostris were decided to breed separately. Parental fishes were separated phenotypicaly on the basis of the morphological signs. PCR-RFLP analysis of the exon 3 to exon 4 part of the lactate dehydrogenase (LDH) C1* gene with restriction endonuclease RsaI was employed to identify the presence of other species representatives or intercrosses in two groups of juvenille fishes. Using this method, we were able to identify two S. trutta representatives in the S. obtusirostris group.
Mutation analysis evaluates a testing or debugging technique by measuring how well it detects mutants, which are systematically seeded, artificial faults. Mutation analysis is inherently expensive ...due to the large number of mutants it generates and due to the fact that many of these generated mutants are not effective; they are redundant, equivalent, or simply uninteresting and waste computational resources. A large body of research has focused on improving the scalability of mutation analysis and proposed numerous optimizations to, e.g., select effective mutants or efficiently execute a large number of tests against a large number of mutants. However, comparatively little research has focused on the costs and benefits of mutation testing, in which mutants are presented as testing goals to a developer, in the context of an industrial-scale software development process. This paper draws on an industrial application of mutation testing, involving 30,000+ developers and 1.9 million change sets, written in 4 programming languages. It shows that mutation testing with productive mutants does not add a significant overhead to the software development process and reports on mutation testing benefits perceived by developers. This paper also quantifies the costs of unproductive mutants, and the results suggest that achieving mutation adequacy is neither practical nor desirable. Finally, this paper describes lessons learned from these studies, highlights the current challenges of efficiently and effectively applying mutation testing in an industrial-scale software development process, and outlines research directions.
Mutation analysis assesses a test suite's adequacy by measuring its ability to detect small artificial faults, systematically seeded into the tested program. Mutation analysis is considered one of ...the strongest test-adequacy criteria. Mutation testing builds on top of mutation analysis and is a testing technique that uses mutants as test goals to create or improve a test suite. Mutation testing has long been considered intractable because the sheer number of mutants that can be created represents an insurmountable problem-both in terms of human and computational effort. This has hindered the adoption of mutation testing as an industry standard. For example, Google has a codebase of two billion lines of code and more than 150,000,000 tests are executed on a daily basis. The traditional approach to mutation testing does not scale to such an environment; even existing solutions to speed up mutation analysis are insufficient to make it computationally feasible at such a scale. To address these challenges, this paper presents a scalable approach to mutation testing based on the following main ideas: (1) mutation testing is done incrementally, mutating only changed code during code review, rather than the entire code base; (2) mutants are filtered, removing mutants that are likely to be irrelevant to developers, and limiting the number of mutants per line and per code review process; (3) mutants are selected based on the historical performance of mutation operators, further eliminating irrelevant mutants and improving mutant quality. This paper empirically validates the proposed approach by analyzing its effectiveness in a code-review-based setting, used by more than 24,000 developers on more than 1,000 projects. The results show that the proposed approach produces orders of magnitude fewer mutants and that context-based mutant filtering and selection improve mutant quality and actionability. Overall, the proposed approach represents a mutation testing framework that seamlessly integrates into the software development workflow and is applicable to industrial settings of any size.
U svijetu se sve veća pozornost posvećuje zdravlju pučanstva i očuvanju bioraznolikosti i prirodne ravnoteže. Briga za zdravlje je sve naglašenija kroz prehranu i vidljiva je kroz porast potražnje ...poljoprivrednih proizvoda i prerađevina proizvedenih po ekološkim principima. Istovremeno, potrebni duži ili kraći odmori često su kombinacija „konzumacije turističkih sadržaja i gastronomije“. Blagodat koju ima Republika Hrvatska u ponudi turističkih sadržaja te u bogatstvu kvalitetnih poljoprivrednih proizvodnih resursa za ekološku poljoprivrednu proizvodnju, prilika su za njihov jači sinergijski učinak. U radu se daje pregled razvitka ekološke poljoprivredne proizvodnje i promjene kod broja turista u Republici Hrvatskoj. U razdoblju od 2007. do 2017. godine površine pod ekološkom proizvodnjom porasle su za trinaest puta ali čine još uvijek mali dio (6,46%) ukupno korištenih poljoprivrednih površina. U 2017. godini ekološkom proizvodnjom se bavilo 4.023 proizvođača te se predviđa da bi u 2020. godini njihov broj porastao za 15,7% odnosno dosegao bi broj od 4.655. Dosadašnji značajan porast broja turista i ukupno ostvarenih noćenja mogao bi nastaviti daljnji porast, te se predviđa da bi u 2020. godini dosegnuo oko 117 milijuna noćenja, što bi u odnosu na 2018. bio porast od 15,4%. Ovo nam ukazuje na potrebu i mogućnost veće ekološke poljoprivredne proizvodnje i njene veće zastupljenosti u ukupnom tržištu poljoprivredno-prehrambenih proizvoda.
PDF
