State of mutation testing at google Petrovic, Goran; Ivankovic, Marko
2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP),
05/2018
Conference Proceeding
Odprti dostop
Mutation testing assesses test suite efficacy by inserting small faults into programs and measuring the ability of the test suite to detect them. It is widely considered the strongest test criterion ...in terms of finding the most faults and it subsumes a number of other coverage criteria. Traditional mutation analysis is computationally prohibitive which hinders its adoption as an industry standard. In order to alleviate the computational issues, we present a diff-based probabilistic approach to mutation analysis that drastically reduces the number of mutants by omitting lines of code without statement coverage and lines that are determined to be uninteresting - we dub these arid lines. Furthermore, by reducing the number of mutants and carefully selecting only the most interesting ones we make it easier for humans to understand and evaluate the result of mutation analysis. We propose a heuristic for judging whether a node is arid or not, conditioned on the programming language. We focus on a code-review based approach and consider the effects of surfacing mutation results on developer attention. The described system is used by 6,000 engineers in Google on all code changes they author or review, affecting in total more than 13,000 code authors as part of the mandatory code review process. The system processes about 30% of all diffs across Google that have statement coverage calculated. About 15% of coverage statement calculations fail across Google.
Does mutation testing improve testing practices? Petrović, Goran; Ivanković, Marko; Fraser, Gordon ...
2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE),
05/2021
Conference Proceeding
Odprti dostop
Various proxy metrics for test quality have been defined in order to guide developers when writing tests. Code coverage is particularly well established in practice, even though the question of how ...coverage relates to test quality is a matter of ongoing debate. Mutation testing offers a promising alternative: Artificial defects can identify holes in a test suite, and thus provide concrete suggestions for additional tests. Despite the obvious advantages of mutation testing, it is not yet well established in practice. Until recently, mutation testing tools and techniques simply did not scale to complex systems. Although they now do scale, a remaining obstacle is lack of evidence that writing tests for mutants actually improves test quality. In this paper we aim to fill this gap: By analyzing a large dataset of almost 15 million mutants, we investigate how these mutants influenced developers over time, and how these mutants relate to real faults. Our analyses suggest that developers using mutation testing write more tests, and actively improve their test suites with high quality tests such that fewer mutants remain. By analyzing a dataset of past fixes of real high-priority faults, our analyses further provide evidence that mutants are indeed coupled with real faults. In other words, had mutation testing been used for the changes introducing the faults, it would have reported a live mutant that could have prevented the bug.
The subject of the research was workforce, its availability and price, and its importance for agriculture development in Bosnia and Herzegovina (BiH). The aim was to determine whether the workforce ...remains to be a comparative advantage or has become a limitation on agricultural development in Bosnia and Herzegovina. The introduction to the selected subject of research was done on the basis of a literature review, followed by an authors’ own survey based on a randomly selected sample of farms. The results confirmed that it is increasingly difficult to find workers for seasonal works in agriculture in Bosnia and Herzegovina and that farms rely primarily on the workforce within families, relatives and friends. The price of labour in agriculture is rising, but it is still lower than in other sectors and abroad, which is the reason why workers are leaving agriculture. Bosnia and Herzegovina compared to three years ago, it is harder for Bosnia and Herzegovina farms to find additional workforce and they pay it more. The future agricultural policy in Bosnia and Herzegovina should seriously count on family farms and their modernization in terms of creating conditions for workforce reduction and substitution.
Mutation testing has been demonstrated to motivate developers to write more tests when presented with undetected, actionable mutants. To facilitate this effect, modern mutation systems aim to ...generate and surface only actionable mutants---few in numbers but highly valuable to the developer. This requires a deeper understanding of the extent to which developers resolve surfaced mutants and how: If they decide not to resolve an undetected mutant, why not? On the other hand, if they do resolve a mutant, do they simply add a test that detects it, or do they also improve the code?
In order to answer these questions we compiled and analyzed a dataset of 1,538 merge requests with corresponding mutants surfaced during the code review phase. Our analysis reveals that determining whether a mutant is indeed resolved during code review is actually a non-trivial problem: for 64% of mutants, the mutated code changes as the merge request evolves, requiring dedicated techniques to precisely resurface the same mutants and to discover which of them remain unresolved after a code change. Overall, our analysis demonstrates that 38% of all surfaced mutants are resolved via code changes or test additions. Out of all mutants that are endorsed by a reviewer, 60% are resolved and result in additional tests, code refactorings, and improved documentation. Unresolved, yet endorsed, mutants stem from developers questioning the value of adding tests for surfaced mutants, later resolving mutants in deferred code changes (atomicity of merge requests), and false positives (mutants being resolved by tests not considered when creating the mutants, e.g., in integration test suites).
The aim of this article is to observe the trade exchange by calculating Relative Trade Advantage index with the wine products (HS 2204) of Bosnia and Herzegovina and the most common destinations ...concerning export and import. The data is used from the trade map data and wine institute data. The trade is based on the former Yugoslavia countries (Serbia, Croatia, North Macedonia, Slovenia, and Montenegro). Export market share and Import market share with these countries ranges from 60% to 95% of overall trade. The most important import partner is Serbia with a stake of 28.2%. The largest export partner is Croatia with 52.3% of all BiH’s export. The calculated RTA index had values from (-0.674) in 2012 to (-0.567) in 2019. Negative values of RTA index represent relative trade disadvantages in Bosnia and Herzegovina’s wine foreign exchange.
The immortelle (Helichrysum italicum /Roth/ G. Don), native to the Mediterranean region grew up and is prevalent on the rockery and rocky grasslands in Bosnia and Herzegovina. The long tradition of ...growing and collecting the medicinal plants in Herzegovina is the reason of its usage in folk medicine according to its antibacterial, antifungal and antioxidant properties. In the last decade we witnessed increased interest of the scientific community and the market demand for this ornamental herb, and the present study evaluated two different rows spacing (0.9 x 0.4 m and 0.8 x 0.4 m) to reveal the economic performance of organically grown immortelle. Results of the research for the both organic immortelle planting spaces revealed that they are economically profitable for producers. Sensitivity analysis with 10% lower yields and with the same retail price revealed that investment is economically efficient.
Code coverage at Google Ivanković, Marko; Petrović, Goran; Just, René ...
Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering,
08/2019
Conference Proceeding
Odprti dostop
Code coverage is a measure of the degree to which a test suite exercises a software system. Although coverage is well established in software engineering research, deployment in industry is often ...inhibited by the perceived usefulness and the computational costs of analyzing coverage at scale. At Google, coverage information is computed for one billion lines of code daily, for seven programming languages. A key aspect of making coverage information actionable is to apply it at the level of changesets and code review. This paper describes Google’s code coverage infrastructure and how the computed code coverage information is visualized and used. It also describes the challenges and solutions for adopting code coverage at scale. To study how code coverage is adopted and perceived by developers, this paper analyzes adoption rates, error rates, and average code coverage ratios over a five-year period, and it reports on 512 responses, received from surveying 3000 developers. Finally, this paper provides concrete suggestions for how to implement and use code coverage in an industrial setting.
There are severale Salmonid species, found in the river Neretva basin, among which S. trutta and S. obtusirostris. Also, natural hybrids such as S. obtusirostris x S. trutta have been observed. In ...one fish farm on the river Neretva, S. trutta and S. obtusirostris were decided to breed separately. Parental fishes were separated phenotypicaly on the basis of the morphological signs. PCR-RFLP analysis of the exon 3 to exon 4 part of the lactate dehydrogenase (LDH) C1* gene with restriction endonuclease RsaI was employed to identify the presence of other species representatives or intercrosses in two groups of juvenille fishes. Using this method, we were able to identify two S. trutta representatives in the S. obtusirostris group.
In modern conditions characterized by the growing importance of foreign trade between the countries, relations of a country with international environment play an increasingly important role in ...economic development. Over the last decade the process of economic integration through the removal of barriers for the free movement of goods, services, money and people has improved job creation and economic growth. Economic cooperation with foreign countries should expect positive effects on the economy of the state if the external economic factor was used in accordance with the plans and programs of economic development of a country. Thereto, there are social forces that can decide relatively and independently to all elements of internal development and cooperation with foreign countries. The objective of this study was to investigate the effect of the CEFTA 2006 on the foreign trade of agricultural products in Bosnia and Herzegovina. In this respect it may be noted that foreign trade of agricultural products has an impact on the agricultural sector in Bosnia and Herzegovina.
Mutation analysis evaluates a testing or debugging technique by measuring how well it detects mutants, which are systematically seeded, artificial faults. Mutation analysis is inherently expensive ...due to the large number of mutants it generates and due to the fact that many of these generated mutants are not effective; they are redundant, equivalent, or simply uninteresting and waste computational resources. A large body of research has focused on improving the scalability of mutation analysis and proposed numerous optimizations to, e.g., select effective mutants or efficiently execute a large number of tests against a large number of mutants. However, comparatively little research has focused on the costs and benefits of mutation testing, in which mutants are presented as testing goals to a developer, in the context of an industrial-scale software development process. This paper draws on an industrial application of mutation testing, involving 30,000+ developers and 1.9 million change sets, written in 4 programming languages. It shows that mutation testing with productive mutants does not add a significant overhead to the software development process and reports on mutation testing benefits perceived by developers. This paper also quantifies the costs of unproductive mutants, and the results suggest that achieving mutation adequacy is neither practical nor desirable. Finally, this paper describes lessons learned from these studies, highlights the current challenges of efficiently and effectively applying mutation testing in an industrial-scale software development process, and outlines research directions.
PDF
