•A survey of 166 sources about smells in software test code is presented.•The survey classifies the knowledge from both industry and academia.•The survey presents the largest catalogue of test ...smells.•Also, guidelines/techniques and the tools available to deal with those smells are reviewed.•This paper will benefit readers by serving as an “index” to the vast body of knowledge.
As a type of anti-pattern, test smells are defined as poorly designed tests and their presence may negatively affect the quality of test suites and production code. Test smells are the subject of active discussions among practitioners and researchers, and various guidelines to handle smells are constantly offered for smell prevention, smell detection, and smell correction. Since there is a vast grey literature as well as a large body of research studies in this domain, it is not practical for practitioners and researchers to locate and synthesize such a large literature. Motivated by the above need and to find out what we, as the community, know about smells in test code, we conducted a ‘multivocal’ literature mapping (classification) on both the scientific literature and also practitioners’ grey literature. By surveying all the sources on test smells in both industry (120 sources) and academia (46 sources), 166 sources in total, our review presents the largest catalogue of test smells, along with the summary of guidelines/techniques and the tools to deal with those smells. This article aims to benefit the readers (both practitioners and researchers) by serving as an “index” to the vast body of knowledge in this important area, and by helping them develop high-quality test scripts, and minimize occurrences of test smells and their negative consequences in large test automation projects.
Software Testing is an integral part of high quality software development process. To enhance quality, IT Companies are spending huge amount of money, labor and time on testing efforts. Test ...Automation tools help speed up testing effort, improve quality and also reduce cost in the long run. Currently, a combination of Open Source and Commercial Test Automation tools are employed to perform Functional Testing activities. With wide adoption of tools like Selenium, Open Source Functional Test Automation tools are accepted as a viable alternative for Commercial Tools. Professionals involved in the selection and purchase of a new Test Automation Tool or replace an existing tool are reviewing and analyzing various factors and features that drive them to select a particular tool. Research Reports, Tool Adoption Trends in Industry and Comparative review of various Test Automation Tools are some of the sources for them to guide their tool selection decision. As Open Source tools are accepted as viable alternative to Commercial tools and are highly economical, an empirical assessment focusing on the factors and features influencing the selection of these Open Source Functional Test Automation tools is attempted. This research aims in identifying the key factors and features that influence the selection of these tools by surveying IT Companies located in India who have already adopted these tools.
Highly Automated Vehicles (HAVs) must undergo strict safety testing before being released to the public. Mileage-based on-road testing suffers from unaffordable time costs and high safety risks. ...Simulated scenario-based testing has been found to be a trustworthy alternative for testing HAVs' built-in algorithms and functionalities. Test automation is typically used to generate target scenarios. This approach facilitates customized testing and avoids wasting time on simple and redundant scenarios. This study aims to review test automation methods and discuss how to accentuate their strengths rather than be trapped in their weaknesses under certain applicable conditions. According to their main purposes, we classify test automation methods into coverage-oriented, unsafe-scenario-oriented, and naturalistic-assessment-oriented categories. To further demonstrate the differences of these methods, we then design numerical experiment to compare the capabilities of seven test automation methods. Finally, we compile our observations to form a comprehensive guide for selecting test automation methods with different test requirements in mind.
The article presents the multifunctional BrowserSpot tool, which serves as an automated environment for testing websites and web applications for Android and iOS systems. It highlights and describes ...the individual stages of research and development work, the issues with solutions currently available on the market, as well as the project's results. The article also discusses the reasons for undertaking work on the tool, its functionalities, and the methods of its usage.
Poor software quality results in avoidable costs of trillion dollars annually in the United States alone. Augmented Reality (AR) applications are a relatively new software category. Currently there ...are no standards to guide the development process and testing is predominantly ad hoc and manual. Consequently, design guidelines and software test automation techniques are intended to remedy the situation. Here, we present a concept for test automation of AR applications. The concept consists of two parts: design guidelines and process model for testing AR applications, and a case study with a prototype application for test automation. The design guidelines and the process model are based on the state-of-the-art. The prototype application presented in this article demonstrates test automation for a multi-platform AR application for Android devices as well as the HoloLens 2. The presented test automation case study is designed to cover a large part of the functions, such as the different interaction variants. This research work shows that by using the proposed process model and test automation techniques, testing of some features of AR applications can be automated. The results of this research can serve as a basis for future research and contribution towards AR application development standardization efforts.
Blockchain technologies have had a rather disruptive impact on many sectors of the contemporary society. The establishment of virtual currencies is probably the most representative case. Nonetheless, ...the inherent support to trustworthy electronic interactions has widened the possible adoption contexts. In the last years, the introduction of Smart Contracts has further increased the potential impact of such technologies. These self-enforcing programs have interesting peculiarities (e.g., code immutability) that require innovative testing strategies. This paper presents a mutation testing approach for assessing the quality of test suites accompanying Smart Contracts written in Solidity, the language used by the Ethereum Blockchain. Specifically, we propose a novel suite of mutation operators capable of simulating a wide variety of traditional programming errors and Solidity-specific faults. The operators come in two flavors: Optimized, for faster mutation testing campaigns, and Non-Optimized, for performing a more thorough adequacy assessment. We implemented our approach in a proof-of-concept work, SuMo (SOlidity MUtator), and we evaluated its effectiveness on a set of real-world Solidity projects. The experiments highlighted a recurrent low Mutation Score for the test suites shipped with the selected applications. Moreover, analyzing the surviving mutants of a selected project helped us to identify faulty test cases and Smart Contract code. These results suggest that SuMo can concretely improve the fault-detection capabilities of a test suite, and help to deliver more reliable Solidity code.
•Mutation testing approach to assess test suites associated to Ethereum applications.•Mutation operators tailored for Solidity smart contracts.•SuMo a fully functional tool for automating mutation testing analysis.•Extensive validation of the proposed approach on real blockchain applications.
The subject of this article is a comparison of selected tools for the implementation of software tests. Seven open source tools were selected for the analysis: Selenium IDE, Selenium WebDriver, ...JMeter, SoapUI, SikuliX, AutoIT, Katalon Studio. The research was carried out using dedicated test applications (desktop, web and web service) with the same functionality.
Summary
Covering all the possible paths of the graphical user interface (GUI) with test scripts would take too much effort and result in serious maintenance issues. We propose complementing scripted ...testing with scriptless test automation using the open‐source testar tool. This paper gives a comprehensive overview of testar and its latest extensions together with the ongoing and future research. With this paper, we hope we can help and encourage other researchers to use testar for their GUI testing‐related research and pave the way for an international research agenda in GUI testing built upon stable and open‐source infrastructure.
This article gives a comprehensive overview of TESTAR, an open source tool for scriptless graphical user interface (GUI) test automation, and presents the latest TESTAR extensions and industrial experiments. The novel extensions include, for example, active learning of state models (model inference) and support for Selenium WebDriver. The inferred state models enable more advanced action selection algorithms, for example, optimizing the GUI exploration with machine learning. Support for Selenium WebDriver improves TESTAR in recognizing widgets and properties of web applications.
Many organizations see software test automation as a solution to decrease testing costs and to reduce cycle time in software development. However, establishment of automated testing may fail if test ...automation is not applied in the right time, right context and with the appropriate approach.
The decisions on when and what to automate is important since wrong decisions can lead to disappointments and major wrong expenditures (resources and efforts). To support decision making on when and what to automate, researchers and practitioners have proposed various guidelines, heuristics and factors since the early days of test automation technologies. As the number of such sources has increased, it is important to systematically categorize the current state-of-the-art and -practice, and to provide a synthesized overview.
To achieve the above objective, we have performed a Multivocal Literature Review (MLR) study on when and what to automate in software testing. A MLR is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). We searched the academic literature using the Google Scholar and the grey literature using the regular Google search engine.
Our MLR and its results are based on 78 sources, 52 of which were grey literature and 26 were formally published sources. We used the qualitative analysis (coding) to classify the factors affecting the when- and what-to-automate questions to five groups: (1) Software Under Test (SUT)-related factors, (2) test-related factors, (3) test-tool-related factors, (4) human and organizational factors, and (5) cross-cutting and other factors. The most frequent individual factors were: need for regression testing (44 sources), economic factors (43), and maturity of SUT (39).
We show that current decision-support in software test automation provides reasonable advice for industry, and as a practical outcome of this research we have summarized it as a checklist that can be used by practitioners. However, we recommend developing systematic empirically-validated decision-support approaches as the existing advice is often unsystematic and based on weak empirical evidence.
PDF
