•A survey of 166 sources about smells in software test code is presented.•The survey classifies the knowledge from both industry and academia.•The survey presents the largest catalogue of test ...smells.•Also, guidelines/techniques and the tools available to deal with those smells are reviewed.•This paper will benefit readers by serving as an “index” to the vast body of knowledge.
As a type of anti-pattern, test smells are defined as poorly designed tests and their presence may negatively affect the quality of test suites and production code. Test smells are the subject of active discussions among practitioners and researchers, and various guidelines to handle smells are constantly offered for smell prevention, smell detection, and smell correction. Since there is a vast grey literature as well as a large body of research studies in this domain, it is not practical for practitioners and researchers to locate and synthesize such a large literature. Motivated by the above need and to find out what we, as the community, know about smells in test code, we conducted a ‘multivocal’ literature mapping (classification) on both the scientific literature and also practitioners’ grey literature. By surveying all the sources on test smells in both industry (120 sources) and academia (46 sources), 166 sources in total, our review presents the largest catalogue of test smells, along with the summary of guidelines/techniques and the tools to deal with those smells. This article aims to benefit the readers (both practitioners and researchers) by serving as an “index” to the vast body of knowledge in this important area, and by helping them develop high-quality test scripts, and minimize occurrences of test smells and their negative consequences in large test automation projects.
Software Testing is an integral part of high quality software development process. To enhance quality, IT Companies are spending huge amount of money, labor and time on testing efforts. Test ...Automation tools help speed up testing effort, improve quality and also reduce cost in the long run. Currently, a combination of Open Source and Commercial Test Automation tools are employed to perform Functional Testing activities. With wide adoption of tools like Selenium, Open Source Functional Test Automation tools are accepted as a viable alternative for Commercial Tools. Professionals involved in the selection and purchase of a new Test Automation Tool or replace an existing tool are reviewing and analyzing various factors and features that drive them to select a particular tool. Research Reports, Tool Adoption Trends in Industry and Comparative review of various Test Automation Tools are some of the sources for them to guide their tool selection decision. As Open Source tools are accepted as viable alternative to Commercial tools and are highly economical, an empirical assessment focusing on the factors and features influencing the selection of these Open Source Functional Test Automation tools is attempted. This research aims in identifying the key factors and features that influence the selection of these tools by surveying IT Companies located in India who have already adopted these tools.
•We present an orchestrated survey of the most prominent techniques for automatic generation of software test cases.•The survey covers symbolic execution, model-based, combinatorial, search-based and ...adaptive random testing techniques.•For each technique, basic concepts and research challenges are reported, and future research directions are envisioned.•Each technique is surveyed by a world renowned expert(s) and active researcher(s) on the topic in a self-contained section.•The experts’ contributions were coordinated and the paper was edited by the orchestrators/editors into a consistent paper.
Test case generation is among the most labour-intensive tasks in software testing. It also has a strong impact on the effectiveness and efficiency of software testing. For these reasons, it has been one of the most active research topics in software testing for several decades, resulting in many different approaches and tools. This paper presents an orchestrated survey of the most prominent techniques for automatic generation of software test cases, reviewed in self-standing sections. The techniques presented include: (a) structural testing using symbolic execution, (b) model-based testing, (c) combinatorial testing, (d) random testing and its variant of adaptive random testing, and (e) search-based testing. Each section is contributed by world-renowned active researchers on the technique, and briefly covers the basic ideas underlying the method, the current state of the art, a discussion of the open research problems, and a perspective of the future development of the approach. As a whole, the paper aims at giving an introductory, up-to-date and (relatively) short overview of research in automatic test case generation, while ensuring a comprehensive and authoritative treatment.
Highly Automated Vehicles (HAVs) must undergo strict safety testing before being released to the public. Mileage-based on-road testing suffers from unaffordable time costs and high safety risks. ...Simulated scenario-based testing has been found to be a trustworthy alternative for testing HAVs' built-in algorithms and functionalities. Test automation is typically used to generate target scenarios. This approach facilitates customized testing and avoids wasting time on simple and redundant scenarios. This study aims to review test automation methods and discuss how to accentuate their strengths rather than be trapped in their weaknesses under certain applicable conditions. According to their main purposes, we classify test automation methods into coverage-oriented, unsafe-scenario-oriented, and naturalistic-assessment-oriented categories. To further demonstrate the differences of these methods, we then design numerical experiment to compare the capabilities of seven test automation methods. Finally, we compile our observations to form a comprehensive guide for selecting test automation methods with different test requirements in mind.
The article presents the multifunctional BrowserSpot tool, which serves as an automated environment for testing websites and web applications for Android and iOS systems. It highlights and describes ...the individual stages of research and development work, the issues with solutions currently available on the market, as well as the project's results. The article also discusses the reasons for undertaking work on the tool, its functionalities, and the methods of its usage.
Poor software quality results in avoidable costs of trillion dollars annually in the United States alone. Augmented Reality (AR) applications are a relatively new software category. Currently there ...are no standards to guide the development process and testing is predominantly ad hoc and manual. Consequently, design guidelines and software test automation techniques are intended to remedy the situation. Here, we present a concept for test automation of AR applications. The concept consists of two parts: design guidelines and process model for testing AR applications, and a case study with a prototype application for test automation. The design guidelines and the process model are based on the state-of-the-art. The prototype application presented in this article demonstrates test automation for a multi-platform AR application for Android devices as well as the HoloLens 2. The presented test automation case study is designed to cover a large part of the functions, such as the different interaction variants. This research work shows that by using the proposed process model and test automation techniques, testing of some features of AR applications can be automated. The results of this research can serve as a basis for future research and contribution towards AR application development standardization efforts.
Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the ...capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized "hallucination" of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.
We present a process driven test automation solution which enables delegating (part of) automation tasks from test automation engineer (expensive resource) to test analyst (non-developer, less ...expensive). In our approach, a test automation engineer implements test steps (or actions) which are executed automatically. Such automated test steps represent user actions in the system under test and specified by a natural language which is understandable by a non-technical person. Then, a test analyst with a domain knowledge organizes automated steps combined with test input to create an automated test case. It should be emphasized that the test analyst does not need to possess programming skills to create, modify or execute automated test cases. We refine benchmark test automation architecture to be better suitable for an effective separation and sharing of responsibilities between the test automation engineer (with coding skills) and test analyst (with a domain knowledge). In addition, we propose a metric to empirically estimate cooperation between test automation engineer and test analyst's works. The proposed automation solution has been defined based on our experience in the development and maintenance of Helsenorg, the national electronic health services in Norway which has had over one million of visits per month past year, and we still use it to automate the execution of regression tests.
The subject of this article is a comparison of selected tools for the implementation of software tests. Seven open source tools were selected for the analysis: Selenium IDE, Selenium WebDriver, ...JMeter, SoapUI, SikuliX, AutoIT, Katalon Studio. The research was carried out using dedicated test applications (desktop, web and web service) with the same functionality.
Many organizations see software test automation as a solution to decrease testing costs and to reduce cycle time in software development. However, establishment of automated testing may fail if test ...automation is not applied in the right time, right context and with the appropriate approach.
The decisions on when and what to automate is important since wrong decisions can lead to disappointments and major wrong expenditures (resources and efforts). To support decision making on when and what to automate, researchers and practitioners have proposed various guidelines, heuristics and factors since the early days of test automation technologies. As the number of such sources has increased, it is important to systematically categorize the current state-of-the-art and -practice, and to provide a synthesized overview.
To achieve the above objective, we have performed a Multivocal Literature Review (MLR) study on when and what to automate in software testing. A MLR is a form of a Systematic Literature Review (SLR) which includes the grey literature (e.g., blog posts and white papers) in addition to the published (formal) literature (e.g., journal and conference papers). We searched the academic literature using the Google Scholar and the grey literature using the regular Google search engine.
Our MLR and its results are based on 78 sources, 52 of which were grey literature and 26 were formally published sources. We used the qualitative analysis (coding) to classify the factors affecting the when- and what-to-automate questions to five groups: (1) Software Under Test (SUT)-related factors, (2) test-related factors, (3) test-tool-related factors, (4) human and organizational factors, and (5) cross-cutting and other factors. The most frequent individual factors were: need for regression testing (44 sources), economic factors (43), and maturity of SUT (39).
We show that current decision-support in software test automation provides reasonable advice for industry, and as a practical outcome of this research we have summarized it as a checklist that can be used by practitioners. However, we recommend developing systematic empirically-validated decision-support approaches as the existing advice is often unsystematic and based on weak empirical evidence.
PDF
