•The paper presents a study of the reliability of two mapping studies.•It is concluded that reliability of systematic studies cannot be taken for granted.•Four conjectures are put forward to increase ...reliability of literature studies.
Systematic literature reviews and systematic mapping studies are becoming increasingly common in software engineering, and hence it becomes even more important to better understand the reliability of such studies.
This paper presents a study of two systematic mapping studies to evaluate the reliability of mapping studies and point out some challenges related to this type of study in software engineering.
The research is based on an in-depth case study of two published mapping studies on software product line testing.
We found that despite the fact that the two studies are addressing the same topic, there are quite a number of differences when it comes to papers included and in terms of classification of the papers included in the two mapping studies.
From this we conclude that although mapping studies are important, their reliability cannot simply be taken for granted. Based on the findings we also provide four conjectures that further research has to address to make secondary studies (systematic mapping studies and systematic literature reviews) even more valuable to both researchers and practitioners.
Several variability representations have been proposed over the years. Software maintenance in the presence of variability is known to be hard. One of the reasons is that maintenance tasks require a ...large amount of cognitive effort for program comprehension. In fact, the different ways of representing variability in source code might influence the comprehension process in different ways. Despite the differences, there is little evidence about how these variability representations – such as conditional-compilation directives or feature-oriented programming – influence program comprehension. Existing research has focused primarily on either understanding how code using modern paradigms evolves compared to the traditional way of realizing variability, namely conditional compilation, or on the aspects influencing the comprehension of conditional compilation only. We used two different programs implemented in Java and each of these variability representations. As Java does not support conditional compilation natively, we relied on the mimicking (i.e., preprocessing annotations in comments) that has been used in the literature. Our results show no significant statistical differences regarding the evaluated measures (correctness, understanding, or response time) in the tasks. Our heterogeneous sample allowed us to produce evidence about the influence of using
CC
and FOP variability representations on the aspects involved in the comprehension of feature-oriented software, while addressing bug-finding tasks.
In software development, Testing is an important mechanism both to identify defects and assure that completed products work as specified. This is a common practice in single-system development, and ...continues to hold in Software Product Lines (SPL). Even though extensive research has been done in the SPL Testing field, it is necessary to assess the current state of research and practice, in order to provide practitioners with evidence that enable fostering its further development.
This paper focuses on Testing in SPL and has the following goals: investigate state-of-the-art testing practices, synthesize available evidence, and identify gaps between required techniques and existing approaches, available in the literature.
A systematic mapping study was conducted with a set of nine research questions, in which 120 studies, dated from 1993 to 2009, were evaluated.
Although several aspects regarding testing have been covered by single-system development approaches, many cannot be directly applied in the SPL context due to specific issues. In addition, particular aspects regarding SPL are not covered by the existing SPL approaches, and when the aspects are covered, the literature just gives brief overviews. This scenario indicates that additional investigation, empirical and practical, should be performed.
The results can help to understand the needs in SPL Testing, by identifying points that still require additional investigation, since important aspects regarding particular points of software product lines have not been addressed yet.
Context: Software product lines (SPL) engineering defines a set of systems that share common features and artifacts to achieve high productivity, quality, market agility, low time to market, and ...cost. An SPL product is derived from a configuration of features which need to be compounded together without violating their particular specifications. While it is easy to identify the behavior of a feature in isolation, specifying and resolving interactions among features may not be a straightforward task. The feature interaction problem has been a challenging subject for decades.
Objective: This study aims at surveying existing research on feature interaction in SPL engineering in order to identify common practices and research trends.
Method: A systematic mapping study was conducted with a set of seven research questions, in which the 35 studies found are mainly classified regarding the feature interaction solution presented: detection, resolution and general analysis.
Results: 43% of the papers deal with feature interaction at early phases of a software lifecycle. The remaining is shared among the other categories: source code detection, resolution and analysis. For each category, it was also identified the main strategies used to deal with interactions.
Conclusions: The findings can help to understand the needs in feature interaction for SPL engineering, and highlight aspects that still demand an additional investigation. For example, often strategies are partial and only address specific points of a feature interaction investigation.
Testing Software Product Lines da Mota Silveira Neto, Paulo Anselmo; Runeson, Per; do Carmo Machado, Ivan ...
IEEE software,
09/2011, Volume:
28, Issue:
5
Journal Article
Peer reviewed
Open access
Two studies of testing practices for software product lines identify gaps between required techniques and existing approaches in the available literature. This Web extra offers extra details for the ...main article (specifically, the bibliography for the two studies described).
Testing plays an important role in the quality assurance process for software product line engineering. There are many opportunities for economies of scope and scale in the testing activities, but ...techniques that can take advantage of these opportunities are still needed.
The objective of this study is to identify testing strategies that have the potential to achieve these economies, and to provide a synthesis of available research on SPL testing strategies, to be applied towards reaching higher defect detection rates and reduced quality assurance effort.
We performed a literature review of two hundred seventy-six studies published from the year 1998 up to the 1st semester of 2013. We used several filters to focus the review on the most relevant studies and we give detailed analyses of the core set of studies.
The analysis of the reported strategies comprised two fundamental aspects for software product line testing: the selection of products for testing, and the actual test of products. Our findings indicate that the literature offers a large number of techniques to cope with such aspects. However, there is a lack of reports on realistic industrial experiences, which limits the inferences that can be drawn.
This study showed a number of leveraged strategies that can support both the selection of products, and the actual testing of products. Future research should also benefit from the problems and advantages identified in this study.
Software Product Lines (SPL) approach has been widely developed in academia and successfully applied in industry. Based on the selection of features, stakeholders can efficiently derive tailor-made ...programs satisfying different requirements. While SPL was very successful at building products based on identified features, achievements and preservation of many nonfunctional properties (NFPs) remain challenging. A knowledge how to deal with NFPs is still not fully obtained. In this paper, we present a systematic literature review of NFPs analysis for SPL products, focusing on runtime NFPs. The goal of the paper is twofold: (i) to present an holistic overview of SPL approaches that have been reported regarding the analysis of runtime NFPs, and (ii) to categorize NFPs treated in the scientific literature regarding development of SPLs. We analyzed 36 research papers, and identified that system performance attributes are typically the most considered. The results also aid future research studies in NFPs analysis by providing an unbiased view of the body of empirical evidence and by guiding future research directions.
► We characterized the software inspection activity on features specifications in an industrial SPL project. ► The inspection activity reported incompleteness as the main non-conformity type found on ...features specifications. ► Correction was the most burdensome SPL Inspection task. ► Optional features presented higher non-conformity density than mandatory features. ► The risk attribute enabled to build a predictive model for estimating non-conformities in features specifications.
In software product lines (SPL), scoping is a phase responsible for capturing, specifying and modeling features, and also their constraints, interactions and variations. The feature specification task, performed in this phase, is usually based on natural language, which may lead to lack of clarity, non-conformities and defects. Consequently, scoping analysts may introduce ambiguity, inconsistency, omissions and non-conformities. In this sense, this paper aims at gathering evidence about the effects of applying an inspection approach to feature specification for SPL. Data from a SPL reengineering project were analyzed in this work and the analysis indicated that the correction activity demanded more effort. Also, Pareto's principle showed that incompleteness and ambiguity reported higher non-conformity occurrences. Finally, the Poisson regression analysis showed that sub-domain risk information can be a good indicator for prioritization of sub-domains in the inspection activity.
MERCI Campos, Denivan; Lima, Crescencio; do Carmo Machado, Ivan
Proceedings of the XVII Brazilian Symposium on Software Quality,
10/2018
Conference Proceeding
Testing a system is a routine activity, and it plays an important role in the software quality assurance process. However, testing highly-configurable systems, such as Software Product Lines (SPL), ...is a rather complex activity, due to the presence of variability in its engineering process, which increases the number of product configurations to test. The underlying idea to make testing feasible in SPL engineering is to select a small but representative subset of products to test, by employing techniques such as combinatorial interaction testing (CIT). This paper presents Method to Evaluate Combinatorial Interaction (MERCI), a novel method to evaluate the adequacy of existing CIT tools for SPL engineering, with respect to three measures: defect detection, test coverage, and test execution length. We carried out an empirical evaluation to compare four CIT tools: ACTS, CATS, PICTMaster and VPTag. The results show that the method may serve as an affordable strategy to evaluate how the CIT tools could behave in an SPL testing scenario.
Understanding how the concepts of sustainability could be incorporated to the Software Engineering (SE) concepts has gained increased attention in the last years, particularly in terms of the ...Software Development Life Cycle (SDLC). Several studies have addressed the impact of sustainability in the SE practice, from a range of perspectives. This study presents a systematic mapping study that aggregates, summarizes and discusses the state-of-the-art approaches for sustainable SE practice. We analyzed 75 relevant primary studies addressing methods, processes, tools and metrics proposed to support the software development in a sustainable way. The included primary studies were selected using inclusion and exclusion criteria applied to studies published prior to 2017. They were analyzed based on a set of classification criteria, including contribution types, SDLC phases, evidence types, research types, application domains, publication venues, distribution between academia and industry and research methods. The results indicated a growing interest by the SE research community in the Green and Sustainable software domain. Besides, there is an observed need for more studies on techniques, tools and metrics covering construction, testing and maintenance. The results also point out a clear view of the SE community about the need for a better alignment between research and practice in this domain.
PDF
