A number of critical configurable systems are implemented using #ifdefs, such as Linux. Some tools and strategies are proposed to avoid these directives. However, these systems still have weaknesses, ...leading to vulnerable code, and may impact millions of users. There is a lack of studies regarding the perception of developers of configurable systems with #ifdefs related to weaknesses, and the strategies and tools they use to identify and remove them. Moreover, few works study the characteristics of weaknesses. To better understand the problem, we conduct two studies. In the first one, we qualitatively analyze 27 variability weaknesses of Apache HTTPD, Linux and OpenSSL reported on their bug trackers. In the second study, we conduct a survey with 110 developers of the previous configurable systems. Overall, our results show evidences that, although developers care about weaknesses, they may not detect some weaknesses reported in the bug trackers, and do not use proper tools to deal with them. They take on median 15 days and 4 discussion messages to solve them. Some weaknesses occur due to two feature interactions, and most of them can be detected by the all macros enabled sampling approach.
Mutation Operators for Code Annotations Pinheiro, Pedro; Viana, José Carlos; Fernandes, Leonardo ...
Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing,
09/2018
Conference Proceeding
Mutation testing is a technique that injects artificial faults to check whether the existing test suite can detect them. To inject the faults, mutation testing tools rely on mutation operators, such ...as Arithmetic Operator Replacement, Relational Operator Replacement, Statement Deletion, and Overriding Method Rename. Most of the proposed mutation operators have focused on imperative language constructs and object-oriented constructs. However, many projects make use of meta-programming through code annotations, and mutation operators to simulate annotation-related faults are missing. To minimize this problem, in this paper we propose a set of mutation operators capable of simulating annotation-related faults. In particular, we define nine operators. To evaluate our operators, we mined Java projects that make heavy use of annotations to identify annotation-related faults. We analyzed 100 faults and categorized them as "misuse", when the developer does not know how to use the annotations in a proper way; and "wrong annotation parsing", when the developer wrongly parses annotation code (by using refection, for example). Our operators are able to simulate 95 out of the 100 mined faults. In particular, three operators are able to simulate up to 82% of the analyzed faults. As implications for practice, our operators can help developers to improve their test suite and the code responsible for parsing annotated code. Also, they can help developers to avoid annotation-related faults.
A product line refinement theory formalizes safe evolution in terms of a refinement notion, which does not rely on particular languages for the elements that constitute a product line. Based on this ...theory, we can derive refinement templates to support safe evolution scenarios. To do so, we need to provide formalizations for particular languages, to specify and prove the templates. Without a systematic approach, this leads to many similar templates and thus repetitive verification tasks. We investigate and explore similarities between these concrete languages, which ultimately results in a product line of theories, where different languages correspond to features, and products correspond to theory instantiations. This also leads to specifying refinement templates at a higher abstraction level, which, in the long run, reduces the specification and proof effort, and also provides the benefits of reusing such templates for additional languages plugged into the theory. We use the Prototype Verification System to encode and prove soundness of the theories and their instantiations. Moreover, we also use the refinement theory to reason about safe evolution of the proposed product line of theories.
The variability of configurable systems may lead to configuration-related issues (i.e., faults and warnings) that appear only when we select certain configuration options. Previous studies found that ...issues related to configurability are harder to detect than issues that appear in all configurations, because variability increases the complexity. However, little effort has been put into understanding configuration-related faults (e.g., undeclared functions and variables) and warnings (e.g., unused functions and variables). To better understand the peculiarities of configuration-related undeclared/unused variables and functions, in this paper we perform an empirical study of 15 systems to answer research questions related to how developers introduce these issues, the number of configuration options involved, and the time that these issues remain in source files. To make the analysis of several projects feasible, we propose a strategy that minimizes the initial setup problems of variability-aware tools. We detect and confirm 2 undeclared variables, 14 undeclared functions, 16 unused variables, and 7 unused functions related to configurability. We submit 30 patches to fix issues not fixed by developers. Our findings support the effectiveness of sampling (i.e., analysis of only a subset of valid configurations) because most issues involve two or less configuration options. Nevertheless, by analyzing the version history of the projects, we observe that a number of issues remain in the code for several years. Furthermore, the corpus of undeclared/unused variables and functions gathered is a valuable source to study these issues, compare sampling algorithms, and test and improve variability-aware tools.
Live feedback on behavioral changes Soares, Gustavo; Murphy-Hill, Emerson; Gheyi, Rohit
2013 1st International Workshop on Live Programming (LIVE)
Conference Proceeding
The costs to find and fix bugs grows over time, to the point where fixing a bug after release may cost as much as 100 times more than before release. To help programmers find bugs as soon as they are ...introduced, we sketch a plugin for an integrated development environment that provides live feedback about behavioral changes to Java programs by continuously generating tests, running the tests on the current and previous versions of the program, and comparing the results. Such a tool would allow programmers to better understand how their changes affect the behavior of their programs. As a proof of concept, we developed a prototype that found a bug that remained undetected by pair programmers working on JHotDraw in a previous study. Had the programmers performed this change with our plugin, they would have been notified about the bug as soon as they introduced it.
Scaling Testing of Refactoring Engines Mongiovi, Melina; Mendes, Gustavo; Gheyi, Rohit ...
2014 IEEE International Conference on Software Maintenance and Evolution,
2014-Sept.
Conference Proceeding
Open access
Proving refactoring sound with respect to a formal semantics is considered a challenge. In practice, developers write test cases to check their refactoring implementations. However, it is difficult ...and time consuming to have a good test suite since it requires complex inputs (programs) and an oracle to check whether it is possible to apply the transformation. If it is possible, the resulting program must preserve the observable behavior. There are some automated techniques for testing refactoring engines. Nevertheless, they may have limitations related to the program generator (exhaustiveness, setup, expressiveness), automation (types of oracles, bug categorization), time consumption or kinds of refactorings that can be tested. In this paper, we extend our previous technique to test refactoring engines. We improve expressiveness of the program generator for testing more kinds of refactorings, such as Extract Function. Moreover, developers just need to specify the input's structure in a declarative language. They may also set the technique to skip some consecutive test inputs to improve performance. We evaluate our technique in 18 refactoring implementations of Java (Eclipse and JRRT) and C (Eclipse). We identify 76 bugs (53 new bugs) related to compilation errors, behavioral changes, and overly strong conditions. We also compare the impact of the skip on the time consumption and bug detection in our technique. By using a skip of 25 in the program generator, it reduces in 96% the time to test the refactoring implementations while missing only 3.9% of the bugs. In a few seconds, it finds the first failure related to compilation error or behavioral change.
Refactoring product lines Alves, Vander; Gheyi, Rohit; Massoni, Tiago ...
Generative Programming And Component Engineering: Proceedings of the 5th international conference on Generative programming and component engineering; 22-26 Oct. 2006,
10/2006
Conference Proceeding
Adoption strategies for Software Product Lines (SPL) frequently involve bootstrapping existing products into a SPL and extending an existing SPL to encompass another product. One way to do that is to ...use program refactorings. However, the traditional notion of refactoring does not handle appropriately feature models (FM), nor transformations involving multiple instances of the same SPL. For instance, it is not desirable to apply a refactoring into a SPL and reduce its configurability. In this paper, we extend the traditional notion of refactoring to an SPL context. Besides refactoring programs, FMs must also be refactored. We present a set of sound refactorings for FMs. We evaluate this extended refactoring definition for SPL in a real case study in the mobile games domain.
Equivalent Mutants in Configurable Systems Carvalho, Luiz; Guimarães, Marcio Augusto; Ribeiro, Márcio ...
Proceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems,
02/2018
Conference Proceeding
Mutation testing is a program-transformation technique that evaluates the quality of test cases by assessing their capability to detect injected artificial faults. The costs of using mutation testing ...are usually high, hindering its use in industry. Previous research has reported that roughly one-third of the mutants generated in single systems are equivalents. In configurable systems, a set of mutation operators that focus on preprocessor directives (e.g., #ifdef) has been proposed. However, there is a lack of existing studies that investigate whether equivalent mutants do occur with these operators. To perform this investigation, we provide a tool that implements the aforementioned mutation operators and we conduct an empirical study using 20 C files of four industrial-scale systems. In particular, we provide examples of equivalent mutants and detailed information, such as which mutation operators generate these mutants and how often they occur. Our preliminary results show that nearly 40% of the generated mutants are equivalent. Hence, testing costs can be drastically reduced if the community comes up with efficient techniques to avoid these equivalent mutants.
Live feedback on behavioral changes Soares, Gustavo; Murphy-Hill, Emerson; Gheyi, Rohit
Proceedings of the 1st International Workshop on Live Programming,
05/2013
Conference Proceeding
The costs to find and fix bugs grows over time, to the point where fixing a bug after release may cost as much as 100 times more than before release. To help programmers find bugs as soon as they are ...introduced, we sketch a plugin for an integrated development environment that provides live feedback about behavioral changes to Java programs by continuously generating tests, running the tests on the current and previous versions of the program, and comparing the results. Such a tool would allow programmers to better understand how their changes affect the behavior of their programs. As a proof of concept, we developed a prototype that found a bug that remained undetected by pair programmers working on JHotDraw in a previous study. Had the programmers performed this change with our plugin, they would have been notified about the bug as soon as they introduced it.
Configurable systems typically use #ifdefs to denote variability. Generating and compiling all configurations may be time-consuming. An alternative consists of using variability-aware parsers, such ...as TypeChef. However, they may not scale. In practice, compiling the complete systems may be costly. Therefore, developers can use sampling strategies to compile only a subset of the configurations. We propose a change-centric approach to compile configurable systems with #ifdefs by analyzing only configurations impacted by a code change (transformation). We implement it in a tool called CHECKCONFIGMX, which reports the new compilation errors introduced by the transformation. We perform an empirical study to evaluate 3,913 transformations applied to the 14 largest files of BusyBox, Apache HTTPD, and Expat configurable systems. CHECKCONFIGMX finds 595 compilation errors of 20 types introduced by 41 developers in 214 commits (5.46% of the analyzed transformations). In our study, it reduces by at least 50% (an average of 99%) the effort of evaluating the analyzed transformations by comparing with the exhaustive approach without considering a feature model. CHECKCONFIGMX may help developers to reduce compilation effort to evaluate fine-grained transformations applied to configurable systems with #ifdefs.
PDF
