Context
Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect.
Objective
We aim at understanding how ...reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper.
Method
We first conducted a series of interviews with recognised reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts.
Results
We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: Nearly half of award-winners (45%) are reviewing 20+ conference papers a year, while 28% of non-award winners conduct that many. The majority of reviewers (88%) are taking more than two hours on journal reviews. We also report on qualitative results. Our findings suggest that the most important criteria of a good review is that it should be factual and helpful, which ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are a clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from papers with overly grandiose claims. Further insights include, if not limited to, that reviewers view data availability and its consistency as being important or that authors need to make their contribution of the work very clear in their paper.
Conclusions
Based on the insights we gained through our study, we conclude our work by compiling a proto-guideline for reviewing. One hope we associate with our work is to contribute to the ongoing debate and contemporary effort to further improve our peer review models in the future.
The advent of Version Control Systems (VCS) in computer science education has significantly improved the learning experience. The Learning Analytics community has started to analyze the interactions ...between students and VCSs to evaluate the behavioral and cognitive aspects of the learning process. Within the aforesaid scope, a promising research direction is the use of Artificial Intelligence (AI) to predict students’ exam outcomes early based on VCS usage data. Previous AI-based solutions have two main drawbacks: (i) They rely on static models, which disregard temporal changes in the student–VCS interactions. (ii) AI reasoning is not transparent to end-users. This paper proposes a time-dependent approach to early predict student performance from VCS data. It applies and compares different classification models trained at various course stages. To gain insights into exam performance predictions it combines classification with explainable AI techniques. It visualizes the explanations of the time-varying performance predictors. The results of a real case study show that the effect of VCS-based features on the exam success rate is relevant much earlier than the end of the course, whereas the timely submission of the first lab assignment is a reliable predictor of the exam grade.
Bias in software systems is a serious threat to human rights: when software makes decisions that allocate resources or opportunities, may disparately impact people based on personal traits (e.g., ...gender, ethnic group, etc.), systematically (dis)advantaging certain social groups. The cause is very often the imbalance of training data, that is, unequal distribution of data between the classes of an attribute. Previous studies showed that lower levels of balance in protected attributes are related to higher levels of unfairness in the output. In this paper we contribute to the current status of knowledge on balance measures as risk indicators of systematic discriminations by studying imbalance on two further aspects: the intersectionality among the classes of protected attributes, and the combination of the target variable with protected attributes. We conduct an empirical study to verify whether: i) it is possible to infer the balance of intersectional attributes from the balance of the primary attributes, ii) measures of balance on intersectional attributes are helpful to detect unfairness in the classification outcome, iii) the computation of balance on the combination of a target variable with protected attributes improves the detection of unfairness. Overall the results reveal positive answers, but not for every combination of balance measure and fairness criterion. For this reason, we recommend selecting the fairness and balance measures that are most suitable to the application context when applying our risk approach to real cases.
Context
Code hardening is meant to fight malicious tampering with sensitive code executed on client hosts. Code splitting is a hardening technique that moves selected chunks of code from client to ...server. Although widely adopted, the effective benefits of code splitting are not fully understood and thoroughly assessed.
Objective
The objective of this work is to compare non protected code vs. code splitting protected code, considering two levels of the chunk size parameter, in order to assess the effectiveness of the protection - in terms of both attack time and success rate - and to understand the attack strategy and process used to overcome the protection.
Method
We conducted an experiment with master students performing attack tasks on a small application hardened with different levels of protection. Students carried out their task working at the source code level.
Results
We observed a statistically significant effect of code splitting on the attack success rate that, on the average, was reduced from 89% with unprotected clear code to 52% with the most effective protection. The protection variant that moved some small-sized code chunks turned out to be more effective than the alternative moving fewer but larger chunks. Different strategies were identified yielding different success rates. Moreover we discovered that successful attacks exhibited different process w.r.t. failed ones.
Conclusions
We found empirical evidence of the effect of code splitting, assessed the relative magnitude, and evaluated the influence of the chunk size parameter. Moreover we extracted the process used to overcome such obfuscation technique.
Energy efficiency is an increasingly important non-functional property of software, especially when it runs on mobile or IoT devices. An engineering approach demands a reliable measurement of energy ...consumption of software while performing computational tasks. In this paper, we describe PowTrAn, an R package supporting the analysis of the power traces of a device executing software tasks. The tool analyzes traces with embedded markers, a non-invasive technique that enables gauging software efficiency based on the energy consumed by the whole device. The package effectively handles large power traces, detects work units, and computes correct energy measures, even in noisy conditions, such as those caused by multiple processes working simultaneously. PowTrAn was validated on applications in realistic conditions and multiple hardware configurations. PowTrAn also provides data visualization that helps the user to assess the measurement consistency, and it also helps to highlight possible energy outliers.
The diffusion of Open Government Data (OGD) in recent years kept a very fast pace. However, evidence from practitioners shows that disclosing data without proper quality control may jeopardize ...dataset reuse and negatively affect civic participation. Current approaches to the problem in literature lack a comprehensive theoretical framework. Moreover, most of the evaluations concentrate on open data platforms, rather than on datasets.
In this work, we address these two limitations and set up a framework of indicators to measure the quality of Open Government Data on a series of data quality dimensions at most granular level of measurement. We validated the evaluation framework by applying it to compare two cases of Italian OGD datasets: an internationally recognized good example of OGD, with centralized disclosure and extensive data quality controls, and samples of OGD from decentralized data disclosure (municipality level), with no possibility of extensive quality controls as in the former case, hence with supposed lower quality.
Starting from measurements based on the quality framework, we were able to verify the difference in quality: the measures showed a few common acquired good practices and weaknesses, and a set of discriminating factors that pertain to the type of datasets and the overall approach. On the basis of this evaluation, we also provided technical and policy guidelines to overcome the weaknesses observed in the decentralized release policy, addressing specific quality aspects.
•We provide a metric-based evaluation framework to assess the quality of Open Government Data (OGD).•We verify the suitability of the framework by applying it to OGD datasets.•We observed that OGD from centralized data release resulted in better quality than the decentralized one.•Most common problems identified by the metrics were lack of metadata, incomplete data, lack of information on data updates.•We provide references to tools and processes, to improve weaknesses in specific quality characteristics.
Android applications do not seem to be tested as thoroughly as desktop ones. In particular, graphical user interface (GUI) testing appears generally limited. Like web-based applications, mobile apps ...suffer from GUI test fragility, i.e., GUI test classes failing or needing updates due to even minor modifications in the GUI or in the application under test. The objective of our study is to estimate the adoption of GUI testing frameworks among Android open-source applications, the quantity of modifications needed to keep test classes up to date, and their amount due to GUI test fragility. We introduce a set of 21 metrics to measure the adoption of testing tools and the evolution of test classes and test methods, and to estimate the fragility of test suites. We computed our metrics for six GUI testing frameworks, none of which achieved a significant adoption among Android projects hosted on GitHub. When present, GUI test methods associated with the considered tools are modified often, and a relevant portion (70% on average) of those modifications is induced by GUI-related fragilities. On average, for the projects considered, more than 7% of the total modified lines of code between consecutive releases belong to test classes developed with the analyzed testing frameworks. The measured percentage was higher on average than the one required by other generic test code, based on the JUnit testing framework. Fragility of GUI tests constitutes a relevant concern, probably an obstacle for developers to adopt test automation. This first evaluation of the fragility of Android scripted GUI testing can constitute a benchmark for developers and testers leveraging the analyzed test tools and the basis for the definition of a taxonomy of fragility causes and guidelines to mitigate the issue.
Context:Kotlin is a new programming language representing an alternative to Java; they both target the same JVM and can safely coexist in the same application. Kotlin is advertised as capable to ...solve several known limitations of Java. Recent surveys show that Kotlin achieved a relevant diffusion among Java developers. Goal:We planned to empirically assess a few typical promises of Kotlin w.r.t. known Java’s limitations, in terms of development effectiveness, maintainability, and ease of development. Method:Our experiment involved 27 teams of 4 people each that completed a set of maintenance tasks (both defect correction and feature addition) on Android apps written in either Java or Kotlin. In addition to the number of fixed defects, effort, and code size, we collected, though a questionnaire, the participants’ perceptions about the avoidance of known pitfalls. Results:We did not observe any significant difference in terms of maintainability between the two languages.We found a significant difference regarding the amount of code written, which constitutes evidence of better conciseness of Kotlin. Concerning ease of development, the frequency of NullPointerExceptions reported by the subjects was significantly lower when developing in Kotlin. On the other hand, no significant difference was found in the occurrence of other common Java pitfalls. Finally, the IDE support was deemed better for Java than Kotlin. Conclusions:Some of the promises of Kotlin to be a ”better Java” have been confirmed by our empirical assessment. Evidence suggests that the effort in transitioning to Kotlin can provide some advantages to Java developers, especially regarding code conciseness.Our results may serve as the basis for further investigations on the properties of the language.
•The automation of decision processes in public sector is rapidly expanding.•Automated decision-making (ADM) discriminates if trained with imbalanced data.•We propose to use balance measures to ...assess the risk of unfairness in ADM systems.•We observed varying abilities of the balance measures to detect discrimination.•The proposal resonates with the EU approach to regulate algorithmic systems.
Automated decision-making (ADM) systems may affect multiple aspects of our lives. In particular, they can result in systematic discrimination of specific population groups, in violation of the EU Charter of Fundamental Rights. One of the potential causes of discriminative behavior, i.e., unfairness, lies in the quality of the data used to train such ADM systems.
Using a data quality measurement approach combined with risk management, both defined in ISO standards, we focus on balance characteristics and we aim to understand how balance indexes (Gini, Simpson, Shannon, Imbalance Ratio) identify discrimination risk in six large datasets containing the classification output of ADM systems.
The best result is achieved using the Imbalance Ratio index. Gini and Shannon indexes tend to assume high values and for this reason they have modest results in both aspects: further experimentation with different thresholds is needed.
In terms of policies, the risk-based approach is a core element of the EU approach to regulate algorithmic systems: in this context, balance measures can be easily assumed as risk indicators of propagation – or even amplification – of bias in the input data of ADM systems.
PDF
