Defect prediction models help software quality assurance teams to allocate their limited resources to the most defect-prone modules. Model validation techniques, such as <inline-formula><tex-math ...notation="LaTeX">k</tex-math> <inline-graphic xlink:href="tantithamthavorn-ieq1-2584050.gif"/> </inline-formula>-fold cross-validation, use historical data to estimate how well a model will perform in the future. However, little is known about how accurate the estimates of model validation techniques tend to be. In this paper, we investigate the bias and variance of model validation techniques in the domain of defect prediction. Analysis of 101 public defect datasets suggests that 77 percent of them are highly susceptible to producing unstable results- - selecting an appropriate model validation technique is a critical experimental design choice. Based on an analysis of 256 studies in the defect prediction literature, we select the 12 most commonly adopted model validation techniques for evaluation. Through a case study of 18 systems, we find that single-repetition holdout validation tends to produce estimates with 46-229 percent more bias and 53-863 percent more variance than the top-ranked model validation techniques. On the other hand, out-of-sample bootstrap validation yields the best balance between the bias and variance of estimates in the context of our study. Therefore, we recommend that future defect prediction studies avoid single-repetition holdout validation, and instead, use out-of-sample bootstrap validation.
A wind turbine’s power curve relates its power production to the wind speed it experiences. The typical shape of a power curve is well known and has been studied extensively. However, power curves of ...individual turbine models can vary widely from one another. This is due to both the technical features of the turbine (power density, cut-in and cut-out speeds, limits on rotational speed and aerodynamic efficiency), and environmental factors (turbulence intensity, air density, wind shear and wind veer). Data on individual power curves are often proprietary and only available through commercial databases. We therefore develop an open-source model for pitch regulated horizontal axis wind turbine which can generate the power curve of any turbine, adapted to the specific conditions of any site. This can employ one of six parametric models advanced in the literature, and accounts for the eleven variables mentioned above. The model is described, the impact of each technical and environmental feature is examined, and it is then validated against the manufacturer power curves of 91 turbine models. Versions of the model are made available in MATLAB, R and Python code for the community.
•A generic power curve model dependent on main turbine characteristics is proposed.•Main environmental parameters are considered.•A statistical analysis of the model inputs is proposed.•The modelled output is validated against numerous manufacturer power curves.•Versions of the model are made available in MATLAB, R and Python.
The damage mechanics challenge (DMC) represents a critical step in predicting the damage evolution and failure in rock-like materials displaying brittle/quasi-brittle characteristics. The phase-field ...fracture (PFF) model is a type of damage mechanics model that is thermodynamically consistent and is well suited for capturing complex crack patterns and interactions in 3D. However, there are two main shortcomings: (1) the definition of the crack driving force function and calibration of model parameters give rise to uncertainty in predictions of load–displacement curves; and (2) the finite element implementation of the PFF model generally necessitates the use of fine meshes, leading to higher computational costs. This study presents a novel numerical methodology that employs h-adaptive algorithms in combination with the stress-based PFF model, and demonstrates its validity against experimental data, as required by the DMC. The core strength of our methodology lies in its computational efficiency derived from dynamically-adaptive local mesh refinement. The potential of our methodology is further demonstrated through calibration, verification, and validation studies. Our 2D and 3D simulation results show good agreement with the benchmark laboratory data from three-point bending experiments, within the bounds of data uncertainty. Our blind prediction of the 3D crack geometry for the final challenge shows good agreement with the corresponding experimental data. We find that the stress-based PFF model simplifies the parameter calibration process to a single critical stress parameter, which reduces uncertainty.
•An adaptive phase-field fracture algorithm to solve the damage mechanics challenge.•Simplified model calibration with a single critical stress parameter reduces uncertainty.•The calibrated model predicts peak load in 3D printed rock with less than 10 % error.•Simulated crack morphology in 2D and 3D matches well with experimental crack surfaces.•Adaptive mesh refinement significantly reduces the cost of fracture simulations.
Review and validation of EnergyPLAN Østergaard, P.A.; Lund, H.; Thellufsen, J.Z. ...
Renewable & sustainable energy reviews,
October 2022, 2022-10-00, Letnik:
168
Journal Article
Recenzirano
Odprti dostop
Energy systems analyses are integrated elements in planning the transition towards renewable energy-based energy systems. This is due to a growing complexity arising from the wider exploitation of ...variable renewable energy sources (VRES) and an increasing reliance on sector integration as an enabler of temporal energy system integration, but it calls for consideration to the validity of modelling tools. This article synthesises EnergyPLAN applications through an analysis of its use both from a bibliometric and a case-geographical point of view and through a review of the evolution in the issues addressed and the results obtained using EnergyPLAN. This synthesis is provided with a view to addressing the validity and contribution of EnergyPLAN-based research. As of July 1st, 2022, EnergyPLAN has been applied in 315 peer-reviewed articles, and we see the very high application as an inferred internal validation. In addition, the review shows how the complexity of energy systems analyses has increased over time with early studies focusing on the role of wind power and the cogeneration of heat and power and later studies addressing contemporarily novel issues like the sector integration offered by using power-to-x in fully integrated renewable energy systems. Important findings developed through the application of EnergyPLAN includes the value of district heating in energy systems, the value of district heating for integration of VRES and more generally the importance of sector integration for resource-efficient renewable energy-based energy systems. The wide application across systems and development stages is interpreted as inferred validation through distributed stepwise replication.
•General and EnergyPLAN-applied model validation approaches.•EnergyPLAN is highly applied with 315 articles in the journal literature (July 2022).•The high application of EnergyPLAN is seen an inferred internal validation.•Review of sector integration and multi-tool EnergyPLAN-based energy systems analyses.•EnergyPLAN's development mirrors a growing complexity in energy transition steps.
Breaking with trends in pre-processing? Engel, Jasper; Gerretzen, Jan; Szymańska, Ewa ...
TrAC, Trends in analytical chemistry (Regular ed.),
10/2013, Letnik:
50
Journal Article
Recenzirano
Odprti dostop
•Pre-processing can make or break data analysis.•Selection of appropriate pre-processing strategies is crucial.•Three types of selection approaches seem commonly used.•All approaches have serious ...drawbacks and can provide misleading results.•Objective approaches to quality parameters are required for future research.
Data pre-processing is an essential part of chemometric data analysis, which aims to remove unwanted variation (such as instrumental artifacts) and thereby focusing on the variation of interest. The choice of an optimal pre-processing method or combination of methods may strongly influence the analysis results, but is far from straightforward, since it depends on the characteristics of the data set and the goal of data analysis. This first critical review is devoted to the selection procedure for appropriate pre-processing strategies. We show that breaking with current trends in pre-processing is essential, as all selection approaches have serious drawbacks and cannot be properly used.
Early identification of patients at risk of developing chronic postsurgical pain (CPSP) is an essential step in reducing pain chronification in postsurgical patients. We aimed to develop and validate ...a prognostic model for the early prediction of CPSP including pain characteristics indicating altered pain processing within 2 weeks after surgery.
A prospective cohort study was conducted in adult patients undergoing orthopaedic, vascular, trauma, or general surgery between 2018 and 2019. Multivariable logistic regression models for CPSP were developed using data from the University Medical Centre (UMC) Utrecht and validated in data from the Erasmus UMC Rotterdam, The Netherlands.
In the development (n=344) and the validation (n=150) cohorts, 28.8% and 21.3% of patients reported CPSP. The best performing model (area under the curve=0.82; 95% confidence interval CI, 0.76–0.87) included preoperative treatment with opioids (odds ratio OR=4.04; 95% CI, 2.13–7.70), bone surgery (OR=2.01; 95% CI, 1.10–3.67), numerical rating scale pain score on postoperative day 14 (OR=1.57; 95% CI, 1.34–1.83), and the presence of painful cold within the painful area 2 weeks after surgery (OR=4.85; 95% CI, 1.85–12.68). Predictive performance was confirmed by external validation.
As only four easily obtainable predictors are necessary for reliable CPSP prediction, the models are useful for the clinician to be alerted to further assess and treat individual patients at risk. Identification of the presence of painful cold within 2 weeks after surgery as a strong predictor supports altered pain processing as an important contributor to CPSP development.
This paper reports the metrics‐based results of the Dst index part of the 2008–2009 GEM Metrics Challenge. The 2008–2009 GEM Metrics Challenge asked modelers to submit results for four geomagnetic ...storm events and five different types of observations that can be modeled by statistical, climatological or physics‐based models of the magnetosphere‐ionosphere system. We present the results of 30 model settings that were run at the Community Coordinated Modeling Center and at the institutions of various modelers for these events. To measure the performance of each of the models against the observations, we use comparisons of 1 hour averaged model data with the Dst index issued by the World Data Center for Geomagnetism, Kyoto, Japan, and direct comparison of 1 minute model data with the 1 minute Dst index calculated by the United States Geological Survey. The latter index can be used to calculate spectral variability of model outputs in comparison to the index. We find that model rankings vary widely by skill score used. None of the models consistently perform best for all events. We find that empirical models perform well in general. Magnetohydrodynamics‐based models of the global magnetosphere with inner magnetosphere physics (ring current model) included and stand‐alone ring current models with properly defined boundary conditions perform well and are able to match or surpass results from empirical models. Unlike in similar studies, the statistical models used in this study found their challenge in the weakest events rather than the strongest events.
Key Points
A large set of models that specify DST have been evaluated
Five skill scores were used to evaluate models
Statistical models perform best but physics-based models can compete
PDF
