Robust scientific evidence shows that human performance predictions are more valid when information is combined mechanically (with a decision rule) rather than holistically (in the decision-maker's ...mind). Yet, information is often combined holistically in practice. One reason is that decision makers lack the knowledge of evidence-based decision making. In a performance prediction task, we tested whether watching an educational video on evidence-based decision making increased decision-makers' use of a decision rule and their prediction accuracy immediately after the manipulation and a month later. Furthermore, we manipulated whether participants earned incentives for accurate predictions. Existing research showed that incentives decrease decision-rule use and prediction accuracy. We hypothesized that this is the case for decision makers who did not receive educational information about evidence-based decision making, but that incentives increase decision-rule use and prediction accuracy for participants who received educational information. Our results showed that educational information increased decision-rule use. This resulted in increased prediction accuracy, but only immediately after receiving the educational information. In contrast to the existing literature, incentives slightly increased decision-rule use. We did not find evidence that this effect was larger for educated participants. Providing decision makers with educational information may be effective to increase decision-rule use in practice.
Public Significance Statement
Combining information with a decision rule results in more valid predictions than combining information holistically in the mind. Yet, decision makers rarely use decision rules in practice. This study suggests that a brief educational intervention can increase decision-makers' use of a decision rule in a human performance prediction task. Consequently, prediction accuracy increased, but only temporarily. Such an educational intervention is easily applicable and may increase evidence-based decision making in practice. But, interventions may need to be repeated for a lasting effect.
A robust finding in psychological research is that combining information with a mechanical rule results in more valid predictions than combining information holistically in the mind. Nevertheless, ...information is typically combined holistically in practice, resulting in suboptimal predictions and decisions. Earlier research showed that decision makers are more likely to use mechanical prediction procedures when they retain autonomy in the decision‐making process. However, it remains largely unknown how different autonomy‐enhancing features affect predictive validity. Therefore, in two preregistered studies (total N = 342), we investigated if and how prediction procedures can be designed such that they satisfy decision makers' autonomy needs and acceptance without reducing predictive validity. Based on archival application data from a university admission procedure, participants predicted applicants' first‐year grade point average and chance of dropout. The results of Bayesian analyses showed that participants preferred prediction procedures in which they retained autonomy by choosing consistent predictor weights of a mechanical rule or by holistically adjusting the predictions of an optimal regression model. In general, these prediction procedures resulted in slightly higher predictive validity compared with fully holistic prediction. Providing participants with predictor validity information slightly increased predictive validity when participants could choose predictor weights but not when making holistic predictions or adjusting optimal model predictions. Giving decision makers a role in designing mechanical rules through choosing weights based on explicit predictive validity information could help promote the implementation and validity of mechanical prediction in practice.
In the past decades, much research has examined the negative effects of stressors on the performance of athletes. However, according to evolutionary biology, organisms may exhibit growth under ...stress, a phenomenon called antifragility. For both coaches and their athletes, a key question is how to design training conditions to help athletes develop the kinds of physical, physiological, and behavioral adaptations underlying antifragility. An answer to this important question requires a better understanding of how individual athletes respond to stress or loads in the context of relevant sports tasks. In order to contribute to such understanding, the present study leverages a theoretical and methodological approach to generate individualized load-response profiles in the context of a climbing task. Climbers (
= 37) were asked to complete different bouldering (climbing) routes with increasing loading (i.e. difficulty). We quantified the behavioral responses of each individual athlete by mathematically combining two measures obtained for each route: (a) maximal performance (i.e. the percentage of the route that was completed) and (b) number of attempts required to achieve maximal performance. We mapped this composite response variable as a function of route difficulty. This procedure resulted in load-response curves that captured each athlete's adaptability to stress, termed phenotypic plasticity (PP), specifically operationalized as the area under the generated curves. The results indicate individual load-response profiles (and by extension PP) for athletes who perform at similar maximum levels. We discuss how these profiles might be used by coaches to systematically select stress loads that may be ideally featured in performance training.
Decision makers often combine multiple pieces of information to make performance predictions and hiring decisions. More valid predictions are made when information is combined algorithmically ...(mechanical prediction) rather than in the decision-maker's mind (holistic prediction). Yet, decision makers rarely use algorithms in practice. One reason is that decision makers are worried about negative evaluations from other stakeholders such as colleagues when using algorithms. We hypothesized that such stakeholders evaluate decision makers more positively when they use autonomy-enhancing algorithmic procedures (AEAPs, holistically adjust predictions from a prescribed algorithm or self-design an algorithm), than when they use a prescribed algorithm. Relatedly, we hypothesized that decision makers who use AEAPs are less worried about negative stakeholder evaluations, and more likely to use algorithms in performance predictions. In Study 1 (N = 582), stakeholders evaluated decision makers more positively when they used AEAPs rather than a prescribed algorithm. In Study 2 (N = 269), decision makers were less worried about negative stakeholder evaluations and more likely to use AEAPs compared to a prescribed algorithm. Importantly, using AEAPs also resulted in substantially higher predictive validity than holistic prediction. We recommend the use of self-designed algorithms to improve perceptions and validity.
Medicine regulators need to judge whether a drug’s favorable effects outweigh its unfavorable effects based on a dossier submitted by an applicant, such as a pharmaceutical company. Because ...scientific knowledge is inherently uncertain, regulators also need to judge the credibility of these effects by identifying and evaluating uncertainties. We performed an ethnographic study of assessment procedures at the Dutch Medicines Evaluation Board (MEB) and describe how regulators evaluate the credibility of an applicant’s claims about the benefits and risks of a drug in practice. Our analysis shows that regulators use an investigative approach, which illustrates the effort required to identify uncertainties. Moreover, we show that regulators’ expectations about the presentation, the design, and the results of studies can shape how they perceive a medicine’s dossier. We highlight the importance of regulatory experience and expertise in the identification and evaluation of uncertainties. In light of our observations, we provide two recommendations to reduce avoidable uncertainty: less reliance on evidence generated by the applicant; and better communication about, and enforcement of, regulatory frameworks toward drug developers.
The psychometric structure of the Brief Symptom Inventory-18 (BSI-18; Derogatis, 2001) was investigated using Mokken scaling and parametric item response theory. Data of 487 outpatients, 266 ...students, and 207 prisoners were analyzed. Results of the Mokken analysis indicated that the BSI-18 formed a strong Mokken scale for outpatients and prisoners, indicating strong unidimensionality. For students, only the depression and anxiety items formed a medium Mokken scale. Parametric item response theory analyses showed that the best discriminating items came from the depression and anxiety subscales. (Contains 3 figures and 6 tables.)
Non-cognitive constructs such as personality traits and behavioral tendencies show predictive validity for academic performance and incremental validity over and above cognitive constructs. ...Therefore, non-cognitive predictors are increasingly used in admission procedures for higher education, typically measured using-self-report instruments. It is well known that self-report instruments are sensitive to self-presentation, especially in high-stakes contexts. However, remarkably few studies investigated the effect of self-presentation on predictive validity. The effect of self-presentation in applicants to an undergraduate psychology program was studied using a repeated measures design. Respondents completed self-report questionnaires measuring non-cognitive predictors of academic performance before admission to the program, and again after admission. Scores were compared between contexts, as well as predictive validity, incremental validity, and potential hiring decisions. Results showed differences in scores between contexts on all scales, attenuated predictive validity for most scales, attenuated incremental validity when scores obtained in the admission context were used, and effects on admission decisions. In conclusion, validity results based on scores measured in low-stakes contexts cannot simply be generalized to high-stakes contexts. Furthermore, results obtained in a high-stakes context may result in self-presentation irrespective of whether participants are informed that their scores are used for selection decisions or not.
•Self-presentation on self-report measures was studied with actual applicants.•Results indicate that self-presentation occurred among applicants.•Self-presentation attenuated predictive validity and incremental validity.•Self-presentation affected hiring decisions and criterion performance.
There is an increasing interest in the use of broadened criteria for admission to higher education, often assessed through noncognitive instruments. We argue that there are several reasons why, ...despite some significant progress, the use of noncognitive predictors to select students is problematic in high-stakes educational selection and why the incremental validity will often be modest, even when studied in low-stakes contexts. Furthermore, we comment on the use of broadened admission criteria in relation to reducing adverse impact of testing on some groups, and we extend the literature by discussing an approach based on behavioral sampling, which showed promising results in Europe. Finally, we provide some suggestions for future research.
In first episode psychosis (FEP) baseline negative symptoms (BNS) and relapse both predict less favorable functional outcome. Relapse-prevention is one of the most important goals of treatment. Apart ...from discontinuation of antipsychotics, natural causes of relapse are unexplained. We hypothesized that BNS, apart from predicting worse functional outcome, might also increase relapse risk.
We performed a post-hoc analysis of 7-year follow-up data of a FEP cohort (n = 103) involved in a dose-reduction/discontinuation (DR) vs. maintenance treatment (MT) trial. We examined: 1) what predicted relapse, 2) what predicted functional outcome, and 3) if BNS predicted relapse, whether MT reduced relapse rates compared to DR. After remission patients were randomly assigned to DR or MT for 18 months. Thereafter, treatment was uncontrolled.
BNS and duration of untreated psychosis (DUP) predicted relapse. Number of relapses, BNS, and treatment strategy predicted functional outcome. BNS was the strongest predictor of relapse, while number of relapses was the strongest predictor of functional outcome above BNS and treatment strategy. Overall and within MT, but not within DR, more severe BNS predicted significantly higher relapse rates. Treatment strategies did not make a difference in relapse rates, regardless of BNS severity.
BNS not only predicted worse functional outcome, but also relapses during follow-up. Since current low dose maintenance treatment strategies did not prevent relapse proneness in patients with more severe BNS, resources should be deployed to find optimal treatment strategies for this particular group of patients.
We explain why invariant item ordering (IIO) is an important property in non-cognitive measurement and we discuss that IIO cannot be easily generalized from dichotomous data to polytomous data, as ...some authors seem to suggest. Methods are discussed to investigate IIO for polytomous items and an empirical example shows how these methods can be used in practice.
PDF
