The preregistration revolution Nosek, Brian A.; Ebersole, Charles R.; DeHaven, Alexander C. ...
Proceedings of the National Academy of Sciences - PNAS,
03/2018, Letnik:
115, Številka:
11
Journal Article
Recenzirano
Odprti dostop
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated ...conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes—a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.
Using a novel technique known as network meta-analysis, we synthesized evidence from 492 studies (87,418 participants) to investigate the effectiveness of procedures in changing implicit measures, ...which we define as response biases on implicit tasks. We also evaluated these procedures' effects on explicit and behavioral measures. We found that implicit measures can be changed, but effects are often relatively weak (|ds| < .30). Most studies focused on producing short-term changes with brief, single-session manipulations. Procedures that associate sets of concepts, invoke goals or motivations, or tax mental resources changed implicit measures the most, whereas procedures that induced threat, affirmation, or specific moods/emotions changed implicit measures the least. Bias tests suggested that implicit effects could be inflated relative to their true population values. Procedures changed explicit measures less consistently and to a smaller degree than implicit measures and generally produced trivial changes in behavior. Finally, changes in implicit measures did not mediate changes in explicit measures or behavior. Our findings suggest that changes in implicit measures are possible, but those changes do not necessarily translate into changes in explicit measures or behavior.
The social world is stratified. Social hierarchies are known but often disavowed as anachronisms or unjust. Nonetheless, hierarchies may persist in social memory. In three studies (total N > ...200,000), we found evidence of social hierarchies in implicit evaluation by race, religion, and age. Participants implicitly evaluated their own racial group most positively and the remaining racial groups in accordance with the following hierarchy: Whites > Asians > Blacks > Hispanics. Similarly, participants implicitly evaluated their own religion most positively and the remaining religions in accordance with the following hierarchy: Christianity > Judaism > Hinduism or Buddhism > Islam. In a final study, participants of all ages implicitly evaluated age groups following this rule: children > young adults > middle-age adults > older adults. These results suggest that the rules of social evaluation are pervasively embedded in culture and mind.
Replication is vital for increasing precision and accuracy of scientific claims. However, when replications "succeed" or "fail," they could have reputational consequences for the claim's originators. ...Surveys of United States adults (N = 4,786), undergraduates (N = 428), and researchers (N = 313) showed that reputational assessments of scientists were based more on how they pursue knowledge and respond to replication evidence, not whether the initial results were true. When comparing one scientist that produced boring but certain results with another that produced exciting but uncertain results, opinion favored the former despite researchers' belief in more rewards for the latter. Considering idealized views of scientific practices offers an opportunity to address incentives to reward both innovation and verification.
Question-and-answer (Q&A) sessions following research talks provide key opportunities for the audience to engage in scientific discourse. Gender inequities persist in academia, where women are ...underrepresented as faculty and their contributions are less valued than men’s. In the present research, we tested how this gender difference translates to face-to-face Q&A-session participation and its psychological correlates. Across two studies examining participation in three conferences, men disproportionately participated in Q&A sessions in a live, recorded conference (N = 189 Q&A interactions), and women were less comfortable participating in Q&A sessions and more likely to fear backlash for their participation (N = 234 conference attendees). Additionally, women were more likely to hold back questions because of anxiety, whereas men were more likely to hold back questions to make space for others to participate. To the extent that men engage more than women in Q&A sessions, men may continue to have more influence over the direction of science.
Scientific Utopia III: Crowdsourcing Science Uhlmann, Eric Luis; Ebersole, Charles R.; Chartier, Christopher R. ...
Perspectives on psychological science,
09/2019, Letnik:
14, Številka:
5
Journal Article
Recenzirano
Odprti dostop
Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently ...as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.
Consistent with power and status differences between men and women in society, men tend to participate more than women do in question-and-answer (Q&A) sessions at in-person academic conferences. This ...gap in participation in scientific discourse may perpetuate the status quo. The current research examines whether this gender gap in participation in Q&A sessions extends to virtual conferences, which have become more prevalent during the COVID-19 pandemic. Due to shifts in conference formats to enable asynchronous, anonymous, and/or simultaneous participation, we examined whether virtual conferences are more inclusive, and mitigate the gender gap in Q&A participation. Across four virtual conferences that varied in gender representation and Q&A structured format, men continued to take a disproportionate amount of time and space in Q&A sessions. Disproportionate participation did not significantly vary between in-person and virtual formats and did not systematically vary by how the Q&A session was organized. In an all-chat virtual conference, gender differences in volubility were attenuated among higher status academics. Gendered participation and volubility were also impacted by which sub-discipline the presentation was in. Discussion considers the theoretical and practical implications of these findings for understanding the persistence of gender inequality in science. We encourage future research that attends to the cultural factors that promote or mitigate gender disparities in participation.
Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important ...stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018–2019 by
Blueprints for Healthy Youth Development
, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. We found that few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.
REPLY TO LEDGERWOOD Nosek, Brian A.; Ebersole, Charles R.; DeHaven, Alexander C. ...
Proceedings of the National Academy of Sciences - PNAS,
11/2018, Letnik:
115, Številka:
45
Journal Article
Recenzirano
Odprti dostop
Nosek et al comment on Ledgerwood's article about preregistration revolution. They argue that to enable theory falsification, a preregistration should offer a prediction derived from theory and ...provide the theoretical context. However, a prediction without an analysis plan is inert for falsification. An analysis plan is necessary to specify how the prediction will be tested with the observed data. So, the position that prediction and analysis plans are conflated is misleading--theory testing requires both.
PDF
