The Value of Direct Replication Simons, Daniel J.
Perspectives on psychological science,
01/2014, Letnik:
9, Številka:
1
Journal Article
Recenzirano
Reproducibility is the cornerstone of science. If an effect is reliable, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. Two of ...the articles in this special section question the value of direct replication by other laboratories. In this commentary, I discuss the problematic implications of some of their assumptions and argue that direct replication by multiple laboratories is the only way to verify the reliability of an effect.
Constraints on Generality (COG) Simons, Daniel J.; Shoda, Yuichi; Lindsay, D. Stephen
Perspectives on psychological science,
11/2017, Letnik:
12, Številka:
6
Journal Article
Recenzirano
Psychological scientists draw inferences about populations based on samples—of people, situations, and stimuli—from those populations. Yet, few papers identify their target populations, and even ...fewer justify how or why the tested samples are representative of broader populations. A cumulative science depends on accurately characterizing the generality of findings, but current publishing standards do not require authors to constrain their inferences, leaving readers to assume the broadest possible generalizations. We propose that the discussion section of all primary research articles specify Constraints on Generality (i.e., a “COG” statement) that identify and justify target populations for the reported findings. Explicitly defining the target populations will help other researchers to sample from the same populations when conducting a direct replication, and it could encourage follow-up studies that test the boundary conditions of the original finding. Universal adoption of COG statements would change publishing incentives to favor a more cumulative science.
To identify pathways involved in adult lung regeneration, we employ a unilateral pneumonectomy (PNX) model that promotes regenerative alveolarization in the remaining intact lung. We show that PNX ...stimulates pulmonary capillary endothelial cells (PCECs) to produce angiocrine growth factors that induce proliferation of epithelial progenitor cells supporting alveologenesis. Endothelial cells trigger expansion of cocultured epithelial cells, forming three-dimensional angiospheres reminiscent of alveolar-capillary sacs. After PNX, endothelial-specific inducible genetic ablation of
Vegfr2 and
Fgfr1 in mice inhibits production of MMP14, impairing alveolarization. MMP14 promotes expansion of epithelial progenitor cells by unmasking cryptic EGF-like ectodomains that activate the EGF receptor (EGFR). Consistent with this, neutralization of MMP14 impairs EGFR-mediated alveolar regeneration, whereas administration of EGF or intravascular transplantation of MMP14
+ PCECs into pneumonectomized
Vegfr2/Fgfr1-deficient mice restores alveologenesis and lung inspiratory volume and compliance function. VEGFR2 and FGFR1 activation in PCECs therefore increases MMP14-dependent bioavailability of EGFR ligands to initiate and sustain alveologenesis.
Display omitted
► Pulmonary capillary endothelial cells (PCECs) support alveologenesis ► Autocrine VEGFR2 and FGFR1 activation in PCECs induces MMP14 expression ► MMP14 unmasks EGF receptor ligands, enhancing epithelial cell proliferation ► Injection of activated PCECs or angiocrine factors accelerates lung regeneration
Capillary endothelial cells support the regeneration of alveolar epithelial cells by secreting a matrix metalloprotease that unmasks cryptic epidermal growth factor receptor ligands.
Incorrect beliefs about memory have wide-ranging implications. We recently reported the results of a survey showing that a substantial proportion of the United States public held beliefs about memory ...that conflicted with those of memory experts. For that survey, respondents answered recorded questions using their telephone keypad. Although such robotic polling produces reliable results that accurately predicts the results of elections, it suffers from four major drawbacks: (1) telephone polling is costly, (2) typically, less than 10 percent of calls result in a completed survey, (3) calls do not reach households without a landline, and (4) calls oversample the elderly and undersample the young. Here we replicated our telephone survey using Amazon Mechanical Turk (MTurk) to explore the similarities and differences in the sampled demographics as well as the pattern of results. Overall, neither survey closely approximated the demographics of the United States population, but they differed in how they deviated from the 2010 census figures. After weighting the results of each survey to conform to census demographics, though, the two approaches produced remarkably similar results: In both surveys, people averaged over 50% agreement with statements that scientific consensus shows to be false. The results of this study replicate our finding of substantial discrepancies between popular beliefs and those of experts and shows that surveys conducted on MTurk can produce a representative sample of the United States population that generates results in line with more expensive survey techniques.
Celotno besedilo
Dostopno za:
DOBA, IZUM, KILJ, NUK, PILJ, PNG, SAZU, SIK, UILJ, UKNU, UL, UM, UPUK
Frequent action video game players often outperform non-gamers on measures of perception and cognition, and some studies find that video game practice enhances those abilities. The possibility that ...video game training transfers broadly to other aspects of cognition is exciting because training on one task rarely improves performance on others. At first glance, the cumulative evidence suggests a strong relationship between gaming experience and other cognitive abilities, but methodological shortcomings call that conclusion into question. We discuss these pitfalls, identify how existing studies succeed or fail in overcoming them, and provide guidelines for more definitive tests of the effects of gaming on cognition.
To draw causal conclusions about the efficacy of a psychological intervention, researchers must compare the treatment condition with a control group that accounts for improvements caused by factors ...other than the treatment. Using an active control helps to control for the possibility that improvement by the experimental group resulted from a placebo effect. Although active control groups are superior to "no-contact" controls, only when the active control group has the same expectation of improvement as the experimental group can we attribute differential improvements to the potency of the treatment. Despite the need to match expectations between treatment and control groups, almost no psychological interventions do so. This failure to control for expectations is not a minor omission—it is a fundamental design flaw that potentially undermines any causal inference. We illustrate these principles with a detailed example from the video-game-training literature showing how the use of an active control group does not eliminate expectation differences. The problem permeates other interventions as well, including those targeting mental health, cognition, and educational achievement. Fortunately, measuring expectations and adopting alternative experimental designs makes it possible to control for placebo effects, thereby increasing confidence in the causal efficacy of psychological interventions.
Those who are less skilled tend to overestimate their abilities more than do those who are more skilled—the so-called
Dunning
–
Kruger
effect. Less-skilled performers presumably have less of the ...knowledge needed to make informed guesses about their relative performance. If so, the Dunning–Kruger effect should vanish when participants do have access to information about their relative ability and performance. Competitive bridge players predicted their results for bridge sessions before playing and received feedback about their actual performance following each session. Despite knowing their own relative skill and showing unbiased memory for their performance, they made overconfident predictions consistent with a Dunning–Kruger effect. This bias persisted even though players received accurate feedback about their predictions after each session. The finding of a Dunning–Kruger effect despite knowledge of relative ability suggests that differential self-knowledge is not a necessary precondition for the Dunning–Kruger effect. At least in some cases, the effect might reflect a different form of irrational optimism.
Incorrect beliefs about the properties of memory have broad implications: the media conflate normal forgetting and inadvertent memory distortion with intentional deceit, juries issue verdicts based ...on flawed intuitions about the accuracy and confidence of testimony, and students misunderstand the role of memory in learning. We conducted a large representative telephone survey of the U.S. population to assess common beliefs about the properties of memory. Substantial numbers of respondents agreed with propositions that conflict with expert consensus: Amnesia results in the inability to remember one's own identity (83% of respondents agreed), unexpected objects generally grab attention (78%), memory works like a video camera (63%), memory can be enhanced through hypnosis (55%), memory is permanent (48%), and the testimony of a single confident eyewitness should be enough to convict a criminal defendant (37%). This discrepancy between popular belief and scientific consensus has implications from the classroom to the courtroom.
Celotno besedilo
Dostopno za:
DOBA, IZUM, KILJ, NUK, PILJ, PNG, SAZU, SIK, UILJ, UKNU, UL, UM, UPUK
People sometimes fail to notice salient unexpected objects when their attention is otherwise occupied, a phenomenon known as inattentional blindness. To explore individual differences in ...inattentional blindness, we employed both static and dynamic tasks that either presented the unexpected object away from the focus of attention (spatial) or near the focus of attention (central). We hypothesized that noticing in central tasks might be driven by the availability of cognitive resources like working memory, and that noticing in spatial tasks might be driven by the limits on spatial attention like attention breadth. However, none of the cognitive measures predicted noticing in the dynamic central task or in either the static or dynamic spatial task. Only in the central static task did working memory capacity predict noticing, and that relationship was fairly weak. Furthermore, whether or not participants noticed an unexpected object in a static task was only weakly associated with their odds of noticing an unexpected object in a dynamic task. Taken together, our results are largely consistent with the notion that noticing unexpected objects is driven more by stochastic processes common to all people than by stable individual differences in cognitive abilities.
Celotno besedilo
Dostopno za:
DOBA, IZUM, KILJ, NUK, PILJ, PNG, SAZU, SIK, UILJ, UKNU, UL, UM, UPUK
Do "Brain-Training" Programs Work? Simons, Daniel J.; Boot, Walter R.; Charness, Neil ...
Psychological science in the public interest,
10/2016, Letnik:
17, Številka:
3
Journal Article
In 2014, two groups of scientists published open letters on the efficacy of brain-training interventions, or "brain games," for improving cognition. The first letter, a consensus statement from an ...international group of more than 70 scientists, claimed that brain games do not provide a scientifically grounded way to improve cognitive functioning or to stave off cognitive decline. Several months later, an international group of 133 scientists and practitioners countered that the literature is replete with demonstrations of the benefits of brain training for a wide variety of cognitive and everyday activities. How could two teams of scientists examine the same literature and come to conflicting "consensus" views about the effectiveness of brain training? In part, the disagreement might result from different standards used when evaluating the evidence. To date, the field has lacked a comprehensive review of the brain-training literature, one that examines both the quantity and the quality of the evidence according to a well-defined set of best practices. This article provides such a review, focusing exclusively on the use of cognitive tasks or games as a means to enhance performance on other tasks. We specify and justify a set of best practices for such brain-training interventions and then use those standards to evaluate all of the published peer-reviewed intervention studies cited on the websites of leading brain-training companies listed on Cognitive Training Data (www.cognitivetrainingdata.org), the site hosting the open letter from brain-training proponents. These citations presumably represent the evidence that best supports the claims of effectiveness. Based on this examination, we find extensive evidence that brain-training interventions improve performance on the trained tasks, less evidence that such interventions improve performance on closely related tasks, and little evidence that training enhances performance on distantly related tasks or that training improves everyday cognitive performance. We also find that many of the published intervention studies had major shortcomings in design or analysis that preclude definitive conclusions about the efficacy of training, and that none of the cited studies conformed to all of the best practices we identify as essential to drawing clear conclusions about the benefits of brain training for everyday activities. We conclude with detailed recommendations for scientists, funding agencies, and policymakers that, if adopted, would lead to better evidence regarding the efficacy of brain-training interventions.
PDF
