We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from ...53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were
or
for 41% of experiments, they were
for 9% of experiments, and
(or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.
Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should ...be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.
As part of the Reproducibility Project: Cancer Biology, we published Registered Reports that described how we intended to replicate selected experiments from 29 high-impact preclinical cancer biology ...papers published between 2010 and 2012. Replication experiments were completed and Replication Studies reporting the results were submitted for 18 papers, of which 17 were accepted and published by
with the rejected paper posted as a preprint. Here, we report the status and outcomes obtained for the remaining 11 papers. Four papers initiated experimental work but were stopped without any experimental outcomes. Two papers resulted in incomplete outcomes due to unanticipated challenges when conducting the experiments. For the remaining five papers only some of the experiments were completed with the other experiments incomplete due to mundane technical or unanticipated methodological challenges. The experiments from these papers, along with the other experiments attempted as part of the Reproducibility Project: Cancer Biology, provides evidence about the challenges of repeating preclinical cancer biology experiments and the replicability of the completed experiments.
As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Lesnik et al., 2016) that described how we intended to replicate selected experiments from the paper 'Melanoma ...exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET' (Peinado et al., 2012). Here we report the results. We regenerated tumor cells stably expressing a short hairpin to reduce Met expression (shMet) using the same highly metastatic mouse melanoma cell line (B16-F10) as the original study, which efficiently downregulated Met in B16F10 cells similar to the original study (Supplementary Figure 5A; Peinado et al., 2012). Exosomes from control cells expressed Met, which was reduced in exosomes from shMet cells; however, we were unable to reliably detect phosphorylated Met in exosomes. We tested the effect of exosome-dependent Met signaling on primary tumor growth and metastasis. Similar to the results in the original study, we did not find a statistically significant change in primary tumor growth. Measuring lung and femur metastases, we found a small increase in metastatic burden with exosomes from control cells that was diminished when Met expression was reduced; however, while the effects were in the same direction as the original study (Figure 4E; Peinado et al., 2012), they were not statistically significant. Differences between the original study and this replication attempt, such as level of knockdown efficiency, cell line genetic drift, sample sizes, study endpoints, and variability of observed metastatic burden, are factors that might have influenced the outcomes. Finally, we report meta-analyses for each result.
As part of the <ext-link ext-link-type="uri" xlink:href="https://osf.io/e81xl/wiki/home/">Reproducibility Project: Cancer Biology</ext-link>, we published a Registered Report (Repass et al., 2016), ...that described how we intended to replicate an experiment from the paper '
infection is prevalent in human colorectal carcinoma' (Castellarin et al., 2012). Here we report the results. When measuring
DNA by qPCR in colorectal carcinoma (CRC), adjacent normal tissue, and separate matched control tissue, we did not detect a signal for
in most samples: 25% of CRCs, 15% of adjacent normal, and 0% of matched control tissue were positive based on quantitative PCR (qPCR) and confirmed by sequencing of the qPCR products. When only samples with detectable
in CRC and adjacent normal tissue were compared, the difference was not statistically significant, while the original study reported a statistically significant increase in
expression in CRC compared to adjacent normal tissue (Figure 2; Castellarin et al., 2012). Finally, we report a meta-analysis of the result, which suggests
expression is increased in CRC, but is confounded by the inability to detect
in most samples. The difference in
expression between CRC and adjacent normal tissues was thus smaller than the original study, and not detected in most samples.
As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Khan et al., 2015), that described how we intended to replicate selected experiments from the paper "A ...coding-independent function of gene and pseudogene mRNAs regulates tumour biology" (Poliseno et al., 2010). Here we report the results. We found
depletion in the prostate cancer cell line DU145 did not detectably impact expression of the corresponding pseudogene
. Similarly, depletion of
did not impact
mRNA levels. The original study reported
or
depletion statistically reduced the corresponding pseudogene or gene (Figure 2G; Poliseno et al., 2010).
and/or
depletion in DU145 cells decreased PTEN protein expression, which was similar to the original study (Figure 2H; Poliseno et al., 2010). Further, depletion of
and/or
increased DU145 proliferation compared to non-targeting siRNA, which was in the same direction as the original study (Figure 2F; Poliseno et al., 2010), but not statistically significant. We found
3'UTR overexpression in DU145 cells did not impact
expression, while the original study reported
3'UTR increased
levels (Figure 4A; Poliseno et al., 2010). Overexpression of
3'UTR also statistically decreased DU145 proliferation compared to controls, which was similar to the findings reported in the original study (Figure 4A; Poliseno et al., 2010). Differences between the original study and this replication attempt, such as level of knockdown efficiency and cellular confluence, are factors that might have influenced the results. Finally, where possible, we report meta-analyses for each result.
Highlights ► Wild adult male zebra finches maintain steady availability of GnRH-I during breeding and non-breeding periods. ► Neither ir-GnIH nor synthesis of GnIH were different during breeding and ...non-breeding periods. ► Wild adult males had more GnRH-II neurons that tended to be bigger during breeding compared to non-breeding periods. ► These data might represent a previously unidentified mechanism by which reproductive flexibility can be achieved.
As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Fiering et al., 2015) that described how we intended to replicate selected experiments from the paper ...'Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis' (Goetz et al., 2011). Here we report the results. Primary mouse embryonic fibroblasts (pMEFs) expressing caveolin 1 (Cav1WT) demonstrated increased extracellular matrix remodeling
compared to Cav1 deficient (Cav1KO) pMEFs, similar to the original study (Goetz et al., 2011).
, we found higher levels of intratumoral stroma remodeling, determined by fibronectin fiber orientation, in tumors from cancer cells co-injected with Cav1WT pMEFs compared to cancer cells only or cancer cells plus Cav1KO pMEFs, which were in the same direction as the original study (Supplemental Figure S7C; Goetz et al., 2011), but not statistically significant. Primary tumor growth was similar between conditions, like the original study (Supplemental Figure S7Ca; Goetz et al., 2011). We found metastatic burden was similar between Cav1WT and Cav1KO pMEFs, while the original study found increased metastases with Cav1WT (Figure 7C; Goetz et al., 2011); however, the duration of our
experiments (45 days) were much shorter than in the study by Goetz et al. (2011) (75 days). This makes it difficult to interpret the difference between the studies as it is possible that the cells required more time to manifest the difference between treatments observed by Goetz et al. We also found a statistically significant negative correlation of intratumoral remodeling with metastatic burden, while the original study found a statistically significant positive correlation (Figure 7Cd; Goetz et al., 2011), but again there were differences between the studies in terms of the duration of the metastasis studies and the imaging approaches that could have impacted the outcomes. Finally, we report meta-analyses for each result.
1. Opportunistic breeding (i.e. breeding whenever favourable conditions occur, regardless of interval or periodicity) is a strategy often adopted by animals breeding in habitats where food resources ...important for offspring survival are unpredictable. 2. One putative adaptation for this breeding strategy is the maintenance of a partially activated reproductive system year-round. Breeding readiness is assumed to be the default state and reproduction is therefore suppressed only when environmental conditions become inhibitory. 3. We tested this assumption in an opportunistic archetype, the Australian zebra finch Taeniopygia guttata, by comparing reproductive activation and suppression in free-living zebra finches in two climatically different habitats: (1) an unpredictable habitat of arid central Australia, where breeding is closely tied to aperiodic rainfall and can occur during any month of the year, and (2) a more predictably seasonal habitat in southern Australia, where breeding occurs during approximately the same months each year. We quantified reproductive activation by measuring gonad size and circulating gonadotrophin concentration during the breeding and nonbreeding seasons in the predictable habitat and within a nonbreeding period during drought in the unpredictable habitat. 4. We found reproductive readiness changed consistently between the breeding and nonbreeding state in the predictable habitat, but more nonbreeding birds in the unpredictable habitat maintained an activated reproductive system even though they had poorer body condition. We found no evidence for suppression of reproductive activity by stress hormones (i.e. elevated glucocorticoids). Furthermore, birds were able to activate the reproductive axis quickly when removed from inhibitory field conditions and responded equally well to various stimulatory cues.
Measuring day length is critical for timing annual changes in physiology and behavior in many species. Recently, rapid changes in several photoperiodically-controlled genes following exposure to a ...single long day have been described. Components of this 'first day release' model have so far only been tested in highly domesticated species: quail, sheep, goats and rodents. Because artificial selection accompanying domestication acts on genes related to photoperiodicity, we must also study this phenomenon in wild organisms for it to be accepted as universal. In a songbird, the great tit (Parus major), we tested whether a) these genes are involved in photoperiodic time measurement (PTM) in a wild species, and b) whether predictable species and population differences in expression patterns exist. Using quantitative RT-PCR, we compared gene expression after a single long day in male great tits from Sweden (57°42'N) with that from a German (47°43'N) population. Hypothalamic gene expression key for PTM changed only in the northern population, and occurred earlier after dawn during the single long day than demonstrated in quail; however, gonadotropins (secretion and synthesis) were stimulated in both populations, albeit with different timing. Our data are the first to show acute changes in gene expression in response to photostimulation in any wild species not selected for study of photoperiodism. The pronounced differences in gene expression in response to a single long day between two populations raise exciting new questions about potential environmental selection on photoperiodic cue sensitivity.
PDF
