There is increasing concern about the replicability of studies in psychology and cognitive neuroscience. Hidden data dredging (also called p-hacking) is a major contributor to this crisis because it ...substantially increases Type I error resulting in a much larger proportion of false positive findings than the usually expected 5%. In order to build better intuition to avoid, detect and criticize some typical problems, here I systematically illustrate the large impact of some easy to implement and so, perhaps frequent data dredging techniques on boosting false positive findings. I illustrate several forms of two special cases of data dredging. First, researchers may violate the data collection stopping rules of null hypothesis significance testing by repeatedly checking for statistical significance with various numbers of participants. Second, researchers may group participants
along potential but unplanned independent grouping variables. The first approach 'hacks' the number of participants in studies, the second approach 'hacks' the number of variables in the analysis. I demonstrate the high amount of false positive findings generated by these techniques with data from true null distributions. I also illustrate that it is extremely easy to introduce strong bias into data by very mild selection and re-testing. Similar, usually undocumented data dredging steps can easily lead to having 20-50%, or more false positives.
We evaluated 1038 of the most cited structural and functional (fMRI) magnetic resonance brain imaging papers (1161 studies) published during 1990–2012 and 270 papers (300 studies) published in top ...neuroimaging journals in 2017 and 2018. 96% of highly cited experimental fMRI studies had a single group of participants and these studies had median sample size of 12, highly cited clinical fMRI studies (with patient participants) had median sample size of 14.5, and clinical structural MRI studies had median sample size of 50. The sample size of highly cited experimental fMRI studies increased at a rate of 0.74 participant/year and this rate of increase was commensurate with the median sample sizes of neuroimaging studies published in top neuroimaging journals in 2017 (23 participants) and 2018 (24 participants). Only 4 of 131 papers in 2017 and 5 of 142 papers in 2018 had pre-study power calculations, most for single t-tests and correlations. Only 14% of highly cited papers reported the number of excluded participants whereas 49% of papers with their own data in 2017 and 2018 reported excluded participants. Publishers and funders should require pre-study power calculations necessitating the specification of effect sizes. The field should agree on universally required reporting standards. Reporting formats should be standardized so that crucial study parameters could be identified unequivocally.
•The most highly cited 1000+ neuroimaging studies had sample size of 12.•A sample of about 300 studies published during 2017 and 2018 had sample size of 23-24.•Sample sizes increase at a rate of ~0.74 participant/year.•Only 3% of recent papers had power calculations, mostly for t-tests and correlations.•Design parameters and power calculations should be reported more clearly.
The relationship between anxiety and mathematics has often been investigated in the literature. Different forms of anxiety have been evaluated, with math anxiety (MA) and test anxiety (TA) ...consistently being associated with various aspects of mathematics. In this meta-analysis, we have evaluated the impact of these forms of anxiety, distinguishing between different types of mathematical tasks. In investigating this relationship, we have also included potential moderators, such as age, gender, working memory, type of task, and type of material. One hundred seventy-seven studies met the inclusion criteria, providing an overall sample of 906,311 participants. Results showed that both MA and TA had a significant impact on mathematics. Sociodemographic factors had modest moderating effects. Working memory (WM) also mediated the relationship between MA and TA with mathematics; however, this indirect effect was weak. Theoretical and educational implications, as well as future directions for research in this field, are discussed.
Individuals with high levels of mathematics anxiety are more likely to have other forms of anxiety, such as general anxiety and test anxiety, and tend to have some math performance decrement compared ...to those with low math anxiety. However, it is unclear how the anxiety forms cluster in individuals, or how the presence of other anxiety forms influences the relationship between math anxiety and math performance.
We measured math anxiety, test anxiety, general anxiety and mathematics and reading performance in 1720 UK students (year 4, aged 8-9, and years 7 and 8, aged 11-13). We conducted latent profile analysis of students' anxiety scores in order to examine the developmental change in anxiety profiles, the demographics of each anxiety profile and the relationship between profiles and academic performance.
Anxiety profiles appeared to change in specificity between the two age groups studied. Only in the older students did clusters emerge with specifically elevated general anxiety or academic anxiety (test and math anxiety). Our findings suggest that boys are slightly more likely than girls to have elevated academic anxieties relative to their general anxiety. Year 7/8 students with specifically academic anxiety show lower academic performance than those who also have elevated general anxiety.
There may be a developmental change in the specificity of anxiety and gender seems to play a strong role in determining one's anxiety profile. The anxiety profiles present in our year 7/8 sample, and their relationships with math performance, suggest a bidirectional relationship between math anxiety and math performance.
Maths anxiety (MA) is a debilitating negative emotional reaction towards mathematics. However, MA research in primary and early secondary school is surprisingly sparse and inconsistent. Here we ...tested primary and secondary students' maths and reading performance and their maths and general anxiety (GA). We examined gender differences, developmental changes regarding the MA/maths performance link and investigated whether MA is linked to other academic domains (reading) and/or to other anxiety-types (GA). Results revealed that girls exhibited higher MA than boys at both educational levels. Whilst there was a reliable negative correlation between MA and secondary students' arithmetic performance, no such relationship was revealed in primary students. Finally, MA was moderately correlated with GA and, when GA was partialled out, MA remained significantly correlated with secondary students' arithmetic performance. MA was not related to reading performance when GA was controlled. It was concluded that the negative MA/maths performance link surfaces later in the educational timeline and MA appears to be both exclusively related to maths and independent of GA.
•We explored MA gender differences, mapped MA developmental changes and investigated the specificity of this anxiety-type.•Girls exhibited higher MA than boys at both educational levels.•The negative relation between MA and maths performance surfaces later in the educational timeline.•MA can be considered a maths specific anxiety-type and is independent of general anxiety.
Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and ...biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.
Mathematics anxiety (MA), a state of discomfort associated with performing mathematical tasks, is thought to affect a notable proportion of the school age population. Some research has indicated that ...MA negatively affects mathematics performance and that girls may report higher levels of MA than boys. On the other hand some research has indicated that boys' mathematics performance is more negatively affected by MA than girls' performance is. The aim of the current study was to measure girls' and boys' mathematics performance as well as their levels of MA while controlling for test anxiety (TA) a construct related to MA but which is typically not controlled for in MA studies.
Four-hundred and thirty three British secondary school children in school years 7, 8 and 10 completed customised mental mathematics tests and MA and TA questionnaires.
No gender differences emerged for mathematics performance but levels of MA and TA were higher for girls than for boys. Girls and boys showed a positive correlation between MA and TA and a negative correlation between MA and mathematics performance. TA was also negatively correlated with mathematics performance, but this relationship was stronger for girls than for boys. When controlling for TA, the negative correlation between MA and performance remained for girls only. Regression analyses revealed that MA was a significant predictor of performance for girls but not for boys.
Our study has revealed that secondary school children experience MA. Importantly, we controlled for TA which is typically not controlled for in MA studies. Girls showed higher levels of MA than boys and high levels of MA were related to poorer levels of mathematics performance. As well as potentially having a detrimental effect on 'online' mathematics performance, past research has shown that high levels of MA can have negative consequences for later mathematics education. Therefore MA warrants attention in the mathematics classroom, particularly because there is evidence that MA develops during the primary school years. Furthermore, our study showed no gender difference in mathematics performance, despite girls reporting higher levels of MA. These results might suggest that girls may have had the potential to perform better than boys in mathematics however their performance may have been attenuated by their higher levels of MA. Longitudinal research is needed to investigate the development of MA and its effect on mathematics performance.
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published ...recently. The reported median effect size was D = 0.93 (interquartile range: 0.64-1.46) for nominally statistically significant results and D = 0.24 (0.11-0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.
Phase entrainment of neuronal oscillations is thought to play a central role in encoding speech. Children with developmental dyslexia show impaired phonological processing of speech, proposed ...theoretically to be related to atypical phase entrainment to slower temporal modulations in speech (<10Hz). While studies of children with dyslexia have found atypical phase entrainment in the delta band (~2Hz), some studies of adults with developmental dyslexia have shown impaired entrainment in the low gamma band (~35–50Hz). Meanwhile, studies of neurotypical adults suggest asymmetric temporal sensitivity in auditory cortex, with preferential processing of slower modulations by right auditory cortex, and faster modulations processed bilaterally. Here we compared neural entrainment to slow (2Hz) versus faster (40Hz) amplitude-modulated noise using fNIRS to study possible hemispheric asymmetry effects in children with developmental dyslexia. We predicted atypical right hemisphere responding to 2Hz modulations for the children with dyslexia in comparison to control children, but equivalent responding to 40Hz modulations in both hemispheres. Analyses of HbO concentration revealed a right-lateralised region focused on the supra-marginal gyrus that was more active in children with dyslexia than in control children for 2Hz stimulation. We discuss possible links to linguistic prosodic processing, and interpret the data with respect to a neural ‘temporal sampling’ framework for conceptualizing the phonological deficits that characterise children with developmental dyslexia across languages.
Mathematics anxiety (MA) can be observed in children from primary school age into the teenage years and adulthood, but many MA rating scales are only suitable for use with adults or older ...adolescents. We have adapted one such rating scale, the Abbreviated Math Anxiety Scale (AMAS), to be used with British children aged 8-13. In this study, we assess the scale's reliability, factor structure, and divergent validity. The modified AMAS (mAMAS) was administered to a very large (
= 1746) cohort of British children and adolescents. This large sample size meant that as well as conducting confirmatory factor analysis on the scale itself, we were also able to split the sample to conduct exploratory and confirmatory factor analysis of items from the mAMAS alongside items from child test anxiety and general anxiety rating scales. Factor analysis of the mAMAS confirmed that it has the same underlying factor structure as the original AMAS, with subscales measuring anxiety about Learning and Evaluation in math. Furthermore, both exploratory and confirmatory factor analysis of the mAMAS alongside scales measuring test anxiety and general anxiety showed that mAMAS items cluster onto one factor (perceived to represent MA). The mAMAS provides a valid and reliable scale for measuring MA in children and adolescents, from a younger age than is possible with the original AMAS. Results from this study also suggest that MA is truly a unique construct, separate from both test anxiety and general anxiety, even in childhood.
PDF
