•We find that cost, from the expectancy-value model, has recently reemerged in the literature.•Student focus groups confirm that cost is salient when considering their motivation.•We redefine cost to ...be consistent with our theoretical review and qualitative inquiry.•Exploratory and confirmatory factor analyses support our theoretical cost structure.•Our cost dimensions are associated with other motivation constructs and student performance.
Although the Expectancy-Value Model offers one of the most influential models for understanding motivation, one component of this model, cost, has been largely ignored in empirical research. Fortunately, recent research is emerging on cost, but no clear consensus has emerged for operationalizing and measuring it. To address this shortcoming, we outline a comprehensive scale development process that builds and extends on prior work. We conducted a literature review of theory and existing measurement, a qualitative study with students, a content alignment with experts, exploratory and confirmatory factor analysis, and a correlational study. In the literature and across our studies, we found that cost was salient to students, separate from expectancy and value components, contained multiple dimensions, and related to student outcomes. This work led to proposing a new, 19 item cost scale with four dimensions: task effort cost, outside effort cost, loss of valued alternatives cost, and emotional cost. In addition, to extend existing cost measures, careful attention was taken to operationalize the cost dimensions such that the scale could be easily used with a wide variety of students in various contexts. Directions for future research and the implications for the study of motivation are discussed.
Measurement invariance-the notion that the measurement properties of a scale are equal across groups, contexts, or time-is an important assumption underlying much of psychology research. The ...traditional approach for evaluating measurement invariance is to fit a series of nested measurement models using multiple-group confirmatory factor analyses. However, traditional approaches are strict, vary across the field in implementation, and present multiplicity challenges, even in the simplest case of two groups under study. The alignment method was recently proposed as an alternative approach. This method is more automated, requires fewer decisions from researchers, and accommodates two or more groups. However, it has different assumptions, estimation techniques, and limitations from traditional approaches. To address the lack of accessible resources that explain the methodological differences and complexities between the two approaches, we introduce and illustrate both, comparing them side by side. First, we overview the concepts, assumptions, advantages, and limitations of each approach. Based on this overview, we propose a list of four key considerations to help researchers decide which approach to choose and how to document their analytical decisions in a preregistration or analysis plan. We then demonstrate our key considerations on an illustrative research question using an open dataset and provide an example of a completed preregistration. Our illustrative example is accompanied by an annotated analysis report that shows readers, step-by-step, how to conduct measurement invariance tests using R and Mplus. Finally, we provide recommendations for how to decide between and use each approach and next steps for methodological research.
Translational AbstractMeasurement invariance refers to the notion that a scale measures a construct the same way across different groups, contexts, or time. If a personality scale measures personality differently for men and women, for example, then men and women cannot be compared on that personality scale because any observed differences could simply be measurement differences. The traditional approaches for assessing measurement invariance use confirmatory factor analyses, require statistical expertise, and become problematic when many groups are tested. The alignment method for assessing measurement invariance is a recently developed alternative to the traditional approaches which addresses some of their disadvantages and works well with many groups but has its own assumptions and limitations. In this tutorial, we introduce and illustrate both approaches for testing measurement invariance to help researchers decide which approach to choose and how to document their analytical decisions in a preregistration or analysis plan. First, we overview the concepts, assumptions, advantages, and limitations of each approach. Based on this overview, we propose a list of four key considerations to help researchers decide which approach to choose. We then illustrate how to use our key considerations by answering a mock research question using an open dataset, which is accompanied by an example of a completed preregistration. We also provide an annotated analysis report with code that shows readers, step-by-step, how to conduct measurement invariance tests using statistical programs R and Mplus. Finally, we provide recommendations for how to decide between and use each approach and next steps for methodological research.
An increased focus on transparency and replication in science has stimulated reform in research practices and dissemination. As a result, the research culture is changing: the use of preregistration ...is on the rise, access to data and materials is increasing, and large-scale replication studies are more common. In this article, I discuss two problems the methodological reform movement is now ready to tackle given the progress thus far and how educational psychology is particularly well suited to contribute. The first problem is that there is a lack of transparency and rigor in measurement development and use. The second problem is caused by the first-replication research is difficult and potentially futile as long as the first problem persists. I describe how to expand transparent practices into measure use and how construct validation can be implemented to bolster the validity of replication studies.
Concerns about the veracity of psychological research have been growing. Many findings in psychological science are based on studies with insufficient statistical power and nonrepresentative samples, ...or may otherwise be limited to specific, ungeneralizable settings or populations. Crowdsourced research, a type of large-scale collaboration in which one or more research projects are conducted across multiple lab sites, offers a pragmatic solution to these and other current methodological challenges. The Psychological Science Accelerator (PSA) is a distributed network of laboratories designed to enable and support crowdsourced research projects. These projects can focus on novel research questions or replicate prior research in large, diverse samples. The PSA’s mission is to accelerate the accumulation of reliable and generalizable evidence in psychological science. Here, we describe the background, structure, principles, procedures, benefits, and challenges of the PSA. In contrast to other crowdsourced research networks, the PSA is ongoing (as opposed to time limited), efficient (in that structures and principles are reused for different projects), decentralized, diverse (in both subjects and researchers), and inclusive (of proposals, contributions, and other relevant input from anyone inside or outside the network). The PSA and other approaches to crowdsourced psychological science will advance understanding of mental processes and behaviors by enabling rigorous research and systematic examination of its generalizability.
Because of the misspecification of models and specificity of operationalizations, many studies produce claims of limited utility. We suggest a path forward that requires taking a few steps back. ...Researchers can retool large-scale replications to conduct the descriptive research which assesses the generalizability of constructs. Large-scale construct validation is feasible and a necessary next step in addressing the generalizability crisis.
Currently, there is little guidance for navigating measurement challenges that threaten construct validity in replication research. To identify common challenges and ultimately strengthen replication ...research, we conducted a systematic review of the measures used in the 100 original and replication studies from the Reproducibility Project: Psychology (Open Science Collaboration, 2015). Results indicate that it was common for scales used in the original studies to have little or no validity evidence. Our systematic review demonstrates and corroborates evidence that issues of construct validity are sorely neglected in original and replicated research. We identify four measurement challenges replicators are likely to face: a lack of essential measurement information, a lack of validity evidence, measurement differences, and translation. Next, we offer solutions for addressing these challenges that will improve measurement practices in original and replication research. Finally, we close with a discussion of the need to develop measurement methodologies for the next generation of replication research.
Public Significance Statement
Over the past decade, psychologists have been calling for methodological reform to increase the rigor and replicability of psychological science, which has been accompanied by progress in improving transparency and statistical practices. This article presents rigorous measurement practices as foundational for generating knowledge from psychological science that can be translated to inform policy, develop interventions, and improve people's lives. We review one of the largest sets of replication studies ever conducted to understand how measurement can be improved and discuss the need to develop measurement practices for the next generation of replication research.
Background: The Psychological Science Accelerator (PSA) recently completed a large-scale moral psychology study using translated versions of the Oxford Utilitarianism Scale (OUS). However, the ...translated versions have no validity evidence. Objective: The study investigated the structural validity evidence of the OUS across 15 translated versions and produced version-specific validity reports. Methods: We analyzed OUS data from the PSA, which was collected internationally on a centralized online questionnaire. We also collected qualitative feedback from experts for each translated version. Results: For each version, we produced version-specific psychometric reports which include the following: (1) descriptive item and demographics analyses, (2) factor structure evidence using confirmatory factor analyses, (3) measurement invariance testing across languages using multiple-group confirmatory factor analyses and alignment optimization, and (4) reliability analyses using coefficients α and ω.
American early childhood education is in the midst of drastic change. In recent years, states have begun the process of overhauling early childhood education systems in response to federal grant ...competitions, bringing an increased focus on assessment and accountability for early learning programs. The assessment of young children is fraught with challenges; psychometricians and educational researchers must work together with the early childhood community to develop these instruments. The purpose of this paper is to present a conceptual framework for the validation of such instrumentation and examine its implications for early childhood educators. We formulate a validity argument for early childhood assessments providing a pivotal link between validity theory and early education practice. Recommendations for the assessment field are also considered.
Multilevel models are used ubiquitously in the social and behavioral sciences and effect sizes are critical for contextualizing results. A general framework of R-squared effect size measures for ...multilevel models has only recently been developed. Rights and Sterba (
2019
) distinguished each source of explained variance for each possible kind of outcome variance. Though researchers have long desired a comprehensive and coherent approach to computing R-squared measures for multilevel models, the use of this framework has a steep learning curve. The purpose of this tutorial is to introduce and demonstrate using a new R package –
r2mlm
– that automates the intensive computations involved in implementing the framework and provides accompanying graphics to visualize all multilevel R-squared measures together. We use accessible illustrations with open data and code to demonstrate how to use and interpret the R package output.
In this article, we propose integrated generalized structured component analysis (IGSCA), which is a general statistical approach for analyzing data with both components and factors in the same ...model, simultaneously. This approach combines generalized structured component analysis (GSCA) and generalized structured component analysis with measurement errors incorporated (GSCAM) in a unified manner and can estimate both factor- and component-model parameters, including component and factor loadings, component and factor path coefficients, and path coefficients connecting factors and components. We conduct 2 simulation studies to investigate the performance of IGSCA under models with both factors and components. The first simulation study assesses how existing approaches for structural equation modeling and IGSCA recover parameters. This study shows that only consistent partial least squares (PLSc) and IGSCA yield unbiased estimates of all parameters, whereas the other approaches always provided biased estimates of several parameters. As such, we conduct a second, extensive simulation study to evaluate the relative performance of the 2 competitors (PLSc and IGSCA), considering a variety of experimental factors (model specification, sample size, the number of indicators per factor/component, and exogenous factor/component correlation). IGSCA exhibits better performance than PLSc under most conditions. We also present a real data application of IGSCA to the study of genes and their influence on depression. Finally, we discuss the implications and limitations of this approach, and recommendations for future research.
Translational Abstract
As psychology and many other sciences become interdisciplinary, there is an ever-increasing need for accommodating two statistical representations of constructs, that is, common factors and components, at the same time and examining their relationships to aid in an understanding of human behavior and cognition from more diverse perspectives. For example, psychologists have increasingly been interested in the influences of genetic variation and/or altered brain activities on the variation of psychological constructs in cognition, personality, or mental disorders. Such psychological constructs have typically been represented by common factors, whereas genetic or imaging constructs, such as genes and brain regions, by components. We thus propose a general statistical approach, called integrated generalized structured component analysis (IGSCA), for estimating structural equation models with both factors and components. This approach combines two versions of generalized structured component analysis in a unified manner to estimate both factor- and component-model parameters, including component and factor loadings, component and factor path coefficients, and path coefficients connecting components and factors. We report on two simulation studies that establish IGSCA as a sensible method for estimating models with both factors and components, as compared with existing approaches. Finally, we demonstrate the potential of IGSCA in real data applications with an investigation of the effects of multiple genes on depression.
PDF
