The purpose of this study was to apply a set of rarely reported psychometric indices that, nevertheless, are important to consider when evaluating psychological measures. All can be derived from a ...standardized loading matrix in a confirmatory bifactor model: omega reliability coefficients, factor determinacy, construct replicability, explained common variance, and percentage of uncontaminated correlations. We calculated these indices and extended the findings of 50 recent bifactor model estimation studies published in psychopathology, personality, and assessment journals. These bifactor derived indices (most not presented in the articles) provided a clearer and more complete picture of the psychometric properties of the assessment instruments. We reached 2 firm conclusions. First, although all measures had been tagged "multidimensional," unit-weighted total scores overwhelmingly reflected variance due to a single latent variable. Second, unit-weighted subscale scores often have ambiguous interpretations because their variance mostly reflects the general, not the specific, trait. Finally, we review the implications of our evaluations and consider the limits of inferences drawn from a bifactor modeling approach.
Thinking twice about sum scores McNeish, Daniel; Wolf, Melissa Gordon
Behavior research methods,
12/2020, Letnik:
52, Številka:
6
Journal Article
Recenzirano
Odprti dostop
A common way to form scores from multiple-item scales is to sum responses of all items. Though sum scoring is often contrasted with factor analysis as a competing method, we review how factor ...analysis and sum scoring both fall under the larger umbrella of latent variable models, with sum scoring being a constrained version of a factor analysis. Despite similarities, reporting of psychometric properties for sum scored or factor analyzed scales are quite different. Further, if researchers use factor analysis to validate a scale but subsequently sum score the scale, this employs a model that differs from validation model. By framing sum scoring within a latent variable framework, our goal is to raise awareness that (a) sum scoring requires rather strict constraints, (b) imposing these constraints requires the same type of justification as any other latent variable model, and (c) sum scoring corresponds to a statistical model and is not a model-free arithmetic calculation. We discuss how unjustified sum scoring can have adverse effects on validity, reliability, and qualitative classification from sum score cut-offs. We also discuss considerations for how to use scale scores in subsequent analyses and how these choices can alter conclusions. The general goal is to encourage researchers to more critically evaluate how they obtain, justify, and use multiple-item scale scores.
Bifactor measurement models are increasingly being applied to personality and psychopathology measures (Reise, 2012). In this work, authors generally have emphasized model fit, and their typical ...conclusion is that a bifactor model provides a superior fit relative to alternative subordinate models. Often unexplored, however, are important statistical indices that can substantially improve the psychometric analysis of a measure. We provide a review of the particularly valuable statistical indices one can derive from bifactor models. They include omega reliability coefficients, factor determinacy, construct reliability, explained common variance, and percentage of uncontaminated correlations. We describe how these indices can be calculated and used to inform: (a) the quality of unit-weighted total and subscale score composites, as well as factor score estimates, and (b) the specification and quality of a measurement model in structural equation modeling.
Decalogue for the Factor Analysis of Test Items.
In the study of the psychometric properties of the items of a test, a fundamental aspect is the analysis of their dimensional structure. The objective ...of this work is to provide some guidelines that allow the factor analysis of the items to be carried out in a rigorous and systematic way.
A review of the recent psychometric literature was carried out to identify the fundamental steps to be followed in order to carry out an adequate factor analysis of the items of a test.
Ten main recommendations were identified to carry out the factorial analysis of the items of a test: adequacy of the data and the sample, univariate statistics, justification of the analysis, selection of the analyzable items, type of model, most appropriate factorial solution, estimation of the parameters, adequacy of the factorial solution, substantive coherence of the model, and final version of the test.
If the ten recommendations proposed in the current psychometric literature are systematically followed, it will be possible to optimize the quality of the tests and the decision-making based on the estimates of the scores obtained through them. These recommendations should be useful to both researchers and practitioners.
Normalization of RNA-sequencing (RNA-seq) data has proven essential to ensure accurate inference of expression levels. Here, we show that usual normalization approaches mostly account for sequencing ...depth and fail to correct for library preparation and other more complex unwanted technical effects. We evaluate the performance of the External RNA Control Consortium (ERCC) spike-in controls and investigate the possibility of using them directly for normalization. We show that the spike-ins are not reliable enough to be used in standard global-scaling or regression-based normalization procedures. We propose a normalization strategy, called remove unwanted variation (RUV), that adjusts for nuisance technical effects by performing factor analysis on suitable sets of control genes (e.g., ERCC spike-ins) or samples (e.g., replicate libraries). Our approach leads to more accurate estimates of expression fold-changes and tests of differential expression compared to state-of-the-art normalization methods. In particular, RUV promises to be valuable for large collaborative projects involving multiple laboratories, technicians, and/or sequencing platforms.
Exploratory graph analysis (EGA) is a new technique that was recently proposed within the framework of network psychometrics to estimate the number of factors underlying multivariate data. Unlike ...other methods, EGA produces a visual guide-network plot-that not only indicates the number of dimensions to retain, but also which items cluster together and their level of association. Although previous studies have found EGA to be superior to traditional methods, they are limited in the conditions considered. These issues are addressed through an extensive simulation study that incorporates a wide range of plausible structures that may be found in practice, including continuous and dichotomous data, and unidimensional and multidimensional structures. Additionally, two new EGA techniques are presented: one that extends EGA to also deal with unidimensional structures, and the other based on the triangulated maximally filtered graph approach (EGAtmfg). Both EGA techniques are compared with 5 widely used factor analytic techniques. Overall, EGA and EGAtmfg are found to perform as well as the most accurate traditional method, parallel analysis, and to produce the best large-sample properties of all the methods evaluated. To facilitate the use and application of EGA, we present a straightforward R tutorial on how to apply and interpret EGA, using scores from a well-known psychological instrument: the Marlowe-Crowne Social Desirability Scale.
Translational Abstract
Understanding the structure and composition of data is an important undertaking for a wide range of scientific domains. An initial step in this endeavor is to determine how the data can be summarized into a smaller set of meaningful variables (i.e., dimensions). In this article, we extend a state-of-the-art network science approach, called exploratory graph analysis (EGA), used to identify the dimensions that exist in multivariate data. Using Monte Carlo methods, we compared EGA with several traditional eigenvalue-based approaches that are commonly used in the psychological literature including parallel analysis. Additionally, the simulation study evaluated the performance of new variants of the EGA method and considered a wider set of realistic conditions, such as unidimensional structures and variables of continuous and categorical levels of measurement. We found that EGA performed as well as or better than the most accurate traditional method (i.e., parallel analysis). Importantly, EGA offers a few advantages over traditional methods: (a) it provides an intuitive visual representation of the results, (b) this representation offers a more complex understanding of the data's structure, and (c) the algorithm is deterministic meaning there are fewer researcher degrees of freedom. In sum, our study demonstrates that EGA can accurately identify the underlying structure of multivariate data, while retaining the complexity of the data's structure. This implies that researchers can meaningfully summarize their data without sacrificing the finer details.
Latent variable modeling is a popular and flexible statistical framework. Concomitant with fitting latent variable models is assessment of how well the theoretical model fits the observed data. ...Although firm cutoffs for these fit indexes are often cited, recent statistical proofs and simulations have shown that these fit indexes are highly susceptible to measurement quality. For instance, a root mean square error of approximation (RMSEA) value of 0.06 (conventionally thought to indicate good fit) can actually indicate poor fit with poor measurement quality (e.g., standardized factors loadings of around 0.40). Conversely, an RMSEA value of 0.20 (conventionally thought to indicate very poor fit) can indicate acceptable fit with very high measurement quality (standardized factor loadings around 0.90). Despite the wide-ranging effect on applications of latent variable models, the high level of technical detail involved with this phenomenon has curtailed the exposure of these important findings to empirical researchers who are employing these methods. This article briefly reviews these methodological studies in minimal technical detail and provides a demonstration to easily quantify the large influence measurement quality has on fit index values and how greatly the cutoffs would change if they were derived under an alternative level of measurement quality. Recommendations for best practice are also discussed.
Model fit assessment is a central component of evaluating confirmatory factor analysis models and the validity of psychological assessments. Fit indices remain popular and researchers often judge fit ...with fixed cutoffs derived by Hu and Bentler (1999). Despite their overwhelming popularity, methodological studies have cautioned against fixed cutoffs, noting that the meaning of fit indices varies based on a complex interaction of model characteristics like factor reliability, number of items, and number of factors. Criticism of fixed cutoffs stems primarily from the fact that they were derived from one specific confirmatory factor analysis model and lack generalizability. To address this, we propose a simulation-based method called dynamic fit index cutoffs such that derivation of cutoffs is adaptively tailored to the specific model and data characteristics being evaluated. Unlike previously proposed simulation-based techniques, our method removes existing barriers to implementation by providing an open-source, Web based Shiny software application that automates the entire process so that users neither need to manually write any software code nor be knowledgeable about foundations of Monte Carlo simulation. Additionally, we extend fit index cutoff derivations to include sets of cutoffs for multiple levels of misspecification. In doing so, fit indices can more closely resemble their originally intended purpose as effect sizes quantifying misfit rather than improperly functioning as ad hoc hypothesis tests. We also provide an approach specifically designed for the nuances of 1-factor models, which have received surprisingly little attention in the literature despite frequent substantive interests in unidimensionality.
Translational AbstractEvaluating confirmatory factor model fit through the lens of "approximate fit" has enjoyed widespread - though not universal - adoption in empirical studies and is valued for its goal to assess whether the model may be practically useful, even if fit is not exact. An obstacle with approximate fit is that there is ambiguity regarding what is considered "practically useful". Hu and Bentler (1999) addressed this issue with an expansive study to provide guidelines for values indicating that a model demonstrates reasonable approximate fit. Though their suggestions remain widely used today and are engrained in the literature, their suggested cutoffs have unfortunately been shown to vary widely depending on context. That is, values that indicate great fit in one context may indicate poor fit in another. The inherent problem is that the Hu and Bentler cutoffs are fixed values, so they arbitrarily benefit some models and arbitrarily punish others. Previous literature has suggested custom simulation methods that take the logic of Hu and Bentler's approach and apply it to the individual model being evaluated. In this way, researchers can obtain values indicative of good approximate fit in their specific circumstances to avoid fixed cutoffs. Though a clever solution, such an approach has seen little uptake presumably because many researchers are not well-versed in conducting simulations. This paper addresses the issue by providing a method and software application that builds and executes a Hu-and- Bentler-style simulation from model output. In this way, researchers can benefit from modern computational resources without a deep programming background.
Dissolved organic matter (DOM) is the precursor of disinfection by-products (DBPs) which is widely found in the aquatic environment. The analysis of DOM in raw water is helpful to evaluate the ...formation potentials of DBPs. However, there is relatively little research on the DOM identification of raw water in northern China. In this study, the variation in DOM in M reservoir water in one year by fluorescence excitation–emission matrix-parallel factor analysis (EEM–PARAFAC) was investigated to evaluate the DBP formation potential (DBPFP). The results suggested that five components, namely, two humic-like substances (C2, C3), two fulvic-like substances (C1, C4) and one protein-like substance (C5), were identified in the DOM of M reservoir water. The content of DOM in autumn and winter was higher than that in spring and summer. The source of DOM in the water body of M reservoir was mainly from terrestrial source, but less from aquatic source. The source, types and humification degree of DOM affect the formation of DBPs. The formation potential of DBPs had the following order: trihalomethanes (THMs) > dichloroacetic acid (TCAA) > trichloroacetic acid (DCAA) > chloral hydrate (CH). The formation potentials of THM and TCAA were strongly correlated with C2 (rTHM = 0.805, rTCAA = 0.857). The formation potential of CH has a good correlation with C1 (r = 0.722). The formation of DCAA has a good correlation with C4 (r = 0.787). DOM and DBPFP were negatively correlated with the biological index (BIX) and fluorescence index (FI) of the raw water, and positively correlated with the humification index (HIX).
Display omitted
•The higher DOM variation in raw reservoir water occurs in autumn and winter.•Humic-like, fulvic-like, and tryptophan-like components were identified by PARAFAC.•DBPFP was significantly correlated with four humic-like and fulvic-like components.•The source, types and humification degree of DOM affect the formation of DBPs.
Technological advances have enabled the profiling of multiple molecular layers at single-cell resolution, assaying cells from multiple samples or conditions. Consequently, there is a growing need for ...computational strategies to analyze data from complex experimental designs that include multiple data modalities and multiple groups of samples. We present Multi-Omics Factor Analysis v2 (MOFA+), a statistical framework for the comprehensive and scalable integration of single-cell multi-modal data. MOFA+ reconstructs a low-dimensional representation of the data using computationally efficient variational inference and supports flexible sparsity constraints, allowing to jointly model variation across multiple sample groups and data modalities.
PDF
