The accumulation of scientific knowledge on calling is limited by the absence of a common theoretical and measurement framework. Many different models of calling have been proposed, and we do not ...know how much research results that refer to a specific model are generalizable to different theoretical accounts of calling. In this article, we investigate whether two leading models of calling tackle the same construct. The two models were merged into a comprehensive framework that measures calling across seven facets: Passion, Purposefulness, Sacrifice, Pervasiveness, Prosocial Orientation, Transcendent Summons, and Identity. We then developed the Unified Multidimensional Calling Scale (UMCS) drawing from previous published items. Across two surveys involving college students (N = 5886) and adult employees (N = 205) the UMCS was proved to be valid and reliable. We also observed that the UMCS is invariant across time and calling domains. Finally, we found that facets of calling have very different relationships with outcomes and concurrent measures, suggesting that results obtained with a smaller set of facets are not generalizable to the higher-order construct of calling or to a different model that does not share the same facets.
This dataset provides de-identified raw responses to a non-anonymous three-wave online survey with a 12-month time lag. Data collection was part of a larger project on the development of career ...calling in Italian college students. The first wave was collected during the fall of 2014. Participants were bachelor's or master's students enrolled in 24 different study domains and 4 different universities. Sample sizes for each wave are NT1 = 5,886, NT2 = 1,700 and NT3 = 881, 434 participants provided valid responses at all the three waves. Consent form was electronic. Dataset and codebook can be found here: https://osf.io/v56du/. The sample is mainly composed of women (63.8%, at Time 1). Participants' mean age at Time 1 was 23.37 years (SD = 5.39). The survey was in Italian and included multiple-item measures of career calling, intrinsic and extrinsic motivation, social support, engaged learning, clarity of professional identity, and quality of mentorship. Socio-demographic information and academic performance indicators are provided. The dataset is necessary to reproduce previously published results (Vianello et al., 2018) and can be useful to 1) investigate cross-cultural differences between college students from Italy and other countries; 2) investigate person-level variability in predictors and consequences of change in the variables collected over time; 3) develop and/or validate new statistical models for longitudinal data; and 4) develop and/or test original theoretical hypotheses.
Estimating the reproducibility of psychological science Aarts, Alexander A.; Nilsonne, Gustav; Zuni, Kellylynn
Science (American Association for the Advancement of Science),
08/2015, Letnik:
349, Številka:
6251
Journal Article
Recenzirano
Odprti dostop
Empirically analyzing empirical evidence
One of the central goals in any scientific endeavor is to understand causality. Experiments that seek to demonstrate a cause/effect relation most often ...manipulate the postulated causal factor. Aarts
et al.
describe the replication of 100 experiments reported in papers published in 2008 in three high-ranking psychology journals. Assessing whether the replication and the original experiment yielded the same result according to several criteria, they find that about one-third to one-half of the original findings were also observed in the replication study.
Science
, this issue
10.1126/science.aac4716
A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired.
INTRODUCTION
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. Scientific claims should not gain credence because of the status or authority of their originator but by the replicability of their supporting evidence. Even research of exemplary quality may have irreproducible empirical findings because of random or systematic error.
RATIONALE
There is concern about the rate and predictors of reproducibility, but limited evidence. Potentially problematic practices include selective reporting, selective analysis, and insufficient specification of the conditions necessary or sufficient to obtain the results. Direct replication is the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding and is the means of establishing reproducibility of a finding with new data. We conducted a large-scale, collaborative effort to obtain an initial estimate of the reproducibility of psychological science.
RESULTS
We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. There is no single standard for evaluating replication success. Here, we evaluated reproducibility using significance and
P
values, effect sizes, subjective assessments of replication teams, and meta-analysis of effect sizes. The mean effect size (r) of the replication effects (
M
r
= 0.197, SD = 0.257) was half the magnitude of the mean effect size of the original effects (
M
r
= 0.403, SD = 0.188), representing a substantial decline. Ninety-seven percent of original studies had significant results (
P
< .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
CONCLUSION
No single indicator sufficiently describes replication success, and the five indicators examined here are not the only ways to evaluate reproducibility. Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes. Moreover, correlational evidence is consistent with the conclusion that variation in the strength of initial evidence (such as original
P
value) was more predictive of replication success than variation in the characteristics of the teams conducting the research (such as experience and expertise). The latter factors certainly can influence replication success, but they did not appear to do so here.
Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not. This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know.
Original study effect size versus replication effect size (correlation coefficients).
Diagonal line represents replication effect size equal to original effect size. Dotted line represents replication effect size of 0. Points below the dotted line were effects in the opposite direction of the original. Density plots are separated by significant (blue) and nonsignificant (red) effects.
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Leaders influence followers in many ways; one way is by eliciting positive emotions. In three studies we demonstrate that the nearly unstudied moral emotion of 'elevation' (a reaction to moral ...excellence) mediates the relations between leaders' and their followers' ethical behavior. Study 1 used scenarios manipulated experimentally; study 2 examined employees' emotional responses to their leaders in a natural work setting; study 3 compared the effects of elevation to those of happiness, serenity, and positive affect. We found that leaders' interpersonal fairness and self-sacrifice are powerful elicitors of elevation, and that this emotion fully mediates leaders' influence on followers' organizational citizenship behavior and affective organizational commitment. In the first study, we also observed a moderation effect of interpersonal fairness on self-sacrifice. Results underline the importance of positive moral emotions in organizations and shed light on the emotional process by which ethical leaders can foster positive organizational outcomes.
There are many open questions concerning the development of calling, and longitudinal empirical evidence is limited. We know that a calling is associated with many beneficial outcomes, but we do not ...know how it changes through time and what predicts these changes. Previous studies have shown that calling is relatively stable at the sample level. We show that, at the individual level, calling shows huge variations through time. We identified nine developmental trajectories that are typical across facets of calling, and we found evidence that the development of a calling is fostered by the extent to which individuals have lived it out. We also observed that the more a calling has grown over a 2-year period, the more it is lived out during the third year. These results provide support for a developmental model of calling according to which having a calling and living it out reciprocally influence each other. The practical and theoretical implications of these results are discussed.
We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples ...and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.
Reproducibility is a defining feature of science. However, because of strong incentives for innovation and weak incentives for confirmation, direct replication is rarely practiced or published. The ...Reproducibility Project is an open, large-scale, collaborative effort to systematically examine the rate and predictors of reproducibility in psychological science. So far, 72 volunteer researchers from 41 institutions have organized to openly and transparently replicate studies published in three prominent psychological journals in 2008. Multiple methods will be used to evaluate the findings, calculate an empirical rate of replication, and investigate factors that predict reproducibility. Whatever the result, a better understanding of reproducibility will ultimately improve confidence in scientific methodology and findings.
We examined the relationship between calling, job‐search clarity, and job‐search intensity in a cross‐sectional study of Italian unemployed job seekers (N = 315). Structural equation modeling with ...observed variables and latent moderated structural equation models were adopted to test whether optimism, self‐esteem, and perseverance moderate the relation between calling, job‐search clarity, and job‐search intensity. Perceiving a calling was positively related with job‐search clarity and intensity, and these relations were stronger in individuals with lower levels of optimism, self‐esteem, and perseverance. This study suggests that perceiving a calling is an important personal resource that is related to a clearer job‐search goal and to more intense job‐search activities and can support job seekers in personal adverse conditions. These results suggest integrating job‐search behaviors in the work‐as‐calling theory and that incorporating the construct of calling into career counselors' practices may increase the efficacy of job‐search activities.
Investigating Variation in Replicability Klein, Richard A.; Ratliff, Kate A.; Vianello, Michelangelo ...
Social psychology (Göttingen, Germany),
01/2014, Letnik:
45, Številka:
3
Journal Article
Recenzirano
Odprti dostop
Although replication is a central tenet of science, direct replications are rare
in psychology. This research tested variation in the replicability of 13 classic and
contemporary effects across 36 ...independent samples totaling 6,344 participants. In the
aggregate, 10 effects replicated consistently. One effect - imagined contact reducing
prejudice - showed weak support for replicability. And two effects - flag priming
influencing conservatism and currency priming influencing system justification - did not
replicate. We compared whether the conditions such as lab versus online or US versus
international sample predicted effect magnitudes. By and large they did not. The results of
this small sample of effects suggest that replicability is more dependent on the effect itself
than on the sample and setting used to investigate the effect.
PDF
