This study provides conceptual and empirical arguments why an assessment of applicants' procedural knowledge about interpersonal behavior via a video-based situational judgment test might be valid ...for academic and postacademic success criteria. Four cohorts of medical students (N = 723) were followed from admission to employment. Procedural knowledge about interpersonal behavior at the time of admission was valid for both internship performance (7 years later) and job performance (9 years later) and showed incremental validity over cognitive factors. Mediation analyses supported the conceptual link between procedural knowledge about interpersonal behavior, translating that knowledge into actual interpersonal behavior in internships, and showing that behavior on the job. Implications for theory and practice are discussed.
Full text
Available for:
CEKLJ, FFLJ, NUK, ODKLJ, PEFLJ
Purpose This study expands upon existing knowledge of response rates by conducting a large-scale quantitative review of published response rates. This allowed a finegrained comparison of response ...rates across respondent groups. Other unique features of this study are the analysis of response enhancing techniques across respondent groups and response rate trends over time. In order to aid researchers in designing surveys, we provide expected response rate percentiles for different survey modalities. Design We analyzed 2,037 surveys, covering 1,251,651 individual respondents, published in 12 journals in I/O Psychology, Management, and Marketing during the period 1995-2008. Expected response rate levels were summarized for different types of respondents and use of response enhancing techniques was coded for each study. Findings First, differences in mean response rate were found across respondent types with the lowest response rates reported for executive respondents and the highest for non-working respondents and non-managerial employees. Second, moderator analyses suggested that the effectiveness of response enhancing techniques was dependent on type of respondents. Evidence for differential prediction across respondent type was found for incentives, salience, identification numbers, sponsorship, and administration mode. When controlling for increased use of response enhancing techniques, a small decline in response rates over time was found. Implications Our findings suggest that existing guidelines for designing effective survey research may not always offer the most accurate information available. Survey researchers should be aware that they may obtain lower/higher response rates depending on the respondent type surveyed and that some response enhancing techniques may be less/more effective in specific samples. Originality/value This study, analyzing the largest set of published response rates to date, offers the first evidence for different response rates and differential functioning of response enhancing techniques across respondent types.
An unanswered question in employee development is how reflection can be used for improving performance in organizations. Drawing from research and theory on dual-process models, we develop and test a ...reflection strategy to stimulate deeper learning after feedback. Results of two studies (
N
=
640 and
N
=
488) showed that reflection combined with feedback enhanced performance improvement on a web-based work simulation better than feedback alone. Reflection without feedback did not lead to performance improvement. Further analyses indicated that the proposed reflection strategy was less effective for individuals low in learning goal orientation, low in need for cognition, and low in personal importance as they engaged less in reflection. Together, these findings provide a theoretical basis for the future study of reflection in organizations and suggest a practical and cost-effective strategy for facilitating employee development after feedback in organizations.
Work effort has been a key concept in management theories and research for more than a century. Maintaining and increasing employee effort also is a persistent concern to managers. The goal of the ...present conceptual and meta-analytic review was to increase clarity and consensus regarding what effort is and how to measure it. First, we reviewed conceptualizations of effort and provided an integrated definition that views effort as a direct outcome of motivation that captures (a) what employees work on, (b) how hard they work, and (c) how long they persist in that work. Second, we identified four main ways researchers have operationalized effort and meta-analytically studied the effects of each operationalization on effort–job performance relationships. For example, measures that assessed multiple dimensions of effort (ρ = .37) tended to relate more strongly to performance than measures that focused on only one dimension (e.g., effort intensity) or on effort more generally (ρ = .18 to .29). Third, we developed and meta-analytically tested a nomological network to gain a better understanding of effort's antecedents (e.g., intrinsic motivation, ρ = .46; performance orientation, ρ = .12) and outcomes (e.g., job performance, ρ = .34; exhaustion, ρ = .04) as well as constructs that appear to overlap with effort (e.g., work engagement, ρ = .48; grit, ρ = .51). Finally, on the basis of our conceptual and meta-analytic reviews, we delineated an agenda for future research on this central, yet often misunderstood, construct.
In case that both the goals of selection quality and diversity are important, a selection system is Pareto-optimal (PO) when its implementation is expected to result in an optimal balance between the ...levels achieved with respect to both these goals. The study addresses the critical issue whether PO systems, as computed from calibration conditions, continue to perform well when applied to a large variety of different validation selection situations. To address the key issue, we introduce two new measures for gauging the achievement of these designs and conduct a large simulation study in which we manipulate 10 factors (related to the selection situation, sensitivity/robustness, and the selection system) that cumulate in a design with 3,888 cells and 24 selection systems. Results demonstrate that PO systems are superior to other, non-PO systems (including unit weighed system designs) both in terms of the achievement measures as well as in terms of yielding more often a better quality/diversity trade-off. The study also identifies a number of conditions that favor the achievement of PO systems in realistic selection situations.
Context Today’s formal medical school admission systems often include only cognitively oriented tests, although most medical school curricula emphasise both cognitive and non‐cognitive factors. ...Situational judgement tests (SJTs) may represent an innovative approach to the formal measurement of interpersonal skills in large groups of candidates in medical school admission processes. This study examined the validity of interpersonal video‐based SJTs in relation to a variety of outcome measures.
Methods This study used a longitudinal and multiple‐cohort design to examine anonymised medical school admissions and medical education data. It focused on data for the Flemish medical school admission examination between 1999 and 2002. Participants were 5444 candidates taking the medical school admission examination. Outcome measures were first‐year grade point average (GPA), GPA in interpersonal communication courses, GPA in non‐interpersonal courses, Bachelor’s degree GPA, Master’s degree GPA and final‐year GPA (after 7 years). For students pursuing careers in general practice, additional outcome measures (9 years after sitting examinations) included supervisor ratings and the results of an interpersonal objective structured clinical examination (OSCE), a general practice knowledge test and a case‐based interview.
Results Interpersonal skills assessment carried out using SJTs had significant added value over cognitive tests for predicting interpersonal GPA throughout the curriculum, doctor performance, and performance on an OSCE and in a case‐based interview. For the other outcomes, cognitive tests emerged as the better predictors. Females significantly outperformed males on the SJT (d = − 0.26). The interpersonal SJT was perceived as significantly more job‐related than the cognitive tests (d = 0.55).
Conclusions Video‐based SJTs as measures of procedural knowledge about interpersonal behaviour show promise as complements to cognitive examination components. The interpersonal skills training received during medical education does not negate the selection of students on the basis of interpersonal skills. Future research is needed to examine the use of SJTs in other cultures and student populations.
Discuss ideas arising from this article at ‘discuss’
In recent years, emotional intelligence and emotional intelligence measures have been used in a plethora of countries and cultures. This is also the case for the Wong and Law Emotional Intelligence ...Scale (WLEIS), highlighting the importance of examining whether the WLEIS is invariant across regions other than the Far Eastern region (China) where it was originally developed. This study investigated the measurement invariance (MI) of the WLEIS scores across two countries, namely Singapore (N= 505) and Belgium (N= 339). Apart from items measuring the factor “use of emotion”, the measurement structure underlying the WLEIS ratings was generally invariant across both countries as there was no departure from MI in terms of factor form and factor loadings. The scalar invariance model (imposing an identical threshold structure) was partially supported. Factor intercorrelations (not involving the factor “use of emotion”) were also identical across countries. These results show promise for the invariance of the WLEIS scores across different countries, yet warn of the non‐invariance of the dimension “use of emotion”. Reducing the motivation‐oriented nature of these items is in order to come to an exact model fit in cross‐cultural comparisons.
This study examines (1) the relationship between the feedback environment and job satisfaction and (2) the mediating role of leader–member exchange in a Belgian context. Results from a sample of 155 ...employees of a governmental service for employment and vocational training supported our hypotheses. A favorable supervisor feedback environment was related to higher levels of job satisfaction 5 months later, and this relationship was fully mediated by the quality of leader–member exchange. These findings highlight the usefulness of diagnosing and assessing the feedback environment for a better understanding of feedback processes and for enhancing feedback interventions in organisations.
Ce travail examine dans un contexte culturel belge 1) la relation entre la satisfaction professionnelle et le feedback provenant de l’environnement 2) le rôle médiateur des échanges leader‐collaborateurs. Nos hypothèses ont été confirmées par les conclusions issues d’un échantillon composé de 155 employés d’un service d’Etat de retour à l’emploi et de formation professionnelle. Un environnement rétroactif positif provenant du chef immédiat était en rapport avec un niveau plus élevé de satisfaction professionnelle cinq mois plus tard, et cette relation était maximisée par la qualité des échanges entre le leader et les membres du groupe. Ces résultats montrent qu’il faut prendre en compte et évaluer la rétroaction environnementale pour une meilleure appréhension des processus de feedback et pour améliorer les interventions dans les organisations portant sur le feedback.
Many scenario-based assessments (e.g., interviews, assessment center exercises, work samples, simulations, and situational judgment tests) use prompts (i.e., cues provided to respondents to increase ...the likelihood that the information received from them is clear, sufficient, and job-related). However, a dilemma for practitioners and researchers is how general or specific one should prompt people's answers. We posit that such differences in prompt-specificity (i.e., extent to which prompts cue performance criteria) have important implications for the predictive validity of scenario-based assessment scores. Drawing on the interplay of situation construal and situational strength theory, we propose that prompt-specificity leads to differential relationships between scenario-based scores and external constructs (personality traits vs. knowledge), which in turn affects the predictive validity of scenario-based assessments. We tested this general hypothesis using intercultural scenarios for predicting effectiveness in multicultural teams. Using a randomized predictive validation design, we contrast scores on these scenarios with general (N = 157) versus specific (N = 158) prompts. As a general conclusion, prompt-specificity mattered: Lesser prompt-specificity augmented the role of perspective taking and openness-to-experience in the intercultural scenario scores and their validity for predicting intercultural performance, whereas greater prompt-specificity increased the role of knowledge in these scores and their validity for predicting in-role performance. This study's theoretical and practical implications go beyond a specific assessment procedure and apply to a broad array of assessment and training approaches that rely on scenarios.
Full text
Available for:
CEKLJ, FFLJ, NUK, ODKLJ, PEFLJ
In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental ...validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT.
Full text
Available for:
CEKLJ, FFLJ, NUK, ODKLJ, PEFLJ
PDF
