Objectives
Within programmatic assessment, the ambition is to simultaneously optimise the feedback and the decision‐making function of assessment. In this approach, individual assessments are ...intended to be low stakes. In practice, however, learners often perceive assessments designed to be low stakes as high stakes. In this study, we explored how learners perceive assessment stakes within programmatic assessment and which factors influence these perceptions.
Methods
Twenty‐six learners were interviewed from three different countries and five different programmes, ranging from undergraduate to postgraduate medical education. The interviews explored learners’ experience with and perception of assessment stakes. An open and qualitative approach to data gathering and analyses inspired by the constructivist grounded theory approach was used to analyse the data and reveal underlying mechanisms influencing learners’ perceptions.
Results
Learners’ sense of control emerged from the analysis as key for understanding learners’ perception of assessment stakes. Several design factors of the assessment programme provided or hindered learners’ opportunities to exercise control over the assessment experience, mainly the opportunities to influence assessment outcomes, to collect evidence and to improve. Teacher–learner relationships that were characterised by learners’ autonomy and in which learners feel safe were important for learners’ believed ability to exercise control and to use assessment to support their learning.
Conclusions
Knowledge of the factors that influence the perception of assessment stakes can help design effective assessment programmes in which assessment supports learning. Learners’ opportunities for agency, a supportive programme structure and the role of the teacher are particularly powerful mechanisms to stimulate the learning value of programmatic assessment.
The authors identified learners’ agency, programme structure and teachers as powerful factors influencing the learning potential of programmatic assessment.
Objectives
Programmatic assessment attempts to facilitate learning through individual assessments designed to be of low‐stakes and used only for high‐stake decisions when aggregated. In practice, ...low‐stake assessments have yet to reach their potential as catalysts for learning. We explored how teachers conceptualise assessments within programmatic assessment and how they engage with learners in assessment relationships.
Methods
We used a constructivist grounded theory approach to explore teachers' assessment conceptualisations and assessment relationships in the context of programmatic assessment. We conducted 23 semi‐structured interviews at two different graduate‐entry medical training programmes following a purposeful sampling approach. Data collection and analysis were conducted iteratively until we reached theoretical sufficiency. We identified themes using a process of constant comparison.
Results
Results showed that teachers conceptualise low‐stake assessments in three different ways: to stimulate and facilitate learning; to prepare learners for the next step, and to use as feedback to gauge the teacher's own effectiveness. Teachers intended to engage in and preserve safe, yet professional and productive working relationships with learners to enable assessment for learning when securing high‐quality performance and achievement of standards. When teachers' assessment conceptualisations were more focused on accounting conceptions, this risked creating tension in the teacher‐learner assessment relationship. Teachers struggled between taking control and allowing learners' independence.
Conclusions
Teachers believe programmatic assessment can have a positive impact on both teaching and student learning. However, teachers' conceptualisations of low‐stake assessments are not focused solely on learning and also involve stakes for teachers. Sampling across different assessments and the introduction of progress committees were identified as important design features to support teachers and preserve the benefits of prolonged engagement in assessment relationships. These insights contribute to the design of effective implementations of programmatic assessment within the medical education context.
Teachers conceptualise programmatic assessment in varied ways, as shown by Schut et al., creating tensions in teacher‐learner assessment relationships.
In some tracked educational systems, track recommendations are formulated by primary school teachers to determine the secondary school level that students will be allocated to. While teachers mostly ...base their track recommendations on students’ prior achievement, the extent to which teachers also consider perceived student attributes, such as students’ perceived work habits or parental involvement, and the extent to which these perceived student attributes are predictive for secondary school performance is unclear. Therefore, we first investigated the extent to which teachers consider their perceptions of student attributes in their track recommendations (RQ1). Differences between students from different backgrounds and differences between teachers were taken into account. Second, we examined the extent to which primary school teachers’ perceptions of student attributes are predictive for their secondary school performance (RQ2). Participants were 17,953 Grade 6 students from 1105 Dutch primary school teachers (RQ1) and 4150 Grade 9 students from 1289 Dutch secondary school classes (RQ2). Data used in this research were analysed using multilevel models. Findings indicated that teacher‐perceived student attributes played only a minor role in track recommendations and secondary school performance. Yet the extent to which these attributes were considered by teachers differed based on students’ background and differed between teachers. For secondary school performance, teacher‐perceived student attributes to have limited predictive value. The limited predictive value of teacher‐perceived student attributes for students’ performance in secondary education suggests that teachers may need to be careful with taking perceived student attributes into account when formulating track recommendations.
Background
There are concerns that school track recommendations that are mostly based on teachers’ judgements of students’ performance (‘judgement‐based recommendations’) are more biased by students’ ...SES than school track recommendations that are mostly based on standardized test results (‘test‐based recommendations’). A recent policy reform of the Dutch educational system has provided us the unique opportunity to compare the effects of students’ SES on these two types of track recommendations.
Aims
The aim of this study was to examine the differences between test‐based and judgement‐based recommendations regarding the direct and indirect effect of students’ SES at student level and school level.
Sample
The sample consisted of 8,639 grade 6 students from 105 Dutch primary schools.
Methods
Data were analysed using two‐level multilevel mediation models.
Results
Track recommendations were higher for high‐SES students. This was mostly due to differences in students’ prior performance. SES also had a small, direct effect on judgement‐based, but not on test‐based recommendations. The effects were partly situated at school level.
Conclusion
Overall, the results indicated that teachers based their track recommendations mostly on students’ prior performance without being biased by students’ SES.
The supervisor-doctoral student interpersonal relationship is important for the success of a PhD-project. Therefore, information about doctoral students' perceptions of their relationship with their ...supervisor can be useful for providing detailed feedback to supervisors aiming at improving the quality of their supervision. This paper describes the development of the questionnaire on supervisor-doctoral student interaction (QSDI). This questionnaire aims at gathering information about doctoral students' perceptions of the interpersonal style of their supervisor. The QSDI appeared to be a reliable and valid instrument. It can be used in research on the relationship between supervisor and doctoral student and can provide supervisors with feedback on their interpersonal style towards a particular student. (HRK / Abstract übernommen).
Introduction
Programmatic assessment was introduced as an approach to design assessment programmes with the aim to simultaneously optimize the decision-making and learning function of assessment. An ...integrative review was conducted to review and synthesize results from studies investigating programmatic assessment in health care professions education in practice.
Methods
The authors systematically searched PubMed, Web of Science, and ERIC to identify studies published since 2005 that reported empirical data on programmatic assessment. Characteristics of the included studies were extracted and synthesized, using descriptive statistics and thematic analysis.
Results
Twenty-seven studies were included, which used quantitative methods (
n
= 10), qualitative methods (
n
= 12) or mixed methods (
n
= 5). Most studies were conducted in clinical settings (77.8%). Programmatic assessment was found to enable meaningful triangulation for robust decision-making and used as a catalyst for learning. However, several problems were identified, including overload in assessment information and the associated workload, counterproductive impact of using strict requirements and summative signals, lack of a shared understanding of the nature and purpose of programmatic assessment, and lack of supportive interpersonal relationships. Thematic analysis revealed that the success and challenges of programmatic assessment were best understood by the interplay between quantity and quality of assessment information, and the influence of social and personal aspects on assessment perceptions.
Conclusion
Although some of the evidence may seem compelling to support the effectiveness of programmatic assessment in practice, tensions will emerge when simultaneously stimulating the development of competencies and assessing its result. The identified factors and inferred strategies provide guidance for navigating these tensions.
The first-year grade point average (FYGPA) is the predominant measure of student success in most studies on university admission. Previous cognitive achievements measured with high school grades or ...standardized tests have been found to be the strongest predictors of FYGPA. For this reason, standardized tests measuring cognitive achievement are widely used as a tool for selective admission to higher education. The FYGPA, however, measures many markedly different aspects of student success. In this article it is shown that when the FYGPA is divided into averages that reflect performance on different types of goals, the predictive value of previous cognitive achievement differs significantly between these disentangled averages. It is therefore important to distinguish between different types of goals when considering what student success is, and which students should be admitted to particular university programmes.
Low-stakes assessments are theorised to stimulate and support self-regulated learning. They are feedback-, not decision-oriented, and should hold little consequences to a learner based on their ...performance. The use of low-stakes assessment as a learning opportunity requires an environment in which continuous improvement is encouraged. This may be hindered by learners’ perceptions of assessment as high-stakes. Teachers play a key role in learners’ assessment perceptions. By investigating assessment perceptions through an interpersonal theory-based perspective of teacher–learner relationships, we aim to better understand the mechanisms explaining the relationship between assessment and learning within medical education. First, twenty-six purposefully selected learners, ranging from undergraduates to postgraduates in five different settings of programmatic assessment, were interviewed about their assessment task perception. Next, we conducted a focussed analysis using sensitising concepts from interpersonal theory to elucidate the influence of the teacher–learner relationship on learners’ assessment perceptions. The study showed a strong relation between learners’ perceptions of the teacher–learner relationship and their assessment task perception. Two important sources for the perception of teachers’ agency emerged from the data: positional agency and expert agency. Together with teacher’s communion level, both types of teachers’ agency are important for understanding learners’ assessment perceptions. High levels of teacher communion had a positive impact on the perception of assessment for learning, in particular in relations in which teachers’ agency was less dominantly exercised. When teachers exercised these sources of agency dominantly, learners felt inferior to their teachers, which could hinder the learning opportunity. To utilise the learning potential of low-stakes assessment, teachers are required to stimulate learner agency in safe and trusting assessment relationships, while carefully considering the influence of their own agency on learners’ assessment perceptions. Interpersonal theory offers a useful lens for understanding assessment relationships. The Interpersonal Circumplex provides opportunities for faculty development that help teachers develop positive and productive relationships with learners in which the potential of low-stakes assessments for self-regulated learning is realised.
Students' use of a rubric for research theses Prins, Frans J.; de Kleijn, Renske; Tartwijk, Jan van
Assessment and evaluation in higher education,
01/2017, Letnik:
42, Številka:
1
Journal Article
Recenzirano
A rubric for research theses was developed, based on the manual of the American Psychological Association, to be used as an assessment tool for teachers and students. The aim was to make students ...aware of what is expected, get familiar with criteria, and interpret teacher and peer feedback. In two studies, it was examined whether students use and value these functions. In the first study, a rubric was provided to 105 Educational Sciences students working on their bachelor's thesis. Questionnaire data indicated that students did value the rubric for the intended functions, although rubric use was not related to ability. In a panel interview, teachers stated that the number of proficiency levels should be increased to be able to distinguish between good and excellent students adequately, and that a criterion concerning student's role during supervision should be added. Therefore, in the second study, 11 teachers were interviewed about their motives to give high grades and about the supervision process. This lead to an extra criterion concerning student's role during supervision and an additional proficiency level to assess excellent performance. It is argued that an adequate course organisation is conditional for the rubric's effectiveness.
Tasks are a powerful instrument for geography teachers, as they let students engage with the subject. To advance the cumulative learning of students, teachers have to make sure that students learn ...how to deal with complex and abstract knowledge structures. In the Netherlands, teachers face a dilemma when it comes to task setting: the intended curriculum aims for a considerable part at (parts of) higher order thinking, whereas the high-stakes exams have a clear focus on the use of thinking strategies. This paper explores the task setting and debriefing of Dutch geography teachers by analyzing twenty-three videotaped lessons in upper secondary education by using the Geography Task Categorization Framework. The results show that Dutch teachers mostly rely on textbooks when setting tasks. The focus lies on reproduction and the use of thinking strategies. Tasks aiming at (parts of) higher order thinking are barely used. Furthermore, teachers use tasks from previous high-stakes exams already used in an early stage of upper secondary education. In the debriefing of tasks, teachers move from simple and concrete to complex and abstract knowledge and vice versa. However, most of these movements aim at simplifying knowledge structures. In the observed lessons, curriculum aims at the level of (parts of) higher order thinking are not achieved. The evaluative rules as set by the high-stakes exams and the type of tasks offered by textbooks seem to be dominant.
PDF
