In the years since Hodges' seminal 2013 paper that placed the phrase in its title, numerous authors have claimed that we are either moving into, or already living in, a 'post-psychometric era' in ...which subjective and qualitative data are increasingly valued. What does this actually mean? There is a post-for most things, from post-modernism to post-humanism to post-rock. Each is a complex and nuanced notion. However, they are bound together, in that 'post' typically indicates an idea that is differentiated from an antithetical conception. Although post-psychometric is different to anti-psychometric, it is rarely articulated as such. In fact, it is barely articulated at all, aside from being 'beyond' psychometrics. Does it supersede the psychometric? Is it a natural evolution that builds on previous epochs? At the time of writing, Hodges' paper has been cited 128 times. Even if authors add a caveat on what they mean by post-psychometric, anti-psychometric conceptions saturate the literature. Moving 'beyond' is too often interpreted as moving 'away' from psychometrics. Hodges identified three imperatives from psychometric discourse that naturally elicited strong reservations from medical educators. Author abstract
The use of decision-making support tools during assessments, such as electronic differential diagnosis in examinations, is just the tip of the iceberg when it comes to how technology is currently ...changing assessment practice. We have
reached a transformative stage in the development of artificial intelligence (AI). We can no longer rely on non-invigilated assessments and submitted 'artefacts' to demonstrate student learning and competence. This is bringing many long-
term demands on educators, course coordinators and curriculum designers, forcing us to rethink assessment approaches. Going forward, we see an important distinction between 'assisted' assessments and 'unassisted' assessments. With the
recent increase and facilitation of virtual assessment through convenient online platforms, and the new challenge to non-invigilated assessment formats posed by AI, we think the time has come for the 'rehabilitation' and re-acceptance of
the oral format as a highly valuable and unique form of assessment in medical education. Nevertheless, generative AI need not threaten the validity or trustworthiness of our assessments in either formative or summative contexts. Rather,
it can add fidelity and nuance to assisted assessment while facilitating a greater focus and purposefulness to unassisted assessment. Author abstract
Practitioners in health sciences education and assessment regularly use a range of psychometric techniques to analyse data, evaluate models, and make crucial progression decisions regarding student ...learning. However, a recent editorial entitled 'Is Psychometrics Science' highlighted some core epistemological and practical problems in psychometrics, and brought its legitimacy into question. This paper attempts to address these issues by applying some key ideas from history and philosophy of science (HPS) discourse. I present some of the conceptual developments in HPS that have bearing on the psychometrics debate. Next, by shifting the focus onto what constitutes the practice of science, I discuss psychometrics in action. Some incorrectly conceptualise science as an assemblage of truths, rather than an assemblage of tools and goals. Psychometrics, however, seems to be an assemblage of methods and techniques. Psychometrics in action represents a range of practices using specific tools in specific contexts. This does not render the practice of psychometrics meaningless or futile. Engaging in debates about whether or not we should regard psychometrics as 'scientific' is, however, a fruitless enterprise. The key question and focus should be whether, on what grounds, and in what contexts, the existing methods and techniques used by psychometricians can be justified or criticised. Author abstract
The way educators think about the nature of competence, the approaches one selects for the assessment of competence, what generated data implies, and what counts as good assessment now involve ...broader and more diverse interpretive
processes. Broadening philosophical positions in assessment has educators applying different interpretations to similar assessment concepts. As a result, what is claimed through assessment, including what counts as quality, can be
different for each of us despite using similar activities and language. This is leading to some uncertainty on how to proceed or worse, provides opportunities for questioning the legitimacy of any assessment activity or outcome. While
some debate in assessment is inevitable, most have been within philosophical positions (e.g., how best to minimize error), whereas newer debates are happening across philosophical positions (e.g., whether error is a useful concept). As
new ways of approaching assessment have emerged, the interpretive nature of underlying philosophical positions has not been sufficiently attended to. We illustrate interpretive processes of assessment in action by (a) summarizing the
current health professions assessment context from a philosophical perspective as a way of describing its evolution; (b) demonstrating implications in practice using two examples (i.e., analysis of assessment work and validity claims);
and (c) examining pragmatism to demonstrate how even within specific philosophical positions opportunities for variable interpretations still exist. Our concern is not that assessment designers and users have different assumptions, but
that practically, educators may unknowingly (or insidiously) apply different assumptions, and methodological and interpretive norms, and subsequently settle on different views on what serves as quality assessment even for the same
assessment program or event. With the state of assessment in health professions in flux, we conclude by calling for a philosophically explicit approach to assessment, and underscore assessment as, fundamentally, an interpretive process -
one which demands the careful elucidation of philosophical assumptions to promote understanding and ultimately defensibility of assessment processes and outcomes. Author abstract
Constructed-response questions (CRQs) require effective marking schemes to ensure that the intended learning objectives and/or professional competencies are appropriately addressed, and valid ...inferences regarding examinee competence
are drawn from such assessments. While the educational literature on writing rubrics has proliferated in recent years, this is largely targeted at classroom use and formative purposes. There is comparatively little guidance on how to
develop appropriate marking schemes for summative assessment contexts. The different purposes mean that different principles and practices apply to mark schemes for examinations. In this article, we draw on the educational literature as
well as our own practical experience of working with medical and health professional educators on their questions and marking schemes to offer 12 key principles or tips for designing and implementing effective marking schemes. Author
abstract
On the use and abuse of metaphors in assessment Pearce, Jacob; Chiavaroli, Neville; Tavares, Walter
Advances in health sciences education : theory and practice,
10/2023, Letnik:
28, Številka:
4
Journal Article
Recenzirano
This paper is motivated by a desire to advance assessment in the health professions through encouraging the judicious and productive use of metaphors. Through five specific examples (pixels, driving ...lesson/test, jury deliberations,
signal processing, and assessment as a toolbox), we interrogate how metaphors are being used in assessment to consider what value they add to understanding and implementation of assessment practices. By unpacking these metaphors in
action, we probe each metaphor's rationale and function, the gains each metaphor makes, and explore the unintended meanings they may carry. In summarizing common uses of metaphors, we elucidate how there may be both advantages and/or
disadvantages. Metaphors can play important roles in simplifying, complexifying, communicating, translating, encouraging reflection, and convincing. They may be powerfully rhetorical, leading to intended consequences, actions, and other
pragmatic outcomes. Although metaphors can be extremely helpful, they do not constitute thorough critique, justified evidence or argumentation. We argue that although metaphors have utility, they must be carefully considered if they are
to serve assessment needs in intended ways. We should pay attention to how metaphors may be misinterpreted, what they ignore or unintentionally signal, and perhaps mitigate against this with anticipated corrections or nuanced
qualifications. Failure to do so may lead to implementing practices that miss underlying and relevant complexities for assessment science and practice. Using metaphors requires careful attention with respect to their role, contributions,
benefits and limitations. We highlight the value that comes from critiquing metaphors, and demonstrate the care required to ensure their continued utility. Author abstract
A programmatic approach to assessment entails gathering and aggregating 'rich information' on candidates to inform progress decisions. However, there is little guidance on how such an approach might ...be implemented in practice.
We describe an approach to aggregating rich information across assessment formats to inform committee decision-making in a specialist medical college.
Each item (n = 272) for every examination was blueprinted to 15 curriculum modules and 7 proficiencies. We developed a six-point holistic rating scale with detailed rubrics outlining expected performance standards for every item. Examiners used this rating scale in making judgements for each item, generating rich performance data for each candidate.
A colour-coded 'mosaic' of patterns of performance across modules and proficiencies was generated along with frequency distributions of ratings. These data allowed examiners to easily visualise candidate performance and to use these data to inform deliberations on borderline candidates. Committee decision-making was facilitated by maintaining the richness of assessment information throughout the process. Moreover, the data facilitated detailed and useful feedback to candidates.
Our study demonstrates that incorporating aspects of programmatic thinking into high-stakes examinations by using a novel approach to aggregating information is a useful first step in reforming an assessment program.
This paper traces the emergence of “why” questions in modern cosmology and the responding proliferation of multiverse discourse in the late twentieth and early twenty-first centuries. Critics who see ...speculative theorizing as delving into the metaphysical are not hard to find. George Ellis’ concern that we are entering a new era of ‘cosmological myth’ resonates with the 1937 debate regarding “cosmythology” and the shifting boundary between physics and metaphysics. However, the charge that multiverse proposals are nothing but speculative metaphysics can be considered in terms other than criteria relating to empirical testability. A historicist reading of what metaphysics represents in this context is presented in order to emphasize that “metaphysical” as a pejorative term in science discourse is a fluid and historically contingent concept. It appears that proposals are being considered metaphysical precisely when there is no consensus on what constitutes empirical testability. Drawing on the work of Nicholas Jardine, Hans-Jörg Rheinberger and Christopher Hookway, I argue that in cosmology during this period, particularly in relation to multiverse proposals, there appears a well-defined “scene of response”, rather than of fully-fledged inquiry. Thus, intelligible questions may be considered metaphysical, but not timelessly so.
Celotno besedilo
Dostopno za:
DOBA, IZUM, KILJ, NUK, PILJ, PNG, SAZU, UILJ, UKNU, UL, UM, UPUK
Setting standards for determining competent performance on examinations is well-established in medical education. Establishing a conceptual boundary that differentiates competent from non-competent ...test-takers typically involves either expert judgement based on test item characteristics (e.g. Angoff), or an empirical process based on examinee performance (e.g. Borderline regression). Standard-setting procedures are a mandatory component of assessment processes in Australian specialist medical training programs. However, small specialist medical Colleges cannot implement existing standard-setting methods for performance-based examinations. Angoff-type methods are often not suited for performance examinations and empirical methods are unsuitable for small cohorts. A reform of assessment processes undertaken by the Royal Australian College of Dental Surgeons (RACDS) Oral and Maxillofacial Surgery (OMS) final examinations required implementation of appropriate standard-setting processes; however, the chosen method had to work with small candidate numbers and small numbers of examiners to support the process. Author abstract
In this article, we examine assessment as conceptualized and enacted in continuing professional development (CPD). Assessment is pervasive throughout the life of an individual health professional, ...serving many different purposes compounded by varied and unique contexts, each with their own drivers and consequences, usually casting the person as the object of assessment. Assessment is often assumed as an included part in CPD development conceptualization. Research on assessment in CPD is often focused on systems, utility, and quality instead of intentionally examining the link between assessment and the person. We present an alternative view of assessment in CPD as person-centered, practice-informed, situated and bound by capability, and enacted in social and material contexts. With this lens of assessment as an inherently personal experience, we introduce the concept of subjectification, as described by educationalist Gert Biesta. We propose that subjectification may be a fruitful way of examining assessment in a CPD context. Although the CPD community, researchers, and educators consider this further, we offer some early implications of adopting a subjectification lens on the design and enactment of assessment in CPD.
