Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored ...trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program.
We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors’ scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability.
7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard.
Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.
The learning environment describes the context and culture in which trainees learn. In order to establish the feasibility and reliability of measuring the anaesthetic learning environment in ...individual departments we implemented a previously developed instrument in hospitals across New South Wales.
We distributed the instrument to trainees from 25 anaesthesia departments and supplied summarized results to individual departments. Exploratory and confirmatory factor analyses were performed to assess internal structure validity and generalizability theory was used to calculate reliability. The number of trainees required for acceptable precision in results was determined using the standard error of measurement.
We received 172 responses (59% response rate). Suitable internal structure validity was confirmed. Measured reliability was acceptable (G-coefficient 0.69) with nine trainees per department. Eight trainees were required for a 95% confidence interval of plus or minus 0.25 in the mean total score. Eight trainees as assessors also allow a 95% confidence interval of approximately plus or minus 0.3 in the subscale mean scores. Results for individual departments varied, with scores below the expected level recorded on individual subscales, particularly the ‘teaching’ subscale.
Our results confirm that, using this instrument, individual departments can obtain acceptable precision in results with achievable trainee numbers. Additionally, with the exception of departments with few trainees, implementation proved feasible across a training region. Repeated use would allow departments or accrediting bodies to monitor their individual learning environment and the impact of changes such as the introduction of new curricular elements, or local initiatives to improve trainee experience.
Changes in work practices have led to a decline in the opportunities for anaesthetic trainees to learn technical procedures in supervised practice. Efforts to mitigate medical error and other changes ...have coincided with the development of alternative training methods so that it is increasingly difficult to justify the traditional model of teaching technical procedures. The range of simulators available for training in technical procedures in anaesthesia continues to expand. While simulation has been widely adopted in anaesthesia for crisis management training, there is little documented evidence of its use for technical skills training. The theoretical basis for the use of simulation to aid the acquisition of psychomotor skills and the development of expertise is now well established. In addition, practical frameworks that allow this theory to be applied in a systematic fashion have been developed and successfully used in other specialties. Using the available simulation equipment and educational tools, trainees can be prepared to begin supervised practice having demonstrated adequate procedural knowledge and expertise in simulation. With the use of simulated patients there is also the opportunity to integrate non-technical skills as well where appropriate. This review summarises the justification for the use of simulation in technical skills training in anaesthesia and the educational theory that supports its use, and outlines one of the available frameworks that can be used to aid its application.
Burnout has a high prevalence among healthcare workers and is increasingly recognised as an environmental problem rather than reflecting a personal inability to cope with work stress. We distributed ...an electronic survey, which included the Maslach Burnout Inventory Health Services Survey and a previously validated learning environment instrument, to 281 Victorian anaesthetic trainees. The response rate was 50%. We found significantly raised rates of burnout in two of three subscales. Ninety-one respondents (67%) displayed evidence of burnout in at least one domain, with 67 (49%) reporting high emotional exhaustion and 57 (42%) reporting high depersonalisation. The clinical learning environment tool demonstrated a significant negative correlation with burnout (r=-0.56, P <0.001). Burnout was significantly more common than when previously measured in Victoria in 2008 (62% versus 38%). Trainees rated examination preparation the most stressful aspect of the training program. There is a high prevalence of burnout among Victorian anaesthetic trainees. We have shown a significant correlation exists between the clinical learning environment measure and the presence of burnout. This correlation supports the development of interventions to improve the clinical learning environment, as a means to improve trainee wellbeing and address the high prevalence of burnout.
A questionnaire on substance abuse was distributed electronically to the heads of 185 Australian and New Zealand College of Anaesthetists accredited anaesthesia departments in Australia and New ...Zealand. The response rate was 57%. From January 2004 to December 2013, 61 cases of substance abuse were identified, giving an estimated incidence of 1.2 cases per 1000 anaesthetist years. Of 44 detailed reports completed, the majority were aged between 30 and 49 years, were male and of specialist grade. However, when corrected for gender and grade, the estimated overall incidence was higher in females and twice as high for trainees compared with specialists. When compared with prior surveys, the pattern of substance abuse in Australia and New Zealand appears to have changed significantly, with a notable increase in propofol and alcohol abuse and a decrease in reported cases of opioid abuse. Common presenting features of abuse included intoxication and witnessed abuse. Seventy percent of cases had more than one comorbid condition, most frequently either mental health or family problems. Only 32% of abusers had made a long-term recovery within the specialty. Death was the eventual outcome in 18% overall, with a particularly high mortality associated with propofol abuse (45%). Trainee suicide from all causes was reported at three times the rate of specialists. The findings indicate that substance abuse remains a significant problem in Australia and New Zealand and is associated with a significant mortality rate.
In reply to Castro-Alves and Kendall Castanelli, D J; Wickramaarachchi, S A; Wallis, S
Anaesthesia and intensive care,
05/2018, Letnik:
46, Številka:
3
Journal Article
Summary
Assessment tools must be investigated for reliability, validity and feasibility before being implemented. In 2013, the Australian and New Zealand College of Anaesthetists introduced ...workplace‐based assessments, including a direct observation of a procedural skills assessment tool. The objective of this study was to evaluate the psychometric properties of this assessment tool for ultrasound‐guided regional anaesthesia. Six experts assessed 30 video‐recorded trainee performances of ultrasound‐guided regional anaesthesia. Inter‐rater reliability, assessed using absolute agreement intraclass correlation coefficients, varied from 0.10 to 0.49 for the nine individual nine‐point scale items, and was 0.25 for a ‘total score’ of all items. Internal consistency was measured by correlation between ‘total score’ and ‘overall performance’ scale item (r = 0.68, p < 0.001). Construct validity was demonstrated by the ‘total score’ correlating with trainee experience (r = 0.51, p = 0.004). The mean time taken to complete assessments was 6 min 35 s.
The learning environment describes the way that trainees perceive the culture of their workplace. We audited the learning environment for trainees throughout Australia and New Zealand in the early ...stages of curriculum reform. A questionnaire was developed and sent electronically to a large random sample of Australian and New Zealand College of Anaesthetists trainees, with a 26% final response rate. This new instrument demonstrated good psychometric properties, with Cronbach's α ranging from 0.81 to 0.91 for each domain. The median score was equivalent to 78%, with the majority of trainees giving scores in the medium range. Introductory respondents scored their learning environment more highly than all other levels of respondents (P=0.001 for almost all comparisons). We present a simple questionnaire instrument that can be used to determine characteristics of the anaesthesia learning environment. The instrument can be used to help assess curricular change over time, alignment of the formal and informal curricula and strengths and weaknesses of individual departments.
The 'Roles in Practice' framework was introduced into the revised ANZCA curriculum in 2013. We conducted an online survey of Supervisors of Training in Australia and New Zealand to assess ...understanding of this framework, and teaching and perceived value of the non-scholar intrinsic roles within the framework. One hundred and forty-three survey responses were received (response rate 60.8%). The majority of respondents (52.1%) reported only a fair understanding of the framework. Formal teaching of all the roles was consistently reported as infrequent, with most teaching occurring through the informal curriculum. The Communicator, Collaborator and Professional Roles were rated as better taught and more important to teach than the roles of Health Advocate and Manager. The Communicator Role was perceived as being the role for which the development of resources would be most valuable. Respondents overwhelmingly nominated small group teaching as the preferred medium for resource development of all intrinsic roles. Our survey indicates that there is a need to increase both the understanding of the Roles in Practice framework and the teaching resources available in the ANZCA Supervisor of Training community.
We surveyed newly qualified consultant anaesthetists and their prospective employers in Victoria, regarding their expectations for the provision of paediatric anaesthesia by anaesthetists who have ...not completed subspecialty training in paediatric anaesthesia (generally-trained anaesthetists). Responses were received from 15 of 19 (79%) eligible Directors and 26 of 37 (70%) newly qualified Australian and New Zealand College of Anaesthetists (ANZCA) Fellows. Of those responding, 80% of Directors and 82% of Fellows would expect a generally-trained anaesthetist to anaesthetise children two years of age or older Regional Directors expected generally-trained anaesthetists to anaesthetise younger children than metropolitan Directors, and Directors' expectations were not influenced by their own practice. Testing these age limits with a series of simple case descriptions showed there is recognition from both Directors and Fellows that the stated age limits would be modified in both directions by case complexity and comorbidities. The new consultants who responded were significantly less confident than Directors in their ability to resuscitate and stabilise a critically ill child prior to transfer if required. Only 50% agreed they still met all the requirements of the ANZCA paediatric module and only 37.5% had the level of confidence they achieved during their paediatric rotations. We suggest that current training provides capacity to routinely anaesthetise well children two years of age or older However it appears more training would be required for most anaesthetists undertaking anaesthesia for younger patients or more complex paediatric cases. This raises the question of subspecialty endorsements within ANZCA Fellowship.
PDF
